江苏龙网

  1. 主页 > 生活百科 > >

基于OGG 实现Oracle到Kafka增量数据实时同步( 二 )

Oracle

3.配置源端Manager
GGSCI (dtproxy) 4> dblogin userid ggsadmin password oracle     Successfully logged into database. GGSCI (dtproxy as ggsadmin@dtstack) 5> edit param ./globals添加
oggschema ggsadmin GGSCI (dtproxy as ggsadmin@dtstack) 6> edit param mgr添加
PORT 7810 --默认监听端口 DYNAMICPORTLIST  7811-7820 --动态端口列表 AUTORESTART EXTRACT *,RETRIES 5,WAITMINUTES 3 --进程有问题，每3分钟重启一次，一共重启五次 PURGEOLDEXTRACTS ./dirdat/*, USECHECKPOINTS, MINKEEPDAYS 7  --*/ LAGREPORTHOURS 1 --每隔一小时检查一次传输延迟情况 LAGINFOMINUTES 30 --传输延时超过30分钟将写入错误日志 LAGCRITICALMINUTES 45 --传输延时超过45分钟将写入警告日志 PURGEMARKERHISTORY MINKEEPDAYS 3, MAXKEEPDAYS 7 --定期清理trail文件 ACCESSRULE, PROG *, IPADDR 172.*.*.*, ALLOW --设定172网段可连接添加同步的表
GGSCI (dtproxy as ggsadmin@dtstack) 9> add trandata baiyang.ora_to_kfk Oracle Goldengate marked following column as key columns on table BAIYANG.ORA_TO_KFK: OBJECT_ID. GGSCI (dtproxy as ggsadmin@dtstack) 10> info trandata baiyang.ora_to_kfk Prepared CSN for table BAIYANG.ORA_TO_KFK: 192881239目标端OGG配置
1.目标端检查环境
GGSCI (172-16-101-242) 1> info all     Program     Status      Group       Lag at Chkpt  Time Since Chkpt     MANAGER     STOPPED 2.创建目录
GGSCI (172-16-101-242) 2> create subdirs     Creating subdirectories under current directory /app/ogg     Parameter file                 /app/ogg/dirprm: created.     Report file                    /app/ogg/dirrpt: created.     Checkpoint file                /app/ogg/dirchk: created.     Process status files           /app/ogg/dirpcs: created.     SQL script files               /app/ogg/dirsql: created.     Database definitions files     /app/ogg/dirdef: created.     Extract data files             /app/ogg/dirdat: created.     Temporary files                /app/ogg/dirtmp: created.     Credential store files         /app/ogg/dircrd: created.     Masterkey wallet files         /app/ogg/dirwlt: created. Dump files                     /app/ogg/dirdmp: created.3.目标端Manager配置
GGSCI (172-16-101-242) 3> edit params mgr添加
PORT 7810     DYNAMICPORTLIST 7811-7820     AUTORESTART EXTRACT *,RETRIES 5,WAITMINUTES 3     PURGEOLDEXTRACTS ./dirdat/*,usecheckpoints, minkeepdays 3        GGSCI (172-16-101-242) 4> edit  param  ./GLOBALSCHECKPOINTTABLE ggsadmin.checkpoint
全量数据同步
1.配置源端数据初始化
配置源端初始化进程
GGSCI (dtproxy as ggsadmin@dtstack) 15> add extract initkfk,sourceistable 配置源端初始化参数
GGSCI (dtproxy as ggsadmin@dtstack) 16> edit params initkfk添加
【基于OGG 实现Oracle到Kafka增量数据实时同步】EXTRACT initkfk    SETENV (NLS_LANG=AMERICAN_AMERICA.AL32UTF8)    USERID ggsadmin,PASSWORD oracle    RMTHOST 172.16.101.242, MGRPORT 7810    RMTFILE ./dirdat/ekfk,maxfiles 999, megabytes 500table baiyang.ora_to_kfk;
2.源端生成表结构define文件
GGSCI (dtproxy as ggsadmin@dtstack) 17> edit param define_kfk添加
defsfile /oradata/oggorcl/ogg/dirdef/define_kfk.txt     userid ggsadmin,password oracle     table baiyang.ora_to_kfk;执行
$./defgen paramfile dirprm/define_kfk.prm -- Definitions generated for 1 table in /oradata/oggorcl/ogg/dirdef/define_kfk.txt将此文件传输到目标段dirdef文件夹


推荐阅读


上一篇：Python爬虫遇到验证码的几种处理方式，文章末尾有源码

下一篇：聊聊HBase分布式数据库设计那些事