1、准备:
github:https://github.com/alibaba/canal
里面有包括canal的文档,server端 client端的 例子 源码包等等。
2、canal概述:
canal是应阿里巴巴存在杭州和美国的双机房部署,存在跨机房同步的业务需求而提出的。
早期,阿里巴巴b2b公司因为存在杭州和美国双机房部署,存在跨机房同步的业务需求。不过早期的数据库同步业务,主要是基于trigger的方式获取增量变更,不过从2010年开始,阿里系公司开始逐步的尝试基于数据库的日志解析,获取增量变更进行同步,由此衍生出了增量订阅&消费的业务,从此开启了一段新纪元。ps. 目前内部使用的同步,已经支持mysql5.x和oracle部分版本的日志解析
基于日志增量订阅&消费支持的业务:
数据库镜像
数据库实时备份
多级索引 (卖家和买家各自分库索引)
search build
业务cache刷新
价格变化等重要业务消息
keyword:数据库同步,增量订阅&消费。
3、canal工作原理:
从上层来看,复制分成三步:
master将改变记录到二进制日志(binary log)中(这些记录叫做二进制日志事件,binary log events,可以通过show binlog events进行查看);
slave将master的binary log events拷贝到它的中继日志(relay log);
slave重做中继日志中的事件,将改变反映它自己的数据。
4、部署canal:
部署canal-server:
(1)开启mysql的binlog功能,并配置binlog模式为row。
在my.cnf 加入如下:
[html] view plain copy
- [mysqld]
- log-bin=mysql-bin #添加这一行就ok
- binlog-format=row #选择row模式
- server_id=1 #配置mysql replaction需要定义,不能和canal的slaveid重复
(2)在mysql中 配置canal数据库管理用户,配置相应权限(repication权限)
[sql] view plain copy
- create user canal identified by 'canal';
- grant select, replication slave, replication client on *.* to 'canal'@'%';
- -- grant all privileges on *.* to 'canal'@'%' ;
- flush privileges;
(3)下载canal https://github.com/alibaba/canal/releases
解压到相应文件夹
[html] view plain copy
- tar -zxvf canal canal
canal 文件目录结构
[html] view plain copy
- drwxr-xr-x 2 jianghang jianghang 136 2013-02-05 21:51 bin
- drwxr-xr-x 4 jianghang jianghang 160 2013-02-05 21:51 conf
- drwxr-xr-x 2 jianghang jianghang 1.3k 2013-02-05 21:51 lib
- drwxr-xr-x 2 jianghang jianghang 48 2013-02-05 21:29 logs
修改配置 instance.properties
[html] view plain copy
- vim canal/conf/example/instance.properties
[html] view plain copy
- #################################################
- ## mysql serverid
- canal.instance.mysql.slaveid = 1234
- # position info,需要改成自己的数据库信息
- canal.instance.master.address = 127.0.0.1:3306
- canal.instance.master.journal.name =
- canal.instance.master.position =
- canal.instance.master.timestamp =
- #canal.instance.standby.address =
- #canal.instance.standby.journal.name =
- #canal.instance.standby.position =
- #canal.instance.standby.timestamp =
- # username/password,需要改成自己的数据库信息
- canal.instance.dbusername = canal
- canal.instance.dbpassword = canal
- canal.instance.defaultdatabasename = canal_test
- canal.instance.connectioncharset = utf-8
- # table regex
- canal.instance.filter.regex = .*\\..*
- #################################################
然后cd到bin目录 启动和停止canal-server
启动
[html] view plain copy
- ./startup.sh
停止
[html] view plain copy
- ./stop.sh
验证启动状态,查看log文件
[html] view plain copy
- vim canal/log/canal/canal.log
[html] view plain copy
- 2014-07-18 10:21:08.525 [main] info com.alibaba.otter.canal.deployer.canallauncher - ## start the canal server.
- 2014-07-18 10:21:08.609 [main] info com.alibaba.otter.canal.deployer.canalcontroller - ## start the canal server[10.12.109.201:11111]
- 2014-07-18 10:21:09.037 [main] info com.alibaba.otter.canal.deployer.canallauncher - ## the canal server is running now ......
上述日志信息显示启动canal成功
运行canal-client实例:
(1)建立实例maven工程
[html] view plain copy
- mvn archetype:create -dgroupid=com.alibaba.otter -dartifactid=canal.sample
(2)添加pom依赖:
[html] view plain copy
- <dependency>
- <groupid>com.alibaba.ottergroupid>
- <artifactid>canal.clientartifactid>
- <version>1.0.12version>
- dependency>
(3)更新依赖 mvn install
(4)canal-client.java 实例代码
[java] view plain copy
- /**
- * created by hp on 14-7-17.
- */
- import java.net.inetsocketaddress;
- import java.util.list;
- import com.alibaba.otter.canal.client.canalconnector;
- import com.alibaba.otter.canal.common.utils.addressutils;
- import com.alibaba.otter.canal.protocol.message;
- import com.alibaba.otter.canal.protocol.canalentry.column;
- import com.alibaba.otter.canal.protocol.canalentry.entry;
- import com.alibaba.otter.canal.protocol.canalentry.entrytype;
- import com.alibaba.otter.canal.protocol.canalentry.eventtype;
- import com.alibaba.otter.canal.protocol.canalentry.rowchange;
- import com.alibaba.otter.canal.protocol.canalentry.rowdata;
- import com.alibaba.otter.canal.client.*;
- import org.jetbrains.annotations.notnull;
- public class clientsample {
- public static void main(string args[]) {
- // 创建链接
- canalconnector connector = canalconnectors.newsingleconnector(new inetsocketaddress(addressutils.gethostip(),
- 11111), "example", "", "");
- int batchsize = 1000;
- int emptycount = 0;
- try {
- connector.connect();
- connector.subscribe(".*\\..*");
- connector.rollback();
- int totalemtrycount = 1200;
- while (emptycount < totalemtrycount) {
- message message = connector.getwithoutack(batchsize); // 获取指定数量的数据
- long batchid = message.getid();
- int size = message.getentries().size();
- if (batchid == -1 || size == 0) {
- emptycount ;
- system.out.println("empty count : " emptycount);
- try {
- thread.sleep(1000);
- } catch (interruptedexception e) {
- e.printstacktrace();
- }
- } else {
- emptycount = 0;
- // system.out.printf("message[batchid=%s,size=%s] \n", batchid, size);
- printentry(message.getentries());
- }
- connector.ack(batchid); // 提交确认
- // connector.rollback(batchid); // 处理失败, 回滚数据
- }
- system.out.println("empty too many times, exit");
- } finally {
- connector.disconnect();
- }
- }
- private static void printentry(@notnull list
entrys) { - for (entry entry : entrys) {
- if (entry.getentrytype() == entrytype.transactionbegin || entry.getentrytype() == entrytype.transactionend) {
- continue;
- }
- rowchange rowchage = null;
- try {
- rowchage = rowchange.parsefrom(entry.getstorevalue());
- } catch (exception e) {
- throw new runtimeexception("error ## parser of eromanga-event has an error , data:" entry.tostring(),
- e);
- }
- eventtype eventtype = rowchage.geteventtype();
- system.out.println(string.format("================> binlog[%s:%s] , name[%s,%s] , eventtype : %s",
- entry.getheader().getlogfilename(), entry.getheader().getlogfileoffset(),
- entry.getheader().getschemaname(), entry.getheader().gettablename(),
- eventtype));
- for (rowdata rowdata : rowchage.getrowdataslist()) {
- if (eventtype == eventtype.delete) {
- printcolumn(rowdata.getbeforecolumnslist());
- } else if (eventtype == eventtype.insert) {
- printcolumn(rowdata.getaftercolumnslist());
- } else {
- system.out.println("-------> before");
- printcolumn(rowdata.getbeforecolumnslist());
- system.out.println("-------> after");
- printcolumn(rowdata.getaftercolumnslist());
- }
- }
- }
- }
- private static void printcolumn(@notnull list
columns) { - for (column column : columns) {
- system.out.println(column.getname() " : " column.getvalue() " update=" column.getupdated());
- }
- }
- }
(5)运行java实例
启动后看到控制端信息:
[html] view plain copy
- empty count : 1
- empty count : 2
- empty count : 3
- empty count : 4
(6)触发数据库变更
[html] view plain copy
- create table test (
- uid int (4) primary key not null auto_increment,
- name varchar(10) not null);
- insert into test (name) values('10');
(7)client 抓取mysql信息:
[html] view plain copy
- ================> binlog[mysql-bin.000016:3281] , name[canal_test,test] , eventtype : insert
- uid : 7 update=false
- name : 10 update=false
- empty count : 1
- empty count : 2
5、部署过程中产生问题:
(1)启动失败,log日志中地址正在使用
1、11111端口正在被占用 可以用 ls -i:11111 查看监听进程谁占用端口 或者 用 ps -ef | grep 11111 查看哪个进程占用端口号 然后 kill -9 进程号 杀掉占用进程
2、可以编辑 canal/conf/canal.properties 中的端口号 ,改为不占用的端口
(2)canal无法抓取mysql触发数据库改变的信息
1、检查mysql是否打开binlog写入功能 检查binlog 是否为行模式。
[html] view plain copy
- show variables like "binlog_format"
2、检查my.cnf 和 instance.properties 等配置文件填写信息是否正确。
3、检查client 代码 调试实例代码
4、版本兼容问题,canal 1.8 换成 canal 1.7 继续测试
5、查看所有日志文件 分析日志