一、mysqlmom简单介绍: 一个同步Mysql数据到Elasticsearch的工具,特色是支持分析 binlog 做实时增量同步，以及支持编写自定义逻辑处理数据后再同步到ES 纯 Python 编写，运行 mysqlsmom 的唯三要求: 它是除了阿里canal+MQ或者kakfa 读取MySQL Binlog文件然后再写入到ES.或者logstash直接从mysql同步数据到ElasticSearch之后的又一个同步MySQL数据到ES的方式。 二、演示环境: 利用MySQLmom程序同步MySQL数据到ES整个部署环境介绍: 3台阿里ECS服务器centos7.6 x86_64位最小化安装
<p>3台服务器默认的python版本号:Python 2.7.5</p>
<p>tidb06 172.16.0.247 部署elasticsearch-5.0.0和elasticsearch-6.2.4(采用的是二进制安装包)</p>
<p>tidb05 172.16.0.246 部署 mysqlsmom工具和redis-3.2.8(redis服务存放MySQL服务的pos位置点和binlog文件名称，是后面做增量同步MySQL数据到ES服务用到)</p>
<p>tidb04 172.16.0.197 部署MySQL5.7.22(MySQL服务没开启gtid,binlog-format的格式为row,)</p>
阿里云安全组要把对应的端口放开，允许内网访问es和mysql安装部署此处省略 单节点ES安装可以参考下面的文档: mysqlsmom官方安装文档: 官方提供的使用文档: 三、安装部署： cd /data/soft/
<p>git clone https://github.com/m358807551/mysqlsmom.git</p>
<p>[root@tidb05 mysqlsmom]# ls</p>
<p>docs mysqlsmom README.md README_OLD.md setup.py</p>
pip install mysqlsmom<p>[root@tidb05 mysqlsmom]# pip install mysqlsmom</p>
<p>DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.</p>
<p>Looking in indexes: http://mirrors.cloud.aliyuncs.com/pypi/simple/</p>
<p>Collecting mysqlsmom</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/e7/ed/3327e96cf31747a98248c833496d9d56349f444c8d5f7168b5bb9aff0925/mysqlsmom-0.1.7-py2-none-any.whl</p>
<p>Collecting idna==2.7 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl (58kB)</p>
<p>|████████████████████████████████| 61kB 4.9MB/s</p>
<p>Collecting asn1crypto==0.24.0 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl (101kB)</p>
<p>|████████████████████████████████| 102kB 8.0MB/s</p>
<p>Collecting six==1.11.0 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl</p>
<p>Collecting Pygments==2.2.0 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/02/ee/b6e02dc6529e82b75bb06823ff7d005b141037cb1416b10c6f00fc419dca/Pygments-2.2.0-py2.py3-none-any.whl (841kB)</p>
<p>|████████████████████████████████| 849kB 8.7MB/s</p>
<p>Collecting wcwidth==0.1.7 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/7e/9f/526a6947247599b084ee5232e4f9190a38f398d7300d866af3ab571a5bfe/wcwidth-0.1.7-py2.py3-none-any.whl</p>
<p>Collecting blinker==1.4 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/1b/51/e2a9f3b757eb802f61dc1f2b09c8c99f6eb01cf06416c0671253536517b6/blinker-1.4.tar.gz (111kB)</p>
<p>|████████████████████████████████| 112kB 65.1MB/s</p>
<p>Collecting decorator==4.3.0 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/bc/bb/a24838832ba35baf52f32ab1a49b906b5f82fb7c76b2f6a7e35e140bac30/decorator-4.3.0-py2.py3-none-any.whl</p>
<p>Collecting pytz==2018.5 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/30/4e/27c34b62430286c6d59177a0842ed90dc789ce5d1ed740887653b898779a/pytz-2018.5-py2.py3-none-any.whl (510kB)</p>
<p>|████████████████████████████████| 512kB 8.3MB/s</p>
<p>Collecting ipython-genutils==0.2.0 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/fa/bc/9bd3b5c2b4774d5f33b2d544f1460be9df7df2fe42f352135381c347c69a/ipython_genutils-0.2.0-py2.py3-none-any.whl</p>
<p>Collecting pathlib2==2.3.2 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/66/a7/9f8d84f31728d78beade9b1271ccbfb290c41c1e4dc13dbd4997ad594dcd/pathlib2-2.3.2-py2.py3-none-any.whl</p>
<p>Collecting tzlocal==1.5.1 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/cb/89/e3687d3ed99bc882793f82634e9824e62499fdfdc4b1ae39e211c5b05017/tzlocal-1.5.1.tar.gz</p>
<p>Collecting scandir==1.7 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/13/bb/e541b74230bbf7a20a3949a2ee6631be299378a784f5445aa5d0047c192b/scandir-1.7.tar.gz</p>
<p>Collecting pycparser==2.18 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/8c/2d/aad7f16146f4197a11f8e91fb81df177adcc2073d36a17b1491fd09df6ed/pycparser-2.18.tar.gz (245kB)</p>
<p>|████████████████████████████████| 256kB 63.1MB/s</p>
<p>Collecting yolk==0.4.3 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/2b/c0/73510e50668b70f3fad25328c58d7a29a300a2e452058dfdcb7c7538af7b/yolk-0.4.3.tar.gz (86kB)</p>
<p>|████████████████████████████████| 92kB 2.0MB/s</p>
<p>Collecting enum34==1.1.6 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl</p>
<p>Requirement already satisfied: chardet==3.0.4 in /usr/lib/python2.7/site-packages (from mysqlsmom) (3.0.4)</p>
<p>Collecting funcsigs==1.0.2 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/69/cb/f5be453359271714c01b9bd06126eaf2e368f1fddfff30818754b5ac2328/funcsigs-1.0.2-py2.py3-none-any.whl</p>
<p>Collecting elasticsearch==2.4.0 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/3f/be/980d79da0fbb14acc29e3c9b724287cbc24be2cb1ede47f2c0eecba60809/elasticsearch-2.4.0-py2.py3-none-any.whl (54kB)</p>
<p>|████████████████████████████████| 61kB 1.8MB/s</p>
<p>Collecting prompt-toolkit==1.0.15 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/d1/b0/1a6c262da35c779dd79550137aa7c298a424987240a28792ec5ccf48f848/prompt_toolkit-1.0.15-py2-none-any.whl (247kB)</p>
<p>|████████████████████████████████| 256kB 63.2MB/s</p>
<p>Collecting backports.shutil-get-terminal-size==1.0.0 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/7d/cd/1750d6c35fe86d35f8562091737907f234b78fdffab42b29c72b1dd861f4/backports.shutil_get_terminal_size-1.0.0-py2.py3-none-any.whl</p>
<p>Collecting ipython==5.0.0 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/eb/b6/3ed54fe22c284108d9676029b7cdca0ce94bcb8dc24487af0c88d57c53a1/ipython-5.0.0-py2.py3-none-any.whl (743kB)</p>
<p>|████████████████████████████████| 747kB 29.8MB/s</p>
<p>Collecting peewee==3.5.2 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/33/07/d4c742d88998fb448e18be41dc25e748376b7f69359d37fedaee09e66617/peewee-3.5.2.tar.gz (1.9MB)</p>
<p>|████████████████████████████████| 1.9MB 86kB/s</p>
<p>Collecting simplegeneric==0.8.1 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/3d/57/4d9c9e3ae9a255cd4e1106bb57e24056d3d0709fc01b2e3e345898e49d5b/simplegeneric-0.8.1.zip</p>
<p>Collecting pickleshare==0.7.4 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/9f/17/daa142fc9be6b76f26f24eeeb9a138940671490b91cb5587393f297c8317/pickleshare-0.7.4-py2.py3-none-any.whl</p>
<p>Collecting cryptography==2.2.2 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/dd/c2/3a5bfefb25690725824ade71e6b65449f0a9f4b29702cce10560f786ebf6/cryptography-2.2.2-cp27-cp27mu-manylinux1_x86_64.whl (2.2MB)</p>
<p>|████████████████████████████████| 2.2MB 2.4MB/s</p>
<p>Collecting futures==3.2.0 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl</p>
<p>Collecting urllib3==1.23 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl (133kB)</p>
<p>|████████████████████████████████| 143kB 66.9MB/s</p>
<p>Collecting PyMySQL==0.9.2 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/a7/7d/682c4a7da195a678047c8f1c51bb7682aaedee1dca7547883c3993ca9282/PyMySQL-0.9.2-py2.py3-none-any.whl (47kB)</p>
<p>|████████████████████████████████| 51kB 6.5MB/s</p>
<p>Collecting ipaddress==1.0.22 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl</p>
<p>Collecting mysql-replication==0.18 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/ff/c3/51cb2bdd50468d0bf37c2ab674bed3fd5939b3ab680c47470ea3a900b721/mysql-replication-0.18.tar.gz</p>
<p>Collecting requests==2.19.1 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/65/47/7e02164a2a3db50ed6d8a6ab1d6d60b69c4c3fdf57a284257925dfc12bda/requests-2.19.1-py2.py3-none-any.whl (91kB)</p>
<p>|████████████████████████████████| 92kB 40.7MB/s</p>
<p>Collecting cffi==1.11.5 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/14/dd/3e7a1e1280e7d767bd3fa15791759c91ec19058ebe31217fe66f3e9a8c49/cffi-1.11.5-cp27-cp27mu-manylinux1_x86_64.whl (407kB)</p>
<p>|████████████████████████████████| 409kB 576kB/s</p>
<p>Collecting traitlets==4.3.2 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/93/d6/abcb22de61d78e2fc3959c964628a5771e47e7cc60d53e9342e21ed6cc9a/traitlets-4.3.2-py2.py3-none-any.whl (74kB)</p>
<p>|████████████████████████████████| 81kB 62.8MB/s</p>
<p>Collecting click==6.7 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/34/c1/8806f99713ddb993c5366c362b2f908f18269f8d792aff1abfd700775a77/click-6.7-py2.py3-none-any.whl (71kB)</p>
<p>|████████████████████████████████| 71kB 58.9MB/s</p>
<p>Collecting ptyprocess==0.6.0 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/d1/29/605c2cc68a9992d18dada28206eeada56ea4bd07a239669da41674648b6f/ptyprocess-0.6.0-py2.py3-none-any.whl</p>
<p>Collecting certifi==2018.4.16 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl (150kB)</p>
<p>|████████████████████████████████| 153kB 64.3MB/s</p>
<p>Collecting pexpect==4.6.0 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/89/e6/b5a1de8b0cc4e07ca1b305a4fcc3f9806025c1b651ea302646341222f88b/pexpect-4.6.0-py2.py3-none-any.whl (57kB)</p>
<p>|████████████████████████████████| 61kB 55.9MB/s</p>
<p>Collecting redis==2.10.6 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/3b/f6/7a76333cf0b9251ecf49efff635015171843d9b977e4ffcf59f9c4428052/redis-2.10.6-py2.py3-none-any.whl (64kB)</p>
<p>|████████████████████████████████| 71kB 52.3MB/s</p>
<p>Collecting APScheduler==3.5.1 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/60/08/fbd377c75a4531d8ef55a65fe3fc316a1cada9b552e290f10705461807fb/APScheduler-3.5.1-py2.py3-none-any.whl (60kB)</p>
<p>|████████████████████████████████| 61kB 38.4MB/s</p>
<p>Collecting appnope==0.1.0 (from mysqlsmom)</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/87/a9/7985e6a53402f294c8f0e8eff3151a83f1fb901fa92909bb3ff29b4d22af/appnope-0.1.0-py2.py3-none-any.whl</p>
<p>Requirement already satisfied: setuptools in /usr/lib/python2.7/site-packages (from yolk==0.4.3->mysqlsmom) (36.4.0)</p>
<p>Building wheels for collected packages: blinker, tzlocal, scandir, pycparser, yolk, peewee, simplegeneric, mysql-replication</p>
<p>Building wheel for blinker (setup.py) ... done</p>
<p>Stored in directory: /root/.cache/pip/wheels/4d/0f/13/a39e525325104948e5ca166374d581961da53d7e8e6d7568ab</p>
<p>Building wheel for tzlocal (setup.py) ... done</p>
<p>Stored in directory: /root/.cache/pip/wheels/8b/02/9e/c2975e524c35630d636e8ee36c50f1802c1e2d5f55808f4eaa</p>
<p>Building wheel for scandir (setup.py) ... done</p>
<p>Stored in directory: /root/.cache/pip/wheels/43/5a/3a/e0601dc9620b1d57d0abf16ad1621245a5223ee90828eadb52</p>
<p>Building wheel for pycparser (setup.py) ... done</p>
<p>Stored in directory: /root/.cache/pip/wheels/73/a0/a2/1d71209495d2e50df76397c4402abdec4f113ac801c112d108</p>
<p>Building wheel for yolk (setup.py) ... done</p>
<p>Stored in directory: /root/.cache/pip/wheels/fc/3f/8f/6a0fe676b8c82fbe6ecf4081469fbc72c9ea11a283bcc01546</p>
<p>Building wheel for peewee (setup.py) ... done</p>
<p>Stored in directory: /root/.cache/pip/wheels/bc/0b/b7/44e1baafcd11ab6251d5e26f2948822960adf2dd0a85156be3</p>
<p>Building wheel for simplegeneric (setup.py) ... done</p>
<p>Stored in directory: /root/.cache/pip/wheels/5e/9c/8b/73a7fb1e4b0ffeb65c9c4341211d09854c86691e746d1af66b</p>
<p>Building wheel for mysql-replication (setup.py) ... done</p>
<p>Stored in directory: /root/.cache/pip/wheels/66/c7/4c/98d1b7d1ad1dc0ea6081ae3324446306e20b2bb65e238ccc40</p>
<p>Successfully built blinker tzlocal scandir pycparser yolk peewee simplegeneric mysql-replication</p>
<p>Installing collected packages: idna, asn1crypto, six, Pygments, wcwidth, blinker, decorator, pytz, ipython-genutils, scandir, pathlib2, tzlocal, pycparser, yolk, enum34, funcsigs, urllib3, elasticsearch, prompt-toolkit, backports.shutil-get-terminal-size, ptyprocess, pexpect, traitlets, simplegeneric, pickleshare, ipython, peewee, cffi, ipaddress, cryptography, futures, PyMySQL, mysql-replication, certifi, requests, click, redis, APScheduler, appnope, mysqlsmom</p>
<p>Found existing installation: idna 2.8</p>
<p>Uninstalling idna-2.8:</p>
<p>Successfully uninstalled idna-2.8</p>
<p>Found existing installation: decorator 3.4.0</p>
<p>Uninstalling decorator-3.4.0:</p>
<p>Successfully uninstalled decorator-3.4.0</p>
<p>Found existing installation: urllib3 1.25.3</p>
<p>Uninstalling urllib3-1.25.3:</p>
<p>Successfully uninstalled urllib3-1.25.3</p>
<p>Found existing installation: certifi 2019.6.16</p>
<p>Uninstalling certifi-2019.6.16:</p>
<p>Successfully uninstalled certifi-2019.6.16</p>
<p>Found existing installation: requests 2.22.0</p>
<p>Uninstalling requests-2.22.0:</p>
<p>Successfully uninstalled requests-2.22.0</p>
<p>Successfully installed APScheduler-3.5.1 PyMySQL-0.9.2 Pygments-2.2.0 appnope-0.1.0 asn1crypto-0.24.0 backports.shutil-get-terminal-size-1.0.0 blinker-1.4 certifi-2018.4.16 cffi-1.11.5 click-6.7 cryptography-2.2.2 decorator-4.3.0 elasticsearch-2.4.0 enum34-1.1.6 funcsigs-1.0.2 futures-3.2.0 idna-2.7 ipaddress-1.0.22 ipython-5.0.0 ipython-genutils-0.2.0 mysql-replication-0.18 mysqlsmom-0.1.7 pathlib2-2.3.2 peewee-3.5.2 pexpect-4.6.0 pickleshare-0.7.4 prompt-toolkit-1.0.15 ptyprocess-0.6.0 pycparser-2.18 pytz-2018.5 redis-2.10.6 requests-2.19.1 scandir-1.7 simplegeneric-0.8.1 six-1.11.0 traitlets-4.3.2 tzlocal-1.5.1 urllib3-1.23 wcwidth-0.1.7 yolk-0.4.3</p>
<p>WARNING: You are using pip version 19.1.1, however version 20.3.4 is available.</p>
<p>You should consider upgrading via the 'pip install --upgrade pip' command.</p>
[root@tidb05 mysqlsmom]#说明：然后指定 elasticsearch 版本（默认支持2.4），支持其它版本请运行（将5.4换成需要的elasticsearch版本 ）<p>[root@tidb05 mysqlsmom]# pip install --upgrade elasticsearch==5.0</p>
<p>DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.</p>
<p>Looking in indexes: http://mirrors.cloud.aliyuncs.com/pypi/simple/</p>
<p>Collecting elasticsearch==5.0</p>
<p>Downloading http://mirrors.cloud.aliyuncs.com/pypi/packages/f3/4a/446018515cbaecaa63465e3af003a41ce6d3c4deb6cbe751b73e1aff1dad/elasticsearch-5.0.0-py2.py3-none-any.whl (57kB)</p>
<p>|████████████████████████████████| 61kB 1.7MB/s</p>
<p>Requirement already satisfied, skipping upgrade: urllib3<2.0,>=1.8 in /usr/lib/python2.7/site-packages (from elasticsearch==5.0) (1.23)</p>
<p>ERROR: mysqlsmom 0.1.7 has requirement elasticsearch==2.4.0, but you'll have elasticsearch 5.0.0 which is incompatible.</p>
<p>Installing collected packages: elasticsearch</p>
<p>Found existing installation: elasticsearch 2.4.0</p>
<p>Uninstalling elasticsearch-2.4.0:</p>
<p>Successfully uninstalled elasticsearch-2.4.0</p>
<p>Successfully installed elasticsearch-5.0.0</p>
<p>WARNING: You are using pip version 19.1.1, however version 20.3.4 is available.</p>
<p>You should consider upgrading via the 'pip install --upgrade pip' command.</p>
<p>[root@tidb05 mysqlsmom]# echo $?</p>
0安装部署到此完成。是不是灰常的简单呢，后面会介绍全量同步MySQL表数据到ES服务 
数据库
发布于：2021-08-03 11:56
|
阅读数：269
|
评论：0
QQ好友和群
QQ空间
