2020-08-06

Redis服务之Redis Cluster

redis cluster 把所有的 redis node 映射到 0-16383 个槽位(slot)上,读写需要到指定的 redis node 上进行操作,因此有多少个 reids node 相当于 redis 并发扩展了多少倍。Redis cluster 预先分配 16384 个(slot)槽位,当需要在 redis 集群中写入一个 key -value 的时候,会使用 CRC16(key) mod 16384 之后的值,决定将 key 写入值哪一个槽位从而决定写入哪一个 Redis 节点上,从而有效解决redis单机读写瓶颈。

  在上一篇博客中我们聊到了redis的高可用组件sentinel的相关配置,回顾请参考https://www.cnblogs.com/qiuhom-1874/p/13429776.html;sentinel在redis主从同步架构中主要起到了监控集群master是否正常,如果master不正常,或者宕机,那么sentinel会提升一个slave当选新的master,从而保证了redis服务的正常使用;但对于redis的单机写入问题还是一直存在;在sentinel+主从同步架构中,程序写数据,始终是把读写请求发送给master(当然,如果有语句路由器读写请求是可以分开的);这样一来对于master来讲它就承担了所有的写操作,很显然这种在写操作非常的频繁的场景,单台master肯定无法承受这么大的压力;为了解决单机master写入数据的瓶颈问题, redis 官方在 redis 3.0 版本之后推出了无中心架构的 redis cluster 机制;所谓无中心架构的cluster意思是集群中的每一个成员都是中心节点,每个成员拥有整个集群的状态元数据,但各自又只存储整个集群的部分数据;对于redis cluster这种架构来讲,所有redis节点使用(ping机制)互联,如果集群中某个节点的失败,是需要整个集群中超过半数的节点监测都失效才算真正的失效;客户端不需要 proxy 即可直接连接 redis,应用程序需要写全部的 redis 服务器 IP。redis cluster 把所有的 redis node 映射到 0-16383 个槽位(slot)上,读写需要到指定的 redis node 上进行操作,因此有多少个 reids node 相当于 redis 并发扩展了多少倍。Redis cluster 预先分配 16384 个(slot)槽位,当需要在 redis 集群中写入一个 key -value 的时候,会使用 CRC16(key) mod 16384 之后的值,决定将 key 写入值哪一个槽位从而决定写入哪一个 Redis 节点上,从而有效解决redis单机读写瓶颈。如下所示

  提示:客户端每次读写都要经过CRC16对读写的KEY做hash计算,然后把hash后的结果再和16384做取模运算,最终结果会落到0-16383的某一个槽位上;我们事先在定义集群时,就把0-5460号槽位都分给了master01,意思就是只要读写的数据的KEY通过CRC16hash计算取得的结果%16384,最后的结果在0-5460这个范围内,就把本次的读/写请求调度到master01上;以此类推,范围在5461-10922就把读/写请求调度到master02,范围在10923-16383就把读/写请求调度到master03;通过上面的图我们也能看出各个master都只保存整个集群的一部分数据,如果某一个master宕机,那么就会导致对应master上分的槽位对应数据丢失;为了解决各个master单点的问题,我们还需要对每个master做主从,如下图

  提示:为了防止某个master宕机后导致数据丢失和写操作不能执行,对应slave应该能够及时的提升为master,类似sentinel的功能,但是我们不需要在部署sentinel了,因为redis cluster就支持主从切换;这样一来我们的集群就变成了3主3从的集群;接下我们就准备一个实验环境来部署上面的架构;

  部署redis集群

  环境说明

角色ip地址端口
Master01192.168.0.416379
Slave01192.168.0.426380
Master02192.168.0.426379
Slave02192.168.0.436380
Master03192.168.0.436379
Slave03192.168.0.416380

 

 

 

 

 

 

 

 

  

  准备启动3台服务器,每个服务器上启动两个redis实例,端口分别是6379(master)和6380(slave),为了防止master和slave落在同一个节点,有意将他们交叉组合,如上表格;

  创建redis cluster我们需要注意每个 redis node 节点尽量采用相同的硬件配置,相同的密码;所有 redis 服务器必须没有任何数据;

  在node01上编译安装redis,有关redis的编译安装和部署请参考https://www.cnblogs.com/qiuhom-1874/p/13378138.html;

  创建目录结构

[root@node01 ~]# tree /usr/local/redis//usr/local/redis/├── 6379│   ├── etc│   │   ├── redis.conf│   │   └── sentinel.conf│   ├── logs│   └── run├── 6380│   ├── etc│   │   ├── redis.conf│   │   └── sentinel.conf│   ├── logs│   └── run└── bin ├── redis-benchmark ├── redis-check-aof ├── redis-check-rdb ├── redis-cli ├── redis-sentinel -> redis-server └── redis-server9 directories, 10 files[root@node01 ~]#

  提示:在/usr/local/redis目录下分别创建6379和6380,然后在其下都创建etc,logs,run 等子目录,如上所示

  修改配置文件

[root@node01 ~]# grep -E "^(bind|requirepass|masterauth|cluster)" /usr/local/redis/6379/etc/redis.conf bind 0.0.0.0masterauth adminrequirepass admincluster-enabled yescluster-config-file redis-cluster_6379.conf[root@node01 ~]# grep -E "^(bind|requirepass|masterauth|cluster)" /usr/local/redis/6380/etc/redis.conf bind 0.0.0.0masterauth adminrequirepass admincluster-enabled yescluster-config-file redis-cluster_6380.conf[root@node01 ~]# 

  提示:6379和6380的配置文件内容都是一样的,唯一区别是端口不同,主要修改bind 监听地址,设置密码,开启集群以及指定对应集群生成的配置文件名称;

  把node01上的/usr/local/redis复制给node02和node03并保存到/usr/local/目录下

  提示:我上面的三台主机都是做了免密登录,所以互相拷贝数据不需要输入密码;有关免密登录的说明和配置可以参考https://www.cnblogs.com/qiuhom-1874/p/11783371.html;

  分别在node01 node02 node03上启动redis

  到此我们就把6个redis实例启动了;接下来我们来创建集群;Redis 3 和 4 版本需要使用到集群管理工具 redis-trib.rb,这个工具是 redis 官方推出的管理 redis 集群的工具,集成在redis 的源码 src 目录下,是基于 redis 提供的集群命令封装成简单、便捷、实用的操作工具,redis-trib.rb是 redis 作者用 ruby 开发完成的,所以我们要使用redis-trib.rb首先要解决ruby环境的问题;

  安装ruby环境

[root@node01 ~]# rzrz waiting to receive. zmodem trl+C ȡ 100% 15621 KB 15621 KB/s 00:00:01  0 Errors[root@node01 ~]# lsredis-4.0.9.tar.gz ruby-2.5.5.tar.gz[root@node01 ~]# tar xf ruby-2.5.5.tar.gz -C /usr/local/src/[root@node01 ~]# cd /usr/local/src/[root@node01 src]# cd ruby-2.5.5/[root@node01 ruby-2.5.5]# ./configure checking for ruby... falsechecking build system type... x86_64-pc-linux-gnuchecking host system type... x86_64-pc-linux-gnuchecking target system type... x86_64-pc-linux-gnuchecking for gcc... gccchecking whether the C compiler works... yeschecking for C compiler default output file name... a.outchecking for suffix of executables... checking whether we are cross compiling... nochecking for suffix of object files... ochecking whether we are using the GNU C compiler... yeschecking whether gcc accepts -g... yeschecking for gcc option to accept ISO C89... none neededchecking for g++... g++checking whether we are using the GNU C++ compiler... yes……省略部分内容……checking for setjmp type... __builtin_setjmpchecking for prefix of external symbols... NONEchecking pthread.h usability... yeschecking pthread.h presence... yeschecking for pthread.h... yeschecking if make is GNU make... yes.ext/include/x86_64-linux/ruby/config.h updatedconfigure: ruby library version = 2.5.0configure: creating ./config.statusconfig.status: creating GNUmakefileconfig.status: creating Makefileconfig.status: creating ruby-2.5.pc---Configuration summary for ruby version 2.5.5 * Installation prefix: /usr/local * exec prefix:   ${prefix} * arch:    x86_64-linux * site arch:   ${arch} * RUBY_BASE_NAME:  ruby * ruby lib prefix:  ${libdir}/${RUBY_BASE_NAME} * site libraries path: ${rubylibprefix}/${sitearch} * vendor path:   ${rubylibprefix}/vendor_ruby * target OS:   linux * compiler:   gcc * with pthread:  yes * enable shared libs: no * dynamic library ext: so * CFLAGS:    ${optflags} ${debugflags} ${warnflags} * LDFLAGS:    -L. -fstack-protector -rdynamic \       -Wl,-export-dynamic * optflags:   -O3 * debugflags:   -ggdb3 * warnflags:   -Wall -Wextra -Wno-unused-parameter \       -Wno-parentheses -Wno-long-long \       -Wno-missing-field-initializers \       -Wno-tautological-compare \       -Wno-parentheses-equality \       -Wno-constant-logical-operand -Wno-self-assign \       -Wunused-variable -Wimplicit-int -Wpointer-arith \       -Wwrite-strings -Wdeclaration-after-statement \       -Wimplicit-function-declaration \       -Wdeprecated-declarations \       -Wno-packed-bitfield-compat \       -Wsuggest-attribute=noreturn \       -Wsuggest-attribute=format * strip command:  strip -S -x * install doc:   yes * man page type:  doc---[root@node01 ruby-2.5.5]# 

  提示:redis-trib.rb这个工具在集群任意一台安装即可,不需要每台都安装;

  编译

[root@node01 ruby-2.5.5]# make -j 2  CC = gcc  LD = ld  LDSHARED = gcc -shared  CFLAGS = -O3 -ggdb3 -Wall -Wextra -Wno-unused-parameter -Wno-parentheses -Wno-long-long -Wno-missing-field-initializers -Wno-tautological-compare -Wno-parentheses-equality -Wno-constant-logical-operand -Wno-self-assign -Wunused-variable -Wimplicit-int -Wpointer-arith -Wwrite-strings -Wdeclaration-after-statement -Wimplicit-function-declaration -Wdeprecated-declarations -Wno-packed-bitfield-compat -Wsuggest-attribute=noreturn -Wsuggest-attribute=format -std=gnu99   XCFLAGS = -D_FORTIFY_SOURCE=2 -fstack-protector -fno-strict-overflow -fvisibility=hidden -fexcess-precision=standard -DRUBY_EXPORT -fPIE  CPPFLAGS = -I. -I.ext/include/x86_64-linux -I./include -I. -I./enc/unicode/10.0.0  DLDFLAGS = -fstack-protector -pie   SOLIBS =   LANG = en_US.UTF-8  LC_ALL =   LC_CTYPE = gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)Copyright (C) 2015 Free Software Foundation, Inc.This is free software; see the source for copying conditions. There is NOwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.compiling ./main.ccompiling dmydln.ccompiling miniinit.ccompiling dmyext.ccompiling miniprelude.cmaking dummy probes.hcompiling bignum.ccompiling class.ccompiling compar.ccompiling compile.ccompiling complex.ccompiling cont.ccompiling debug.ccompiling debug_counter.ccompiling dir.ccompiling dln_find.ccompiling encoding.ccompiling enum.ccompiling enumerator.c……省略部分内容……*** Fix the problems, then remove these directories and try again if you want.make[1]: Leaving directory `/usr/local/src/ruby-2.5.5'Generating RDoc documentationParsing sources...100% [871/871] vsnprintf.c                   Generating RI format into /usr/local/src/ruby-2.5.5/.ext/rdoc... Files:  871 Classes:  1324 ( 565 undocumented) Modules:  286 ( 121 undocumented) Constants: 2181 ( 555 undocumented) Attributes: 1066 ( 251 undocumented) Methods: 10080 (2161 undocumented) Total:  14937 (3653 undocumented) 75.54% documented Elapsed: 25.0s[root@node01 ruby-2.5.5]# 

  安装ruby环境

[root@node01 ruby-2.5.5]# make install  CC = gcc  LD = ld  LDSHARED = gcc -shared  CFLAGS = -O3 -ggdb3 -Wall -Wextra -Wno-unused-parameter -Wno-parentheses -Wno-long-long -Wno-missing-field-initializers -Wno-tautological-compare -Wno-parentheses-equality -Wno-constant-logical-operand -Wno-self-assign -Wunused-variable -Wimplicit-int -Wpointer-arith -Wwrite-strings -Wdeclaration-after-statement -Wimplicit-function-declaration -Wdeprecated-declarations -Wno-packed-bitfield-compat -Wsuggest-attribute=noreturn -Wsuggest-attribute=format -std=gnu99   XCFLAGS = -D_FORTIFY_SOURCE=2 -fstack-protector -fno-strict-overflow -fvisibility=hidden -fexcess-precision=standard -DRUBY_EXPORT -fPIE  CPPFLAGS = -I. -I.ext/include/x86_64-linux -I./include -I. -I./enc/unicode/10.0.0  DLDFLAGS = -fstack-protector -pie   SOLIBS =   LANG = en_US.UTF-8  LC_ALL =   LC_CTYPE = gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)Copyright (C) 2015 Free Software Foundation, Inc.This is free software; see the source for copying conditions. There is NOwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.generating enc.mkmaking srcs under encmake[1]: Entering directory `/usr/local/src/ruby-2.5.5'make[1]: Nothing to be done for `srcs'.make[1]: Leaving directory `/usr/local/src/ruby-2.5.5'generating transdb.htransdb.h unchangedgenerating makefiles ext/configure-ext.mkext/configure-ext.mk updatedmake[1]: Entering directory `/usr/local/src/ruby-2.5.5'make[1]: Leaving directory `/usr/local/src/ruby-2.5.5'generating makefile exts.mkexts.mk updatedmake[1]: Entering directory `/usr/local/src/ruby-2.5.5'make[2]: Entering directory `/usr/local/src/ruby-2.5.5/ext/fcntl'make[2]: Leaving directory `/usr/local/src/ruby-2.5.5/ext/fcntl'make[2]: Entering directory `/usr/local/src/ruby-2.5.5/ext/rubyvm'……省略部分内容……installing binary commands:   /usr/local/bininstalling base libraries:   /usr/local/libinstalling arch files:    /usr/local/lib/ruby/2.5.0/x86_64-linuxinstalling pkgconfig data:   /usr/local/lib/pkgconfiginstalling command scripts:   /usr/local/bininstalling library scripts:   /usr/local/lib/ruby/2.5.0installing common headers:   /usr/local/include/ruby-2.5.0installing manpages:    /usr/local/share/man/man1installing extension objects:  /usr/local/lib/ruby/2.5.0/x86_64-linuxinstalling extension objects:  /usr/local/lib/ruby/site_ruby/2.5.0/x86_64-linuxinstalling extension objects:  /usr/local/lib/ruby/vendor_ruby/2.5.0/x86_64-linuxinstalling extension headers:  /usr/local/include/ruby-2.5.0/x86_64-linuxinstalling extension scripts:  /usr/local/lib/ruby/2.5.0installing extension scripts:  /usr/local/lib/ruby/site_ruby/2.5.0installing extension scripts:  /usr/local/lib/ruby/vendor_ruby/2.5.0installing extension headers:  /usr/local/include/ruby-2.5.0/rubyinstalling default gems from lib: /usr/local/lib/ruby/gems/2.5.0 (build_info, cache, doc, extensions, gems, specifications)         cmath 1.0.0         csv 1.0.0         fileutils 1.0.2         ipaddr 1.2.0         rdoc 6.0.1         scanf 1.0.0         webrick 1.4.2installing default gems from ext: /usr/local/lib/ruby/gems/2.5.0 (build_info, cache, doc, extensions, gems, specifications)         bigdecimal 1.3.4         date 1.0.0         etc 1.0.0         fcntl 1.0.0         fiddle 1.0.0         io-console 0.4.6         json 2.1.0         psych 3.0.2         sdbm 1.0.0         stringio 0.0.1         strscan 1.0.0installing bundled gems:   /usr/local/lib/ruby/gems/2.5.0 (build_info, cache, doc, extensions, gems, specifications)         minitest 5.10.3         rake 12.3.0         

  提示:到此ruby环境就安装好了,这里注意一点,yum安装的ruby版本有点低,所以我们这里选择编译安装;

  gem安装redis包

[root@node01 ruby-2.5.5]# gem install redisERROR: Loading command: install (LoadError)  cannot load such file -- zlibERROR: While executing gem ... (NoMethodError) undefined method `invoke_with_build_args' for nil:NilClass[root@node01 ruby-2.5.5]#

  提示:gem安装提示我们没有zlib,遇到这个问题我们用yum安装zlib-devel包,然后在把zlib集成到ruby环境即可;

[root@node01 ruby-2.5.5]# yum install zlib-develLoaded plugins: fastestmirrorLoading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.comResolving Dependencies--> Running transaction check---> Package zlib-devel.x86_64 0:1.2.7-18.el7 will be installed--> Processing Dependency: zlib = 1.2.7-18.el7 for package: zlib-devel-1.2.7-18.el7.x86_64--> Running transaction check---> Package zlib.x86_64 0:1.2.7-17.el7 will be updated---> Package zlib.x86_64 0:1.2.7-18.el7 will be an update--> Finished Dependency ResolutionDependencies Resolved=========================================================================================================== Package     Arch     Version      Repository   Size===========================================================================================================Installing: zlib-devel     x86_64     1.2.7-18.el7     base     50 kUpdating for dependencies: zlib      x86_64     1.2.7-18.el7     base     90 kTransaction Summary===========================================================================================================Install 1 PackageUpgrade    ( 1 Dependent package)Total download size: 140 kIs this ok [y/d/N]: yDownloading packages:Delta RPMs disabled because /usr/bin/applydeltarpm not installed.(1/2): zlib-devel-1.2.7-18.el7.x86_64.rpm           | 50 kB 00:00:00  /2): zlib-1.2.7-18.el7.x86_64.rpm             | 90 kB 00:00:00  -----------------------------------------------------------------------------------------------------------Total                  509 kB/s | 140 kB 00:00:00  Running transaction checkRunning transaction testTransaction test succeededRunning transaction Updating : zlib-1.2.7-18.el7.x86_64                1/3 Installing : zlib-devel-1.2.7-18.el7.x86_64               2/3 Cleanup : zlib-1.2.7-17.el7.x86_64                3/3 Verifying : zlib-devel-1.2.7-18.el7.x86_64               1/3 Verifying : zlib-1.2.7-18.el7.x86_64                2/3 Verifying : zlib-1.2.7-17.el7.x86_64                3/3 Installed: zlib-devel.x86_64 0:1.2.7-18.el7                   Dependency Updated: zlib.x86_64 0:1.2.7-18.el7                    Complete![root@node01 ruby-2.5.5]#

  集成zlib库到Ruby环境

[root@node01 ruby-2.5.5]# cd ext/zlib/[root@node01 zlib]# ruby extconf.rbchecking for deflateReset() in -lz... yeschecking for zlib.h... yeschecking for crc32_combine() in zlib.h... yeschecking for adler32_combine() in zlib.h... yeschecking for z_crc_t in zlib.h... yescreating Makefile[root@node01 zlib]# make make: *** No rule to make target `/include/ruby.h', needed by `zlib.o'. Stop.[root@node01 zlib]#

  提示:遇到这种问题,我们需要打开上面生成的Makefile 把zlib.o: $(top_srcdir)/include/ruby.h替换为zlib.o: ../../include/ruby.h即可

  提示:修改好Makefile,再次make

"Makefile" 282L, 8468C written[root@node01 zlib]# makecompiling zlib.clinking shared-object zlib.so[root@node01 zlib]# make install/usr/bin/install -c -m 0755 zlib.so /usr/local/lib/ruby/site_ruby/2.5.0/x86_64-linux[root@node01 zlib]#

  再次使用gem安装redis包

[root@node01 zlib]# gem install redisERROR: While executing gem ... (Gem::Exception) Unable to require openssl, install OpenSSL and rebuild Ruby (preferred) or use non-HTTPS sources[root@node01 zlib]#

  提示:它提示我们缺少openssl库,处理方法安装openssl-devel包,然后把openssl库集成到ruby环境中;

  安装openssl-devel包

[root@node01 zlib]# yum install openssl-develLoaded plugins: fastestmirrorLoading mirror speeds from cached hostfile * base: mirrors.aliyun.com * extras: mirrors.aliyun.com * updates: mirrors.aliyun.comResolving Dependencies--> Running transaction check---> Package openssl-devel.x86_64 1:1.0.2k-19.el7 will be installed--> Processing Dependency: openssl-libs(x86-64) = 1:1.0.2k-19.el7 for package: 1:openssl-devel-1.0.2k-19.el7.x86_64--> Processing Dependency: krb5-devel(x86-64) for package: 1:openssl-devel-1.0.2k-19.el7.x86_64--> Running transaction check---> Package krb5-devel.x86_64 0:1.15.1-46.el7 will be installed……省略部分内容……Installed: openssl-devel.x86_64 1:1.0.2k-19.el7                  Dependency Installed: keyutils-libs-devel.x86_64 0:1.5.8-3.el7    krb5-devel.x86_64 0:1.15.1-46.el7     libcom_err-devel.x86_64 0:1.42.9-17.el7     libkadm5.x86_64 0:1.15.1-46.el7     libselinux-devel.x86_64 0:2.5-15.el7     libsepol-devel.x86_64 0:2.5-10.el7     libverto-devel.x86_64 0:0.2.5-4.el7      pcre-devel.x86_64 0:8.32-17.el7     Dependency Updated: e2fsprogs.x86_64 0:1.42.9-17.el7     e2fsprogs-libs.x86_64 0:1.42.9-17.el7     krb5-libs.x86_64 0:1.15.1-46.el7     libcom_err.x86_64 0:1.42.9-17.el7      libselinux.x86_64 0:2.5-15.el7      libselinux-python.x86_64 0:2.5-15.el7     libselinux-utils.x86_64 0:2.5-15.el7    libsepol.x86_64 0:2.5-10.el7       libss.x86_64 0:1.42.9-17.el7      openssl.x86_64 1:1.0.2k-19.el7      openssl-libs.x86_64 1:1.0.2k-19.el7    Complete![root@node01 zlib]#

  集成openssl到ruby环境中

[root@node01 zlib]# cd ..[root@node01 ext]# lsbigdecimal  date  fcntl json  psych  ripper  Setup.nt -test-cgi    dbm  fiber nkf  pty  rubyvm  socket win32configure-ext.mk digest fiddle objspace racc  sdbm   stringio win32olecontinuation  etc  gdbm openssl rbconfig Setup   strscan zlibcoverage   extmk.rb io  pathname readline Setup.atheos syslog[root@node01 ext]# cd openssl/[root@node01 openssl]# lsdepend    ossl_asn1.c ossl_digest.c ossl_ocsp.c  ossl_pkey_rsa.c  ossl_x509ext.cdeprecation.rb  ossl_asn1.h ossl_digest.h ossl_ocsp.h  ossl_rand.c   ossl_x509.hextconf.rb   ossl_bio.c  ossl_engine.c ossl_pkcs12.c ossl_rand.h   ossl_x509name.cexts.mk   ossl_bio.h  ossl_engine.h ossl_pkcs12.h ossl_ssl.c   ossl_x509req.cHistory.md   ossl_bn.c  ossl.h   ossl_pkcs7.c  ossl_ssl.h   ossl_x509revoked.clib    ossl_bn.h  ossl_hmac.c  ossl_pkcs7.h  ossl_ssl_session.c ossl_x509store.cMakefile   ossl.c   ossl_hmac.h  ossl_pkey.c  ossl_version.h  ruby_missing.hmkmf.log   ossl_cipher.c ossl_kdf.c  ossl_pkey_dh.c ossl_x509attr.copenssl.gemspec ossl_cipher.h ossl_kdf.h  ossl_pkey_dsa.c ossl_x509.copenssl_missing.c ossl_config.c ossl_ns_spki.c ossl_pkey_ec.c ossl_x509cert.copenssl_missing.h ossl_config.h ossl_ns_spki.h ossl_pkey.h  ossl_x509crl.c[root@node01 openssl]# ruby extconf.rb checking for t_open() in -lnsl... nochecking for socket() in -lsocket... nochecking for openssl/ssl.h... yeschecking for OpenSSL version is 1.0.1 or later... yeschecking for RAND_egd()... yes……省略部分内容……checking for X509_get0_notBefore()... nochecking for SSL_SESSION_get_protocol_version()... nochecking for EVP_PBE_scrypt()... nocreating extconf.hcreating Makefile[root@node01 openssl]# makecompiling openssl_missing.cmake: *** No rule to make target `/include/ruby.h', needed by `ossl.o'. Stop.[root@node01 openssl]# 

  提示:这个错误和刚才继承zlib库到ruby环境中类似,处理方式同上面的处理方式一样修改Makefile文件,在makefile定义变量的位置增加top_srcdir = ../..

  再次make

Makefile" 1458L, 49182C written[root@node01 openssl]# makecompiling ossl.ccompiling ossl_asn1.ccompiling ossl_bio.ccompiling ossl_bn.ccompiling ossl_cipher.ccompiling ossl_config.ccompiling ossl_digest.ccompiling ossl_engine.ccompiling ossl_hmac.ccompiling ossl_kdf.ccompiling ossl_ns_spki.ccompiling ossl_ocsp.ccompiling ossl_pkcs12.ccompiling ossl_pkcs7.ccompiling ossl_pkey.ccompiling ossl_pkey_dh.ccompiling ossl_pkey_dsa.ccompiling ossl_pkey_ec.ccompiling ossl_pkey_rsa.ccompiling ossl_rand.ccompiling ossl_ssl.ccompiling ossl_ssl_session.ccompiling ossl_x509.ccompiling ossl_x509attr.ccompiling ossl_x509cert.ccompiling ossl_x509crl.ccompiling ossl_x509ext.ccompiling ossl_x509name.ccompiling ossl_x509req.ccompiling ossl_x509revoked.ccompiling ossl_x509store.clinking shared-object openssl.so[root@node01 openssl]# make install/usr/bin/install -c -m 0755 openssl.so /usr/local/lib/ruby/site_ruby/2.5.0/x86_64-linuxinstalling default openssl libraries[root@node01 openssl]# 

  再次用gem安装redis包

[root@node01 openssl]# gem install redisFetching: redis-4.2.1.gem (100%)Successfully installed redis-4.2.1Parsing documentation for redis-4.2.1Installing ri documentation for redis-4.2.1Done installing documentation for redis after 0 seconds1 gem installed[root@node01 openssl]# 

  提示:到此ruby环境就准备好了,redis包也安装好了,接下来我们把redis-trib.rb连接到/usr/bin/下,然后执行redis-trib.rb,如果能够正常执行说明ruby环境没有问题,如果不能则还需要调整

  软连接redis-trib.rb到/usr/bin/

[root@node01 openssl]# find /usr/local/src/redis-4.0.9/ -name "redis-trib.rb"/usr/local/src/redis-4.0.9/src/redis-trib.rb[root@node01 openssl]# ln -s /usr/local/src/redis-4.0.9/src/redis-trib.rb /usr/bin/[root@node01 openssl]# redis-trib.rb Usage: redis-trib <command> <options> <arguments ...> create   host1:port1 ... hostN:portN     --replicas <arg> check   host:port info   host:port fix    host:port     --timeout <arg> reshard   host:port     --from <arg>     --to <arg>     --slots <arg>     --yes     --timeout <arg>     --pipeline <arg> rebalance  host:port     --weight <arg>     --auto-weights     --use-empty-masters     --timeout <arg>     --simulate     --pipeline <arg>     --threshold <arg> add-node  new_host:new_port existing_host:existing_port     --slave     --master-id <arg> del-node  host:port node_id set-timeout  host:port milliseconds call   host:port command arg arg .. arg import   host:port     --from <arg>     --copy     --replace help   (show this help)For check, fix, reshard, del-node, set-timeout you can specify the host and port of any working node in the cluster.[root@node01 openssl]#

  提示:可以看到redis-trib.rb可以正常打印出帮助信息,说明当前ruby环境能够支持redis-trib.rb运行;

  添加密码到刚才下载的redis包的库文件中

  提示:这里需要把对应密码字符串用引号把它引起来;

  创建集群

[root@node01 ~]# redis-trib.rb create --replicas 1 192.168.0.41:6379 192.168.0.42:6379 192.168.0.43:6379 192.168.0.41:6380 192.168.0.42:6380 192.168.0.43:6380>>> Creating cluster>>> Performing hash slots allocation on 6 nodes...Using 3 masters:192.168.0.41:6379192.168.0.42:6379192.168.0.43:6379Adding replica 192.168.0.42:6380 to 192.168.0.41:6379Adding replica 192.168.0.43:6380 to 192.168.0.42:6379Adding replica 192.168.0.41:6380 to 192.168.0.43:6379M: 8c785e6ec3f8f7ff4fb7768765da8b8a93f26855 192.168.0.41:6379 slots:0-5460 (5461 slots) masterM: e99b0b450e78719d63520cb6efc068d5e8d4d081 192.168.0.42:6379 slots:5461-10922 (5462 slots) masterM: a7ace08c36f7d55c4f28463d72865aa1ff74829e 192.168.0.43:6379 slots:10923-16383 (5461 slots) masterS: 62ece0b80b83c0f1f078b07fc1687bb8376f76b3 192.168.0.41:6380 replicates a7ace08c36f7d55c4f28463d72865aa1ff74829eS: dbfff4c49a94c0ee55d14401ccc9245af3655427 192.168.0.42:6380 replicates 8c785e6ec3f8f7ff4fb7768765da8b8a93f26855S: 91169e71359deed96f8778cf31c823dbd6ded350 192.168.0.43:6380 replicates e99b0b450e78719d63520cb6efc068d5e8d4d081Can I set the above configuration? (type 'yes' to accept): yes>>> Nodes configuration updated>>> Assign a different config epoch to each node>>> Sending CLUSTER MEET messages to join the clusterWaiting for the cluster to join......>>> Performing Cluster Check (using node 192.168.0.41:6379)M: 8c785e6ec3f8f7ff4fb7768765da8b8a93f26855 192.168.0.41:6379 slots:0-5460 (5461 slots) master 1 additional replica(s)S: 91169e71359deed96f8778cf31c823dbd6ded350 192.168.0.43:6380 slots: (0 slots) slave replicates e99b0b450e78719d63520cb6efc068d5e8d4d081M: e99b0b450e78719d63520cb6efc068d5e8d4d081 192.168.0.42:6379 slots:5461-10922 (5462 slots) master 1 additional replica(s)M: a7ace08c36f7d55c4f28463d72865aa1ff74829e 192.168.0.43:6379 slots:10923-16383 (5461 slots) master 1 additional replica(s)S: dbfff4c49a94c0ee55d14401ccc9245af3655427 192.168.0.42:6380 slots: (0 slots) slave replicates 8c785e6ec3f8f7ff4fb7768765da8b8a93f26855S: 62ece0b80b83c0f1f078b07fc1687bb8376f76b3 192.168.0.41:6380 slots: (0 slots) slave replicates a7ace08c36f7d55c4f28463d72865aa1ff74829e[OK] All nodes agree about slots configuration.>>> Check for open slots...>>> Check slots coverage...[OK] All 16384 slots covered.[root@node01 ~]# 

  提示:从上面的信息可以了解到,我们给指定了6个redis实例的地址和端口,它给我们创建了3个master3个slave,并且在3个master上平均分配了16384个槽位;如果能看到后面的ok all 16384 slots covered,说明集群创建成功;--replicates用来指定集群中每个master的副本数量,1表示每个master有一个副本;

  查看集群状态信息

[root@node01 ~]# redis-trib.rb info 192.168.0.41:6379     192.168.0.41:6379 (8c785e6e...) -> 0 keys | 5461 slots | 1 slaves.192.168.0.42:6379 (e99b0b45...) -> 0 keys | 5462 slots | 1 slaves.192.168.0.43:6379 (a7ace08c...) -> 0 keys | 5461 slots | 1 slaves.[OK] 0 keys in 3 masters.0.00 keys per slot on average.[root@node01 ~]# redis-trib.rb check 192.168.0.41:6379  >>> Performing Cluster Check (using node 192.168.0.41:6379)M: 8c785e6ec3f8f7ff4fb7768765da8b8a93f26855 192.168.0.41:6379 slots:0-5460 (5461 slots) master 1 additional replica(s)S: 91169e71359deed96f8778cf31c823dbd6ded350 192.168.0.43:6380 slots: (0 slots) slave replicates e99b0b450e78719d63520cb6efc068d5e8d4d081M: e99b0b450e78719d63520cb6efc068d5e8d4d081 192.168.0.42:6379 slots:5461-10922 (5462 slots) master 1 additional replica(s)M: a7ace08c36f7d55c4f28463d72865aa1ff74829e 192.168.0.43:6379 slots:10923-16383 (5461 slots) master 1 additional replica(s)S: dbfff4c49a94c0ee55d14401ccc9245af3655427 192.168.0.42:6380 slots: (0 slots) slave replicates 8c785e6ec3f8f7ff4fb7768765da8b8a93f26855S: 62ece0b80b83c0f1f078b07fc1687bb8376f76b3 192.168.0.41:6380 slots: (0 slots) slave replicates a7ace08c36f7d55c4f28463d72865aa1ff74829e[OK] All nodes agree about slots configuration.>>> Check for open slots...>>> Check slots coverage...[OK] All 16384 slots covered.[root@node01 ~]# 

  提示:从上面的信息可以看到192.168.0.41/42/43的6379端口都是master,并且都有一个slave节点;

  确认master状态

[root@node01 ~]# redis-cli -h 192.168.0.41192.168.0.41:6379> AUTH adminOK192.168.0.41:6379> CLUSTER INFOcluster_state:okcluster_slots_assigned:16384cluster_slots_ok:16384cluster_slots_pfail:0cluster_slots_fail:0cluster_known_nodes:6cluster_size:3cluster_current_epoch:6cluster_my_epoch:1cluster_stats_messages_ping_sent:606cluster_stats_messages_pong_sent:586cluster_stats_messages_sent:1192cluster_stats_messages_ping_received:581cluster_stats_messages_pong_received:606cluster_stats_messages_meet_received:5cluster_stats_messages_received:1192192.168.0.41:6379> info replication# Replicationrole:masterconnected_slaves:1slave0:ip=192.168.0.42,port=6380,state=online,offset=854,lag=0master_replid:303943ae1e9f54e3525abda2fd1efff905bf08e3master_replid2:0000000000000000000000000000000000000000master_repl_offset:854second_repl_offset:-1repl_backlog_active:1repl_backlog_size:1048576repl_backlog_first_byte_offset:1repl_backlog_histlen:854192.168.0.41:6379> quit[root@node01 ~]# redis-cli -h 192.168.0.42192.168.0.42:6379> auth adminOK192.168.0.42:6379> info replication# Replicationrole:masterconnected_slaves:1slave0:ip=192.168.0.43,port=6380,state=online,offset=896,lag=0master_replid:f4e1b75b8f30be622868814fc4618a4584f3aa32master_replid2:0000000000000000000000000000000000000000master_repl_offset:896second_repl_offset:-1repl_backlog_active:1repl_backlog_size:1048576repl_backlog_first_byte_offset:1repl_backlog_histlen:896192.168.0.42:6379> CLUSTER INFOcluster_state:okcluster_slots_assigned:16384cluster_slots_ok:16384cluster_slots_pfail:0cluster_slots_fail:0cluster_known_nodes:6cluster_size:3cluster_current_epoch:6cluster_my_epoch:2cluster_stats_messages_ping_sent:662cluster_stats_messages_pong_sent:645cluster_stats_messages_meet_sent:5cluster_stats_messages_sent:1312cluster_stats_messages_ping_received:644cluster_stats_messages_pong_received:667cluster_stats_messages_meet_received:1cluster_stats_messages_received:1312192.168.0.42:6379> quit[root@node01 ~]# redis-cli -h 192.168.0.43192.168.0.43:6379> AUTH adminOK192.168.0.43:6379> CLUSTER INFOcluster_state:okcluster_slots_assigned:16384cluster_slots_ok:16384cluster_slots_pfail:0cluster_slots_fail:0cluster_known_nodes:6cluster_size:3cluster_current_epoch:6cluster_my_epoch:3cluster_stats_messages_ping_sent:683cluster_stats_messages_pong_sent:787cluster_stats_messages_meet_sent:4cluster_stats_messages_sent:1474cluster_stats_messages_ping_received:786cluster_stats_messages_pong_received:687cluster_stats_messages_meet_received:1cluster_stats_messages_received:1474192.168.0.43:6379> info replication# Replicationrole:masterconnected_slaves:1slave0:ip=192.168.0.41,port=6380,state=online,offset=952,lag=1master_replid:7cbc7d52452717cd49a0e543f6efdccafe5fe1bdmaster_replid2:0000000000000000000000000000000000000000master_repl_offset:952second_repl_offset:-1repl_backlog_active:1repl_backlog_size:1048576repl_backlog_first_byte_offset:1repl_backlog_histlen:952192.168.0.43:6379> quit[root@node01 ~]# 

  提示:可以看到集群状态都是正常ok的,并且41的6379对应42的6380,42的6379对应43的6380,43的6379对应41的6380,刚好master和slave都是错开的;

  查看集群各node对应关系

  提示:上面的node对应是依据id号来的。到此redis集群就正常的跑起来了;

  验证:在master上任意写入一个数据看看会发生什么?

  提示:在master01上写入k1,它提示我们到192.168.0.43:6379上去写;这是因为我们写入k1计算的槽位是12706,这个槽位是在master03上,所以在master01上就不能正常的写入;这同时也告诉我们,要使用redis集群,客户端还必须能够智能去理解redis协议,通过集群返回的信息,把对应请求调度到对应master上执行;

  验证读请求

  提示:从上面的信息可以确定一点,这三个主节点他们互相数据是完全隔离的,也就说master01不知道master02上到底有没有对应的数据,它只知道应该去master02上去查看才知道;这就是我们之前说的,每个节点拥有整个集群的全部状态信息(元数据),而每个节点只拥有部分数据在本节点;

  验证:把任意一个master节点宕机,看看对应slave是否能够被提升为master,接管master呢?

  提示:可以看到当master01宕机以后,对应slave就被提升为master;

  查看master01的配置文件是否有改动?

[root@node01 ~]# cat /usr/local/redis/redis-cluster_6379.conf 91169e71359deed96f8778cf31c823dbd6ded350 192.168.0.43:6380@16380 master - 0 1596648345000 8 connected 5461-10922dbfff4c49a94c0ee55d14401ccc9245af3655427 192.168.0.42:6380@16380 slave 8c785e6ec3f8f7ff4fb7768765da8b8a93f26855 0 1596648345510 9 connected62ece0b80b83c0f1f078b07fc1687bb8376f76b3 192.168.0.41:6380@16380 slave a7ace08c36f7d55c4f28463d72865aa1ff74829e 0 1596648343893 4 connectede99b0b450e78719d63520cb6efc068d5e8d4d081 192.168.0.42:6379@16379 slave 91169e71359deed96f8778cf31c823dbd6ded350 0 1596648344903 8 connected8c785e6ec3f8f7ff4fb7768765da8b8a93f26855 192.168.0.41:6379@16379 myself,master - 0 1596648345000 9 connected 0-5460a7ace08c36f7d55c4f28463d72865aa1ff74829e 192.168.0.43:6379@16379 master - 0 1596648345000 3 connected 10923-16383vars currentEpoch 9 lastVoteEpoch 0[root@node01 ~]# ps -ef |grep redisroot  15980  1 0 Aug05 ?  00:00:17 redis-server 0.0.0.0:6380 [cluster]root  30867  1 0 01:19 ?  00:00:02 redis-server 0.0.0.0:6379 [cluster]root  30949 14690 0 01:30 pts/0 00:00:00 grep --color=auto redis[root@node01 ~]# redis-cli -a admin127.0.0.1:6379> info replication# Replicationrole:masterconnected_slaves:1slave0:ip=192.168.0.42,port=6380,state=online,offset=4476,lag=0master_replid:ed690ba2766e6e334ffc298fc6aba8be80465aa8master_replid2:776b5f996d78171e7c5ab5419d65c6c14a253bdamaster_repl_offset:4476second_repl_offset:4071repl_backlog_active:1repl_backlog_size:1048576repl_backlog_first_byte_offset:3595repl_backlog_histlen:882127.0.0.1:6379> quit[root@node01 ~]# kill -9 30867[root@node01 ~]# redis-cli -h 192.168.0.42 -p 6380192.168.0.42:6380> AUTH adminOK192.168.0.42:6380> info replication# Replicationrole:masterconnected_slaves:0master_replid:d46c0a3761b61352b1fc75986ab3343e04a89b83master_replid2:ed690ba2766e6e334ffc298fc6aba8be80465aa8master_repl_offset:4518second_repl_offset:4519repl_backlog_active:1repl_backlog_size:1048576repl_backlog_first_byte_offset:4071repl_backlog_histlen:448192.168.0.42:6380> quit[root@node01 ~]# cat /usr/local/redis/redis-cluster_6379.conf91169e71359deed96f8778cf31c823dbd6ded350 192.168.0.43:6380@16380 master - 0 1596648345000 8 connected 5461-10922dbfff4c49a94c0ee55d14401ccc9245af3655427 192.168.0.42:6380@16380 slave 8c785e6ec3f8f7ff4fb7768765da8b8a93f26855 0 1596648345510 9 connected62ece0b80b83c0f1f078b07fc1687bb8376f76b3 192.168.0.41:6380@16380 slave a7ace08c36f7d55c4f28463d72865aa1ff74829e 0 1596648343893 4 connectede99b0b450e78719d63520cb6efc068d5e8d4d081 192.168.0.42:6379@16379 slave 91169e71359deed96f8778cf31c823dbd6ded350 0 1596648344903 8 connected8c785e6ec3f8f7ff4fb7768765da8b8a93f26855 192.168.0.41:6379@16379 myself,master - 0 1596648345000 9 connected 0-5460a7ace08c36f7d55c4f28463d72865aa1ff74829e 192.168.0.43:6379@16379 master - 0 1596648345000 3 connected 10923-16383vars currentEpoch 9 lastVoteEpoch 0[root@node01 ~]# grep slaveof /usr/local/redis/6379/etc/redis.conf # Master-Slave replication. Use slaveof to make a Redis instance a copy of# slaveof <masterip> <masterport>[root@node01 ~]# 

  提示:从上面的信息可以看到,在master01没有宕机前和宕机以后slave被提升为master后,对应的配置信息都没有发生变化;我上面为了验证配置文件是否发生变化,先是把master01上线,然后把对应slave下线在上线,还原master01为master角色;

  重新把master上线,看看它的配置文件有什么变化?角色还会是master吗?

[root@node01 ~]# redis-server /usr/local/redis/6379/etc/redis.conf  [root@node01 ~]# ss -tnlState  Recv-Q Send-Q          Local Address:Port              Peer Address:Port    LISTEN  0  511              *:6379                 *:*     LISTEN  0  511              *:6380                 *:*     LISTEN  0  128              *:22                 *:*     LISTEN  0  100            127.0.0.1:25                 *:*     LISTEN  0  511              *:16379                 *:*     LISTEN  0  511              *:16380                 *:*     LISTEN  0  128              :::22                 :::*     LISTEN  0  100              ::1:25                 :::*     [root@node01 ~]# cat /usr/local/redis/redis-cluster_6379.conf 91169e71359deed96f8778cf31c823dbd6ded350 192.168.0.43:6380@16380 master - 1596649311329 1596649311326 8 connected 5461-109228c785e6ec3f8f7ff4fb7768765da8b8a93f26855 192.168.0.41:6379@16379 myself,slave dbfff4c49a94c0ee55d14401ccc9245af3655427 0 1596649311326 9 connectede99b0b450e78719d63520cb6efc068d5e8d4d081 192.168.0.42:6379@16379 slave 91169e71359deed96f8778cf31c823dbd6ded350 0 1596649311331 8 connecteda7ace08c36f7d55c4f28463d72865aa1ff74829e 192.168.0.43:6379@16379 master - 0 1596649311332 3 connected 10923-1638362ece0b80b83c0f1f078b07fc1687bb8376f76b3 192.168.0.41:6380@16380 slave a7ace08c36f7d55c4f28463d72865aa1ff74829e 0 1596649311331 4 connecteddbfff4c49a94c0ee55d14401ccc9245af3655427 192.168.0.42:6380@16380 master - 0 1596649311332 10 connected 0-5460vars currentEpoch 10 lastVoteEpoch 0[root@node01 ~]# redis-cli 127.0.0.1:6379> AUTH adminOK127.0.0.1:6379> info replication# Replicationrole:slavemaster_host:192.168.0.42master_port:6380master_link_status:upmaster_last_io_seconds_ago:5master_sync_in_progress:0slave_repl_offset:4686slave_priority:100slave_read_only:1connected_slaves:0master_replid:d46c0a3761b61352b1fc75986ab3343e04a89b83master_replid2:0000000000000000000000000000000000000000master_repl_offset:4686second_repl_offset:-1repl_backlog_active:1repl_backlog_size:1048576repl_backlog_first_byte_offset:4519repl_backlog_histlen:168127.0.0.1:6379> 

  提示:可以看到,重新把master01上线以后,它自动降级为slave了,并且配置文件也从master需改成slave了;这应该是redis cluster在检测到192.168.0.41:6379上线后,触发把内存中的配置同步到对应文件,使得对应配置文件也被修改了;到此一个redis cluster 就搭建测试完成了;

Redis服务之Redis Clustere邮包贸发局订单处理2018年欧洲电商市场将达6020亿欧元,西欧成欧洲电商主战场!多国欲出"数字税",亚马逊等互联网巨头将遭重击?日本站点问题不断,更有专业卖家账户惨遭降级!去沙头角购物要不要办证?西冲烧烤场地费需要多少钱?锦绣中华春节期间有什么活动?

No comments:

Post a Comment