Notes on Redis Operation

Some common problems and solution to containerized Redis deployment.

· 1 min read

Background saving terminated by signal 9

If Redis fails due to being killed:

10866:M 05 Sep 20:15:19.711 # Background saving terminated by signal 9
10866:M 05 Sep 20:18:43.898 # Background saving terminated by signal 9
10866:M 05 Sep 20:26:46.434 # Background saving terminated by signal 9
10866:M 05 Sep 20:34:49.161 # Background saving terminated by signal 9
10866:M 05 Sep 20:42:52.406 # Background saving terminated by signal 9
10866:M 05 Sep 20:42:55.332 # Background saving terminated by signal 9

It is typically caused by OOM. Log in the host and check the debug messages

# dmesg -T | grep redis
....
[三 12月 9 21:43:16 2020] redis-server invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=872
[三 12月 9 21:43:16 2020] redis-server cpuset=docker-c008dd75edd4484901b4192cb9379d69129ca00804a7214a991783aa284b807b.scope
mems_allowed=0
[三 12月 9 21:43:16 2020] CPU: 0 PID: 59217 Comm: redis-server Kdump: loaded Tainted: G ------------ T 3.10.0-
957.10.1.el7.x86_64 #1
[三 12月 9 21:43:17 2020] [27930] 1001 27930 2262690 2079645 4234 0 872 redis-server
[三 12月 9 21:43:17 2020] [59217] 1001 59217 2337478 2092550 4261 0 872 redis-server
[三 12月 9 21:43:17 2020] Memory cgroup out of memory: Kill process 59217 (redis-server) score 1871 or sacrifice child
[三 12月 9 21:43:17 2020] Killed process 59217 (redis-server) total-vm:9349912kB, anon-rss:8369816kB, file-rss:384kB, shmem-
rss:0kB

Notice the "Memory cgroup out of memory", it could be inferred that the memory limit set for the Redis container is too low. Increasing the limit will solve the problem.

Overcoming output buffer limit

If Redis fails to initialize with log lines:

5101:M 20 Feb 18:14:29.130 # Client id=4500196 addr=71.459.815.760:43872 fd=533 name= age=127 idle=127 flags=S db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=13997 oll=1227 omem=192281275 events=rw cmd=psync scheduled to be closed ASAP for overcoming of output buffer limits.

This means that slave buffer is not enough and you should increase it (at master):

redis-cli -a $REDIS_PASSWORD config set client-output-buffer-limit "slave 836870912 836870912 0"

More information: https://redis.io/topics/clients