Error connection reset by peer redis

Similar problem here, with python 2.7.14.

@venthur @azdkj532 Are you running under Kubernetes?

Hi, sorry I haven’t responded to this issue earlier. Upgrading to version 3.3.x should improve your experience. 3.3.x added a new option health_check_interval. You can use health checks like:

client = redis.Redis(..., health_check_interval=30)

The value of health_check_interval specifies the time in seconds a connection can be idle before its health needs to be checked. In the above case, if the connection is idle for more than 30 seconds, a round trip PING/PONG will be attempted just before your next command. If the PING/PONG fails, the connection will reestablished behind the scenes before your intended command is executed.

I have this problem since 3.0 got released…

@dejlek upgrade to the latest 3.3.x and turn on health checks.

@dejlek for now the health checks are opt in.

We are currently testing GCP MemoryStore as a replacement for memcache.

Using GAE Python 2.7, we get random «ConnectionError: Error while reading from socket: (104, ‘Connection reset by peer’)». Typically this error occurs after about 120 seconds of waiting for some redis command to complete (delete, set, get), so they can be hard to handle with a backoff mechanism.

We use version 3.3.8. I’ve been testing using health_check_interval=30 and lower values than 30 (down to 2, currently). This seems to have made the errors less frequent, but they still occur often enough to be of concern.

Perhaps this is a purely MemoryStore/redis server issue, however.

I have the same issue as @ulodciv. I’m considering whether it has anything to do with the beta VPC needed to talk to MemoryStore. I tried the same setup with Redis running on a normal Container and the same thing happens.

Hi @andymccurdy sorry, for the late reply. We’re testing it now, it might take some time to say with confidence if it worked. But looking at the code I’m quite positive.

I am also running with GCP MemoryStore, standard environment with a vpc access connector, and I get seemingly random 104 Connection reset errors (a string of 4 or 5 within a few seconds, about once a day). I’ve tried the health_check_interval fix, tcp_keepalive, nothing seems to work. What does seem to work is switching to the flex environment (so dropping the vpc access connector) or dropping back down to redis-py 3.0.1 (so the socket reconnect in redis-py still works). I can’t readily put in reconnection logic into my code because the cookie management library I’m using (flask-session) uses redis. I’d really be happy if the socket-reconnect logic got put back into redis-py, if only as an option.

@gimmeao It sounds like something in the VPC access connector environment is severing your connections. We’d need more info to figure out what’s going on there.

What value are you supplying to health_check_interval?

It looks like the VPC access connector times out idle connections, but this is configurable. You should make sure that health_check_interval is less than the idle timeout value in your environment.

The socket reconnect logic was a bug and can lead to data loss/corruption. There’s no plan to reintroduce that. However, if that’s really what you want to do, you could subclass the Connection class and do the retry yourself, assuming that you can supply connection options to the library that’s using redis-py.

I tried 30 and 15 for health_check_interval. I’m not sure that the timeout
is configurable on Google’s side, at least if it is I haven’t found where
and I’ve looked quite hard. In any case I’m pretty much just using redis as
a database cache that I rarely write to, so I think I can live with the
potential issues. Could be it’s a bug on G’s side, I’ll probably try again
in a few months and see if anything has changed. Thanks for the response.

…

Closing this as 3.3.x+ has provided options to resolve this issue. It seems like the only outstanding issue is specific to Google Cloud.

If I’ve missed something please open a new issue.

The socket reconnect logic was a bug and can lead to data loss/corruption.

Is that just because commands might get executed more than once if the connection is severed after the server received the command? Or is there something more insidious that I should be aware of if I plan to implement my own retry logic for idempotent commands?

@bmerry Just the duplicate command execution. The severity depends on the command. Retrying a SET is probably not a problem as the key just gets overwritten with the same value. Other commands, such as all the list mutation commands, are much more problematic.

Thanks. Our application uses only idempotent commands (SETNX, HSETNX, ZADD) so I’ll look at implementing retries at the application level.

Do you have any recommendations for application-level retry logic that ensures that a fresh connection will be attempted? I’m worrying about the case where I have a pool with lots of connections and something in the network severs all of them. If I just retry N times (fixed N, which is less than the size of the pool) on the redis.Redis object, it could presumably just cycle through the dead pool connections without ever trying to establish a new one?

It’s easier than that. If you get a TimeoutError or ConnectionError, you can simply call .disconnect() followed by .connect() on the connection that raised the error. You could do this by subclassing Redis.execute_command and putting your retry logic there.

Thanks, I think I understand. I’ll try that if health_check_interval doesn’t fix my issues — still not sure what causes connections to drop in the first place since we’re running a bare-metal network and nothing should be monkeying around at the TCP layer.

Related to GCP issues, it appears to have gone away after I changed to Redis version 5 inside MemoryStore. I have only been testing it for a day on version 5, but so far I haven’t seen this error again and I used to see it all the time.

For all of you who had issues with GCP, are you guys still facing this problem? We have health check intervals setup and are on Redis 5 — not sure what else to try. We keep getting hundreds of these errors each day.

For all of you who had issues with GCP, are you guys still facing this problem? We have health check intervals setup and are on Redis 5 — not sure what else to try. We keep getting hundreds of these errors each day.

I haven’t had any more issues, I’m running redis v5 and have it running in the same region as my app.

I no longer have the issue, I made a few changes. Not totally sure that my changes are what fixed it but it’s possible. In my main library’s init I added a connect_to_cache method:

def connect_to_cache():
    global r
    r = redis.Redis(
            host=redis_host, port=redis_port, health_check_interval=10,
            socket_timeout=10, socket_keepalive=True,
            socket_connect_timeout=10, retry_on_timeout=True
            )

and I added a route for GCP’s warmup request:

@app.route('/_ah/warmup/', methods=['GET'])
def warmup():
    shfl.connect_to_cache()
    return ('OK', 200)

@tastypackets Yes, every few minutes this error appears, the config changes suggested here don’t do the trick for me.

I’ve thrown the kitchen sink at this problem with an application I’m working on. It has not yet seen enough action to conclude one way or other, but I am posting it here on off chance it may help someone. The big point is configuring keepalive options beyond the boolean socket_keepalive. In particular, the default value for TCP_KEEPIDLE is 7200 seconds (my understanding is that there is an RFC specifying 2 hours as the minimum acceptable default for TCP).

            if platform == "linux":
                ka_options = {
                    socket.TCP_KEEPIDLE: 10,
                    socket.TCP_KEEPINTVL: 5,
                    socket.TCP_KEEPCNT: 5
                }
            elif platform == "darwin":
                # python 3.10 will support socket.TCP_KEEPALIVE on MacOS which
                # is direct analog to TCP_KEEPIDLE on Linux
                ka_options = {socket.TCP_KEEPINTVL: 5, socket.TCP_KEEPCNT: 5}
            else:
                ka_options = {}

            connection = redis.StrictRedis(host=Config.REDIS_HOST,
                                           port=Config.REDIS_PORT,
                                           health_check_interval=15,
                                           socket_keepalive=True,
                                           socket_keepalive_options=ka_options)

I’m able to reproduce this scenario with following setup.

import json
import redis
import time

from redis.retry import Retry
from redis.exceptions import (TimeoutError, ConnectionError)
from redis.backoff import ExponentialBackoff, NoBackoff

"""
Timeout Error Investigation
"""
redis_client = redis.StrictRedis(
    retry=Retry(NoBackoff(), 5), retry_on_timeout=True,
    health_check_interval=5,
    retry_on_error=[TimeoutError, ConnectionError]
)

# reproduce error with connection reset
print(redis_client.execute_command('CONFIG SET timeout 10'))
print(redis_client.set("spock", "vulcan"))
print(redis_client.get("spock"))
print(redis_client.execute_command('RESET'))
time.sleep(121)
try:
    print(redis_client.set("spock", "vulcan"))
except ConnectionResetError:
    # it will be raised
    print("Connection reset")
# this time it will be successful
print(redis_client.set("spock", "vulcan"))

configuration

redis server - 6.2.6
python redis - 4.1.0

UPDATE: I’ve updated my sample code that reproduces connect reset by peer error.

Hey guys, there is no fix for the issue?

I am also on GCP and also having random issues. I am using django_cache which is using redis-py under the hood. I am providing socket_keepalive and health_check_interval but still having mentioned issue from time to time, at least once a day.

If you are using NDB to interact with the Datastore, make sure to pass to NDB the Redis client you constructed yourself with the above suggested parameters instead of using ndb.RedisCache.from_environment()
This was what solved this problem for us.

Hello guys, after debugging the code with TCP dump and ipdb inside python I finally discovered why the provided here solutions were not working or working but not always.

The next portion of discussion will be related to the people who are using django-redis as their CACHE backend configuration and not redis-py directly.

The problem appeared in the configuration of the RedisClient, as we should setup the configuration for the CONNECTION_POOL_KWARGS and not for REDIS_CLIENT_KWARGS.

Currently we are having hte following configuration and health-check with PING-PONG commands is working correctly.

CACHES = {
    'default': {
        ...
        'OPTIONS': {
            'CLIENT_CLASS': 'django_redis.client.DefaultClient',
            'SOCKET_CONNECT_TIMEOUT': 10,
            'SOCKET_TIMEOUT': 60,
            'CONNECTION_POOL_KWARGS': {  # <-- note that here we are using the kwargs for connection pool
                'socket_keepalive': True,
                'health_check_interval': 30,  # <-- interval for the health-check if the last command executed longer that 30 seconds ago
                'retry_on_timeout': True,  # <-- perform retry on timeout (just in case)
                'retry': Retry(FullJitterBackoff(cap=5, base=1), 5),  # <-- which retry logic to apply, if you will not specify it, you will have 0 retries, meaning that if `PING-PONG` query is failed, it will hard fail instantly and you will see the traceback of the health-check command
            },
        },
    }
}

Not everything is working and retried for 5 times in case of ConnectionError or TimeoutError. Also if you need to handle special exceptions, you can specify them using special kwarg.

Is there any way to reproduce Connection reset by peer?
client kill not working at my environment
os: mac, redis on docker