Haproxy socket error connection reset by peer

Hi.

New occurrence of this problem today. We have recover using the same procedure (pause, restart, resume).

This time we were running patroni in debug. We don’t see anything unusual except from:

• INFO: Lock owner: patroni-48-node-[1-2]; I am patroni-48-node-[1-2]
• INFO: no action.
• DEBUG: Sending request
• DEBUG: Received response

Why do we say that we don’t have anything unusual? The outage was at 3:15 PM. Using journalctl we have extract a subset of log

patroni-48-node-1 ~# journalctl -u patroni | grep 'Nov 19 15:' > patroni_freeze_20181119

And we proceed by filtering the resulting file.

Here is an example of Sending request/Received response lines

Nov 19 15:00:49 patroni-48-node-1 patroni[11424]: 2018-11-19 15:00:49,732 DEBUG: Sending request(xid=55709): SetData(path='/patroni/cluster48/optime/leader', data=b'6893385174560', version=-1)
Nov 19 15:00:49 patroni-48-node-1 patroni[11424]: 2018-11-19 15:00:49,734 DEBUG: Received response(xid=55709): ZnodeStat(czxid=17198457179, mzxid=17224239511, ctime=1535978271044, mtime=1542636049735, version=658873, cversion=0, aversion=0, ephemeralOwner=0, dataLength=13, numChildren=0, pzxid=17198457179)
--8<--
Nov 19 15:01:09 patroni-48-node-1 patroni[11424]: 2018-11-19 15:01:09,734 DEBUG: Sending request(xid=55714): SetData(path='/patroni/cluster48/members/patroni-48-node-1', data=b'{"api_url":"https://x.x.x.x:8008/patroni","conn_url":"postgres://x.x.x.x:5432/postgres","xlog_location":6893463973832,"state":"running","timeline":1,"role":"master"}', version=-1)
Nov 19 15:01:09 patroni-48-node-1 patroni[11424]: 2018-11-19 15:01:09,737 DEBUG: Received response(xid=55714): ZnodeStat(czxid=17223154622, mzxid=17224239591, ctime=1542358192025, mtime=1542636069738, version=27797, cversion=0, aversion=0, ephemeralOwner=99873831030490191, dataLength=173, numChildren=0, pzxid=17223154622)

The remaining lines are our probres (specially the one check the validity of the ssl cert) and the logs of patroni pause/restart/resume.

Nov 19 15:07:01 patroni-48-node-1 patroni[11424]: 2018-11-19 15:07:01,438 DEBUG: API thread: 127.0.0.1 - - [19/Nov/2018 15:07:01] "GET / HTTP/1.1" 200 -
--8<--
Nov 19 15:24:58 patroni-48-node-1 patroni[11424]: 2018-11-19 15:24:58,257 DEBUG: Received EVENT: Watch(type=3, state=3, path='/patroni/cluster48/config')
Nov 19 15:24:58 patroni-48-node-1 patroni[11424]: 2018-11-19 15:24:58,270 INFO: PAUSE: no action.  i am the leader with the lock
Nov 19 15:24:58 patroni-48-node-1 patroni[11424]: 2018-11-19 15:24:58,273 INFO: Changed DateStyle from ISO, MDY to iso, mdy
Nov 19 15:24:58 patroni-48-node-1 patroni[11424]: 2018-11-19 15:24:58,273 INFO: PostgreSQL configuration items changed, reloading configuration.
Nov 19 15:24:58 patroni-48-node-1 patroni[11424]: server signaled
Nov 19 15:25:08 patroni-48-node-1 patroni[11424]: 2018-11-19 15:25:08,267 INFO: PAUSE: no action.  i am the leader with the lock
Nov 19 15:25:18 patroni-48-node-1 patroni[11424]: 2018-11-19 15:25:18,264 INFO: PAUSE: no action.  i am the leader with the lock
Nov 19 15:25:18 patroni-48-node-1 systemd[1]: Stopping Runners to orchestrate a high-availability PostgreSQL...
Nov 19 15:25:26 patroni-48-node-1 patroni[11424]: 2018-11-19 15:25:26,948 DEBUG: Received EVENT: Watch(type=4, state=3, path='/patroni/cluster48/members')
Nov 19 15:25:48 patroni-48-node-1 systemd[1]: patroni.service: State 'stop-sigterm' timed out. Killing.
Nov 19 15:25:48 patroni-48-node-1 systemd[1]: patroni.service: Killing process 11424 (patroni) with signal SIGKILL.
Nov 19 15:25:48 patroni-48-node-1 systemd[1]: patroni.service: Main process exited, code=killed, status=9/KILL
Nov 19 15:25:48 patroni-48-node-1 systemd[1]: Stopped Runners to orchestrate a high-availability PostgreSQL.
Nov 19 15:25:48 patroni-48-node-1 systemd[1]: patroni.service: Unit entered failed state.
Nov 19 15:25:48 patroni-48-node-1 systemd[1]: patroni.service: Failed with result 'timeout'.
Nov 19 15:25:48 patroni-48-node-1 systemd[1]: Started Runners to orchestrate a high-availability PostgreSQL.
Nov 19 15:25:48 patroni-48-node-1 patroni[40256]: 2018-11-19 15:25:48,842 INFO: Failed to import patroni.dcs.consul
Nov 19 15:25:48 patroni-48-node-1 patroni[40256]: 2018-11-19 15:25:48,843 INFO: Failed to import patroni.dcs.etcd
Nov 19 15:25:48 patroni-48-node-1 patroni[40256]: 2018-11-19 15:25:48,860 INFO: Failed to import patroni.dcs.kubernetes
Nov 19 15:25:48 patroni-48-node-1 patroni[40256]: 2018-11-19 15:25:48,861 INFO: Connecting to x.x.x.x:xxxx
Nov 19 15:25:48 patroni-48-node-1 patroni[40256]: 2018-11-19 15:25:48,864 INFO: Zookeeper connection established, state: CONNECTED
Nov 19 15:25:48 patroni-48-node-1 patroni[40256]: 2018-11-19 15:25:48,877 INFO: I am leader but not owner of the session. Removing leader node
Nov 19 15:25:48 patroni-48-node-1 patroni[40256]: 2018-11-19 15:25:48,880 DEBUG: Received EVENT: Watch(type=4, state=3, path='/patroni/cluster48')
Nov 19 15:25:48 patroni-48-node-1 patroni[40256]: 2018-11-19 15:25:48,882 INFO: establishing a new patroni connection to the postgres cluster
Nov 19 15:25:48 patroni-48-node-1 patroni[40256]: server signaled
Nov 19 15:25:48 patroni-48-node-1 patroni[40256]: 2018-11-19 15:25:48,909 DEBUG: Received EVENT: Watch(type=4, state=3, path='/patroni/cluster48')
Nov 19 15:25:48 patroni-48-node-1 patroni[40256]: 2018-11-19 15:25:48,917 DEBUG: Received EVENT: Watch(type=4, state=3, path='/patroni/cluster48/members')

Prerequisites

To follow this tutorial, it’s assumed that you have already set up an OpenConnect VPN server with Let’s Encrypt TLS server certificate. If not, please follow one of the following tutorials.

• Set Up OpenConnect VPN Server (ocserv) on Ubuntu 20.04 with Let’s Encrypt
• Set Up OpenConnect VPN Server (ocserv) on Ubuntu 16.04/18.04 with Let’s Encrypt
• Set Up OpenConnect VPN Server (ocserv) on Debian 10 Buster with Let’s Encrypt
• Set Up OpenConnect VPN Server (ocserv) on CentOS 8/RHEL 8 with Let’s Encrypt

Make OpenConnect VPN server and web server use port 443 at the same time

By default, OpenConnect VPN server listens on port 443. If you already have Apache/Nginx listening on port 443, then ocserv can’t bind to port 443. You can configure ocserv to listen on another port, but it will require end-users to specify the port in client software, which you should avoid if you care about user experience. Also, TLS traffic on TCP port 443 usually enjoys higher priority in QoS (Quality of Service), so you will have better speed.

Normally a port can only be used by one process. However, we can use HAproxy (High Availability Proxy) and SNI (Server Name Indication) to make ocserv and Apache/Nginx use port 443 at the same time.

Ocserv Configuration

First, edit ocserv configuration file.

sudo nano /etc/ocserv/ocserv.conf

Uncomment the following line. This will allow ocserv to obtain the client IP address instead of HAproxy IP address.

listen-proxy-proto = true

Then find the following line.

#listen-host = [IP|HOSTNAME]

Change it to

listen-host = 127.0.0.1

This will make ocserv listen on 127.0.0.1 because later HAproxy will need to listen on the public IP address. Save and close the file. Then restart ocserv.

sudo systemctl restart ocserv

Next, we also need to make the web server listen on localhost only, instead of listening on public IP address.

Nginx Configuration

If you use Nginx, edit the server block file.

sudo nano /etc/nginx/conf.d/example.com.conf

In the SSL server block, find the following directive.

listen 443 ssl;

Change it to

listen 127.0.0.2:443 ssl;

This time we make it listen on 127.0.0.2:443 because 127.0.0.1:443 is already taken by ocserv. Save and close the file. The Nginx main configuration file /etc/nginx/nginx.conf and the default server block /etc/nginx/sites-enabled/default might include a default virtual host listening on 443, so you might need to edit this file too.

Then restart Nginx.

sudo systemctl restart nginx

Apache Configuration

If you use Apache web server, edit your virtual host file.

Debian/Ubuntu

sudo nano /etc/apache2/sites-enabled/example.com.conf

CentOS/RHEL

sudo nano /etc/httpd/conf.d/example.com.conf

In the SSL virtual host, change

<VirtualHost *:443>

To

<VirtualHost 127.0.0.2:443>

This time we make it listen on 127.0.0.2:443 because 127.0.0.1:443 is already taken by ocserv. Save and close the file.

Then edit the /etc/apache2/ports.conf file on Debian/Ubuntu.

sudo nano /etc/apache2/ports.conf

Edit the/etc/httpd/conf.d/ssl.conf file on CentOS/RHEL.

sudo nano /etc/httpd/conf.d/ssl.conf

Change

Listen 443

To

Listen 127.0.0.2:443

Save and close the file. Restart Apache.

sudo systemctl restart apache2

or

sudo systemctl restart httpd

HAProxy Configuration

Now install HAproxy.

sudo apt install haproxy

or

sudo dnf install haproxy

Start HAProxy

sudo systemctl start haproxy

Edit configuration file.

sudo nano /etc/haproxy/haproxy.cfg

If you use Nginx, copy and paste the following lines to the end of the file. Replace 12.34.56.78 with the public IP address of your server. Replace vpn.example.com with the domain name used by ocserv and www.example.com with the domain name used by your web server.

frontend https
   bind 12.34.56.78:443
   mode tcp
   tcp-request inspect-delay 5s
   tcp-request content accept if { req_ssl_hello_type 1 }

   use_backend ocserv if { req_ssl_sni -i vpn.example.com }
   use_backend nginx if { req_ssl_sni -i www.example.com }
   use_backend nginx if { req_ssl_sni -i example.com }

   default_backend ocserv

backend ocserv
   mode tcp
   option ssl-hello-chk
   # pass requests to 127.0.0.1:443. Proxy protocol (v2) header is required by ocserv.
   server ocserv 127.0.0.1:443 send-proxy-v2

backend nginx
   mode tcp
   option ssl-hello-chk
   server nginx 127.0.0.2:443 check

If you use Apache, copy and paste the following lines to the end of the file. Replace 12.34.56.78 with the public IP address of your server. Replace vpn.example.com with the domain name used by ocserv and www.example.com with the domain name used by your web server.

frontend https
   bind 12.34.56.78:443
   mode tcp
   tcp-request inspect-delay 5s
   tcp-request content accept if { req_ssl_hello_type 1 }

   use_backend ocserv if { req_ssl_sni -i vpn.example.com }
   use_backend apache if { req_ssl_sni -i www.example.com }
   use_backend apache if { req_ssl_sni -i example.com }

   default_backend ocserv

backend ocserv
   mode tcp
   option ssl-hello-chk
   # pass requests to 127.0.0.1:443. Proxy protocol (v2) header is required by ocserv.
   server ocserv 127.0.0.1:443 send-proxy-v2

backend apache
    mode tcp
    option ssl-hello-chk
    server apache 127.0.0.2:443 check

Save and close the file. Then restart HAproxy.

sudo systemctl restart haproxy

In the configuration above, we utilized the SNI (Server Name Indication) feature in TLS to differentiate VPN traffic and normal HTTPS traffic.

• When vpn.example.com is in the TLS Client Hello, HAProxy redirect traffic to the ocserv backend.
• When www.example.com is in the TLS Client Hello, HAProxy redirect traffic to the apache/nginx backend.
• If the client doesn’t specify the server name in TLS Client Hello, then HAproxy will use the default backend (ocserv).

You can test this setup with the openssl tool. First, run the following command multiple times.

echo | openssl s_client -connect your-server-IP:443 | grep subject

We didn’t specify server name in the above command, so HAproxy will always pass the request to the default backend (ocserv), and its certificate will be sent to the client. Next, run the following two commands.

echo | openssl s_client -servername www.example.com -connect your-server-IP:443 | grep subject

echo | openssl s_client -servername vpn.example.com -connect your-server-IP:443 | grep subject

Now we specified the server name in the commands, so HAproxy will pass requests according to the SNI rules we defined. Note that the Cisco AnyConnect App doesn’t support TLS SNI, so it’s better to set ocserv as the default backend in HAProxy configuration file.

When renewing Let’s Encrypt certificate for your website, it’s recommended that you use the http-01 challenge instead of tls-alpn-01 challenge, because HAproxy is listening on port 443 of the public IP address, so it can interfere with the renewal process.

sudo certbot renew --preferred-challenges http-01

Fixing HAproxy Error

If your Apache/Nginx website doesn’t show up in your browser and you see the following messages in haproxy log (/var/log/haproxy.log)

Server nginx/nginx is DOWN, reason: Socket error, info: "Connection reset by peer

backend nginx has no server available!

Layer6 invalid response

It might be your backend Nginx web server is using a TLS certificate with OCSP must staple extension. Nginx doesn’t send the OCSP staple information on the first HTTP request. To make it work, be sure to add a resolver in your Nginx virtual host configuration like below.

{
     ....
     ssl_trusted_certificate /etc/letsencrypt/live/www.example.com/chain.pem;
     ssl_stapling on;
     ssl_stapling_verify on;

    resolver 8.8.8.8;
    ....
}

Save and close the file. Then restart Nginx.

sudo systemctl restart nginx

Also, consider removing the health check for the backend server in HAproxy. So change

server nginx 127.0.0.2:443 check

To

server nginx 127.0.0.2:443

Save and close the file. Then restart HAproxy.

sudo systemctl restart haproxy

How to Enable IPv6 in ocserv with HAProxy

First, create AAAA record for vpn.example.com in your DNS zone editor, so when you finish setting up IPv6 in ocserv, the DNS record should be propagated to the Internet.

Testing IPv6 Connectivity

To establish VPN tunnel in IPv6 protocol, make sure the VPN server has a public IPv6 address. (The VPN client doesn’t have to have a public IPv6 address.) To find out, run the following command.

ip addr

Locate the main network interface. If you can find a inet6 .... scope global line like below, then you have a public IPv6 address. The inet6 address with scope link is a private IPv6 address.

Then go to https://test-ipv6.com/ to check your IPv6 connectivity. If the VPN client has a public IPv6 address, it might tell you that your VPN is only protecting one protocol, not both. That’s because we didn’t enable IPv6 in ocserv.

Enable IPv6 in ocserv

To enable IPv6 in ocserv, edit ocserv configuration file.

sudo nano /etc/ocserv/ocserv.conf

Find the following two lines and uncomment them, so VPN clients will be given private IPv6 addresses.

ipv6-network = fda9:4efe:7e3b:03ea::/48
ipv6-subnet-prefix = 64

If you see the following line

ipv6-network = fda9:4efe:7e3b:03ea::/64

Please change it to:

ipv6-network = fda9:4efe:7e3b:03ea::/48

Save and close the file. Restart ocserv for the change to take effect.

sudo systemctl restart ocserv

Enable IP Forwarding for IPv6

Then we need to enable IP forwarding for IPv6 in the Linux kernel. Edit sysctl.conf file.

sudo nano /etc/sysctl.conf

Add the following line at the end of this file.

net.ipv6.conf.all.forwarding=1

Save and close the file. Then apply the changes with the below command.

sudo sysctl -p

Set Up IPv6 in Firewall (Debian, Ubuntu)

Next, we need to set up IPv6 masquerading in the UFW firewall, so that the server becomes a virtual router for VPN clients.

sudo nano /etc/ufw/before6.rules

By default, there are some rules for the filter table. Add the following lines at the end of this file. Replace ens3 with your own network interface name. In Nano text editor, you can go to the end of the file by pressing Ctrl+W, then Ctrl+V.

# NAT table rules
*nat
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -o ens3 -j MASQUERADE

# End each table with the 'COMMIT' line or these rules won't be processed
COMMIT

By default, UFW forbids packet forwarding. We can allow forwarding for our private IPv6 network. Find the ufw6-before-forward chain in this file and add the following 3 lines, which will accept packet forwarding if the source IP or destination IP is in the fda9:4efe:7e3b:03ea::/48 range.

# allow forwarding for VPN
-A ufw6-before-forward -s fda9:4efe:7e3b:03ea::/48 -j ACCEPT
-A ufw6-before-forward -d fda9:4efe:7e3b:03ea::/48 -j ACCEPT

Save and close the file. We also need to allow IPv6 VPN clients in the firewall’s INPUT chain.

sudo ufw allow in from fda9:4efe:7e3b:03ea::/48

Restart UFW for the change to take effect.

sudo systemctl restart ufw

Now if you list the rules in the POSTROUTING chain of the NAT table by using the following command:

sudo ip6tables -t nat -L POSTROUTING

You can see the Masquerade rule.

Disconnect the current VPN connection, add an AAAA record for vpn.example.com and re-establish VPN connection. Then go to https://test-ipv6.com/ to check your IPv6 connectivity.

Set Up IPv6 in Firewall (CentOS)

Enable masquerading for IPv6.

sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv6" source address="fda9:4efe:7e3b:03ea::/48" masquerade'

Allow VPN clients in the INPUT chain.

sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv6" source address="fda9:4efe:7e3b:03ea::/48" accept'

Reload firewalld for the changes to take effect.

sudo systemctl reload firewalld

Configure IPv6 in BIND Resolver

If you run your own BIND DNS resolver on the VPN server, you can add the following line in /etc/ocserv/ocserv.conf file to set the VPN server as the DNS resolver for VPN clients.

dns = fda9:4efe:7e3b::1

Save and close the file. To query DNS names in IPv6, we need to configure BIND to allow IPv6 VPN clients.

Debian/Ubuntu

sudo nano /etc/bind/named.conf.options

Find the allow-recursion parameter and change it to:

allow-recursion { 127.0.0.1; 10.10.10.0/24; fda9:4efe:7e3b:03ea::/48; };

Save and close the file. Restart BIND9.

sudo systemctl restart bind9

CentOS

sudo nano /etc/named.conf

Find the allow-query parameter and change it to:

allow-query { 127.0.0.1; 10.10.10.0/24; fda9:4efe:7e3b:03ea::/48; };

Save and close the file. Restart BIND9.

sudo systemctl restart named

Set Up IPv6 in HAProxy

Edit the HAProxy configuration file.

sudo nano /etc/haproxy/haproxy.cfg

Make the https frontend listen on both IPv4 and IPv6 addresses. Obviously you need to use your own server’s public IPv6 address.

frontend https
   bind 12.34.56.78:443
   bind 2607:f8b0:4006:810::200e:443
   mode tcp
   tcp-request inspect-delay 5s
   tcp-request content accept if { req_ssl_hello_type 1 }

Then find the ocserv backend and add an IPv6 server.

backend ocserv
   mode tcp
   option ssl-hello-chk
   server ocserv 127.0.0.1:443 send-proxy-v2
   server ocserv6 [::1]:443 send-proxy-v2

Save and close the file.

To make ocserv listen on both 127.0.0.1 and ::1, edit the /etc/hosts file.

sudo nano /etc/hosts

Edit the entry for 127.0.0.1 and ::1 like below, so the vpn.example.com hostname can be resolved to both addresses.

127.0.0.1   localhost vpn.example.com

::1         ip6-localhost ip6-loopback vpn.example.com

Save and close the file. Then edit ocserv configuration file.

sudo nano /etc/ocserv/ocserv.conf

Find the following line.

listen-host = 127.0.0.1

Change it to

listen-host  = vpn.example.com

ocserv will find the IPv4 and IPv6 addresses of vpn.example.com in the /etc/hosts file and bind to both 127.0.0.1 and ::1 addresses. Save and close the file. Then restart ocserv and HAProxy

sudo systemctl restart ocserv
sudo systemctl restart haproxy

Now run the following command to check the listening status of ocserv. You will see that it’s listening on both 127.0.0.1 and ::1.

sudo ss -lnpt | grep ocserv

Testing IPv6 Connectivity

Restart your VPN client and go to https://test-ipv6.com/ to check your IPv6 connectivity. If everything goes well, you should see your VPN server’s IPv4 and IPv6 addresses in the test result. And the warning “VPN is only protecting one protocol” should be gone.

If you don’t see your VPN server’s IPv6 address in the test result, perhaps you need to reboot the VPN clients and re-establish VPN connection.

Note: The VPN client doesn’t have to have a public IPv6 address. It can use IPv6 over the IPv4 VPN tunnel.

Wrapping Up

I hope this tutorial helped you run OpenConnect VPN server and Apache/Nginx on the same box. As always, if you found this post useful, then subscribe to our free newsletter to get more tips and tricks. Take care 🙂

Мы используем haproxy перед бэкэндом netty-3.6. Мы обрабатываем огромное количество соединений, некоторые из которых могут быть многолетними.

Теперь проблема в том, что когда haproxy закрывает соединение для средств перебалансировки, он делает это, отправляя tcp-RST. Когда класс sun.nio.ch, используемый netty, видит это, он выдает исключение IOException: «Connection reset by peer».

Трассировка:

sun.nio.ch.FileDispatcherImpl.read0(Native Method):1 in ""
sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39):1 in ""
sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225):1 in ""
sun.nio.ch.IOUtil.read(IOUtil.java:193):1 in ""
sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375):1 in ""
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64):1 in ""
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109):1 in ""
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312):1 in ""
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90):1 in ""
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178):1 in ""
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145):1 in ""
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615):1 in ""
java.lang.Thread.run(Thread.java:724):1 in ""

Это вызывает следующие проблемы для каждой конфигурации:

опция http-pretend-keepalive

Это то, что работает лучше всего (поскольку haproxy, похоже, закрывает большинство соединений с FIN, а не RST), но все же производит около 3 исключений на сервер в секунду. Кроме того, это эффективно neuters loadbalancing, потому что некоторые входящие соединения очень долговечны с очень высокой пропускной способностью: с pretend-keepalive, они никогда не получают перебалансировку с другим сервером с помощью haproxy.

опция http-keep-alive

Поскольку наш бэкэнд ожидает, что соединения keep-alive действительно будут сохранены (и, следовательно, не закрывают их сами по себе), этот параметр сводится к каждому соединению, в конечном счете, объединяющему одно исключение, что, в свою очередь, приводит к сбоям наших серверов. Мы попробовали добавить prefer-last-server, но это мало помогает.

опция http-server-close

Это теоретически должно работать как для правильного балансировки нагрузки, так и для исключений. Тем не менее, похоже, что после того, как наши серверные серверы ответят, есть гонка относительно того, какая сторона отправляет свой RST в первую очередь: haproxy или наш зарегистрированный ChannelFutureListener.CLOSE. На практике мы по-прежнему получаем слишком много исключений, и наши серверы вылетают.

Интересно, что исключения обычно получают больше, чем больше работников мы поставляем нашим каналам. Я думаю, это ускоряет чтение больше, чем писать.

В любом случае, я уже некоторое время читал о различных каналах и socketoptions в netty, а также о haproxy, и на самом деле не нашел ничего похожего на решение (или работал, когда я его пробовал).

Connection Reset by peer means the remote side is terminating the session. This error is generated when the OS receives notification of TCP Reset (RST) from the remote peer.

Understanding Connection Reset by peer

Connection reset by peer means the TCP stream was abnormally closed from the other end. A TCP RST was received and the connection is now closed. This occurs when a packet is sent from our end of the connection but the other end does not recognize the connection; it will send back a packet with the RST bit set in order to forcibly close the connection.

“Connection reset by peer” is the TCP/IP equivalent of slamming the phone back on the hook. It’s more polite than merely not replying, leaving one hanging. But it’s not the FIN-ACK expected of the truly polite TCP/IP.

Understanding RST TCP Flag

RST is used to abort connections. It is very useful to troubleshoot a network connection problem.

RST (Reset the connection). Indicates that the connection is being aborted. For active connections, a node sends a TCP segment with the RST flag in response to a TCP segment received on the connection that is incorrect, causing the connection to fail.

The sending of an RST segment for an active connection forcibly terminates the connection, causing data stored in send and receive buffers or in transit to be lost. For TCP connections being established, a node sends an RST segment in response to a connection establishment request to deny the connection attempt. The sender will get Connection Reset by peer error.

Understanding TCP Flags SYN ACK RST FIN URG PSH

Check network connectivity 

The “ping” command is a tool used to test the availability of a network resource. The “ping” command sends a series of packets to a network resource and then measures the amount of time it takes for the packets to return.

If you want to ping a remote server, you can use the following command: ping <remote server>

In this example, “<remote server>” is the IP address or hostname of the remote server that you want to ping.

Ping the remote host we were connected to. If it doesn’t respond, it might be offline or there might be a network problem along the way. If it does respond, this problem might have been a transient one (so we can reconnect now)

If you are experiencing packet loss when pinging a remote server, there are a few things that you can do to troubleshoot the issue.

The first thing that you can do is check the network interface on the remote server. To do this, use the “ifconfig” command. The output of the “ifconfig” command will show you the status of all network interfaces on the system. If there is a problem with one of the interfaces, it will be shown in the output.

You can also use the “ip route” command to check routing information. The output of the “ip route” command will show you a list of all routes on the system. If there is a problem with one of the routes, it will be shown in the output.

If you are still experiencing packet loss, you can try to use a different network interface. To do this, use the “ping” command with the “-i” option. For example, the following command will use the eth0 interface:

ping -i eth0 google.com

Check remote service port is open

A port is a logical entity which acts as a endpoint of communication associated with an application or process on an Linux operating system. We can use some Linux commands to check remote port status.

Commands like nc, curl can be used to check if remote ports are open or not. For example, the following command will check if port 80 is open on google.com:

nc -zv google.com 80

The output of the above command should look something like this: Connection to google.com port 80 [tcp/80] succeeded!

This means that the port is open and we can establish a connection to it.

6 ways to Check a remote port is open in Linux

Check application log on remote server

For example, if the error is related with SSH. we can debug this on the remote server from sshd logs. The log entries will be in one of the files in the /var/log directory. SSHD will be logging something every time it drops our session.

Oct 22 12:09:10 server internal-sftp[4929]: session closed for local user fred from [192.0.2.33]

Check related Linux kernel parameters

Kernel parameter is also related to Connection Reset by peer error. The keepalive concept is very simple: when we set up a TCP connection, we associate a set of timers. Some of these timers deal with the keepalive procedure. When the keepalive timer reaches zero, we send our peer a keepalive probe packet with no data in it and the ACK flag turned on.

we can do this because of the TCP/IP specifications, as a sort of duplicate ACK, and the remote endpoint will have no arguments, as TCP is a stream-oriented protocol. On the other hand, we will receive a reply from the remote host (which doesn’t need to support keepalive at all, just TCP/IP), with no data and the ACK set.

If we receive a reply to we keepalive probe, we can assert that the connection is still up and running without worrying about the user-level implementation. In fact, TCP permits us to handle a stream, not packets, and so a zero-length data packet is not dangerous for the user program.

we usually use tcp keepalive for two tasks:

• Checking for dead peers
• Preventing disconnection due to network inactivity

Check Application heartbeat configuration

Connection Reset by peer error is also related to the application. Certain networking tools (HAproxy, AWS ELB) and equipment (hardware load balancers) may terminate “idle” TCP connections when there is no activity on them for a certain period of time. Most of the time it is not desirable.

We will use rabbitmq as an example. When heartbeats are enabled on a connection, it results in periodic light network traffic. Therefore heartbeats have a side effect of guarding client connections that can go idle for periods of time against premature closure by proxies and load balancers.

With a heartbeat timeout of 30 seconds the connection will produce periodic network traffic roughly every 15 seconds. Activity in the 5 to 15 second range is enough to satisfy the defaults of most popular proxies and load balancers. Also see the section on low timeouts and false positives above.

Check OS metric on peer side

Connection Reset by peer can be triggered by a busy system. we can setup a monitoring for our Linux system to the metrics like CPU, memory, network etc. If the system is too busy, the network will be impacted by this.

For example, we can use the “top” command to check the CPU usage. The output of the “top” command will show us the list of processes sorted by CPU usage. If there is a process which is using a lot of CPU, we can investigate this further to see if it is causing the network issues.

We can also use the “netstat” command to check network statistics. The output of the “netstat” command will show us a list of active network connections. If there are too many connections established, this could be causing the network issues.

We can use these commands to troubleshoot network issues on a Linux system. By using these commands, we can narrow down the root cause of the issue and fix it.

Monitoring Linux System with Telegraf Influxdb Grafana

Troubleshoot Network Slow Problems In Linux