Linux Server Closing Connections

[McChikenhanger]

Dear masters of Linux.

 

I have a situation were one of our Linux servers keeps closing all remote connections to itself throughout the working day. I could be connected through it on SSH, only for it to close my connection whenever it feels like it.

 

Is there a way of finding out why this is happening and how to stop it if it is a setting on that server somewhere?

 

Thanks in advance to all of the masters out there.

[GTX]

Is this a VM or physical box?

[pete]

Is this on the local LAN or are there firewalls between your SSH client and the server?

 

You can fudge it by using something like

 

Host *
       ServerAliveInterval     60

 

to your ~/.ssh/config if there's a firewall in-between, but as the above poster says - a bit more info is needed.

 

When did it start happening?

What's changed on the server/client and anything in-between?

[McChikenhanger]

The server is a VM.

[McChikenhanger]

This is happening on the local LAN, and there are no firewalls in between these machines. We have changed our default router over to a Juniper box.

[JackCPickup]

What OS and version is it?

 

Could there be anything getting the same IP as it?

[McChikenhanger]

The OS version is CentOS 6.8 and I've double checked and there's nothing using the same IP address anywhere. But thank you for suggesting that, it wasn't one of the things that I had checked yet.

[bossman]

What model of switch is the linux box running through?

 

Could be the port on that switch is closing down through a nic error on the linux box?

[McChikenhanger]

Hi Bossman,

 

The Linux box is running through a Netgear GS724T, and the machine is virtualised on a VMWare host. Don't think it could be the port on the switch, as all the other VMs on the same host don't have this issue that I can see. Plus if you re-open the connection to the machine either through SSH or any other protocol everything starts working fine, until the connection gets closed again by something.

[pete]

Did connections start dropping as soon as you swapped in the Juniper box?

 

Is local routing happening on the Juniper box or your main switch/switch stack?

[McChikenhanger]

Hi Pete,

 

Yeah they must have, as that's the only thing that we have changed over here recently. It wasn't until a while afterwards that some of the users started reporting this problem. Routing is being done by the Juniper Box.

[pete]

So the router may be dropping the remote connection based on whatever the idle TCP session timeout setting is. A quick Google suggests this is 30 minutes for Juniper routers. You probably don't want to fiddle with this unless you can say something like:

 

"For SSH connections to ServerA, use a longer timeout. For everything else, use the default."

 

Don't apply a longer timeout across the board - you'll break stuff (the state table may fill up faster than it empties).

 

On the server side....

 

edit /etc/ssh/sshd_config and add the following line:

 

ClientAliveInterval 60

 

On the client connecting via SSH, edit the ~/.ssh/config file as show above if you're using Linux/OS X to connect. If you're using Putty on Windows, go into Connection and tick the "Enable TCP Keepalives" box. See attached screenshot.

 

 

Finally, use tmux after logging into the server so that if the connection does drop you can type tmux attach to reattach to the disconnected session.

 

Now leave a SSH session alone for 40 mins and see if it's still there when you come back.

Edited July 3, 2017 by pete

[McChikenhanger]

Hi Pete,

 

Will give that a go. Sorry one more question. What do you mean by using tmux? I just tried running that on the Linux server but it doesn't recognize the command.

 

Regards.

[pete]

tmux is a terminal multiplexer - you may need to install it (it's there by default on Ubuntu). Alternatively you can use "screen" (which I'm pretty sure should be installed on Centos).

 

You SSH to the server, type "tmux" or "screen" and you're then in a pseudo-terminal session (or sessions) that still exists even if the SSH connection drops.

 

screen -r will re-attach disconnected screen sessions.

 

 

I prefer tmux to screen, but that's mainly because it's got a few more features. For what you're doing (making your working session recoverable if it disconnects), either will do the job.

 

typing man screen will give you more info on usage.

[McChikenhanger]

Ah I see, cool. Ok will give that a go now.

 

Thank you Pete.

[pete]

Forgot to mention - after changing the SSH configuration (/etc/ssh/sshd_config) on the server, you'll need to restart the SSH service in order for it to pick up the configuration change.

 

typing service sshd restart as root (or sudo service sshd restart if you're using sudo) should do that.

 

 

The SSH service might be called something different (ssh, openssh-server), but you should be able to tab-complete the command to find the correct one.