Running `docker pull rayproject/ray-ml:latest-gpu`įull command is `ssh -tt -i /home/chris_chiasson/.ssh/ray-autoscaler_gcp_us-central1_ting-1-3_ubuntu_0.pem -o StrictHostKe圜hecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -o ExitOnForwardFailure=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=3 -o ControlMaster=auto -o ControlPath=/tmp/ray_ssh_8ec3601252/c21f969b5f/%C -o ControlPersist=10s -o ConnectTimeout=120s bash -login -c -i 'true & source ~/.bashrc & export OMP_NUM_THREADS=1 PYTHONWARNINGS=ignore & (docker pull rayproject/ray-ml:latest-gpu)'` Note I have altered the project name in my copy/paste to protect myself. (On the latest GPU VMs I am working with, it does not always fail at this part.) It’s late, but tomorrow I will try to isolate it as requested. When I re-ran it (forgetting to add the -vvv to the ssh), it succeeded. I ran it just now and it failed on the docker pull. Status: Downloaded newer image for rayproject/ray-ml:latest-gpu If there is still no response, the SSH connection will be disconnected.Why does ray like to repeatedly die with messages like this? I’ve been using ray for a few months and have gotten it to work, but I have noticed the amount of time spent setting up and tearing down VMs is ridiculous due to issues like this.

If there is still no response from the client, it will send another alive message at 600 seconds. If there is no response from the client, it will again send an alive message at 400 seconds.

Set the parameters ClientAliveInterval and ClientAliveCountMax to the desired values.įor example, ClientAliveInterval=200 and ClientAliveCountMax=3 means that the server will send an alive message after 200 seconds.

Open the sshd config file located at /etc/ssh/sshd_config.Set the parameter in the config file. For example:Įcho 'ServerAliveInterval 600' > ~/.ssh/config.Create the ssh config file (if it does not exist):.When connecting to a server, use the -o option: ssh -o ServerAliveInterval=600 value 600 represents 600 seconds i.e., 10 minutes.Īlternatively, add it to your ssh config file: Set it on your machine if you connect to multiple servers via SSH.Īnd several users complain about frequent SSH connection disconnect, You can address this problem on either the client or the server. David's answer is OK, but a more comprehensive solution is explained below.

Xquartz ssh broken pipe