ssh

a.k.a. “Secure shell”, which is not actually a shell but an encrypted network protocol that people like to shell with

2014-12-27 — 2026-03-01

Wherein sundry secure conveyances are effected by client contrivance, the EDITOR is set by SSH_CONNECTION, and the session is suspended by Enter ~ and Ctrl‑Z.

computers are awful
computers are awful together
confidentiality
encryption
plain text
POSIX

Assumed audience:

People who need to run programs on remote machines without sweating the details

SSH, Secure Shell, is the Swiss Army knife of the internet. It provides methods that let us shunt data from one place to another in a huge number of different ways. There are similar tools that are even less fuss, but they’re gaping security holes and we should not use them.

It is weird that it is named after almost the one thing that it is not: a shell. But other pedants have complained about that, I am sure.

I’m writing from the perspective of someone using SSH as a client, not maintaining a server.

Figure 1

1 Setting vars specially when we log in over .ssh

For example, we might want to set up $EDITOR to use a GUI normally, but not when we’re logging in over SSH:

~/.config/fish/config.fish:

if status is-login
    if set -q SSH_CONNECTION
        set -x EDITOR vim
    else
        set -x EDITOR "code -w"
    end
end

~/.zshrc/~/.bashrc

if [[ -n $SSH_CONNECTION ]]; then
    export EDITOR='vim'
else
    export EDITOR='code -w'
fi

2 Suspending ssh

Use the escape character. By default, this is Enter ~. Right after typing the escape character, Ctrl-Z will suspend the SSH client.

3 Extra security

SSH has historically shipped with some insecure authentication and encryption options. We should probably secure SSH by shutting down unnecessarily weak crypto options, disabling unneeded ports, and so on. If I want to be extra sensible, I could secure it to modern cryptography standards, such as elliptic-curve ciphers, and smart defaults that we suspect are less vulnerable to everyday NSA attacks. (With these settings we’re still screwed when cheap quantum factorisation becomes a thing, though, but let us set that aside for now.)

There are a lot of other commands needed to make stuff go. Recommended /etc/ssh/sshd_config (for the server):

KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
PasswordAuthentication no
PubkeyAuthentication yes
ChallengeResponseAuthentication no
AllowGroups ssh-user
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com

We recommend /etc/ssh/ssh_config for our clients:

# Github needs diffie-hellman-group-exchange-sha1 some of the time but not always.
#Host github.com
#    KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1
Host *
    KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
    PasswordAuthentication no
    ChallengeResponseAuthentication no
    PubkeyAuthentication yes
    HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ssh-rsa
    Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
    MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
    UseRoaming no

See especially dolmen/github-keygen: Easy creation and upgrade of secure SSH configuration for your GitHub account(s).

4 SSH identities

It’s fiddly to have multiple identities for the same host—especially when using git, since we need to manage this at the configuration level via host aliases. A guide to that is here. See also the lengthy Stack Overflow discussion. For bonus paranoia, Mac users can store SSH identities in their secure enclave doodad.

5 Key encryption with ssh-agent

If you don’t want someone to steal your private key you can encrypt it. Doing this even if you have filesystem encryption would be wise if you were in a hostile environment such as a cloud server, or use features such as ssh agent forwarding (long story), or are just generally cautious.

Typing a password to decrypt the key all the time is annoying; for this I use ssh-agent. It is somewhat magical and and I worry I’m endangering myself by not understanding how the subtleties of how it decrypts stuff for me.

tl;dr On Bash, use

eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519

Inside tmux, we need

# Update SSH_AUTH_SOCK in tmux sessions
if [ -n "$TMUX" ]; then
    export SSH_AUTH_SOCK=$(tmux show-environment | grep '^SSH_AUTH_SOCK' | cut -d= -f2-)
fi

Content: * GitHub’s guide * paranoid mode * How to use the ssh-agent with fish also breaks down the ssh-agent commands without resorting to magic. * SSH Agent Explained

5.1 At startup

Do we want to start ssh-agent on startup, but not if it’s already running? See Joseph M. Reagle’s solution:

SSH_ENV="$HOME/.ssh/environment"

function start_agent {
    echo "Initialising new SSH agent…"
    /usr/bin/ssh-agent | sed ’s/^echo/#echo/' > "${SSH_ENV}"
    echo succeeded
    chmod 600 "${SSH_ENV}"
    . "${SSH_ENV}" > /dev/null
    /usr/bin/ssh-add;
}

# Source SSH settings, if applicable
if [ -f "${SSH_ENV}" ]; then
    . "${SSH_ENV}" > /dev/null
    #ps ${SSH_AGENT_PID} doesn’t work under cywgin
    ps -ef | grep ${SSH_AGENT_PID} | \
        grep ssh-agent$ > /dev/null || {
            start_agent;
    }
else
    start_agent;
fi

I’m not sure how much of this we can avoid with a modern setup.

5.2 ssh-agent with fish

Easy enough; see the fish shell ssh-agent, which installs the fish_ssh_agent command.

How to use the ssh-agent with fish explains how things are supposed to work, but I found the fish_ssh_agent solution was the one that actually worked for me.

5.3 ssh-agent on macOS

How does ssh-agent work with the macOS keychain? Should we let it, or does that invite hostile actors in?

Things are weird on macOS because we can store keys in ssh-agent, in the macOS keychain, or via weird hybrid options that make my eyes cross. Apple’s Explanation is sorta clear, but it’s changed (and gotten more confusing) over time. GitHub has an opinion on it. Old macOS SSH behaviour for the vexed. The best, most up-to-date summary seems to be Awesome macOS CLI.

Prior to macOS Sierra, ssh would present a dialog asking for your passphrase and would offer the option to store it into the keychain. This UI was deprecated some time ago and has been removed.

Instead, a new UseKeychain option was introduced in macOS Sierra allowing users to specify whether they would like for the passphrase to be stored in the keychain. This option was enabled by default on macOS Sierra, which caused all passphrases to be stored in the keychain.

This was not the intended default behaviour, so this has been changed in macOS 10.12.2. …

ssh-add -K /path/to/private_key

Then add this to ~/.ssh/config:

Host server.example.com
    IdentityFile /path/to/private_key
    UseKeychain yes

Update:

ssh-add --apple-use-keychain /path/to/private_key

6 SSH as VPN

sshuttle (manual) is a VPN work-alike built on SSH. As far as I can tell, it’s easy for both the client and server to set up a VPN this way, so I’m not sure why it’s not more common. Possibly because setting up SSH shells on various servers is, in itself, easy to do in a way that leaves the server insecure? Anyway, if we have a login, we might as well use it.

Installation options:

pip install sshuttle # or
brew install sshuttle
# etc

Run:

sshuttle --dns -r username@sshserver 0/0

7 Over HTTPS

A classic trick is to use corkscrew, which sneaks SSH over HTTPS through hostile, web-only firewalls (like the ones at airports).

8 Over QUIC

QUIC is a modern Internet transport protocol that runs over UDP and is designed to make connections faster and more reliable than the traditional TCP+TLS stack for many applications (notably HTTP/3). The reliability part is especially interesting to me. SSH connections are irritating to maintain over spotty links, and QUIC could help with that, maybe?

tsuna/quicssh is a QUIC proxy that allows us to use QUIC to connect to an SSH server without needing to patch the client or the server. It does need a proxy server started on the server side, though.

9 Tunnels

sshtunnelmanager (macOS) assembles the right command-line arguments to make “tunnels” (i.e. forwarding connections via other hosts) without needing to check the manual every time.

ProxyJump is a two-step proxy to make double tunnelling easier.

ssh -J your.jump.host remote.internal.host

or

scp -o 'ProxyJump your.jump.host' \
  myfile.txt remote.internal.host:/my/dir

We can dynamically add tunnels!:

<enter>~Cto bring up a console with your local SSH client (not the server). The provided console accepts a few of the ssh commands options, including -R, -L.

So, for example, if I wanted to suddenly access some service running on port 4321 on my local machine from the server, I could type <enter>~C-R 1234:localhost:4321<enter> and I would immediately have access to that resource from the server on localhost:4321 (that’s the server’s localhost).

9.1 via Microsoft’s dev tunnels

Microsoft’s Dev tunnels do not appear to be an SSH tool, but the tunnels look similar to SSH tunnels, with a few advantages, including: * Persistent SSH sessions over unreliable connections using Microsoft’s own SSH variant, microsoft/dev-tunnels-ssh: SSH library for dev-tunnels.

10 autossh

  • autossh is a program to start a copy of ssh and monitor it, restarting it as necessary should it die or stop passing traffic. The idea is from rstunnel (Reliable SSH Tunnel), but implemented in C.
  • The author’s view is that it is not as fiddly as rstunnel to get to work.
  • Connection monitoring using a loop of port forwardings or a remote echo service.
  • Backs off on rate of connection attempts when experiencing rapid failures such as connection refused.
  • Compiled and tested on OpenBSD, Linux, Solaris, Mac OS X, Cygwin, and AIX; should work on other BSDs.
  • Freeware.

See a worked example.

11 Terminal multiplexing

screen, tmux, and so on. Handy for remote admin over SSH.

See terminal multiplexers.

12 Network connection multiplexing

We can share a single connection between multiple clients. Here is a worked example:

# set up master connection
ssh -f -N -M -S ~/.ssh/S.user@host.com -D 9000 user@host.com
# recycle it for a new session
ssh -S ~/.ssh/S.user@host.com dummyhost

13 Alternatives/extensions

13.1 Dropbear

Dropbear SSH is a minimal SSH implementation.

13.2 EternalTerminal

Eternal Terminal integrates some kind of session management into the terminal.

13.3 mosh

mosh (“mobile shell”) is also, like SSH, not a shell. It is a terminal with SSH-style tunnelling for intermittent connections, including optimistic screen updates and graceful packet loss.

How is mosh better than tmux + autossh though? I need a Venn diagram of features here. Not terribly actively developed since 2017.

13.4 Eternal Terminal

An SSH extension that supports resumable terminal sessions via clever network footwork. It also integrates with tmux. (In fact, what does it offer that tmux does not?) It needs a special server process, so I probably will not get around to running this everywhere I would need it.

13.5 Teleport

“Enterprise” SSH? teleport claims to offer that.

14 Incoming