Intro

The remote access to 'Beethoven' is realised by means of (an) ssh tunnel(s). This means the traffic is encrypted and access is limited to personnel who's public keys have been copied to their respective accounts on 'Beethoven' (we'll refer to 'Beethoven' as beethoven from now on)

Below you will find a desciption how this was implemented. Basically it boils down to configuring the restricted host to create an ssh tunnel to a middle man, a.k.a. jump server. This tunnel is persistent en kept alive using the autossh tool. One can subsequently login from home on the jump server and connect through to the restricted host. If so configured, it is also possible to be directly forwarded to the restricted host. In our case the jump server is called dashy (117.204.107.243) and is a JISTARC controlled host that is specifically used for cases where we need full internet access. Users that want to make use of the facility described here naturally need an account on dashy.

NB: the numbers in the sections below denote the order in which the steps are to be taken!

Restricted host

This is the host that we want to reach from the outside (e.g. from home). This host is properly firewalled but is able to ssh to the outside world. In our case this is beethoven.

#JBR lines like this are comments

step 1
Execute the following sequence of commands:

apt install openssh-server
apt install autossh
useradd -s /bin/true -m autotunnel
su - autotunnel -s /bin/bash
ssh-keygen -t rsa
ssh-keygen -t dsa
ssh-keygen -t ed25519
#JBR first make sure account on dashy exists (see step 2)
ssh-copy-id autotunnel@dashy

This installs the autossh program, adds the autotunnel user which will not have a default login shell. To still be able to execute commands as autotunnel we have to specify a shell to execute, i.e. /bin/bash. This enables us to generate the keys and copy them to the middleman server dashy. The ssh-copy-id command copies the most recent id_*.pub file. As of recent the ed25519 is considered safest (and quick). NB: the autotunnel account password needs to be known (temporarily set) see step 2. NB: The last command of step 2 can only be executed when step 2 below has been completed! Also temporarily enable logins using a password  by setting PasswordAuthentication yes in the /etc/ssh/sshd_config file and restart the daemon (don't forget to restore original setitngs and restart again).

step 3

This step is to test some aspects of the setup:

First try a regular ssh session:

ssh autotunnel@dashy

This should work without supplying a password and relies on the public keys we copied in step 1. Exit this after determining that it works. Now start an autossh one:

autossh -M0 autotunnel@dashy -N -R 8082:localhost:22 -vvv

Due to the -vvv option this will cause a lot of output on screen. Switch over to the Middleman server execute step 4 and try to connect.

Middle man

This is a server somewhere on the internet that we control. In our case this is dashy.

step 2

useradd -s /bin/bash -m autotunnel
#JBR generate a password
pw=`date|md5sum|cut -c1-10` && echo $pw
#JBR sets the password for the autotunnel user
echo -e "$pw\n$pw"| passwd autotunnel

Next edit the /etc/ssh/sshd_config file to include the 

GatewayPorts clientspecified

line. This enables the client to specify an IP address from which connections to the port are allowed (it proved crucial in getting the tunnel to work like I wanted). Next we need to restart the ssh service for this to take effect.

systemctl status sshd.service 

Make sure the firewall allows connection on port 8082 (or whatever port you chose).

ufw allow 8082

step 4

There are two ways to test the connection. First is simple and aimed at finding out if it connects at all (if you have a telnet client installed).

telnet localhost 8082

This should generate some output on the restricted client due to the -vvv option we supplied. When this indeed is the case just quit and commence with the following actual test.

ssh -p 8082 localhost
#JBR or alternatively for instance:
ssh -p 8082 jan@localhost

This should log you in on the restricted host. If you need to supply a password it means you did not copy your own public keys or the permission of the authorized_keys file in set incorrectly. These keys are different (personal) public keys that you should have on your home pc/laptop account. Again this should generate respective verbose output on the restricted host. When succesful you can logout on the middle man host and also stop the autossh running on the restricted host. If you cannot login from an outside host (not localhost) it might be because the provider that hosts your middle man server has a firewall that need to be adjusted (i.e. amazon, oracle, etc).

Automation and persistence

In step 1-4 we have demonstrated the sshtunnel. Now we need to have it start automatically and make it persistent. This can be accomplished using appropriate systemd scripts on the restricted host, as indicated below.

Put the script below in /etc/systemd/system and name it autossh-dashy.service

[Unit]
Description=AutoSSH tunnel service to dashy
After=network-online.target

[Service]
Restart=always
RuntimeMaxSec=86400
Environment="AUTOSSH_GATETIME=0"
ExecStart=/bin/su -s /bin/bash autotunnel -c 'autossh -M 0 -q -N -C -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -o "ConnectTimeout 10" -o "ExitOnForwardFailure yes" -R *:8082:localhost:22 autotunnel@117.204.107.243'

[Install]
WantedBy=multi-user.target

Then activate it and start it by using the following commands:

systemctl enable autossh-dashy.service
systemctl start autossh-dashy.service

We also want to do the same to the web related ports 80,443. Put the script below in /etc/systemd/system and name it autossh-dashy-web.service

[Unit]
Description=AutoSSH tunnel for web-service to dashy a.k.a. zij.informeer.de
After=network-online.target

[Service]
Restart=always
RuntimeMaxSec=86400
Environment="AUTOSSH_GATETIME=0"
#JBRv this is different than the other autossh-dashy script because it is run as root! 
#OFF ExecStart=autossh -M 0 -q -N -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -o "ConnectTimeout 10" -o "ExitOnForwardFailure yes" -R :80:localhost:80 -R *:81:localhost:81 -R *:443:localhost:443 root@zij.informeer.de
#JBRv the entry for port 81 is only needed once and can be disabled after the admin has secured and configured the access over https
ExecStart=autossh -M 0 -q -N -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -o "ConnectTimeout 10" -o "ExitOnForwardFailure yes" -R :80:localhost:80 -R *:443:localhost:443 root@117.204.107.243

[Install]
WantedBy=multi-user.target

Then activate it and start it by using the following commands:

systemctl enable autossh-dashy-web.service
systemctl start autossh-dashy-web.service

Important! 

• For this to work ports 80,443 cannot be occupied by other processes. 
• Because these are privileged ports it needs root access! (hence the root@zij.informeer.de entry above)
• The root public keys need to be copied to the middle man server (i.e. ssh-copyid or manually into /.ssh/authorized_keys)
• The host fingerprint needs to be accepted. This can be achieved by fist executing a manual ssh login from the restricted host to the middle man server and manually accept the fingerprint when prompted.
• We initially open port 81 because this is the admin port of nginxproxymanager. This can be disabled (commented out) after nginxproxymanager has been set up (webinterface https)
• When you (temporariy) edit/change the script you need to restart thedaemon i.e.: systemctl daemon-reload

Troubleshooting

Usually the SSH tunnel works fine. Problems can potentially occur when disconnecting and reconnecting the hosts liszt and beeethoven. Typically issues consist of:

• authentication problems (i.e. wrong keys of missing keys)
• connection problems (i.e. ports that are occupied or misdirected, unintended fail2ban jails)

Sometimes problems can simply be solved by restarting the ssh daemon on dashy and the autossh daemon that maintain the ssh tunnel(s) in liszt and beethoven.

#JBR on beethoven and liszt
systemctl restart autossh-dashy.service
systemctl restart autossh-dashy-web.service

#JBR on dashy
systemctl restart sshd.service

When using ssh and entering a wrong password multiple times in quick succession fail2ban (temporarily) blacklists the originating host IP. This so called jailing can be undone if you can login by another means (i.e. from a different host). Alternately one can permanently whitelist a specific trusted host to prevent an unintended lock-out situation. NB: fail2ban naturally runs on dashy the middleman server.