path: root/_drafts/ssh-proxy.md

---
title: "SSH Proxy"
description: "How to establish a secure tunnel between two firewalled machines."
published: 2020-06-22
---

Consumer grade internet connections typically prevent inbound connections by means of network address translation (NAT) or firewalls.
While this protects consumers to some degree from the evil outside, it also hinders them from providing network services.
Luckily, people came up with several [NAT traversal techniques](https://en.wikipedia.org/w/index.php?title=NAT_traversal&oldid=950406393#Techniques) including the widespread [SOCKS](https://en.wikipedia.org/w/index.php?title=SOCKS&oldid=963014782) protocol and the martial [UDP hole punching](https://en.wikipedia.org/w/index.php?title=UDP_hole_punching&oldid=957144154).
However, I found that [OpenSSH](https://www.openssh.com/)'s little-known remote port forwarding feature enables a simple and yet secure alternative that I would like to share in this post.

We'll use the [OpenSSH](https://www.openssh.com/) client and server here because its a free, battle-tested, and portable implementation of the SSH protocol.
Chances are, that your operating system ships with OpenSSH built-in or packaged.
In fact, even [macOS](https://support.apple.com/guide/remote-desktop/about-systemsetup-apd95406b8d/mac) and [Windows 10](https://docs.microsoft.com/en-us/windows-server/administration/openssh/openssh_overview) include OpenSSH, activatable from the command-line.

Contents:

* TCP port forwarding. Anyone on the proxy may access the exposed service.
* End-to-end encrypted tunnel. Servers needs to run local SSH server.
* Unix domain socket forwarding. Protected socket instead of TCP port.

## TL;DR

Basically, you can ...

You've got three different options to share a local network service by means of a publically reachable OpenSSH proxy:

1. Create a remote port forwarding from the proxy to the server and 

Authentication and authorization aside, you can share a local network service in three steps:

1. Create a remote port forwarding from a publically reachable proxy to the firewalled server:

       $ ssh -nNTR 8080:localhost:80 proxy.example.com

1. Create a local port forwarding from the client to the proxy:

       $ ssh -nNTL 8080:localhost:8080 proxy.example.com

1. Access the exposed network service from the client:

       $ curl http://localhost:8080/

So far so good, but the real work is of course to setup the public key authentication and to restrict the keys' permissions.
Thus, the remainder of this blog post describes a typical use case in detail.

## Screen sharing example

Let's say, Alex wants to share his screen with Tyler.
Sure enough, Alex can run a local [RFB](https://www.iana.org/assignments/rfb/rfb.xml) server such as [TigerVNC](https://tigervnc.org/) or [Apple Remote Desktop](https://support.apple.com/remote-desktop) on port 5900.
But thanks to their internet provider's NAT, neither of the two may connect to the other directly.
Fortunately, Tyler has access to a publicly reachable OpenSSH server that they can use as a forward proxy as follows.

Note:
OpenSSH is developed as a part of [OpenBSD](https://www.openbsd.org/).
Thus, it comes with excellent [man pages](https://www.openssh.com/manual.html).
So please _read the fine manual_ to understand the various options that we'll use.
I cannot describe them better.

First, Alex starts an OpenSSH server on his machine.
By default, the SSH server should listen on port 22 of all local addresses and permit both, password and public key authentication, see [`sshd_config(5)`](https://man.openbsd.org/sshd_config).
This is sufficient for our setup.
However, the truly paranoid may safely bind the SSH server to the loopback interface only and forbid password authentication:

    ListenAddress 127.0.0.1
    ListenAddress ::1
    Port 22
    PasswordAuthentication no
    PubkeyAuthentication yes

Second, Alex prepares a [`known_hosts`](https://man.openbsd.org/sshd#SSH_KNOWN_HOSTS_FILE_FORMAT) file for Tyler:

    $ cat /etc/ssh/ssh_host_*_key.pub \
      | sed -e 's/^/alex_workstation /' \
      | tee -a alex_public_host_keys
    alex_workstation ssh-dss AAAAB3N... root@alex.localdomain
    alex_workstation ssh-ed25519 AAAAC3N... root@alex.localdomain
    alex_workstation ssh-rsa AAAAB3N... root@alex.localdomain

Third, Alex generates an SSH key to authenticate at the proxy.
We'll generate a dedicated key here, so you can follow this guide without using your normal SSH key:

    $ ssh-keygen -f ~/.ssh/id_proxy

Fourth, Tyler imports Alex' public host keys on his machine:

    $ cat alex_public_host_keys >> ~/.ssh/known_hosts

Fifth, Tyler authorizes Alex to connect to the proxy.
That is, Tyler adds Alex' public user key to his personal [`authorized_keys`](https://man.openbsd.org/sshd#AUTHORIZED_KEYS_FILE_FORMAT) file on the proxy, applying the following restrictions to prevent Alex from misusing Tyler's account:

    restrict,command="echo 'ssh command restricted by authorized_keys'",port-forwarding,permitlisten="2222" ssh-rsa AAAAB3N...

The truly paranoid may wish to create a dedicated [`nologin(8)`](https://man.openbsd.org/nologin) user account on the proxy instead and apply the above restrictions by means of the [`sshd_config(5)`](https://man.openbsd.org/sshd_config):

    Match User port-forward-only
      DisableForwarding yes
      ForceCommand echo 'ssh command forced by sshd_config'
      PermitTTY no
      AllowTcpForwarding remote
      PermitListen 2222

Sixth, Tyler prepares a [`known_hosts`](https://man.openbsd.org/sshd#SSH_KNOWN_HOSTS_FILE_FORMAT) file for Alex:

    $ cat /etc/ssh/ssh_host_*_key.pub \
      | sed -e 's/^/[proxy.example.com]:22 /' \
      | tee -a proxy_public_host_keys
    [proxy.example.com]:22 ssh-dss AAAAB3N... root@proxy.example.com
    [proxy.example.com]:22 ssh-ed25519 AAAAC3N... root@proxy.example.com
    [proxy.example.com]:22 ssh-rsa AAAAB3N... root@proxy.example.com

Seventh, Alex imports the public host keys of the proxy on his machine:

    $ cat proxy_public_host_keys >> ~/.ssh/known_hosts

Eighth, Alex authorizes Tyler to connect to his machine.
That is, Alex adds Tyler's public user key to his personal [`authorized_keys`](https://man.openbsd.org/sshd#AUTHORIZED_KEYS_FILE_FORMAT) file on his machine, applying the following restrictions to prevent Tyler from misusing Alex' account:

    restrict,command="echo 'ssh command restricted by authorized_keys'",port-forwarding,permitopen="5900" ssh-rsa AAAAB3N...

Again, the truly paranoid may wish to create a dedicated user account as described above.

Ninth, Alex connects to the proxy and forwards port 2222 to port 22 on his machine:

    $ ssh \
      -nNT \
      -R 2222:localhost:22 \
      -i ~/.ssh/id_proxy \
      -o "IdentitiesOnly yes" \
      -o "StrictHostKeyChecking yes" \
      -o "ExitOnForwardFailure yes" \
      -l tyler \
      proxy.example.com

Alternatively, Alex may add the following settings to his personal [`ssh_config(5)`](https://man.openbsd.org/ssh_config) and run `ssh -nN proxy`:

    Host proxy
      Hostname proxy.example.com
      Port 22
      User alex
      StrictHostKeyChecking yes
      IdentitiesFile ~/.ssh/id_proxy
      IdentitiesOnly yes
      RemoteForward 2222 localhost:22
      ExitOnForwardFailure yes
      RequestTTY no
    
    Host *
      Protocol 2
      AddKeysToAgent yes
      IgnoreUnknown UseKeychain
      UseKeychain yes


Tenth, Tyler connects to Alex' machine and forwards port 5900 from his machine to Alex' machine:

    $ ssh \
      -J tyler@proxy.example.com \
      -L 5900:localhost:5900 \
      -o "ExitOnForwardFailure yes" \
      -o "StrictHostKeyChecking yes" \
      -l alex \
      -p 2222 \
      -o "HostKeyAlias alex_workstation" \
      localhost

Alternatively, Tyler may add the following settings to his personal [`ssh_config(5)`](https://man.openbsd.org/ssh_config) and run `ssh alex_workstation`:

    Host alex_workstation
      ProxyJump tyler@proxy.example.com
      User alex
      Hostname localhost
      Port 2222
      HostKeyAlias alex_workstation
      StrictHostKeyChecking yes
      ExitOnForwardFailure yes
      LocalForward 5900 localhost:5900
    
    Host *
      AddKeysToAgent yes
      IgnoreUnknown UseKeychain
      UseKeychain yes

Finally, Tyler accesses the tunneled RFD server from his machine.
For example, using TigerVNC's [`vncviewer(1)`](https://tigervnc.org/doc/vncviewer.html)[^vncviewer]

    $ vncviewer localhost

[^vncviewer]:
Apparently, the [TigerVNC](https://tigervnc.org/)'s client, [`vncviewer(1)`](https://tigervnc.org/doc/vncviewer.html), may not connect to an [Apple Remote Desktop](https://support.apple.com/remote-desktop) agent, even if you enable the legacy VNC option of the latter.
However, you can use [FreeRDP](https://www.freerdp.com/) instead, or one of its graphical front-ends like [Gnome Boxes](https://wiki.gnome.org/Apps/Boxes).

TODO: permitlisten="none",permitopen="none"

## Conclusion

We've established a secure, end-to-end encrypted tunnel between two otherwise disconnected machines through an OpenSSH forward proxy.
Moreover, the two parties need minimal trust in each other because we've restricted the keys' permissions.

OpenSSH enables even more fine grained control if need be.
For example:

* Apply an `expiry-time` to your `authorized_keys` to grant temporary access.

* Add a trusted `cert-authority` to your `authorized_keys` instead of individual keys.

* Enable `VerifyHostKeyDNS` to automatically trust host keys with a corresponding SSHFP resource record in the DNS.

* Use [Unix domain sockets](https://en.wikipedia.org/wiki/Unix_domain_socket) instead of network ports to further restrict access to the tunnel from the proxy to the actual server.[^socket]

[^socket]:
Simply replace the port number with an absolute path like `/home/tyler/proxy.sock`.
Do not use the `~` to represent the user's home directory.

# Rewrite

## Goal

Forward a (protected) Unix domain socket on the proxy to a local port.
Such that nobody else on the proxy may use the forwarding, as with a forwarded TCP port.

Serve the current directory at http://127.0.0.1:8080/:

    python3 -m http.server --bind 127.0.0.1 8080

Using explicit loopback address 127.0.0.1 (or ::1) instead of localhost lest socat or ssh shoud bind to a non-loopback address.

Want either end-to-end encryption (using SSH server on the server) or protected address (Unix domain socket instead of TCP port) on the proxy.

## OpenSSH RemoteForward

Works as advertised.
Missing `PermitListen` (and `permitlisten`) equivalent to restrict the name of the socket.

## OpenBSD netcat

Idea: manually bind to remote socket using netcat.
Problem: Doesn't even work locally. Neither on Arch Linux nor on OpenBSD.

Forward Unix domain socket test.sock to TCP port 8080 using OpenBSD's [`nc`](https://man.openbsd.org/nc):

    #! /bin/sh
    rm -f backpipe
    mkfifo backpipe
    nc -lkU test.sock 0<backpipe \
      | nc 127.0.0.1 8080 1>backpipe

Retrieve the home page http://127.0.0.1:8080/ via the socket:

    printf "GET / HTTP/1.0\r\n\r\n" \
      | nc -UN test.sock

Unfortunately, this doesn't work reliably.
The retrieval command prints the response at most once, ofter on the second invocation and NOT on the first invocation.

I found neither the cause of this behavior nor a workaround.
I suspect, the problem is a combination of the following:

* Netcat closes connection to early because of an early EOF

* Shell pipe errors

* Shell buffers standard input/output

Moreover, each Linux appears to implement a slightly different version of OpenBSD's netcat.

## socat

Forward remote socket test.sock to local port 8080:

    socat EXEC:'ssh -T engine.skreutz.com socat "UNIX-LISTEN:test.sock,fork,unlink-early" STDIO' TCP4:127.0.0.1:8080,fork

TODO: Test local nc and remote socat, because it might be easier to find a nc implementation for Windows than socat.

Forward local port 8081 to remote socket test.sock:

    ssh -vnNT -L 127.0.0.1:8081:/home/stefan/test.sock -o "ExitOnForwardFailure yes" engine.skreutz.com

Note: Specify the local bind address 127.0.0.1 and option `ExitOnForwardFailure` to make `ssh` fail if `GatewayPorts` is set to `yes`.

Noto: Specify the absolute path of the socket. Do not rely on the ~.

Retrieve home page:

    printf "GET / HTTP/1.0\r\n\r\n" \
      | nc -N 127.0.0.1 8081

Alternatively:

    curl http://127.0.0.1:8081/

TODO: Restrict authorized_keys
TODO: Inspect ports with netstat.
TODO: permitlisten="none",permitopen="none".
TODO: Request OpenSSH feature PermitStreamLocalListen/Open analoguous to PermitListen/Open.
TODO: Test tcpserver instead of socat: https://cr.yp.to/ucspi-tcp.html