11 KiB

Raw Blame History

Reverse SSH Tunnel Server for Traefik

A lightweight Go system that lets remote Docker hosts expose HTTP services through a central Traefik reverse proxy via SSH tunnels. The server SSHs into the Traefik ingress host to dynamically manage route configuration.

Architecture

Remote Host                        Tunnel Server              Traefik Host (ingress.nixc.us)
+--------------+                   +---------------------+    +---------------------+
| tunnel-client| ---SSH tunnel---> | tunnel-server       |    | Traefik             |
| TUNNEL_KEY   |                   | allocates port,     |    | watches dynamic/    |
| TUNNEL_DOMAIN|                   | SSHs into Traefik   |--->| routes HTTPS to     |
| TUNNEL_PORT  |                   | host to write config |    | tunnel-server ports |
+--------------+                   +---------------------+    +---------------------+

Flow

Client connects to tunnel-server via SSH using a private key
Client sends domain metadata ({"domain":"myapp.example.com"}) over a custom channel
Server allocates a port from the pool and sets up a reverse port forward
Server SSHs into ingress.nixc.us and writes a Traefik dynamic config file
Traefik detects the new config and routes HTTPS traffic to tunnel-server:<port>
Traffic flows: Internet -> Traefik -> tunnel-server:port -> SSH tunnel -> client -> local service
When the client disconnects, the config file is removed and the port is freed

Quick Start

1. Generate SSH Keys

mkdir -p keys

# Server host key (for the SSH server that clients connect to)
ssh-keygen -t ed25519 -f keys/host_key -N ""

# Client key (for tunnel clients)
ssh-keygen -t ed25519 -f keys/id_ed25519 -N ""

# Authorize the client (install this file as the tunnel user's authorized_keys on ingress.nixc.us)
cat keys/id_ed25519.pub > keys/authorized_keys

# Deploy key for SSHing into the Traefik host
# (use an existing key that has root access to ingress.nixc.us,
#  or generate one and add its .pub to the Traefik host's authorized_keys)
cp ~/.ssh/ingress_deploy_key keys/traefik_deploy_key

2. Enable Traefik File Provider

On ingress.nixc.us, ensure Traefik has the file provider enabled. If Traefik is run via Docker stack with CLI args, add:

--providers.file.directory=/root/traefik/dynamic
--providers.file.watch=true

Or via SSH (non-interactive):

ssh root@ingress.nixc.us "mkdir -p /root/traefik/dynamic"

Then update your Traefik stack/service to mount the directory and add the CLI args.

3. Start the Tunnel Server

docker compose up -d

4. Run a Client

On any remote host where your service is running:

Using a mounted key file:

docker run -d \
  --name tunnel-client \
  -e TUNNEL_SERVER=ingress.nixc.us:2222 \
  -e TUNNEL_DOMAIN=myapp.example.com \
  -e TUNNEL_PORT=8080 \
  -e TUNNEL_KEY=/keys/id_ed25519 \
  -v /path/to/keys:/keys:ro \
  --network host \
  tunnel-client

Using raw PEM key content in an envvar:

docker run -d \
  --name tunnel-client \
  -e TUNNEL_SERVER=ingress.nixc.us:2222 \
  -e TUNNEL_DOMAIN=myapp.example.com \
  -e TUNNEL_PORT=8080 \
  -e "TUNNEL_KEY=$(cat /path/to/id_ed25519)" \
  --network host \
  tunnel-client

Environment Variables

Server

Variable	Description	Default
`SSH_PORT`	SSH listen port for tunnel clients	`2222`
`PORT_RANGE_START`	First allocatable tunnel port	`10000`
`PORT_RANGE_END`	Last allocatable tunnel port	`10100`
`SSH_HOST_KEY`	Path to SSH host private key	`/keys/host_key`
`AUTHORIZED_KEYS`	Path to authorized_keys file (tunnel user’s keys from ingress.nixc.us)	`/keys/authorized_keys`
`TRAEFIK_SSH_HOST`	Traefik host to SSH into (required)	-
`TRAEFIK_SSH_USER`	SSH user on the Traefik host	`root`
`TRAEFIK_SSH_KEY`	SSH key for Traefik host (path or PEM) (required)	-
`TRAEFIK_CONFIG_DIR`	Remote path for dynamic configs	`/root/traefik/dynamic`
`TRAEFIK_ENTRYPOINT`	Traefik entrypoint name	`websecure`
`TRAEFIK_CERT_RESOLVER`	Traefik TLS cert resolver	`letsencryptresolver`

Client

Variable	Description	Default
`TUNNEL_SERVER`	Server host:port (required)	-
`TUNNEL_DOMAIN`	Public domain to expose (required)	-
`TUNNEL_PORT`	Local port of your service	`8080`
`TUNNEL_KEY`	SSH private key — file path or raw PEM (required)	`/keys/id_ed25519`

Generated Traefik Config

When a client connects requesting myapp.example.com, the server writes this to the Traefik host:

# /root/traefik/dynamic/tunnel-myapp-example-com.yml
http:
  routers:
    tunnel-myapp-example-com-router:
      rule: "Host(`myapp.example.com`)"
      entryPoints:
        - websecure
      tls:
        certResolver: letsencryptresolver
      service: tunnel-myapp-example-com-service
  services:
    tunnel-myapp-example-com-service:
      loadBalancer:
        servers:
          - url: "http://tunnel-server:10042"

Building

# Build both binaries locally (commit and push these for remote deployment)
go build -o tunnel-server ./cmd/server/
go build -o tunnel-client ./cmd/client/

# Build Docker images
docker compose build                                # server image
docker build --target client -t tunnel-client .     # client image

Binaries and systemd (bare metal)

The repo ships compiled tunnel-server and tunnel-client for remote hosts that run without Docker. Use the included systemd units under systemd/.

Keys: Do not reuse the host’s SSH keys or share one key between hosts or tunnels. Generate a dedicated ed25519 key per tunnel (or per host). Add that key’s public half to the tunnel user’s authorized_keys on ingress.nixc.us (the reverse tunnel server).

Multiple tunnels: Run one systemd instance per tunnel (different env file and optional unit name), e.g. tunnel-client@app1.service and tunnel-client@app2.service each with their own env and key.

Install from git.nixc.us (HTTPS raw)

From a host with curl, install the binary and systemd unit directly from the repo (replace main with your branch if needed):

REPO=https://git.nixc.us/colin/better-argo-tunnels/raw/branch/main
sudo curl -o /usr/local/bin/tunnel-client -L "$REPO/client"
sudo chmod +x /usr/local/bin/tunnel-client
sudo curl -o /etc/systemd/system/tunnel-client.service -L "$REPO/systemd/tunnel-client.service"

Then add a dedicated key for this tunnel, env file, and authorize on the server:

sudo mkdir -p /etc/tunnel-client
sudo ssh-keygen -t ed25519 -f /etc/tunnel-client/id_ed25519 -N ""
# Add the public key to the tunnel user's authorized_keys on ingress.nixc.us:
sudo cat /etc/tunnel-client/id_ed25519.pub
# On ingress.nixc.us as tunnel user: append that line to ~tunnel/.ssh/authorized_keys
sudo cp systemd/tunnel-client.env.example /etc/tunnel-client.env   # or curl from $REPO/systemd/tunnel-client.env.example
sudo edit /etc/tunnel-client.env   # set TUNNEL_SERVER, TUNNEL_DOMAIN, TUNNEL_KEY=/etc/tunnel-client/id_ed25519
sudo systemctl daemon-reload && sudo systemctl enable --now tunnel-client

Client on a remote host (clone or copy)

Install the binary (from repo clone or raw URL above; repo has client/server):

sudo cp client /usr/local/bin/tunnel-client && sudo chmod +x /usr/local/bin/tunnel-client

Copy the systemd unit and create env file:

sudo cp systemd/tunnel-client.service /etc/systemd/system/
sudo cp systemd/tunnel-client.env.example /etc/tunnel-client.env
sudo edit /etc/tunnel-client.env   # set TUNNEL_SERVER, TUNNEL_DOMAIN, TUNNEL_KEY

Use a dedicated ed25519 key for this tunnel (not the host’s keys). Put the private key on the host (e.g. /etc/tunnel-client/id_ed25519) and set TUNNEL_KEY in env. Add the matching public key to the tunnel user’s ~/.ssh/authorized_keys on ingress.nixc.us.

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now tunnel-client
sudo journalctl -u tunnel-client -f

For a second tunnel on the same host, use a separate env and key (e.g. /etc/tunnel-client-app2.env, /etc/tunnel-client/app2_id_ed25519) and a second unit (e.g. copy to tunnel-client-app2.service with EnvironmentFile=/etc/tunnel-client-app2.env).

Server (optional, bare metal)

If you run the tunnel server without Docker:

Install binary and keys under e.g. /etc/tunnel-server/ (host_key, authorized_keys from the tunnel user on ingress.nixc.us, traefik_deploy_key).
Copy systemd/tunnel-server.service to /etc/systemd/system/ and systemd/tunnel-server.env.example to /etc/tunnel-server.env. Set TRAEFIK_SSH_HOST, TRAEFIK_SSH_KEY, and paths to keys.
systemctl enable --now tunnel-server.

Troubleshooting

"unable to authenticate, attempted methods [none publickey]"

This means the tunnel-server rejected the client's key. There are exactly two causes:

1. The client's public key is not in authorized_keys.

The server reads /home/tunnel/.ssh/authorized_keys on ingress.nixc.us. Every client key must have its .pub contents in that file. To check and fix:

# See what keys the server knows about:
ssh root@ingress.nixc.us "cat /home/tunnel/.ssh/authorized_keys"

# Add a missing client key:
ssh root@ingress.nixc.us "cat >> /home/tunnel/.ssh/authorized_keys" < YOUR_CLIENT_KEY.pub

2. The tunnel-server was not restarted after adding the key.

The tunnel-server loads authorized_keys into memory once at startup. Editing the file on disk does nothing until the container restarts. After adding a key, always restart:

ssh root@ingress.nixc.us "docker service update --force better-argo-tunnels_tunnel-server"

Then restart the client so it reconnects immediately instead of waiting for its backoff timer.

Checklist for a new tunnel client

Generate an ed25519 key (or reuse an existing one).
Append the .pub to /home/tunnel/.ssh/authorized_keys on ingress.nixc.us.
Restart the tunnel-server: docker service update --force better-argo-tunnels_tunnel-server.
Start the client container with the private key mounted and TUNNEL_KEY pointing at it.

If all four steps happen, the tunnel will connect. If any step is skipped, it won't.

Security Notes

Only clients whose public keys are in the tunnel user’s authorized_keys on ingress.nixc.us can connect
The server uses a stable host key for client verification
SSH tunnels encrypt all traffic between client and server
The server authenticates to the Traefik host with a separate deploy key
Traefik handles TLS termination with automatic Let's Encrypt certificates

11 KiB Raw Blame History Unescape Escape