Reverse SSH Tunnel Server for Traefik

A lightweight Go system that lets remote Docker hosts expose HTTP services through a central Traefik reverse proxy via SSH tunnels. The server SSHs into the Traefik ingress host to dynamically manage route configuration.

Architecture

Remote Host                        Tunnel Server              Traefik Host (ingress.nixc.us)
+--------------+                   +---------------------+    +---------------------+
| tunnel-client| ---SSH tunnel---> | tunnel-server       |    | Traefik             |
| TUNNEL_KEY   |                   | allocates port,     |    | watches dynamic/    |
| TUNNEL_DOMAIN|                   | SSHs into Traefik   |--->| routes HTTPS to     |
| TUNNEL_PORT  |                   | host to write config |    | tunnel-server ports |
+--------------+                   +---------------------+    +---------------------+

Flow

1. Client connects to tunnel-server via SSH using a private key
2. Client sends domain metadata ({"domain":"myapp.example.com"}) over a custom channel
3. Server allocates a port from the pool and sets up a reverse port forward
4. Server SSHs into ingress.nixc.us and writes a Traefik dynamic config file
5. Traefik detects the new config and routes HTTPS traffic to tunnel-server:<port>
6. Traffic flows: Internet -> Traefik -> tunnel-server:port -> SSH tunnel -> client -> local service
7. When the client disconnects, the config file is removed and the port is freed

Quick Start

1. Generate SSH Keys

mkdir -p keys

# Server host key (for the SSH server that clients connect to)
ssh-keygen -t ed25519 -f keys/host_key -N ""

# Client key (for tunnel clients)
ssh-keygen -t ed25519 -f keys/id_ed25519 -N ""

# Authorize the client (install this file as the tunnel user's authorized_keys on ingress.nixc.us)
cat keys/id_ed25519.pub > keys/authorized_keys

# Deploy key for SSHing into the Traefik host
# (use an existing key that has root access to ingress.nixc.us,
#  or generate one and add its .pub to the Traefik host's authorized_keys)
cp ~/.ssh/ingress_deploy_key keys/traefik_deploy_key

2. Enable Traefik File Provider

On ingress.nixc.us, ensure Traefik has the file provider enabled. If Traefik is run via Docker stack with CLI args, add:

--providers.file.directory=/root/traefik/dynamic
--providers.file.watch=true

Or via SSH (non-interactive):

ssh root@ingress.nixc.us "mkdir -p /root/traefik/dynamic"

Then update your Traefik stack/service to mount the directory and add the CLI args.

3. Start the Tunnel Server

docker compose up -d

4. Run a Client

On any remote host where your service is running:

Using a mounted key file:

docker run -d \
  --name tunnel-client \
  -e TUNNEL_SERVER=ingress.nixc.us:2222 \
  -e TUNNEL_DOMAIN=myapp.example.com \
  -e TUNNEL_PORT=8080 \
  -e TUNNEL_KEY=/keys/id_ed25519 \
  -v /path/to/keys:/keys:ro \
  --network host \
  tunnel-client

Using raw PEM key content in an envvar:

docker run -d \
  --name tunnel-client \
  -e TUNNEL_SERVER=ingress.nixc.us:2222 \
  -e TUNNEL_DOMAIN=myapp.example.com \
  -e TUNNEL_PORT=8080 \
  -e "TUNNEL_KEY=$(cat /path/to/id_ed25519)" \
  --network host \
  tunnel-client

Environment Variables

Server

Variable	Description	Default
SSH_PORT	SSH listen port for tunnel clients	2222
PORT_RANGE_START	First allocatable tunnel port	10000
PORT_RANGE_END	Last allocatable tunnel port	10100
SSH_HOST_KEY	Path to SSH host private key	/keys/host_key
AUTHORIZED_KEYS	Path to authorized_keys file (tunnel user’s keys from ingress.nixc.us)	/keys/authorized_keys
TRAEFIK_SSH_HOST	Traefik host to SSH into (required)	-
TRAEFIK_SSH_USER	SSH user on the Traefik host	root
TRAEFIK_SSH_KEY	SSH key for Traefik host (path or PEM) (required)	-
TRAEFIK_ENTRYPOINT	Traefik entrypoint name	websecure
TRAEFIK_CERT_RESOLVER	Traefik TLS cert resolver	letsencryptresolver
PURGE_STALE_LABELS_ON_START	Remove all managed tunnel labels during startup; normally leave disabled so routes survive restarts	false
HEALTHCHECK_ADDR	TCP address checked by tunnel-server healthcheck	127.0.0.1:$SSH_PORT
HEALTHCHECK_TIMEOUT_SECONDS	Healthcheck TCP dial timeout	2

Client

Variable	Description	Default
TUNNEL_SERVER	Server host:port (required)	-
TUNNEL_DOMAIN	Public domain to expose (required)	-
TUNNEL_PORT	Local port of your service	8080
TUNNEL_KEY	SSH private key — file path or raw PEM (required)	/keys/id_ed25519
TUNNEL_AUTH_USER	Optional HTTP Basic auth username (tunnel ingress)	(none)
TUNNEL_AUTH_PASS	Optional HTTP Basic auth password	(none)

Set TUNNEL_AUTH_USER / TUNNEL_AUTH_PASS via an env file; never commit passwords.

Credentials and .env

For WebDAV stacks (macmini, logos) and any client using HTTP Basic Auth at the tunnel: copy .env.example to .env, set TUNNEL_AUTH_USER and TUNNEL_AUTH_PASS, and never commit .env. Compose files reference ${TUNNEL_AUTH_USER} and ${TUNNEL_AUTH_PASS}; Docker Compose reads .env from the project directory automatically.

WebDAV stacks (macmini / logos) — in archive/

• Macmini: from repo root: docker compose -f archive/docker-compose-macmini.yml up -d --build — ensure .env has tunnel auth set.
• Logos: docker compose -f archive/docker-compose-logos.yml -p logos up -d --build — same .env or separate; images/SVG-only uploads.

See archive/README.md for other archived files (Woodpecker CI, reference Traefik YAML, old root binaries).

Generated Traefik Config

When a client connects requesting myapp.example.com, the server writes this to the Traefik host:

# /root/traefik/dynamic/tunnel-myapp-example-com.yml
http:
  routers:
    tunnel-myapp-example-com-router:
      rule: "Host(`myapp.example.com`)"
      entryPoints:
        - websecure
      tls:
        certResolver: letsencryptresolver
      service: tunnel-myapp-example-com-service
  services:
    tunnel-myapp-example-com-service:
      loadBalancer:
        servers:
          - url: "http://tunnel-server:10042"

Building

# Build both binaries locally
go build -o tunnel-server ./cmd/server/
go build -o tunnel-client ./cmd/client/

# Build Docker images
docker compose build                                # server image
docker build --target client -t tunnel-client .     # client image

Release to GitHub + GHCR (multi-arch): ./ship.sh "your message". Does not run docker login — use credentials you already configured (docker login ghcr.io, credential helper, etc.), then ship buildx push + git push. git remote (default github, or git config ship.remote) must be github.com unless ship.registryBase is set. Optional: ship.registryBase, ship.tagServer, ship.tagClient, ship.platforms, ship.buildxBuilder.

Binaries and systemd (bare metal)

Build tunnel-server / tunnel-client with go build. Use the included systemd units under systemd/.

Keys: Do not reuse the host’s SSH keys or share one key between hosts or tunnels. Generate a dedicated ed25519 key per tunnel (or per host). Add that key’s public half to the tunnel user’s authorized_keys on ingress.nixc.us (the reverse tunnel server).

Multiple tunnels: Run one systemd instance per tunnel (different env file and optional unit name), e.g. tunnel-client@app1.service and tunnel-client@app2.service each with their own env and key.

Install from git.nixc.us (HTTPS raw)

From a host with curl, install the binary and systemd unit directly from the repo (replace main with your branch if needed):

REPO=https://git.nixc.us/colin/better-argo-tunnels/raw/branch/main
sudo curl -o /usr/local/bin/tunnel-client -L "$REPO/client"
sudo chmod +x /usr/local/bin/tunnel-client
sudo curl -o /etc/systemd/system/tunnel-client.service -L "$REPO/systemd/tunnel-client.service"

Then add a dedicated key for this tunnel, env file, and authorize on the server:

sudo mkdir -p /etc/tunnel-client
sudo ssh-keygen -t ed25519 -f /etc/tunnel-client/id_ed25519 -N ""
# Add the public key to the tunnel user's authorized_keys on ingress.nixc.us:
sudo cat /etc/tunnel-client/id_ed25519.pub
# On ingress.nixc.us as tunnel user: append that line to ~tunnel/.ssh/authorized_keys
sudo cp systemd/tunnel-client.env.example /etc/tunnel-client.env   # or curl from $REPO/systemd/tunnel-client.env.example
sudo edit /etc/tunnel-client.env   # set TUNNEL_SERVER, TUNNEL_DOMAIN, TUNNEL_KEY=/etc/tunnel-client/id_ed25519
sudo systemctl daemon-reload && sudo systemctl enable --now tunnel-client

Client on a remote host (clone or copy)

1. Install the binary (from repo clone, go build -o tunnel-client ./cmd/client/, GHCR image extract, or raw URL):

   sudo cp tunnel-client /usr/local/bin/tunnel-client && sudo chmod +x /usr/local/bin/tunnel-client
   

2. Copy the systemd unit and create env file:

   sudo cp systemd/tunnel-client.service /etc/systemd/system/
   sudo cp systemd/tunnel-client.env.example /etc/tunnel-client.env
   sudo edit /etc/tunnel-client.env   # set TUNNEL_SERVER, TUNNEL_DOMAIN, TUNNEL_KEY
   

3. Use a dedicated ed25519 key for this tunnel (not the host’s keys). Put the private key on the host (e.g. /etc/tunnel-client/id_ed25519) and set TUNNEL_KEY in env. Add the matching public key to the tunnel user’s ~/.ssh/authorized_keys on ingress.nixc.us.

4. Enable and start:

   sudo systemctl daemon-reload
   sudo systemctl enable --now tunnel-client
   sudo journalctl -u tunnel-client -f
   

For a second tunnel on the same host, use a separate env and key (e.g. /etc/tunnel-client-app2.env, /etc/tunnel-client/app2_id_ed25519) and a second unit (e.g. copy to tunnel-client-app2.service with EnvironmentFile=/etc/tunnel-client-app2.env).

Server (optional, bare metal)

If you run the tunnel server without Docker:

1. Install binary and keys under e.g. /etc/tunnel-server/ (host_key, authorized_keys from the tunnel user on ingress.nixc.us, traefik_deploy_key).
2. Copy systemd/tunnel-server.service to /etc/systemd/system/ and systemd/tunnel-server.env.example to /etc/tunnel-server.env. Set TRAEFIK_SSH_HOST, TRAEFIK_SSH_KEY, and paths to keys.
3. systemctl enable --now tunnel-server.

Troubleshooting

Self-healing after ingress restarts

The tunnel-server keeps Traefik tunnel labels across normal restarts and reconciles them on startup instead of deleting every route. Tunnel ports are chosen from a stable hash of the domain, so a reconnecting client gets the same port and preserved labels point back to the right listener once the client returns.

Use a non-interactive healthcheck to verify the SSH ingress port:

tunnel-server healthcheck

For Swarm deployments, configure the service healthcheck to run the same command inside the container. The production stack example includes this healthcheck without requiring a Dockerfile change.

If you intentionally need to clear every managed Traefik tunnel route, start the server with PURGE_STALE_LABELS_ON_START=true once, then disable it again.

"unable to authenticate, attempted methods [none publickey]"

This means the tunnel-server rejected the client's key. There are exactly two causes:

1. The client's public key is not in authorized_keys.

The server reads /home/tunnel/.ssh/authorized_keys on ingress.nixc.us. Every client key must have its .pub contents in that file. To check and fix:

# See what keys the server knows about:
ssh root@ingress.nixc.us "cat /home/tunnel/.ssh/authorized_keys"

# Add a missing client key:
ssh root@ingress.nixc.us "cat >> /home/tunnel/.ssh/authorized_keys" < YOUR_CLIENT_KEY.pub

Checklist for a new tunnel client

1. Generate an ed25519 key (or reuse an existing one).
2. Append the .pub to /home/tunnel/.ssh/authorized_keys on ingress.nixc.us.
3. Start the client container with the private key mounted and TUNNEL_KEY pointing at it.

The tunnel-server reloads authorized_keys on each auth attempt, so new keys do not require a server restart.

Security Notes

• Only clients whose public keys are in the tunnel user’s authorized_keys on ingress.nixc.us can connect
• The server uses a stable host key for client verification
• SSH tunnels encrypt all traffic between client and server
• The server authenticates to the Traefik host with a separate deploy key
• Traefik handles TLS termination with automatic Let's Encrypt certificates