19 KiB

Raw Blame History

Reverse SSH Tunnel Server for Traefik

A lightweight Go system that lets remote Docker hosts expose HTTP services through a central Traefik reverse proxy via SSH tunnels. The server SSHs into the Traefik ingress host to dynamically manage route configuration.

Architecture

Remote Host                        Tunnel Server              Traefik Host (ingress.nixc.us)
+--------------+                   +---------------------+    +---------------------+
| tunnel-client| ---SSH tunnel---> | tunnel-server       |    | Traefik             |
| TUNNEL_KEY   |                   | allocates port,     |    | watches dynamic/    |
| TUNNEL_DOMAIN|                   | SSHs into Traefik   |--->| routes HTTPS to     |
| TUNNEL_PORT  |                   | host to write config |    | tunnel-server ports |
+--------------+                   +---------------------+    +---------------------+

Flow

Client connects to tunnel-server via SSH using a private key
Client sends domain metadata ({"domain":"myapp.example.com"}) over a custom channel
Server allocates a port from the pool and sets up a reverse port forward
Server SSHs into ingress.nixc.us and writes a Traefik dynamic config file
Traefik detects the new config and routes HTTPS traffic to tunnel-server:<port>
Traffic flows: Internet -> Traefik -> tunnel-server:port -> SSH tunnel -> client -> local service
When the client disconnects, the config file is removed and the port is freed

Quick Start

1. Generate SSH Keys

mkdir -p keys

# Server host key (for the SSH server that clients connect to)
ssh-keygen -t ed25519 -f keys/host_key -N ""

# Client key (for tunnel clients)
ssh-keygen -t ed25519 -f keys/id_ed25519 -N ""

# Authorize the client (install this file as the tunnel user's authorized_keys on ingress.nixc.us)
cat keys/id_ed25519.pub > keys/authorized_keys

# Deploy key for SSHing into the Traefik host
# (use an existing key that has root access to ingress.nixc.us,
#  or generate one and add its .pub to the Traefik host's authorized_keys)
cp ~/.ssh/ingress_deploy_key keys/traefik_deploy_key

2. Enable Traefik File Provider

On ingress.nixc.us, ensure Traefik has the file provider enabled. If Traefik is run via Docker stack with CLI args, add:

--providers.file.directory=/root/traefik/dynamic
--providers.file.watch=true

Or via SSH (non-interactive):

ssh root@ingress.nixc.us "mkdir -p /root/traefik/dynamic"

Then update your Traefik stack/service to mount the directory and add the CLI args.

3. Start the Tunnel Server

docker compose up -d

4. Run a Client

On any remote host where your service is running:

Using a mounted key file:

docker run -d \
  --name tunnel-client \
  -e TUNNEL_SERVER=ingress.nixc.us:2222 \
  -e TUNNEL_DOMAIN=myapp.example.com \
  -e TUNNEL_PORT=8080 \
  -e TUNNEL_KEY=/keys/id_ed25519 \
  -v /path/to/keys:/keys:ro \
  --network host \
  tunnel-client

Using raw PEM key content in an envvar:

docker run -d \
  --name tunnel-client \
  -e TUNNEL_SERVER=ingress.nixc.us:2222 \
  -e TUNNEL_DOMAIN=myapp.example.com \
  -e TUNNEL_PORT=8080 \
  -e "TUNNEL_KEY=$(cat /path/to/id_ed25519)" \
  --network host \
  tunnel-client

Environment Variables

Server

Variable	Description	Default
`SSH_PORT`	SSH listen port for tunnel clients	`2222`
`PORT_RANGE_START`	First allocatable tunnel port	`10000`
`PORT_RANGE_END`	Last allocatable tunnel port	`10100`
`SSH_HOST_KEY`	Path to SSH host private key	`/keys/host_key`
`AUTHORIZED_KEYS`	Path to authorized_keys file (tunnel user’s keys from ingress.nixc.us)	`/keys/authorized_keys`
`TRAEFIK_SSH_HOST`	Traefik host to SSH into (required)	-
`TRAEFIK_SSH_USER`	SSH user on the Traefik host	`root`
`TRAEFIK_SSH_KEY`	SSH key for Traefik host (path or PEM) (required)	-
`TRAEFIK_SSH_KNOWN_HOSTS`	known_hosts file for verifying the Traefik SSH host; auto-detects `/keys/traefik_known_hosts` or `/keys/known_hosts` when present	(none)
`TRAEFIK_SSH_STRICT_HOST_KEY`	Fail startup if the Traefik SSH host cannot be verified from known_hosts	`false`
`SSH_MAX_AUTH_TRIES`	Max SSH auth attempts per tunnel-client connection	`3`
`TRAEFIK_ENTRYPOINT`	Traefik entrypoint name	`websecure`
`TRAEFIK_CERT_RESOLVER`	Traefik TLS cert resolver	`letsencryptresolver`
`PURGE_STALE_LABELS_ON_START`	Remove all managed tunnel labels during startup; normally leave disabled so routes survive restarts	`false`
`HEALTHCHECK_ADDR`	TCP address checked by `tunnel-server healthcheck`	`127.0.0.1:$SSH_PORT`
`HEALTHCHECK_TIMEOUT_SECONDS`	Healthcheck TCP dial timeout	`2`

Client

Variable	Description	Default
`TUNNEL_SERVER`	Server host:port (required)	-
`TUNNEL_DOMAIN`	Public domain to expose (required)	-
`TUNNEL_PORT`	Local port of your service	`8080`
`TUNNEL_BACKEND_TLS`	Backend connection mode: `auto` tries verified TLS then plain TCP, `plain` skips TLS probing, `tls` requires verified TLS	`auto`
`TUNNEL_KEY`	SSH private key — file path or raw PEM (required)	`/keys/id_ed25519`
`TUNNEL_KNOWN_HOSTS`	known_hosts file for verifying `TUNNEL_SERVER`; auto-detects `/keys/known_hosts` when present	(none)
`TUNNEL_STRICT_HOST_KEY`	Fail startup if `TUNNEL_SERVER` cannot be verified from known_hosts	`false`
`TUNNEL_AUTH_USER`	Optional HTTP Basic auth username (tunnel ingress)	(none)
`TUNNEL_AUTH_PASS`	Optional HTTP Basic auth password	(none)
`TUNNEL_LABELS`	Optional custom Traefik labels for this tunnel, as JSON or newline/semicolon-separated `key=value` entries	(none)
`TUNNEL_LABELS_FILE`	Optional file containing custom labels in the same format as `TUNNEL_LABELS`; env labels override file labels	(none)

Set TUNNEL_AUTH_USER / TUNNEL_AUTH_PASS via an env file; never commit passwords.

Use TUNNEL_BACKEND_TLS=plain for normal HTTP backends to avoid the client probing the service with a TLS handshake before forwarding traffic. Keep the default auto when you want compatibility with either HTTP or HTTPS backends. Use TUNNEL_BACKEND_TLS=tls only when the backend presents a certificate trusted by the client image and valid for TUNNEL_HOST / TUNNEL_BACKEND_HOST.

SSH security posture

The tunnel client, tunnel server, and server-to-Traefik SSH client all use golang.org/x/crypto/ssh supported algorithms only, excluding legacy SHA-1-era algorithm sets returned by ssh.InsecureAlgorithms(). The server also limits client auth attempts with SSH_MAX_AUTH_TRIES (default 3).

The Docker images run as non-root UID/GID 10001. Mounted private keys and known_hosts files must be readable by that user, for example by using a readable group, ACL, Docker secret mode, or a deployment-specific user: override while file ownership is being migrated.

For host authenticity, mount known_hosts files and enable strict mode:

services:
  tunnel-client:
    environment:
      TUNNEL_KNOWN_HOSTS: /keys/known_hosts
      TUNNEL_STRICT_HOST_KEY: "true"
    volumes:
      - ./known_hosts:/keys/known_hosts:ro

Generate the client known_hosts entry from a trusted network path:

ssh-keyscan -p 2222 ingress.nixc.us > known_hosts

For the tunnel server's SSH connection to the Traefik/Swarm manager:

environment:
  TRAEFIK_SSH_KNOWN_HOSTS: /keys/traefik_known_hosts
  TRAEFIK_SSH_STRICT_HOST_KEY: "true"
volumes:
  - ./traefik_known_hosts:/keys/traefik_known_hosts:ro

Without known_hosts, the code keeps a logged compatibility fallback so existing deployments continue to work, but it logs the unverified host fingerprint on each SSH connection. Treat strict known_hosts as the desired production mode.

Custom client labels

Clients can request extra Traefik labels for their own generated router, service, or tunnel-scoped middlewares. Existing clients remain compatible: if TUNNEL_LABELS and TUNNEL_LABELS_FILE are unset, the client sends the same metadata as before.

Use placeholders so clients do not need to know the sanitized tunnel key:

Placeholder	Expands to
`{router}`	Generated router name, e.g. `tunnel-myapp-example-com-router`
`{service}`	Generated service name, e.g. `tunnel-myapp-example-com-service`
`{middleware}`	Generated Basic Auth middleware name, e.g. `tunnel-myapp-example-com-auth`
`{tunKey}`	Sanitized tunnel key, e.g. `myapp-example-com`
`{domain}`	Requested `TUNNEL_DOMAIN`
`{port}`	Allocated tunnel-server port

Example Compose block scalar:

environment:
  TUNNEL_LABELS: |
    traefik.http.routers.{router}.priority=100
    traefik.http.middlewares.tunnel-{tunKey}-headers.headers.customrequestheaders.X-Tunnel-Domain={domain}
    traefik.http.routers.{router}.middlewares=tunnel-{tunKey}-headers

Example JSON:

TUNNEL_LABELS='{"traefik.http.routers.{router}.priority":"100"}'

Attach an existing middleware already available to Traefik, such as an Authentik forward-auth middleware defined on ingress:

environment:
  TUNNEL_LABELS: |
    traefik.http.routers.{router}.middlewares=authentik@docker

If the middleware is defined by the file provider instead, use its provider suffix:

environment:
  TUNNEL_LABELS: |
    traefik.http.routers.{router}.middlewares=authentik@file

Define a tunnel-scoped middleware by naming it tunnel-{tunKey}-.... For example, to use the Trackleware Traefik plugin from ../trackleware and inject Notomo tracking into proxied HTML:

environment:
  TUNNEL_LABELS: |
    traefik.http.routers.{router}.middlewares=tunnel-{tunKey}-trackleware
    traefik.http.middlewares.tunnel-{tunKey}-trackleware.plugin.trackleware.notomoBaseURL=https://notomo.colinknapp.com
    traefik.http.middlewares.tunnel-{tunKey}-trackleware.plugin.trackleware.siteIDMode=host
    traefik.http.middlewares.tunnel-{tunKey}-trackleware.plugin.trackleware.siteIDPrefix=tw-

Use a fixed Trackleware site ID when a host-derived ID is not appropriate:

environment:
  TUNNEL_LABELS: |
    traefik.http.routers.{router}.middlewares=tunnel-{tunKey}-trackleware
    traefik.http.middlewares.tunnel-{tunKey}-trackleware.plugin.trackleware.notomoBaseURL=https://notomo.colinknapp.com
    traefik.http.middlewares.tunnel-{tunKey}-trackleware.plugin.trackleware.siteIDMode=static
    traefik.http.middlewares.tunnel-{tunKey}-trackleware.plugin.trackleware.siteID=my-static-site-id

Traefik must already have the Trackleware plugin installed/enabled on the ingress side; TUNNEL_LABELS only attaches and configures middleware labels for the client's generated tunnel route.

Custom labels are limited to this tunnel's generated router, generated service, or middleware names beginning with tunnel-{tunKey}-. The server rejects custom labels that try to replace managed route labels such as rule, entrypoints, tls, service, or loadbalancer.server.port. If Basic Auth is enabled and a custom router.middlewares label is supplied, the server keeps the auth middleware and appends the requested middlewares.

Credentials and .env

For WebDAV stacks (macmini, logos) and any client using HTTP Basic Auth at the tunnel: copy .env.example to .env, set TUNNEL_AUTH_USER and TUNNEL_AUTH_PASS, and never commit .env. Compose files reference ${TUNNEL_AUTH_USER} and ${TUNNEL_AUTH_PASS}; Docker Compose reads .env from the project directory automatically.

WebDAV stacks (macmini / logos) — in `archive/`

Macmini: from repo root: docker compose -f archive/docker-compose-macmini.yml up -d --build — ensure .env has tunnel auth set.
Logos: docker compose -f archive/docker-compose-logos.yml -p logos up -d --build — same .env or separate; images/SVG-only uploads.

See archive/README.md for other archived files (Woodpecker CI, reference Traefik YAML, old root binaries).

Generated Traefik Config

When a client connects requesting myapp.example.com, the server writes this to the Traefik host:

# /root/traefik/dynamic/tunnel-myapp-example-com.yml
http:
  routers:
    tunnel-myapp-example-com-router:
      rule: "Host(`myapp.example.com`)"
      entryPoints:
        - websecure
      tls:
        certResolver: letsencryptresolver
      service: tunnel-myapp-example-com-service
  services:
    tunnel-myapp-example-com-service:
      loadBalancer:
        servers:
          - url: "http://tunnel-server:10042"

Building

# Build both binaries locally
go build -o tunnel-server ./cmd/server/
go build -o tunnel-client ./cmd/client/

# Build Docker images
docker compose build                                # server image
docker build --target client -t tunnel-client .     # client image

Release to GitHub + GHCR (multi-arch): ./ship.sh "your message". Does not run docker login — use credentials you already configured (docker login ghcr.io, credential helper, etc.), then ship buildx push + git push. git remote (default github, or git config ship.remote) must be github.com unless ship.registryBase is set. Optional: ship.registryBase, ship.tagServer, ship.tagClient, ship.platforms, ship.buildxBuilder.

Binaries and systemd (bare metal)

Build tunnel-server / tunnel-client with go build. Use the included systemd units under systemd/.

Keys: Do not reuse the host’s SSH keys or share one key between hosts or tunnels. Generate a dedicated ed25519 key per tunnel (or per host). Add that key’s public half to the tunnel user’s authorized_keys on ingress.nixc.us (the reverse tunnel server).

Multiple tunnels: Run one systemd instance per tunnel (different env file and optional unit name), e.g. tunnel-client@app1.service and tunnel-client@app2.service each with their own env and key.

Install from git.nixc.us (HTTPS raw)

From a host with curl, install the binary and systemd unit directly from the repo (replace main with your branch if needed):

REPO=https://git.nixc.us/colin/better-argo-tunnels/raw/branch/main
sudo curl -o /usr/local/bin/tunnel-client -L "$REPO/client"
sudo chmod +x /usr/local/bin/tunnel-client
sudo curl -o /etc/systemd/system/tunnel-client.service -L "$REPO/systemd/tunnel-client.service"

Then add a dedicated key for this tunnel, env file, and authorize on the server:

sudo mkdir -p /etc/tunnel-client
sudo ssh-keygen -t ed25519 -f /etc/tunnel-client/id_ed25519 -N ""
# Add the public key to the tunnel user's authorized_keys on ingress.nixc.us:
sudo cat /etc/tunnel-client/id_ed25519.pub
# On ingress.nixc.us as tunnel user: append that line to ~tunnel/.ssh/authorized_keys
sudo cp systemd/tunnel-client.env.example /etc/tunnel-client.env   # or curl from $REPO/systemd/tunnel-client.env.example
sudo edit /etc/tunnel-client.env   # set TUNNEL_SERVER, TUNNEL_DOMAIN, TUNNEL_KEY=/etc/tunnel-client/id_ed25519
sudo systemctl daemon-reload && sudo systemctl enable --now tunnel-client

Client on a remote host (clone or copy)

Install the binary (from repo clone, go build -o tunnel-client ./cmd/client/, GHCR image extract, or raw URL):
```
sudo cp tunnel-client /usr/local/bin/tunnel-client && sudo chmod +x /usr/local/bin/tunnel-client
```

Copy the systemd unit and create env file:

sudo cp systemd/tunnel-client.service /etc/systemd/system/
sudo cp systemd/tunnel-client.env.example /etc/tunnel-client.env
sudo edit /etc/tunnel-client.env   # set TUNNEL_SERVER, TUNNEL_DOMAIN, TUNNEL_KEY

Use a dedicated ed25519 key for this tunnel (not the host’s keys). Put the private key on the host (e.g. /etc/tunnel-client/id_ed25519) and set TUNNEL_KEY in env. Add the matching public key to the tunnel user’s ~/.ssh/authorized_keys on ingress.nixc.us.

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now tunnel-client
sudo journalctl -u tunnel-client -f

For a second tunnel on the same host, use a separate env and key (e.g. /etc/tunnel-client-app2.env, /etc/tunnel-client/app2_id_ed25519) and a second unit (e.g. copy to tunnel-client-app2.service with EnvironmentFile=/etc/tunnel-client-app2.env).

Server (optional, bare metal)

If you run the tunnel server without Docker:

Install binary and keys under e.g. /etc/tunnel-server/ (host_key, authorized_keys from the tunnel user on ingress.nixc.us, traefik_deploy_key).
Copy systemd/tunnel-server.service to /etc/systemd/system/ and systemd/tunnel-server.env.example to /etc/tunnel-server.env. Set TRAEFIK_SSH_HOST, TRAEFIK_SSH_KEY, and paths to keys.
systemctl enable --now tunnel-server.

Troubleshooting

Self-healing after ingress restarts

The tunnel-server keeps Traefik tunnel labels across normal restarts and reconciles them on startup instead of deleting every route. Tunnel ports are chosen from a stable hash of the domain, so a reconnecting client gets the same port and preserved labels point back to the right listener once the client returns.

Use a non-interactive healthcheck to verify the SSH ingress port:

tunnel-server healthcheck

For Swarm deployments, configure the service healthcheck to run the same command inside the container. The production stack example includes this healthcheck without requiring a Dockerfile change.

If you intentionally need to clear every managed Traefik tunnel route, start the server with PURGE_STALE_LABELS_ON_START=true once, then disable it again.

Split DNS: HTTP on `sms.taylor-co.com`, tunnel on `ingress.nixc.us`

If public DNS for a name points at the SMS host but Traefik and tunnel-server run on ingress, browsers get 404 on the public URL. Use the same Pi keys; see deploy/sms/README.md for an optional SSH TCP relay (TUNNEL_SERVER=sms.taylor-co.com:2222) and a small Caddy edge that proxies HTTPS to ingress so Host rules still match.

"unable to authenticate, attempted methods [none publickey]"

This means the tunnel-server rejected the client's key. There are exactly two causes:

1. The client's public key is not in authorized_keys.

The server reads /home/tunnel/.ssh/authorized_keys on ingress.nixc.us. Every client key must have its .pub contents in that file. To check and fix:

# See what keys the server knows about:
ssh root@ingress.nixc.us "cat /home/tunnel/.ssh/authorized_keys"

# Add a missing client key:
ssh root@ingress.nixc.us "cat >> /home/tunnel/.ssh/authorized_keys" < YOUR_CLIENT_KEY.pub

Checklist for a new tunnel client

Generate an ed25519 key (or reuse an existing one).
Append the .pub to /home/tunnel/.ssh/authorized_keys on ingress.nixc.us.
Start the client container with the private key mounted and TUNNEL_KEY pointing at it.

The tunnel-server reloads authorized_keys on each auth attempt, so new keys do not require a server restart.

Security Notes

Only clients whose public keys are in the tunnel user’s authorized_keys on ingress.nixc.us can connect
The server uses a stable host key for client verification
SSH tunnels encrypt all traffic between client and server
The server authenticates to the Traefik host with a separate deploy key
Traefik handles TLS termination with automatic Let's Encrypt certificates

19 KiB Raw Blame History Unescape Escape