451 lines
19 KiB
Markdown
451 lines
19 KiB
Markdown
# Reverse SSH Tunnel Server for Traefik
|
||
|
||
A lightweight Go system that lets remote Docker hosts expose HTTP services through a central Traefik reverse proxy via SSH tunnels. The server SSHs into the Traefik ingress host to dynamically manage route configuration.
|
||
|
||
## Architecture
|
||
|
||
```
|
||
Remote Host Tunnel Server Traefik Host (ingress.nixc.us)
|
||
+--------------+ +---------------------+ +---------------------+
|
||
| tunnel-client| ---SSH tunnel---> | tunnel-server | | Traefik |
|
||
| TUNNEL_KEY | | allocates port, | | watches dynamic/ |
|
||
| TUNNEL_DOMAIN| | SSHs into Traefik |--->| routes HTTPS to |
|
||
| TUNNEL_PORT | | host to write config | | tunnel-server ports |
|
||
+--------------+ +---------------------+ +---------------------+
|
||
```
|
||
|
||
### Flow
|
||
|
||
1. Client connects to tunnel-server via SSH using a private key
|
||
2. Client sends domain metadata (`{"domain":"myapp.example.com"}`) over a custom channel
|
||
3. Server allocates a port from the pool and sets up a reverse port forward
|
||
4. Server SSHs into `ingress.nixc.us` and writes a Traefik dynamic config file
|
||
5. Traefik detects the new config and routes HTTPS traffic to `tunnel-server:<port>`
|
||
6. Traffic flows: Internet -> Traefik -> tunnel-server:port -> SSH tunnel -> client -> local service
|
||
7. When the client disconnects, the config file is removed and the port is freed
|
||
|
||
## Quick Start
|
||
|
||
### 1. Generate SSH Keys
|
||
|
||
```bash
|
||
mkdir -p keys
|
||
|
||
# Server host key (for the SSH server that clients connect to)
|
||
ssh-keygen -t ed25519 -f keys/host_key -N ""
|
||
|
||
# Client key (for tunnel clients)
|
||
ssh-keygen -t ed25519 -f keys/id_ed25519 -N ""
|
||
|
||
# Authorize the client (install this file as the tunnel user's authorized_keys on ingress.nixc.us)
|
||
cat keys/id_ed25519.pub > keys/authorized_keys
|
||
|
||
# Deploy key for SSHing into the Traefik host
|
||
# (use an existing key that has root access to ingress.nixc.us,
|
||
# or generate one and add its .pub to the Traefik host's authorized_keys)
|
||
cp ~/.ssh/ingress_deploy_key keys/traefik_deploy_key
|
||
```
|
||
|
||
### 2. Enable Traefik File Provider
|
||
|
||
On `ingress.nixc.us`, ensure Traefik has the file provider enabled.
|
||
If Traefik is run via Docker stack with CLI args, add:
|
||
|
||
```
|
||
--providers.file.directory=/root/traefik/dynamic
|
||
--providers.file.watch=true
|
||
```
|
||
|
||
Or via SSH (non-interactive):
|
||
|
||
```bash
|
||
ssh root@ingress.nixc.us "mkdir -p /root/traefik/dynamic"
|
||
```
|
||
|
||
Then update your Traefik stack/service to mount the directory and add the CLI args.
|
||
|
||
### 3. Start the Tunnel Server
|
||
|
||
```bash
|
||
docker compose up -d
|
||
```
|
||
|
||
### 4. Run a Client
|
||
|
||
On any remote host where your service is running:
|
||
|
||
**Using a mounted key file:**
|
||
```bash
|
||
docker run -d \
|
||
--name tunnel-client \
|
||
-e TUNNEL_SERVER=ingress.nixc.us:2222 \
|
||
-e TUNNEL_DOMAIN=myapp.example.com \
|
||
-e TUNNEL_PORT=8080 \
|
||
-e TUNNEL_KEY=/keys/id_ed25519 \
|
||
-v /path/to/keys:/keys:ro \
|
||
--network host \
|
||
tunnel-client
|
||
```
|
||
|
||
**Using raw PEM key content in an envvar:**
|
||
```bash
|
||
docker run -d \
|
||
--name tunnel-client \
|
||
-e TUNNEL_SERVER=ingress.nixc.us:2222 \
|
||
-e TUNNEL_DOMAIN=myapp.example.com \
|
||
-e TUNNEL_PORT=8080 \
|
||
-e "TUNNEL_KEY=$(cat /path/to/id_ed25519)" \
|
||
--network host \
|
||
tunnel-client
|
||
```
|
||
|
||
## Environment Variables
|
||
|
||
### Server
|
||
|
||
| Variable | Description | Default |
|
||
|---|---|---|
|
||
| `SSH_PORT` | SSH listen port for tunnel clients | `2222` |
|
||
| `PORT_RANGE_START` | First allocatable tunnel port | `10000` |
|
||
| `PORT_RANGE_END` | Last allocatable tunnel port | `10100` |
|
||
| `SSH_HOST_KEY` | Path to SSH host private key | `/keys/host_key` |
|
||
| `AUTHORIZED_KEYS` | Path to authorized_keys file (tunnel user’s keys from ingress.nixc.us) | `/keys/authorized_keys` |
|
||
| `TRAEFIK_SSH_HOST` | Traefik host to SSH into **(required)** | - |
|
||
| `TRAEFIK_SSH_USER` | SSH user on the Traefik host | `root` |
|
||
| `TRAEFIK_SSH_KEY` | SSH key for Traefik host (path or PEM) **(required)** | - |
|
||
| `TRAEFIK_SSH_KNOWN_HOSTS` | known_hosts file for verifying the Traefik SSH host; auto-detects `/keys/traefik_known_hosts` or `/keys/known_hosts` when present | (none) |
|
||
| `TRAEFIK_SSH_STRICT_HOST_KEY` | Fail startup if the Traefik SSH host cannot be verified from known_hosts | `false` |
|
||
| `SSH_MAX_AUTH_TRIES` | Max SSH auth attempts per tunnel-client connection | `3` |
|
||
| `TRAEFIK_ENTRYPOINT` | Traefik entrypoint name | `websecure` |
|
||
| `TRAEFIK_CERT_RESOLVER` | Traefik TLS cert resolver | `letsencryptresolver` |
|
||
| `PURGE_STALE_LABELS_ON_START` | Remove all managed tunnel labels during startup; normally leave disabled so routes survive restarts | `false` |
|
||
| `HEALTHCHECK_ADDR` | TCP address checked by `tunnel-server healthcheck` | `127.0.0.1:$SSH_PORT` |
|
||
| `HEALTHCHECK_TIMEOUT_SECONDS` | Healthcheck TCP dial timeout | `2` |
|
||
|
||
### Client
|
||
|
||
| Variable | Description | Default |
|
||
|---|---|---|
|
||
| `TUNNEL_SERVER` | Server host:port **(required)** | - |
|
||
| `TUNNEL_DOMAIN` | Public domain to expose **(required)** | - |
|
||
| `TUNNEL_PORT` | Local port of your service | `8080` |
|
||
| `TUNNEL_KEY` | SSH private key — file path or raw PEM **(required)** | `/keys/id_ed25519` |
|
||
| `TUNNEL_KNOWN_HOSTS` | known_hosts file for verifying `TUNNEL_SERVER`; auto-detects `/keys/known_hosts` when present | (none) |
|
||
| `TUNNEL_STRICT_HOST_KEY` | Fail startup if `TUNNEL_SERVER` cannot be verified from known_hosts | `false` |
|
||
| `TUNNEL_AUTH_USER` | Optional HTTP Basic auth username (tunnel ingress) | (none) |
|
||
| `TUNNEL_AUTH_PASS` | Optional HTTP Basic auth password | (none) |
|
||
| `TUNNEL_LABELS` | Optional custom Traefik labels for this tunnel, as JSON or newline/semicolon-separated `key=value` entries | (none) |
|
||
| `TUNNEL_LABELS_FILE` | Optional file containing custom labels in the same format as `TUNNEL_LABELS`; env labels override file labels | (none) |
|
||
|
||
Set `TUNNEL_AUTH_USER` / `TUNNEL_AUTH_PASS` via an env file; never commit passwords.
|
||
|
||
### SSH security posture
|
||
|
||
The tunnel client, tunnel server, and server-to-Traefik SSH client all use
|
||
`golang.org/x/crypto/ssh` supported algorithms only, excluding legacy SHA-1-era
|
||
algorithm sets returned by `ssh.InsecureAlgorithms()`. The server also limits
|
||
client auth attempts with `SSH_MAX_AUTH_TRIES` (default `3`).
|
||
|
||
The Docker images run as non-root UID/GID `10001`. Mounted private keys and
|
||
known_hosts files must be readable by that user, for example by using a readable
|
||
group, ACL, Docker secret mode, or a deployment-specific `user:` override while
|
||
file ownership is being migrated.
|
||
|
||
For host authenticity, mount known_hosts files and enable strict mode:
|
||
|
||
```yaml
|
||
services:
|
||
tunnel-client:
|
||
environment:
|
||
TUNNEL_KNOWN_HOSTS: /keys/known_hosts
|
||
TUNNEL_STRICT_HOST_KEY: "true"
|
||
volumes:
|
||
- ./known_hosts:/keys/known_hosts:ro
|
||
```
|
||
|
||
Generate the client known_hosts entry from a trusted network path:
|
||
|
||
```bash
|
||
ssh-keyscan -p 2222 ingress.nixc.us > known_hosts
|
||
```
|
||
|
||
For the tunnel server's SSH connection to the Traefik/Swarm manager:
|
||
|
||
```yaml
|
||
environment:
|
||
TRAEFIK_SSH_KNOWN_HOSTS: /keys/traefik_known_hosts
|
||
TRAEFIK_SSH_STRICT_HOST_KEY: "true"
|
||
volumes:
|
||
- ./traefik_known_hosts:/keys/traefik_known_hosts:ro
|
||
```
|
||
|
||
Without known_hosts, the code keeps a logged compatibility fallback so existing
|
||
deployments continue to work, but it logs the unverified host fingerprint on
|
||
each SSH connection. Treat strict known_hosts as the desired production mode.
|
||
|
||
### Custom client labels
|
||
|
||
Clients can request extra Traefik labels for their own generated router, service,
|
||
or tunnel-scoped middlewares. Existing clients remain compatible: if
|
||
`TUNNEL_LABELS` and `TUNNEL_LABELS_FILE` are unset, the client sends the same
|
||
metadata as before.
|
||
|
||
Use placeholders so clients do not need to know the sanitized tunnel key:
|
||
|
||
| Placeholder | Expands to |
|
||
|---|---|
|
||
| `{router}` | Generated router name, e.g. `tunnel-myapp-example-com-router` |
|
||
| `{service}` | Generated service name, e.g. `tunnel-myapp-example-com-service` |
|
||
| `{middleware}` | Generated Basic Auth middleware name, e.g. `tunnel-myapp-example-com-auth` |
|
||
| `{tunKey}` | Sanitized tunnel key, e.g. `myapp-example-com` |
|
||
| `{domain}` | Requested `TUNNEL_DOMAIN` |
|
||
| `{port}` | Allocated tunnel-server port |
|
||
|
||
Example Compose block scalar:
|
||
|
||
```yaml
|
||
environment:
|
||
TUNNEL_LABELS: |
|
||
traefik.http.routers.{router}.priority=100
|
||
traefik.http.middlewares.tunnel-{tunKey}-headers.headers.customrequestheaders.X-Tunnel-Domain={domain}
|
||
traefik.http.routers.{router}.middlewares=tunnel-{tunKey}-headers
|
||
```
|
||
|
||
Example JSON:
|
||
|
||
```bash
|
||
TUNNEL_LABELS='{"traefik.http.routers.{router}.priority":"100"}'
|
||
```
|
||
|
||
Attach an existing middleware already available to Traefik, such as an Authentik
|
||
forward-auth middleware defined on ingress:
|
||
|
||
```yaml
|
||
environment:
|
||
TUNNEL_LABELS: |
|
||
traefik.http.routers.{router}.middlewares=authentik@docker
|
||
```
|
||
|
||
If the middleware is defined by the file provider instead, use its provider
|
||
suffix:
|
||
|
||
```yaml
|
||
environment:
|
||
TUNNEL_LABELS: |
|
||
traefik.http.routers.{router}.middlewares=authentik@file
|
||
```
|
||
|
||
Define a tunnel-scoped middleware by naming it `tunnel-{tunKey}-...`. For
|
||
example, to use the Trackleware Traefik plugin from `../trackleware` and inject
|
||
Notomo tracking into proxied HTML:
|
||
|
||
```yaml
|
||
environment:
|
||
TUNNEL_LABELS: |
|
||
traefik.http.routers.{router}.middlewares=tunnel-{tunKey}-trackleware
|
||
traefik.http.middlewares.tunnel-{tunKey}-trackleware.plugin.trackleware.notomoBaseURL=https://notomo.colinknapp.com
|
||
traefik.http.middlewares.tunnel-{tunKey}-trackleware.plugin.trackleware.siteIDMode=host
|
||
traefik.http.middlewares.tunnel-{tunKey}-trackleware.plugin.trackleware.siteIDPrefix=tw-
|
||
```
|
||
|
||
Use a fixed Trackleware site ID when a host-derived ID is not appropriate:
|
||
|
||
```yaml
|
||
environment:
|
||
TUNNEL_LABELS: |
|
||
traefik.http.routers.{router}.middlewares=tunnel-{tunKey}-trackleware
|
||
traefik.http.middlewares.tunnel-{tunKey}-trackleware.plugin.trackleware.notomoBaseURL=https://notomo.colinknapp.com
|
||
traefik.http.middlewares.tunnel-{tunKey}-trackleware.plugin.trackleware.siteIDMode=static
|
||
traefik.http.middlewares.tunnel-{tunKey}-trackleware.plugin.trackleware.siteID=my-static-site-id
|
||
```
|
||
|
||
Traefik must already have the Trackleware plugin installed/enabled on the ingress
|
||
side; `TUNNEL_LABELS` only attaches and configures middleware labels for the
|
||
client's generated tunnel route.
|
||
|
||
Custom labels are limited to this tunnel's generated router, generated service,
|
||
or middleware names beginning with `tunnel-{tunKey}-`. The server rejects custom
|
||
labels that try to replace managed route labels such as `rule`, `entrypoints`,
|
||
`tls`, `service`, or `loadbalancer.server.port`. If Basic Auth is enabled and a
|
||
custom `router.middlewares` label is supplied, the server keeps the auth
|
||
middleware and appends the requested middlewares.
|
||
|
||
### Credentials and .env
|
||
|
||
For WebDAV stacks (macmini, logos) and any client using HTTP Basic Auth at the tunnel: copy `.env.example` to `.env`, set `TUNNEL_AUTH_USER` and `TUNNEL_AUTH_PASS`, and **never commit `.env`**. Compose files reference `${TUNNEL_AUTH_USER}` and `${TUNNEL_AUTH_PASS}`; Docker Compose reads `.env` from the project directory automatically.
|
||
|
||
#### WebDAV stacks (macmini / logos) — in `archive/`
|
||
|
||
- **Macmini:** from repo root: `docker compose -f archive/docker-compose-macmini.yml up -d --build` — ensure `.env` has tunnel auth set.
|
||
- **Logos:** `docker compose -f archive/docker-compose-logos.yml -p logos up -d --build` — same `.env` or separate; images/SVG-only uploads.
|
||
|
||
See `archive/README.md` for other archived files (Woodpecker CI, reference Traefik YAML, old root binaries).
|
||
|
||
## Generated Traefik Config
|
||
|
||
When a client connects requesting `myapp.example.com`, the server writes this to the Traefik host:
|
||
|
||
```yaml
|
||
# /root/traefik/dynamic/tunnel-myapp-example-com.yml
|
||
http:
|
||
routers:
|
||
tunnel-myapp-example-com-router:
|
||
rule: "Host(`myapp.example.com`)"
|
||
entryPoints:
|
||
- websecure
|
||
tls:
|
||
certResolver: letsencryptresolver
|
||
service: tunnel-myapp-example-com-service
|
||
services:
|
||
tunnel-myapp-example-com-service:
|
||
loadBalancer:
|
||
servers:
|
||
- url: "http://tunnel-server:10042"
|
||
```
|
||
|
||
## Building
|
||
|
||
```bash
|
||
# Build both binaries locally
|
||
go build -o tunnel-server ./cmd/server/
|
||
go build -o tunnel-client ./cmd/client/
|
||
|
||
# Build Docker images
|
||
docker compose build # server image
|
||
docker build --target client -t tunnel-client . # client image
|
||
```
|
||
|
||
Release to GitHub + GHCR (multi-arch): `./ship.sh "your message"`. **Does not run `docker login`** — use credentials you already configured (`docker login ghcr.io`, credential helper, etc.), then ship **buildx push** + **git push**. **`git remote`** (default `github`, or `git config ship.remote`) must be **github.com** unless `ship.registryBase` is set. Optional: `ship.registryBase`, `ship.tagServer`, `ship.tagClient`, `ship.platforms`, `ship.buildxBuilder`.
|
||
|
||
## Binaries and systemd (bare metal)
|
||
|
||
Build `tunnel-server` / `tunnel-client` with `go build`. Use the included systemd units under `systemd/`.
|
||
|
||
**Keys:** Do not reuse the host’s SSH keys or share one key between hosts or tunnels. Generate a dedicated ed25519 key per tunnel (or per host). Add that key’s **public** half to the **tunnel** user’s `authorized_keys` on **ingress.nixc.us** (the reverse tunnel server).
|
||
|
||
**Multiple tunnels:** Run one systemd instance per tunnel (different env file and optional unit name), e.g. `tunnel-client@app1.service` and `tunnel-client@app2.service` each with their own env and key.
|
||
|
||
### Install from git.nixc.us (HTTPS raw)
|
||
|
||
From a host with curl, install the binary and systemd unit directly from the repo (replace `main` with your branch if needed):
|
||
|
||
```bash
|
||
REPO=https://git.nixc.us/colin/better-argo-tunnels/raw/branch/main
|
||
sudo curl -o /usr/local/bin/tunnel-client -L "$REPO/client"
|
||
sudo chmod +x /usr/local/bin/tunnel-client
|
||
sudo curl -o /etc/systemd/system/tunnel-client.service -L "$REPO/systemd/tunnel-client.service"
|
||
```
|
||
|
||
Then add a dedicated key for this tunnel, env file, and authorize on the server:
|
||
|
||
```bash
|
||
sudo mkdir -p /etc/tunnel-client
|
||
sudo ssh-keygen -t ed25519 -f /etc/tunnel-client/id_ed25519 -N ""
|
||
# Add the public key to the tunnel user's authorized_keys on ingress.nixc.us:
|
||
sudo cat /etc/tunnel-client/id_ed25519.pub
|
||
# On ingress.nixc.us as tunnel user: append that line to ~tunnel/.ssh/authorized_keys
|
||
sudo cp systemd/tunnel-client.env.example /etc/tunnel-client.env # or curl from $REPO/systemd/tunnel-client.env.example
|
||
sudo edit /etc/tunnel-client.env # set TUNNEL_SERVER, TUNNEL_DOMAIN, TUNNEL_KEY=/etc/tunnel-client/id_ed25519
|
||
sudo systemctl daemon-reload && sudo systemctl enable --now tunnel-client
|
||
```
|
||
|
||
### Client on a remote host (clone or copy)
|
||
|
||
1. Install the binary (from repo clone, `go build -o tunnel-client ./cmd/client/`, GHCR image extract, or raw URL):
|
||
|
||
```bash
|
||
sudo cp tunnel-client /usr/local/bin/tunnel-client && sudo chmod +x /usr/local/bin/tunnel-client
|
||
```
|
||
|
||
2. Copy the systemd unit and create env file:
|
||
|
||
```bash
|
||
sudo cp systemd/tunnel-client.service /etc/systemd/system/
|
||
sudo cp systemd/tunnel-client.env.example /etc/tunnel-client.env
|
||
sudo edit /etc/tunnel-client.env # set TUNNEL_SERVER, TUNNEL_DOMAIN, TUNNEL_KEY
|
||
```
|
||
|
||
3. Use a **dedicated** ed25519 key for this tunnel (not the host’s keys). Put the private key on the host (e.g. `/etc/tunnel-client/id_ed25519`) and set `TUNNEL_KEY` in env. Add the matching **public** key to the **tunnel** user’s `~/.ssh/authorized_keys` on **ingress.nixc.us**.
|
||
|
||
4. Enable and start:
|
||
|
||
```bash
|
||
sudo systemctl daemon-reload
|
||
sudo systemctl enable --now tunnel-client
|
||
sudo journalctl -u tunnel-client -f
|
||
```
|
||
|
||
For a second tunnel on the same host, use a separate env and key (e.g. `/etc/tunnel-client-app2.env`, `/etc/tunnel-client/app2_id_ed25519`) and a second unit (e.g. copy to `tunnel-client-app2.service` with `EnvironmentFile=/etc/tunnel-client-app2.env`).
|
||
|
||
### Server (optional, bare metal)
|
||
|
||
If you run the tunnel server without Docker:
|
||
|
||
1. Install binary and keys under e.g. `/etc/tunnel-server/` (host_key, authorized_keys from the tunnel user on ingress.nixc.us, traefik_deploy_key).
|
||
2. Copy `systemd/tunnel-server.service` to `/etc/systemd/system/` and `systemd/tunnel-server.env.example` to `/etc/tunnel-server.env`. Set `TRAEFIK_SSH_HOST`, `TRAEFIK_SSH_KEY`, and paths to keys.
|
||
3. `systemctl enable --now tunnel-server`.
|
||
|
||
## Troubleshooting
|
||
|
||
### Self-healing after ingress restarts
|
||
|
||
The tunnel-server keeps Traefik tunnel labels across normal restarts and
|
||
reconciles them on startup instead of deleting every route. Tunnel ports are
|
||
chosen from a stable hash of the domain, so a reconnecting client gets the same
|
||
port and preserved labels point back to the right listener once the client
|
||
returns.
|
||
|
||
Use a non-interactive healthcheck to verify the SSH ingress port:
|
||
|
||
```bash
|
||
tunnel-server healthcheck
|
||
```
|
||
|
||
For Swarm deployments, configure the service healthcheck to run the same command
|
||
inside the container. The production stack example includes this healthcheck
|
||
without requiring a Dockerfile change.
|
||
|
||
If you intentionally need to clear every managed Traefik tunnel route, start the
|
||
server with `PURGE_STALE_LABELS_ON_START=true` once, then disable it again.
|
||
|
||
### Split DNS: HTTP on `sms.taylor-co.com`, tunnel on `ingress.nixc.us`
|
||
|
||
If public DNS for a name points at the **SMS** host but Traefik and tunnel-server
|
||
run on **ingress**, browsers get 404 on the public URL. Use the same Pi keys; see
|
||
[deploy/sms/README.md](deploy/sms/README.md) for an optional SSH TCP relay
|
||
(`TUNNEL_SERVER=sms.taylor-co.com:2222`) and a small Caddy edge that proxies HTTPS
|
||
to `ingress` so `Host` rules still match.
|
||
|
||
### "unable to authenticate, attempted methods [none publickey]"
|
||
|
||
This means the tunnel-server rejected the client's key. There are exactly two causes:
|
||
|
||
**1. The client's public key is not in `authorized_keys`.**
|
||
|
||
The server reads `/home/tunnel/.ssh/authorized_keys` on `ingress.nixc.us`. Every client key must have its `.pub` contents in that file. To check and fix:
|
||
|
||
```bash
|
||
# See what keys the server knows about:
|
||
ssh root@ingress.nixc.us "cat /home/tunnel/.ssh/authorized_keys"
|
||
|
||
# Add a missing client key:
|
||
ssh root@ingress.nixc.us "cat >> /home/tunnel/.ssh/authorized_keys" < YOUR_CLIENT_KEY.pub
|
||
```
|
||
|
||
### Checklist for a new tunnel client
|
||
|
||
1. Generate an ed25519 key (or reuse an existing one).
|
||
2. Append the `.pub` to `/home/tunnel/.ssh/authorized_keys` on `ingress.nixc.us`.
|
||
3. Start the client container with the private key mounted and `TUNNEL_KEY` pointing at it.
|
||
|
||
The tunnel-server reloads `authorized_keys` on each auth attempt, so new keys do
|
||
not require a server restart.
|
||
|
||
## Security Notes
|
||
|
||
- Only clients whose public keys are in the tunnel user’s `authorized_keys` on ingress.nixc.us can connect
|
||
- The server uses a stable host key for client verification
|
||
- SSH tunnels encrypt all traffic between client and server
|
||
- The server authenticates to the Traefik host with a separate deploy key
|
||
- Traefik handles TLS termination with automatic Let's Encrypt certificates
|