# Reverse SSH Tunnel Server for Traefik

A lightweight Go system that lets remote Docker hosts expose HTTP services through a central Traefik reverse proxy via SSH tunnels. The server SSHs into the Traefik ingress host to dynamically manage route configuration.

## Architecture

```
Remote Host                        Tunnel Server              Traefik Host (ingress.nixc.us)
+--------------+                   +---------------------+    +---------------------+
| tunnel-client| ---SSH tunnel---> | tunnel-server       |    | Traefik             |
| TUNNEL_KEY   |                   | allocates port,     |    | watches dynamic/    |
| TUNNEL_DOMAIN|                   | SSHs into Traefik   |--->| routes HTTPS to     |
| TUNNEL_PORT  |                   | host to write config |    | tunnel-server ports |
+--------------+                   +---------------------+    +---------------------+
```

### Flow

1. Client connects to tunnel-server via SSH using a private key
2. Client sends domain metadata (`{"domain":"myapp.example.com"}`) over a custom channel
3. Server allocates a port from the pool and sets up a reverse port forward
4. Server SSHs into `ingress.nixc.us` and writes a Traefik dynamic config file
5. Traefik detects the new config and routes HTTPS traffic to `tunnel-server:<port>`
6. Traffic flows: Internet -> Traefik -> tunnel-server:port -> SSH tunnel -> client -> local service
7. When the client disconnects, the config file is removed and the port is freed

## Quick Start

### 1. Generate SSH Keys

```bash
mkdir -p keys

# Server host key (for the SSH server that clients connect to)
ssh-keygen -t ed25519 -f keys/host_key -N ""

# Client key (for tunnel clients)
ssh-keygen -t ed25519 -f keys/id_ed25519 -N ""

# Authorize the client (install this file as the tunnel user's authorized_keys on ingress.nixc.us)
cat keys/id_ed25519.pub > keys/authorized_keys

# Deploy key for SSHing into the Traefik host
# (use an existing key that has root access to ingress.nixc.us,
#  or generate one and add its .pub to the Traefik host's authorized_keys)
cp ~/.ssh/ingress_deploy_key keys/traefik_deploy_key
```

### 2. Enable Traefik File Provider

On `ingress.nixc.us`, ensure Traefik has the file provider enabled.
If Traefik is run via Docker stack with CLI args, add:

```
--providers.file.directory=/root/traefik/dynamic
--providers.file.watch=true
```

Or via SSH (non-interactive):

```bash
ssh root@ingress.nixc.us "mkdir -p /root/traefik/dynamic"
```

Then update your Traefik stack/service to mount the directory and add the CLI args.

### 3. Start the Tunnel Server

```bash
docker compose up -d
```

### 4. Run a Client

On any remote host where your service is running:

**Using a mounted key file:**
```bash
docker run -d \
  --name tunnel-client \
  -e TUNNEL_SERVER=ingress.nixc.us:2222 \
  -e TUNNEL_DOMAIN=myapp.example.com \
  -e TUNNEL_PORT=8080 \
  -e TUNNEL_KEY=/keys/id_ed25519 \
  -v /path/to/keys:/keys:ro \
  --network host \
  tunnel-client
```

**Using raw PEM key content in an envvar:**
```bash
docker run -d \
  --name tunnel-client \
  -e TUNNEL_SERVER=ingress.nixc.us:2222 \
  -e TUNNEL_DOMAIN=myapp.example.com \
  -e TUNNEL_PORT=8080 \
  -e "TUNNEL_KEY=$(cat /path/to/id_ed25519)" \
  --network host \
  tunnel-client
```

## Environment Variables

### Server

| Variable | Description | Default |
|---|---|---|
| `SSH_PORT` | SSH listen port for tunnel clients | `2222` |
| `PORT_RANGE_START` | First allocatable tunnel port | `10000` |
| `PORT_RANGE_END` | Last allocatable tunnel port | `10100` |
| `SSH_HOST_KEY` | Path to SSH host private key | `/keys/host_key` |
| `AUTHORIZED_KEYS` | Path to authorized_keys file (tunnel user’s keys from ingress.nixc.us) | `/keys/authorized_keys` |
| `TRAEFIK_SSH_HOST` | Traefik host to SSH into **(required)** | - |
| `TRAEFIK_SSH_USER` | SSH user on the Traefik host | `root` |
| `TRAEFIK_SSH_KEY` | SSH key for Traefik host (path or PEM) **(required)** | - |
| `TRAEFIK_SSH_KNOWN_HOSTS` | known_hosts file for verifying the Traefik SSH host; auto-detects `/keys/traefik_known_hosts` or `/keys/known_hosts` when present | (none) |
| `TRAEFIK_SSH_STRICT_HOST_KEY` | Fail startup if the Traefik SSH host cannot be verified from known_hosts | `false` |
| `SSH_MAX_AUTH_TRIES` | Max SSH auth attempts per tunnel-client connection | `3` |
| `TRAEFIK_ENTRYPOINT` | Traefik entrypoint name | `websecure` |
| `TRAEFIK_CERT_RESOLVER` | Traefik TLS cert resolver | `letsencryptresolver` |
| `PURGE_STALE_LABELS_ON_START` | Remove all managed tunnel labels during startup; normally leave disabled so routes survive restarts | `false` |
| `HEALTHCHECK_ADDR` | TCP address checked by `tunnel-server healthcheck` | `127.0.0.1:$SSH_PORT` |
| `HEALTHCHECK_TIMEOUT_SECONDS` | Healthcheck TCP dial timeout | `2` |

### Client

| Variable | Description | Default |
|---|---|---|
| `TUNNEL_SERVER` | Server host:port **(required)** | - |
| `TUNNEL_DOMAIN` | Public domain to expose **(required)** | - |
| `TUNNEL_PORT` | Local port of your service | `8080` |
| `TUNNEL_KEY` | SSH private key — file path or raw PEM **(required)** | `/keys/id_ed25519` |
| `TUNNEL_KNOWN_HOSTS` | known_hosts file for verifying `TUNNEL_SERVER`; auto-detects `/keys/known_hosts` when present | (none) |
| `TUNNEL_STRICT_HOST_KEY` | Fail startup if `TUNNEL_SERVER` cannot be verified from known_hosts | `false` |
| `TUNNEL_AUTH_USER` | Optional HTTP Basic auth username (tunnel ingress) | (none) |
| `TUNNEL_AUTH_PASS` | Optional HTTP Basic auth password | (none) |
| `TUNNEL_LABELS` | Optional custom Traefik labels for this tunnel, as JSON or newline/semicolon-separated `key=value` entries | (none) |
| `TUNNEL_LABELS_FILE` | Optional file containing custom labels in the same format as `TUNNEL_LABELS`; env labels override file labels | (none) |

Set `TUNNEL_AUTH_USER` / `TUNNEL_AUTH_PASS` via an env file; never commit passwords.

### SSH security posture

The tunnel client, tunnel server, and server-to-Traefik SSH client all use
`golang.org/x/crypto/ssh` supported algorithms only, excluding legacy SHA-1-era
algorithm sets returned by `ssh.InsecureAlgorithms()`. The server also limits
client auth attempts with `SSH_MAX_AUTH_TRIES` (default `3`).

The Docker images run as non-root UID/GID `10001`. Mounted private keys and
known_hosts files must be readable by that user, for example by using a readable
group, ACL, Docker secret mode, or a deployment-specific `user:` override while
file ownership is being migrated.

For host authenticity, mount known_hosts files and enable strict mode:

```yaml
services:
  tunnel-client:
    environment:
      TUNNEL_KNOWN_HOSTS: /keys/known_hosts
      TUNNEL_STRICT_HOST_KEY: "true"
    volumes:
      - ./known_hosts:/keys/known_hosts:ro
```

Generate the client known_hosts entry from a trusted network path:

```bash
ssh-keyscan -p 2222 ingress.nixc.us > known_hosts
```

For the tunnel server's SSH connection to the Traefik/Swarm manager:

```yaml
environment:
  TRAEFIK_SSH_KNOWN_HOSTS: /keys/traefik_known_hosts
  TRAEFIK_SSH_STRICT_HOST_KEY: "true"
volumes:
  - ./traefik_known_hosts:/keys/traefik_known_hosts:ro
```

Without known_hosts, the code keeps a logged compatibility fallback so existing
deployments continue to work, but it logs the unverified host fingerprint on
each SSH connection. Treat strict known_hosts as the desired production mode.

### Custom client labels

Clients can request extra Traefik labels for their own generated router, service,
or tunnel-scoped middlewares. Existing clients remain compatible: if
`TUNNEL_LABELS` and `TUNNEL_LABELS_FILE` are unset, the client sends the same
metadata as before.

Use placeholders so clients do not need to know the sanitized tunnel key:

| Placeholder | Expands to |
|---|---|
| `{router}` | Generated router name, e.g. `tunnel-myapp-example-com-router` |
| `{service}` | Generated service name, e.g. `tunnel-myapp-example-com-service` |
| `{middleware}` | Generated Basic Auth middleware name, e.g. `tunnel-myapp-example-com-auth` |
| `{tunKey}` | Sanitized tunnel key, e.g. `myapp-example-com` |
| `{domain}` | Requested `TUNNEL_DOMAIN` |
| `{port}` | Allocated tunnel-server port |

Example Compose block scalar:

```yaml
environment:
  TUNNEL_LABELS: |
    traefik.http.routers.{router}.priority=100
    traefik.http.middlewares.tunnel-{tunKey}-headers.headers.customrequestheaders.X-Tunnel-Domain={domain}
    traefik.http.routers.{router}.middlewares=tunnel-{tunKey}-headers
```

Example JSON:

```bash
TUNNEL_LABELS='{"traefik.http.routers.{router}.priority":"100"}'
```

Attach an existing middleware already available to Traefik, such as an Authentik
forward-auth middleware defined on ingress:

```yaml
environment:
  TUNNEL_LABELS: |
    traefik.http.routers.{router}.middlewares=authentik@docker
```

If the middleware is defined by the file provider instead, use its provider
suffix:

```yaml
environment:
  TUNNEL_LABELS: |
    traefik.http.routers.{router}.middlewares=authentik@file
```

Define a tunnel-scoped middleware by naming it `tunnel-{tunKey}-...`. For
example, to use the Trackleware Traefik plugin from `../trackleware` and inject
Notomo tracking into proxied HTML:

```yaml
environment:
  TUNNEL_LABELS: |
    traefik.http.routers.{router}.middlewares=tunnel-{tunKey}-trackleware
    traefik.http.middlewares.tunnel-{tunKey}-trackleware.plugin.trackleware.notomoBaseURL=https://notomo.colinknapp.com
    traefik.http.middlewares.tunnel-{tunKey}-trackleware.plugin.trackleware.siteIDMode=host
    traefik.http.middlewares.tunnel-{tunKey}-trackleware.plugin.trackleware.siteIDPrefix=tw-
```

Use a fixed Trackleware site ID when a host-derived ID is not appropriate:

```yaml
environment:
  TUNNEL_LABELS: |
    traefik.http.routers.{router}.middlewares=tunnel-{tunKey}-trackleware
    traefik.http.middlewares.tunnel-{tunKey}-trackleware.plugin.trackleware.notomoBaseURL=https://notomo.colinknapp.com
    traefik.http.middlewares.tunnel-{tunKey}-trackleware.plugin.trackleware.siteIDMode=static
    traefik.http.middlewares.tunnel-{tunKey}-trackleware.plugin.trackleware.siteID=my-static-site-id
```

Traefik must already have the Trackleware plugin installed/enabled on the ingress
side; `TUNNEL_LABELS` only attaches and configures middleware labels for the
client's generated tunnel route.

Custom labels are limited to this tunnel's generated router, generated service,
or middleware names beginning with `tunnel-{tunKey}-`. The server rejects custom
labels that try to replace managed route labels such as `rule`, `entrypoints`,
`tls`, `service`, or `loadbalancer.server.port`. If Basic Auth is enabled and a
custom `router.middlewares` label is supplied, the server keeps the auth
middleware and appends the requested middlewares.

### Credentials and .env

For WebDAV stacks (macmini, logos) and any client using HTTP Basic Auth at the tunnel: copy `.env.example` to `.env`, set `TUNNEL_AUTH_USER` and `TUNNEL_AUTH_PASS`, and **never commit `.env`**. Compose files reference `${TUNNEL_AUTH_USER}` and `${TUNNEL_AUTH_PASS}`; Docker Compose reads `.env` from the project directory automatically.

#### WebDAV stacks (macmini / logos) — in `archive/`

- **Macmini:** from repo root: `docker compose -f archive/docker-compose-macmini.yml up -d --build` — ensure `.env` has tunnel auth set.
- **Logos:** `docker compose -f archive/docker-compose-logos.yml -p logos up -d --build` — same `.env` or separate; images/SVG-only uploads.

See `archive/README.md` for other archived files (Woodpecker CI, reference Traefik YAML, old root binaries).

## Generated Traefik Config

When a client connects requesting `myapp.example.com`, the server writes this to the Traefik host:

```yaml
# /root/traefik/dynamic/tunnel-myapp-example-com.yml
http:
  routers:
    tunnel-myapp-example-com-router:
      rule: "Host(`myapp.example.com`)"
      entryPoints:
        - websecure
      tls:
        certResolver: letsencryptresolver
      service: tunnel-myapp-example-com-service
  services:
    tunnel-myapp-example-com-service:
      loadBalancer:
        servers:
          - url: "http://tunnel-server:10042"
```

## Building

```bash
# Build both binaries locally
go build -o tunnel-server ./cmd/server/
go build -o tunnel-client ./cmd/client/

# Build Docker images
docker compose build                                # server image
docker build --target client -t tunnel-client .     # client image
```

Release to GitHub + GHCR (multi-arch): `./ship.sh "your message"`. **Does not run `docker login`** — use credentials you already configured (`docker login ghcr.io`, credential helper, etc.), then ship **buildx push** + **git push**. **`git remote`** (default `github`, or `git config ship.remote`) must be **github.com** unless `ship.registryBase` is set. Optional: `ship.registryBase`, `ship.tagServer`, `ship.tagClient`, `ship.platforms`, `ship.buildxBuilder`.

## Binaries and systemd (bare metal)

Build `tunnel-server` / `tunnel-client` with `go build`. Use the included systemd units under `systemd/`.

**Keys:** Do not reuse the host’s SSH keys or share one key between hosts or tunnels. Generate a dedicated ed25519 key per tunnel (or per host). Add that key’s **public** half to the **tunnel** user’s `authorized_keys` on **ingress.nixc.us** (the reverse tunnel server).

**Multiple tunnels:** Run one systemd instance per tunnel (different env file and optional unit name), e.g. `tunnel-client@app1.service` and `tunnel-client@app2.service` each with their own env and key.

### Install from git.nixc.us (HTTPS raw)

From a host with curl, install the binary and systemd unit directly from the repo (replace `main` with your branch if needed):

```bash
REPO=https://git.nixc.us/colin/better-argo-tunnels/raw/branch/main
sudo curl -o /usr/local/bin/tunnel-client -L "$REPO/client"
sudo chmod +x /usr/local/bin/tunnel-client
sudo curl -o /etc/systemd/system/tunnel-client.service -L "$REPO/systemd/tunnel-client.service"
```

Then add a dedicated key for this tunnel, env file, and authorize on the server:

```bash
sudo mkdir -p /etc/tunnel-client
sudo ssh-keygen -t ed25519 -f /etc/tunnel-client/id_ed25519 -N ""
# Add the public key to the tunnel user's authorized_keys on ingress.nixc.us:
sudo cat /etc/tunnel-client/id_ed25519.pub
# On ingress.nixc.us as tunnel user: append that line to ~tunnel/.ssh/authorized_keys
sudo cp systemd/tunnel-client.env.example /etc/tunnel-client.env   # or curl from $REPO/systemd/tunnel-client.env.example
sudo edit /etc/tunnel-client.env   # set TUNNEL_SERVER, TUNNEL_DOMAIN, TUNNEL_KEY=/etc/tunnel-client/id_ed25519
sudo systemctl daemon-reload && sudo systemctl enable --now tunnel-client
```

### Client on a remote host (clone or copy)

1. Install the binary (from repo clone, `go build -o tunnel-client ./cmd/client/`, GHCR image extract, or raw URL):

   ```bash
   sudo cp tunnel-client /usr/local/bin/tunnel-client && sudo chmod +x /usr/local/bin/tunnel-client
   ```

2. Copy the systemd unit and create env file:

   ```bash
   sudo cp systemd/tunnel-client.service /etc/systemd/system/
   sudo cp systemd/tunnel-client.env.example /etc/tunnel-client.env
   sudo edit /etc/tunnel-client.env   # set TUNNEL_SERVER, TUNNEL_DOMAIN, TUNNEL_KEY
   ```

3. Use a **dedicated** ed25519 key for this tunnel (not the host’s keys). Put the private key on the host (e.g. `/etc/tunnel-client/id_ed25519`) and set `TUNNEL_KEY` in env. Add the matching **public** key to the **tunnel** user’s `~/.ssh/authorized_keys` on **ingress.nixc.us**.

4. Enable and start:

   ```bash
   sudo systemctl daemon-reload
   sudo systemctl enable --now tunnel-client
   sudo journalctl -u tunnel-client -f
   ```

For a second tunnel on the same host, use a separate env and key (e.g. `/etc/tunnel-client-app2.env`, `/etc/tunnel-client/app2_id_ed25519`) and a second unit (e.g. copy to `tunnel-client-app2.service` with `EnvironmentFile=/etc/tunnel-client-app2.env`).

### Server (optional, bare metal)

If you run the tunnel server without Docker:

1. Install binary and keys under e.g. `/etc/tunnel-server/` (host_key, authorized_keys from the tunnel user on ingress.nixc.us, traefik_deploy_key).
2. Copy `systemd/tunnel-server.service` to `/etc/systemd/system/` and `systemd/tunnel-server.env.example` to `/etc/tunnel-server.env`. Set `TRAEFIK_SSH_HOST`, `TRAEFIK_SSH_KEY`, and paths to keys.
3. `systemctl enable --now tunnel-server`.

## Troubleshooting

### Self-healing after ingress restarts

The tunnel-server keeps Traefik tunnel labels across normal restarts and
reconciles them on startup instead of deleting every route. Tunnel ports are
chosen from a stable hash of the domain, so a reconnecting client gets the same
port and preserved labels point back to the right listener once the client
returns.

Use a non-interactive healthcheck to verify the SSH ingress port:

```bash
tunnel-server healthcheck
```

For Swarm deployments, configure the service healthcheck to run the same command
inside the container. The production stack example includes this healthcheck
without requiring a Dockerfile change.

If you intentionally need to clear every managed Traefik tunnel route, start the
server with `PURGE_STALE_LABELS_ON_START=true` once, then disable it again.

### Split DNS: HTTP on `sms.taylor-co.com`, tunnel on `ingress.nixc.us`

If public DNS for a name points at the **SMS** host but Traefik and tunnel-server
run on **ingress**, browsers get 404 on the public URL. Use the same Pi keys; see
[deploy/sms/README.md](deploy/sms/README.md) for an optional SSH TCP relay
(`TUNNEL_SERVER=sms.taylor-co.com:2222`) and a small Caddy edge that proxies HTTPS
to `ingress` so `Host` rules still match.

### "unable to authenticate, attempted methods [none publickey]"

This means the tunnel-server rejected the client's key. There are exactly two causes:

**1. The client's public key is not in `authorized_keys`.**

The server reads `/home/tunnel/.ssh/authorized_keys` on `ingress.nixc.us`. Every client key must have its `.pub` contents in that file. To check and fix:

```bash
# See what keys the server knows about:
ssh root@ingress.nixc.us "cat /home/tunnel/.ssh/authorized_keys"

# Add a missing client key:
ssh root@ingress.nixc.us "cat >> /home/tunnel/.ssh/authorized_keys" < YOUR_CLIENT_KEY.pub
```

### Checklist for a new tunnel client

1. Generate an ed25519 key (or reuse an existing one).
2. Append the `.pub` to `/home/tunnel/.ssh/authorized_keys` on `ingress.nixc.us`.
3. Start the client container with the private key mounted and `TUNNEL_KEY` pointing at it.

The tunnel-server reloads `authorized_keys` on each auth attempt, so new keys do
not require a server restart.

## Security Notes

- Only clients whose public keys are in the tunnel user’s `authorized_keys` on ingress.nixc.us can connect
- The server uses a stable host key for client verification
- SSH tunnels encrypt all traffic between client and server
- The server authenticates to the Traefik host with a separate deploy key
- Traefik handles TLS termination with automatic Let's Encrypt certificates