Connecting Hermes Desktop to a Raspberry Pi Backend Over SSH

If you’ve installed Hermes Agent on a Raspberry Pi and have been talking to it through Telegram, email, or other messaging bridges, there’s a cleaner option for working at your desk: Hermes Desktop, the GUI frontend. It connects straight to your Pi over an SSH tunnel — no exposed ports, no API keys flying across your network, and no third-party messaging platform in the loop for local use.

I set this up over a weekend and hit exactly one non-obvious snag that cost me an hour. This post walks through the whole thing so you can skip that hour. So I show you now how to connect Hermes Desktop to a Raspberry Pi.

The mental model that matters

The single most important thing to understand before you start: Hermes Desktop is not a replacement for Hermes Agent. It’s a frontend. Your Pi keeps running the agent backend; the Desktop app just gives you a nice window into it.

Even more important — and this is the part that tripped me up — the hermes gateway process actually wears two hats:

1. The messaging gateway — cron jobs, the kanban dispatcher, and any chat platforms you’ve wired up (Telegram, Discord, etc.).
2. The OpenAI-compatible API server on port 8642 — the HTTP endpoint that Hermes Desktop (and tools like Open WebUI) talk to.

The catch: the second one is disabled by default. You can have a perfectly healthy gateway showing “active (running)” while nothing is listening on 8642. Hold that thought — it’s the punchline later.

What you’ll need

• A Raspberry Pi already running Hermes Agent.
• A desktop machine (mine is Windows) with Hermes Desktop installed.
• Both on the same local network.

Step 1 — Set up passwordless SSH

The tunnel relies on key-based SSH auth, so the first job is making sure you can SSH from your desktop to the Pi without typing a password.

On Windows, open PowerShell and generate a key:

ssh-keygen -t rsa -b 4096

Press Enter through the prompts to accept the default path (~/.ssh/id_rsa) and an empty passphrase. This creates two files: id_rsa (private — stays on your PC) and id_rsa.pub (public — goes to the Pi).

Copy the public key to the Pi in one shot:

type $env:USERPROFILE\.ssh\id_rsa.pub | ssh youruser@your-pi-ip "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"

Enter your Pi password one last time, then test:

ssh youruser@your-pi-ip

If it logs in without asking for a password, you’re done with this step.

If it still asks for a password

This is worth a callout because it’s the most common stumbling block. Two things to check on the Pi:

The key must be on a single line. Pasting a key into nano often wraps it across multiple lines, which silently breaks it. Verify with:

wc -l ~/.ssh/authorized_keys

It should report 1 per key. If it’s higher, the key got mangled — clear the file and re-copy it using the PowerShell pipe method above rather than a manual paste.

Permissions must be tight. SSH refuses keys if the files or your home directory are too open:

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
chmod 755 ~

A loose home directory (group- or world-writable) will make SSH reject the key without a useful error message. The verbose flag — ssh -v youruser@your-pi-ip — is your friend for diagnosing what the server is actually doing.

Step 2 — Enable the API server on the Pi

Here’s the snag I mentioned. Starting the gateway is not enough. You have to explicitly turn on the API server that listens on port 8642.

Edit the environment file on the Pi:

nano ~/.hermes/.env

Add:

API_SERVER_ENABLED=true
API_SERVER_KEY=pick-a-secret-key-here

A few notes:

• The key is yours to choose — just remember it, you’ll need it in the Desktop app.
• You can leave API_SERVER_HOST and API_SERVER_PORT unset; they default to 127.0.0.1 and 8642.
• Because we’re tunneling over SSH, binding to localhost is the right call. The tunnel handles remote reach for you, so there’s no reason to expose the port to the wider network.

Save (Ctrl+O, Enter, Ctrl+X) and restart the gateway:

hermes gateway restart

Step 3 — Verify the port is actually open

This is the verification step I wish I’d run first. Check that something is now listening:

ss -tlnp | grep 8642
curl http://127.0.0.1:8642/health

Before enabling the API server, ss returned nothing and curl gave “connection refused” — even though hermes gateway status happily reported the service as active. After flipping the flag, curl returns {"status": "ok"}. That contrast is the whole lesson: a running gateway and a listening API server are two different conditions.

Step 4 — Connect from Hermes Desktop

In Hermes Desktop, choose the Connect via SSH option and fill in:

Field	Value
SSH Host	your Pi’s local IP (e.g. 192.168.X.XXX)
SSH Port	22
Username	your Pi username
Private Key Path	C:\Users\<You>\.ssh\id_rsa
Remote Hermes Port	8642

If there’s a field for the API key, enter the same API_SERVER_KEY you set in Step 2. Hit connect, and the Desktop app tunnels through SSH to the API server on the Pi.

Do you still need Telegram, email, and friends?

No — and that’s the nicest part. Those messaging gateways are optional. They exist so you can reach the agent from your phone when you’re away from your desk. For local work, the Desktop Chat screen is your interface, talking directly to the Pi backend. You get chat, sessions, memory, skills, and schedules in one window, with nothing exposed to the internet.

Keep the messaging bridges around if you want to poke the agent from your phone on the train. Otherwise, leave them off.

Lessons learned

• Hermes Desktop is a frontend, not a reinstall. Your Pi backend stays exactly where it is.
• The SSH tunnel is cleaner than exposing port 8642. No open ports, no API key on the wire.
• hermes gateway runs two services in one process — the messaging gateway and the API server — and the API server is off by default.
• “Gateway running” ≠ “port 8642 listening.” When in doubt, ss -tlnp | grep 8642 and curl .../health tell you the truth in two seconds.
• Most SSH key failures are a wrapped key or loose permissions. Check wc -l on authorized_keys and the chmod values before assuming anything fancier.

Once it’s wired up, a Pi humming away in the corner running your agent, driven from a clean desktop GUI over an encrypted tunnel, is a genuinely satisfying little setup. Happy tinkering.