Self-hosted · voice
One trusted, one public Running
Two self-hosted voices on the same GPU. Atlas is private: it watches my homelab and tells me what needs me. Jarvis is public: a voice bot my friends talk to in Discord, built so that a full compromise yields no key to my house or my servers, only the same media requests any friend can already make.
- 18
- tools the public bot has
- 0
- Home Assistant or Seerr keys it holds
- 0.66 s
- median model reply in testing, was 1.6 s
- 60 s
- between Atlas health checks
Two assistants, one boundary
In one line: the public bot is safe because it holds nothing worth stealing, not because it was told to keep secrets.
Jarvis answers anyone in my friends' Discord voice channel. It has 18 tools: web search, music, media search and requests, voice-channel moderation, timers, polls, dice and remembered preferences. None of them touch my house or my homelab. Try to break it below, then flip to v1, the version I built first and got wrong.
Desk agent
private · Home Assistant
Only I talk to it. It may know everything. v1 reused it.
possesses
- Home Assistant
- infra APIs
- personal context
- live host sensors
Jarvis
Discord voice
Friends talk to it. Untrusted audience by design.
possesses
Why it's built this way
A system prompt that says "don't reveal X" is a request. It fails the first time someone is clever, insistent, or claims to be an admin. v1 reused my private Home Assistant agent for the public bot and failed three ways, each enough on its own:
- The system prompt was the leak. The private agent's prompt named me and my job. Asking it to repeat its instructions got all of it.
- Home Assistant exposes entities to every conversation agent, not per agent. A second agent with a safer prompt would have contained nothing: my friends could have read a household to-do list and driven a media player.
- It needed a full-privilege, 10-year token in a container that untrusted speech drives. Home Assistant tokens cannot be scoped.
v2 drops Home Assistant entirely: Discord voice, an on-device wake word, Whisper, Gemini, Kokoro, back to Discord. Three rules hold it together:
- Authorization lives in code. Moving, muting or disconnecting someone checks the asker's own Discord permission before any API call. The model can be talked into trying; it cannot be talked past a check it does not run.
- Least privilege for every API. Since 10-31 the bot holds no Seerr key. A small gateway adds the key and allows exactly two calls, search and request; everything else returns 403. Tested: search 200, settings 403, user list 403, listing requests 403.
- The model never sees secrets or raw responses. Keys stay in the process environment, and tool output is whitelisted field by field.
What it hears, and what it keeps
In one line: nothing is transcribed until the wake word fires on the bot's own hardware, and no audio is ever written to disk.
- The wake word runs locally on each person's audio. Without it, audio is dropped.
- After a reply it keeps listening to that one person for 8 seconds, up to 4 follow-ups, so they can answer without the wake word.
- It holds the last 6 exchanges in memory to follow a conversation; a restart forgets them.
- On disk it keeps preferences people explicitly ask it to remember, when each person last spoke (a time, never the words), and a log capped at 30 MB that rotates out. Since 10-31 that log records who spoke, timings, lengths and every tool call with its arguments as an audit trail, but not what anyone said or what Jarvis answered.
On 10-31 I turned off the one feature that broke this rule: it transcribed the last 15 seconds of anyone who left the channel, without the wake word. The stored sign-offs were cleared and the setting is now zero.
Measure, then fix the slow stage
In one line: the language model was the slow stage, so I benchmarked models instead of tuning anything else.
| Stage | Median | Range |
|---|---|---|
| Speech to text (Whisper, GPU) | 0.91 s | 0.83-1.05 s |
| Language model | 2.56 s | 0.77-4.45 s |
| Text to speech (Kokoro, GPU) | 0.30 s | 0.13-0.49 s |
| You stop talking to first sound | 4.1 s | 1.8-5.8 s |
Text to speech, the stage that sounds slow, is the fastest one. So I ran the same five requests through each candidate model three times, with the bot's real prompt and tool list. The model it used answered in 1.6 s at the median, 6.9 s at the 90th percentile, and one call hung for 45 s. Its successor answered in 0.66 s at the median and 0.79 s at the 90th percentile, with no errors. The bot switched on 10-31 and keeps the old model only as a hedge for slow answers.
The private one
In one line: Atlas tells me what needs me, on whichever device I am actually using.
- A watcher on the hypervisor checks health, security and self-heal results every 60 seconds; Home Assistant decides what is worth saying.
- One speaker at a time: my PC when I am at it (unlocked, input in the last 5 minutes, no camera or microphone involved), otherwise my phone.
- Quiet from 01:00 to 10:30 unless it is urgent. Low-value lines are visual only.
- Down alerts carry a signed Restart button that hands the fix to self-heal.
- On the phone it speaks through my own Android app, which checks Do Not Disturb live: the Home Assistant app's sensor was minutes stale and the system killed it mid-sentence.
Replies in the demo are scripted illustrations of each architecture's real capabilities, not live model output. The capability boundaries, the permission check and the numbers on this page are real.
what it demonstrates
Threat modeling for an LLM agent that strangers can talk to, authorization enforced in code rather than in the prompt, least-privilege gateways for third-party APIs, privacy by design for always-on audio, and latency work driven by per-stage measurement.