Why it exists

I wanted a voice assistant that answers to me, not to a vendor's cloud, and that can actually do things: check the homelab, answer questions, control the house. Wiring the speech pipeline was the easy half. The hard half was letting friends talk to one too, without handing strangers a microphone into my infrastructure.

Two assistants

Friday knows my infrastructure and can act on it. Jarvis, the Discord bot, has zero infrastructure knowledge, zero personal context, and exactly three whitelisted capabilities. They are isolated at the architecture level, not by politeness.

Try to break it. Send the same message to both, then flip the architecture to v1, the version I built first and got wrong, and send it again.

Why it's built this way

The isolation is structural, not textual. A system prompt that says "don't reveal X" is a request, and it fails the first time someone is clever, insistent, or claims to be an admin. Prompt-level defenses are a speed bump for an adversary and an illusion for the operator.

v1 called Home Assistant's conversation API and reused the desk agent. It looked elegant: same brain, same tools, no duplication. It failed three separate ways, each sufficient on its own.

• The system prompt was the leak. The desk agent's prompt opened by naming me and my occupation. Anyone asking it to repeat its instructions got all of it.
• Home Assistant exposes entities to the conversation API globally, not per agent. This is the one that defeats the obvious fix. Standing up "a second agent with a safer prompt" would have contained nothing: every agent sees the same exposed-entity set. My friends would have been able to read a household to-do list and drive a media player. Not theoretical. That is what the wiring did.
• It needed a 10-year, full-privilege token living in a container that untrusted speech drives. Home Assistant tokens have no scoping; they inherit the minting user's full rights. A safety guard blocked me from creating it, and that block was correct. It should have been read as a design smell, not an obstacle to route around.

v2 drops Home Assistant entirely: Discord voice, to Whisper, to Gemini called directly, to Kokoro, back to Discord. No HA token, no code path to it. Even if that container were compromised, it holds no credential worth taking. Its whole capability is three functions (web search, media search, media request) and there is no fourth. Secrets live in the process environment where Python uses them and the model never sees them, and tool output is whitelisted field by field, so raw API bodies never reach the model either.

It is told, truthfully, that it has no access to any computer or private information and cannot get any. So when it declines, it isn't role-playing a refusal. It's stating a fact about its situation.

Making it fast

The only interval a human actually feels is from the moment you stop talking to the first sound coming back. I instrumented every stage on the live bot rather than guessing, and the profile was not what I expected.

Build a pipeline and listen to it. The playback is real time: 3.4 seconds of waiting is 3.4 seconds of waiting. That gap is the whole problem, and no bar chart has ever conveyed it.

Speech-to-text and the silence wait are 60% of the delay between them. Text to speech, the stage that sounds like it should be the slow one, is the fastest thing in the pipeline. Spending a night optimising it would have bought nothing, which is the entire argument for measuring before you optimise.

So I cut the silence wait to 600ms and dropped Whisper's beam search to greedy: beam search is worth it on a forty-minute podcast, not on "play Funkytown", where it pays double the decode time to fix a word you were never going to get wrong. I also lowered the wake threshold after the logs showed real wake attempts scoring just under it and being rejected, which is the most infuriating failure this thing has. That took it to about 2.6 seconds. The remaining path to roughly 1.2 seconds is a faster ASR model, streaming the model's reply instead of waiting for all of it, and semantic turn detection that predicts you've finished speaking instead of waiting out a fixed silence.

What I built

• In-browser wake word, on-device. No always-on cloud microphone.
• Whisper speech-to-text on a CUDA GPU, shared with media transcoding.
• Friday: an LLM control brain with tool access to my infrastructure APIs, plus live host sensors (CPU, memory, uptime, storage pool health) wired in from monitoring, so it reads real numbers instead of guessing.
• Kokoro text-to-speech, streaming from the first sentence.
• Jarvis: a Discord voice interface built on per-user audio streams, so it hears individual people in a call.
• Requests attributed to whoever actually spoke, rather than every request in the channel landing under my name.

Responses in the demo are scripted illustrations of each architecture's real capabilities, not live model output. The capability boundaries and the latency figures are the actual ones.

← all projects