Dansday
People assume the voice is just the chat model with speech bolted on top. It is not. There are two AIs in this bot and they share a configuration table and nothing else. One is a text model I can point at any OpenAI-compatible endpoint. The other is Google's Gemini Live API running in a completely separate process, speaking a different audio format at a different sample rate. Why they are separate is the most interesting thing about how this was built.
Start with chat. The bot answers when you mention it, or when you reply to something it said. That second one is what makes it feel like a conversation instead of a command line.
Before any of that happens, a message has to get past three guards. The first rejects a duplicate delivery from Discord's gateway. The second writes a key to Redis so that when several bot processes get the same event, only one of them answers. The third is an in-flight set keyed on server plus member, which stops one person triggering two overlapping replies by mentioning the bot twice. I release that last one in a finally block. That sounds obvious, but every early return in the handler would otherwise leave someone permanently unable to talk to the bot, and I would rather not find that out from a bug report.
Memory is a rolling window. Every member gets their own conversation per server. I store 40 messages and send the last 10 to the model. The whole thing expires 30 minutes after someone stops talking, and every read pushes that expiry back out. It lives in Redis as a list I trim and re-expire on write, with an in-process Map as a fallback if Redis is not there. That fallback matters more than it sounds. Redis is optional in this project, so everything that depends on it has to degrade instead of fail.
I deliberately do not pick the model. A server owner puts an endpoint URL, a model name and a key into the panel, and the bot talks to whatever is on the other end. Gemini, OpenAI, a self-hosted GLM or Qwen, anything that speaks the chat completions shape.
The cost of that is one function I am not proud of. Every model family wants reasoning turned on in a different way and there is no standard for it, so my code guesses the family by searching for substrings in the model name. GLM wants one set of template arguments. Nemotron wants a reasoning budget of minus one. Qwen and DeepSeek and Kimi each want something slightly different. Gemini just wants an effort level. It works, but it is a guess. If somebody serves a model under a renamed identifier it silently gets no reasoning at all, and nothing anywhere tells me that happened. There is a second function that strips the think and reasoning blocks back out of replies, because several of those families leak their scratchpad into the answer.
Then there is the tool loop, which is where the time goes. I offer the model a set of tools. If it asks for one, the bot runs it, appends the result and calls the model again. That repeats up to five times, and on the last pass I deliberately take the tools away so it has to answer with what it has instead of looping forever. Every pass re-sends the whole growing message array, so the fifth round pays for everything the first four piled up. Tools inside a round used to run one after another, which meant a question needing the leaderboard and the shop and your own bag cost all three lookups added together. They run at the same time now, so it costs the slowest one. If the model asks the same thing twice in a turn, I only pay once.
The tools split into three kinds, and that split is about privacy more than architecture. Server tools read what is already on public web pages: leaderboards across twenty-one metrics, the shop with its prices and timings, live statistics, running giveaways. Account tools read the account of whoever is asking, and only that person. There is no tool anywhere that takes a member name and gives you their data. The caller's Discord id resolves to a member row and everything comes off that row, so "what is in my bag" works and "what is in theirs" has no code path at all. That was not the obvious way to build it. The obvious way is a lookup with a name parameter and it would have been half the code.
All of it respects the server's own module switches. If an owner turns off minigames, those tools stop being offered instead of being offered and refusing. That difference matters. A model that knows about a capability it cannot use will keep trying anyway and then explain to the member why it failed. Making the tool invisible is the only way I found to make the subject actually disappear.
The web tools work the same way from the other side. Search, page fetch and image generation each need their own endpoint, model and key, and each stays completely hidden until all three fields are filled in. A half-configured integration is inactive rather than broken, which quietly removes a whole category of confusing failure. Generated images go up to Discord as files instead of links, because provider URLs expire and an image that dies in three hours is worse than no image.
The wiki tool is the one I would actually show someone. It reads any MediaWiki site including Fandom, and it does not use the API's summary extract. It pulls the fully rendered page and parses that, so infoboxes and tables and changelogs come through. On most game wikis that is the entire content and a summary throws all of it away. Pages cache for ten minutes, capped at 200 entries so it cannot grow forever. There is also a relay, because some wikis sit behind a Cloudflare check that rejects datacenter addresses outright. The bot ships a small PHP script you host somewhere that gets accepted, and it proxies those requests through. That is an ugly answer to somebody else's policy and I have not found a better one.
Voice is a different program entirely. The process that reads Discord messages never joins a voice channel. A separate worker does that. When the chat model decides to join a call it does not join anything, it publishes a command to Redis and reads back a shared state. There is a real answer the model can receive that says the voice worker is unavailable, and it explains that in its own words. The voice state has a sixty second time to live and the worker refreshes it on a heartbeat at half that, so a crashed worker stops advertising itself within a minute instead of leaving a ghost session nothing can clear.
The audio arithmetic is the least glamorous code I have written. Discord speaks 48 kHz stereo in twenty millisecond frames. The model listens at 16 kHz mono and answers at 24 kHz. So there is a converter in the middle that I wrote by hand. It walks the output buffer, picks the nearest input sample by integer division, averages the channels and clamps. That is nearest-neighbour resampling with no filter, and it aliases. Taking every third sample to get from 48 down to 16 folds everything above 8 kHz back into the audible range as distortion. A proper resampling library would do this better. Mine is thirty lines and it is good enough for speech, and I would rather write that down than pretend I did the correct thing.
The wake word runs entirely on the machine. Three ONNX models ship in the repository, a mel spectrogram front end, an embedding model and the trained classifier, and audio gets scored locally against a threshold before anything leaves. I did that for cost and for noise in equal measure. A busy voice channel would otherwise stream continuously to a paid API just to discover nobody was talking to the bot. With local detection there is no charge for audio nobody meant to send, and a room full of people is not constantly waking it up.
Turn-taking is a pile of constants I found rather than derived, and I am not going to pretend otherwise. Speech is detected by RMS energy with a floor of 900, a ceiling of 2,200 and a release at 550, needing three frames in a row before it counts as someone starting and holding for half a second after they stop. The noise floor adapts, rising slowly and falling fast, so a room with a fan in it does not read as constant speech. A turn ends after 500 milliseconds of silence. It was 700 until recently, and that difference looks like nothing on paper and is immediately obvious in a real conversation. Barge-in has its own guard so the first fraction of a second of the bot's own voice does not register as a human interrupting it.
Two behaviours exist purely because silence is unbearable in a voice call. If a lookup takes longer than 600 milliseconds I prompt the model to say something while it waits, so a wiki search sounds like thinking instead of a dropped connection. Chat has no equivalent and probably should. And when a session hits its budget the bot does not just hang up. It asks the model for a short natural goodbye and to leave politely, with an explicit instruction not to explain why. So I wrote a prompt telling a machine to be vague about its reasons so that a spending limit would feel like a person with somewhere else to be. I think that is the right product decision. I also want it written down that it is a small instructed deception.
Observability sits in an odd place and it is worth being exact about where. The container starts with an OpenTelemetry import that hooks every console call and ships it as a structured log record, so everything this part of the bot logs gets collected. What it does not do is trace. There are no spans and no timers around the model call itself, so nothing inside the process knows how long a completion took or which phase of a turn was slow. That data does exist, but it lives one layer out at the gateway the endpoint URL points at. Latency, token counts and cost per call are the gateway's job and it does that job better than in-process timing would. So the problem is not missing data. It is that the gateway sees a stream of completions with no idea which Discord message each one belongs to, and the bot sees a Discord message with no idea what its completion cost. Nothing joins the two. Joining them is a request id, not a tracing stack.
The other gap is simpler. None of this has a single automated test. Every claim I just made, the caller scoping on the account tools, the three guards, the wake threshold, the resampler, is something I verified by reading the code. Nothing would catch a regression in any of it except me noticing. That matters more here than in the rest of the bot for three reasons. Anyone in a server can use it, so a bug reaches people I have never met. It stores their messages for half an hour, so a bug can show the wrong person's conversation. And every call is billed, so a loop that retries when it should stop costs real money. Those three are what I would want tests around before I build anything else on top of this.