LLMGate
I'm building an OpenAI-compatible gateway in Go to manage access to self-hosted inference and work toward predictable behavior under load.
- Built with
- Go · llama.cpp · PostgreSQL · Redis
- Status
- Active development · not production-ready
Between the client and the model
An inference server loads a model and generates tokens. The serving layer has other jobs: checking who can call it, limiting requests, and handling clients that disconnect while a response is still streaming. LLMGate is where I'm building those controls.
The gateway exposes OpenAI-compatible model listing and chat endpoints in front of llama.cpp. It authenticates API keys, checks request limits, and forwards either a JSON response or a server-sent event (SSE) stream. PostgreSQL stores key hashes and inference usage. Redis holds the per-key request credits.
I'm using this project to learn inference backend engineering by following requests through the whole serving path, including the parts that fail.
Bound the work around each stream
Carry cancellation upstream
The streaming proxy flushes response bytes as they arrive and carries the client's cancellation context to the backend. Connection, response-header, and idle-read deadlines bound stalled work without imposing a fixed total duration on a healthy stream. Whether cancellation stops model decoding still depends on the inference server.
Make request admission atomic
Two requests can arrive with one credit left. The Redis Lua script refills the bucket, checks its balance, and spends a credit in one atomic operation. An exhausted bucket returns 429 with retry guidance. If Redis is unavailable, new requests receive 503 rather than bypassing the limit.
Keep credentials and usage separate from content
API keys are stored as hashes, and the proxy removes gateway credentials before forwarding. Usage records capture timings, outcomes, and backend token counts when available, without storing prompt or response text. Missing token counts stay unknown. Counting stream events would not give an accurate token count.
Designing for overload and backend failures
A request-rate limit doesn't tell us whether an inference server has room for another long-running stream. Phase 2 is planned around that distinction: multiple backends, bounded queues, and a fixed number of active requests per backend.
The plan is to route toward healthy, less-loaded backends, reject work when queues fill, and expire requests that wait too long. Retries would only be allowed before any response bytes reach the client. Once streaming starts, switching backends cannot safely continue the same response.
Health tracking, circuit breakers, and Prometheus and Grafana visibility are part of that planned work. I want to check queue depth, tail latency, and client-visible errors under overload and when a backend dies. These are design goals, not completed features or measured results.