An AI assistant that never sees a secret

The chat button on this site answers questions about my projects, stack and prices, and a signed-in client can ask about their own requests. Behind it is a Python service with FastAPI and the Anthropic SDK, running Claude Haiku. The interesting part is what that service is not allowed to do.

The shape

Browser ──► NestJS API ──► assistant (Python, internal) ──► Claude
              │
              └─ checks limits, builds the context

The Python service is reachable only from the API, with an internal key. It never sees the visitor’s session token, never connects to the database, and holds no config secrets.

The API builds the context

Every message goes through the API first, in this order:

A kill switch, so the assistant can be turned off without a deploy.

The message length and the shape of the conversation.

A rate limit per visitor and per signed-in client, and a daily cap for the whole site.

The context: published projects, the stack, quote options and profile links for everyone, and the latest requests of a signed-in client.

If something is not in the context, the model cannot leak it. That is a stronger promise than any instruction in a prompt.

Streaming through

The Python side streams server-sent events. The API pipes them to the browser unchanged.

@app.post("/chat", dependencies=[Depends(require_internal_key)])
def chat(body: ChatRequest, request: Request) -> StreamingResponse:
    async def events() -> AsyncIterator[str]:
        try:
            async for text in stream_reply(client, settings, knowledge, body):
                yield format_event("delta", {"text": text})
        except Exception:
            # The stream has already started with a 200, so the failure goes out as an event.
            yield format_event("error", {"code": "UPSTREAM_ERROR"})
            return
        yield format_event("done", {})

    return StreamingResponse(events(), media_type="text/event-stream")

The internal key is compared with secrets.compare_digest, so its check takes the same time whatever the guess. The service logs token counts and latency, never the conversation.

Evals before changes

A small set of eval cases asks real questions in both languages and checks the answers: does "how do I start a project?" point to the quote form, does a Mongolian question get a Mongolian answer. They run before any prompt change ships.

The safest data in a prompt is the data you never put there.