AI Applications

  1. 1 Prompt Engineering for Production "I treat a prompt as a spec: role and task, rules including what to do when unsure, an exact output format, a few examples of hard cases, then delimited context and the input. Stable parts go first for caching. I keep an eval set of real inputs, change one thing at a time, version the prompt with... 4 min read
  2. 2 Structured Output: Getting Reliable JSON from an LLM "There are three levels: prompt instructions, JSON mode or tool calling for valid syntax, and schema-constrained decoding, where the decoder masks any token that would break the schema. I'd use schema-constrained output, keep the schema simple, allow null for missing values, and still validate in... 5 min read
  3. 3 Streaming and Latency: TTFT and TPOT "LLM latency has two parts: TTFT, which covers queueing and prefill and grows with prompt length, and TPOT, the time per generated token, set by model size, batch load and memory bandwidth. Total time is roughly TTFT plus output tokens times TPOT. I'd stream over SSE so users read early, measure ... 4 min read
  4. 4 Conversation Memory for LLM Apps "The model is stateless, so memory is built by the app. I'd use three layers: recent turns verbatim, a running summary for long sessions, and a long-term store of extracted facts and preferences that I retrieve per message. Writing is a policy decision: add, update, delete or ignore, never store ... 4 min read
  5. 5 Vector Search: HNSW, IVF and Approximate Nearest Neighbours "Exact search is linear in the number of vectors, so at scale we use ANN. IVF clusters vectors and searches the nprobe nearest clusters; HNSW builds a layered proximity graph and walks it greedily, tuned by M and efSearch. PQ or int8 compression shrinks memory for very large collections, with re-... 5 min read
  6. 6 RAG End to End "RAG has an offline part: ingest, clean, chunk, embed and index with metadata and permissions; and an online part: rewrite the query, retrieve candidates with hybrid search filtered by access, rerank to a few passages, and generate an answer grounded in them with citations and a 'not found' optio... 4 min read
  7. 7 Chunking Strategies for RAG "Chunks are the unit of retrieval, so each should be self-contained, on one topic, and carry its title and section path. I'd default to structure-aware chunks of a few hundred tokens with light overlap, keep tables and code intact, and consider parent–child: search small chunks, send the parent s... 4 min read
  8. 8 Hybrid Search and Reranking "I'd run keyword search (BM25) and vector search in parallel, because one catches exact terms and the other catches meaning, and merge them with reciprocal rank fusion, which only needs ranks. Then a cross-encoder reranks the top 50 or so and the best 3–10 go to the LLM. I'd measure recall of the... 5 min read
  9. 9 Evaluating a RAG System "I'd build a test set of real questions with gold passages and reference answers, including unanswerable ones. Then I'd measure retrieval with recall@k and MRR, and generation with faithfulness, relevance and correctness, scored by an LLM judge calibrated against human labels. Every change runs a... 5 min read
  10. 10 Evals and LLM-as-Judge "Evals are a test suite: a dataset of real and adversarial cases, a scorer, and a runner that compares versions case by case. I use code checks for anything objective, LLM-as-judge with a specific rubric for open-ended quality, and humans for high-stakes cases and for calibrating the judge. I con... 4 min read
  11. 11 Prompt Injection and Guardrails "Prompt injection works because instructions and data share one token stream, so the model can't reliably tell them apart. Direct injection comes from the user; indirect injection hides in content the system reads, which is worse for agents. I'd defend in layers: input checks, delimiting untruste... 5 min read
  12. 12 Semantic Caching for LLM Apps "A semantic cache embeds each query, looks up the nearest past query, and returns its answer if similarity passes a tuned threshold; otherwise it calls the model and stores the result. The risk is reusing an answer for a query that looks similar but means something different, so I'd tune the thre... 4 min read
  13. 13 Token Rate Limits and Quotas "For LLMs I limit tokens, not just requests, because one request can be a hundred thousand tokens. I'd use a token bucket per tenant: reserve input tokens plus max_tokens at admission, reconcile with actual output after, refill at the plan's rate. Limits stack per user, per tenant and per deploym... 4 min read