Building AI apps

Shipping an AI feature is shipping a system with a non-deterministic component, a per-request bill, and an external dependency that can degrade without an outage. Plan for all three from day one: evals gate changes, cost is tracked per request, and every AI feature has an off switch.

Build vs buy, per layer: framework, evals, observability, prompt management. Own only what differentiates your product.

Token optimization

Prompt caching

Caching reuses a stable prompt prefix across requests. Cached reads cost about a tenth of normal input tokens. Cache writes cost about a quarter more. It pays off after a couple of requests with the same prefix.

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

const response = await client.messages.create({
  model: "claude-sonnet-5",
  max_tokens: 16000,
  cache_control: { type: "ephemeral" },  // cache the stable prefix
  system: longSystemPrompt,               // keep this byte-for-byte stable
  messages: [{ role: "user", content: userQuery }],
});

console.log(response.usage.cache_creation_input_tokens); // written to cache
console.log(response.usage.cache_read_input_tokens);     // served from cache

Caching is a prefix match. Any byte change invalidates everything after it. Short prefixes (below a model-dependent minimum) don't cache at all.

If cache reads stay at zero across repeated requests, something in the prefix is changing. Find it.

Model routing

Route by task, not by habit. Most traffic doesn't need the most capable model.

// Keep model IDs in config so upgrades are one change
const MODELS = {
  fast: "claude-haiku-4-5",   // classification, extraction, summaries
  default: "claude-sonnet-5", // most product traffic
  deep: "claude-opus-5",      // hard reasoning, agentic work
};

function selectModel(task: "classify" | "answer" | "agent"): string {
  if (task === "classify") return MODELS.fast;
  if (task === "agent") return MODELS.deep;
  return MODELS.default;
}

Before building a cascade of models, measure the simpler option: one strong model at lower effort. Newer models at low effort often match older ones at high effort, and one model means one cache.

Routing strategies:

Track cost per request and per user from the first release. Model spend grows with success, and finance should not learn about it from the invoice.

Agent frameworks

Start with the thinnest layer that works. A framework is a dependency your team has to understand, upgrade, and debug. Reach for one when it clearly saves work, not by default.

Claude Agent SDK: Claude Code's harness as a library, with built-in tools (read, edit, bash, search), context management, hooks, subagents, and permissions.

import { query } from "@anthropic-ai/claude-agent-sdk";

for await (const message of query({
  prompt: "Refactor the auth module. Done when tests pass.",
  options: { allowedTools: ["Read", "Edit", "Bash"] },
})) {
  if (message.type === "result") console.log(message.result);
}

bun add @anthropic-ai/claude-agent-sdk   # TypeScript
pip install claude-agent-sdk             # Python

Docs: https://code.claude.com/docs/en/agent-sdk

LangGraph fits when the workflow is a graph with state: conditional branches on model output, human-in-the-loop checkpoints, and resumable runs.

LangChain fits when you want its loaders, splitters, and integrations for RAG and document processing.

Tools for AI

Follow the provider's tool spec. Keep tool descriptions short and precise. They are prompts too. Design tools narrow and least-privilege (see Tool Design Best Practices).

Prompt management

Prompts are code. Version them, review changes, and don't hardcode them where only a deploy can change them. Hosted tools (LangSmith, LangFuse, Vellum) add versioning and A/B testing. Every prompt change goes through evals, like a code change goes through tests.

Observability

Trace every model call: inputs, outputs, latency, tokens, cost, and the user feedback it got. PostHog LLM tracing is the simple option. LangSmith and LangFuse go further. Collect thumbs-up and thumbs-down ratings from day one. They are the cheapest eval data you'll ever get.

Eval (testing AI)

Evals are tests for AI behavior, and they gate every prompt and model change.

What to test: correctness, groundedness, PII leakage, hallucination, toxicity, and tone. Tools: LangFuse, LangSmith, PromptFoo.

LangFuse (open source): traces, datasets, prompt versioning, feedback.

# Self-hosted with its docker compose file, or use https://cloud.langfuse.com
docker compose up -d

LangSmith: automated eval runs, human labeling, prompt regression tests, production monitoring.

Eval types:

Example eval setup:

from langfuse import Langfuse

langfuse = Langfuse()
dataset = langfuse.create_dataset("customer-support-eval")

dataset.create_item(
    input={"query": "What's your refund policy?"},
    expected_output="We offer 30-day refunds...",
)

for item in dataset.items:
    response = my_llm_app(item.input)
    langfuse.score(
        trace_id=response.trace_id,
        name="correctness",
        value=1 if matches_expected(response, item.expected_output) else 0,
    )

Where eval data comes from: real traffic (with PII scrubbed), past incidents, and adversarial cases your team writes. Synthetic data fills gaps. It doesn't replace real examples.

Fine tuning

Avoid it unless you've exhausted prompting, retrieval, and model choice, and have the budget to redo it. A fine-tuned model can't move to next quarter's better base model without repeating the work.

Example prompts for building AI apps:

Tools:
"Create a read-only tool for querying our Postgres database. Anthropic tool spec,
JSON results, table schema in the description, parameterized queries only."

RAG:
"Implement docs search: Docling chunks, Voyage embeddings, PGVector with HNSW,
top 5 chunks with sources, ACL filter per user."

Evals:
"Eval suite for the support bot: product facts against the catalog, no PII leakage
with fake customer data, refuses questions about products we don't sell."

Plan for degradation. Model outages, rate limits, and quietly worse answers all happen. Every AI feature needs a fallback path and a kill switch.