Module 1 · Lesson 1 of 24The LLM Application StackFree preview
The LLM Application Stack
What sits between your users and a language model, how one API call works end to end, and the thin instrumented wrapper you will build on for the rest of the course.
On this page
Calling a large language model (LLM) API takes about five lines of code. You send some text, you get some text back, and the demo works on the first try. Then you ship it, and the real questions arrive. Why did the bill triple last Tuesday? Why does the support bot sometimes invent a refund policy? Why did a customer's pasted email convince your assistant to ignore its instructions? Why did a model upgrade break the invoice parser nobody touched?
None of those problems live inside the model. They live in the code around it. That code is what this course is about: the LLM application stack, the layers you build between your users and a model so the feature is accurate, safe, affordable, and maintainable. In this lesson you will map that stack, make a first well-instrumented API call in Python and TypeScript, and write the small wrapper module you will extend throughout the course.
Learning objectives
- Describe the layers of a production LLM application and the failure each layer prevents
- Explain why an LLM API behaves like a stateless, non-deterministic, metered function
- Make a first call with the Anthropic Python SDK and read every part of the response
- Translate the same call to the OpenAI SDK and to TypeScript
- Write a thin wrapper that records tokens, latency, and stop reasons for every call
The model is a stateless function
Start with the right mental model. For engineering purposes, a hosted LLM API is a function:
f(model, instructions, conversation, tools, parameters)
-> (content blocks, stop reason, token usage)
Three properties of that function shape everything you build:
- It is stateless. The model does not remember your last request. If you want a conversation, you send the whole conversation every time. If you want the model to know your refund policy, you put the policy, or the relevant part of it, in the request.
- It is non-deterministic. The same input can produce different outputs, and a new model version can change behavior you relied on. You test it with evaluations (Module 6), not with
assertEqualon prose. - It is metered. You pay per token in and per token out, and you wait roughly in proportion to the tokens out. Every design decision is also a cost and latency decision (Lesson 1.3).
Watch out: "Stateless" describes the model call. Some providers also offer stateful conveniences, such as stored conversations or hosted agents. For example, OpenAI's Responses API can store a response and continue from it by ID. Those features are useful, but they move state into someone else's system, which affects data retention, debugging, and portability. Know which one you are using.
The stack, layer by layer
Here is the stack you will build in this course. Every production LLM feature has some version of each layer, even if it is three lines of code today.
+-----------------------------------------------+
Users ---> | Product surface: web UI, API, Slack bot, job |
+-----------------------------------------------+
| Orchestration |
| - prompt and context assembly |
| - retrieval (RAG) <--- your documents |
| - tool calls / agents <--- your APIs |
| - output validation |
+-----------------------------------------------+
| Gateway: auth, routing, retries, quotas, |
| caching, redaction, cost accounting |
+-----------------------------------------------+
| Model providers: hosted APIs or self-hosted |
+-----------------------------------------------+
Cross-cutting: evals | observability | security | cost control
| Layer | Its job | What breaks without it | Module |
|---|---|---|---|
| Model selection | Match each task to a model on capability, cost, latency, and data rules | Overpaying for simple tasks, or failing hard ones | 1 |
| API client | Messages, parameters, streaming, retries, timeouts | Hung requests, cascading failures during provider incidents | 2 |
| Structured output and tools | Turn text into validated data and actions | Parsers that break on a stray comma; tools called with bad arguments | 3 |
| Retrieval (RAG) | Ground answers in your current, private data | Confident answers that are out of date or invented | 4 |
| Agents | Multi-step tool use with budgets and stopping rules | Runaway loops, runaway bills, unsafe actions | 5 |
| Evals and observability | Measure quality and see what happened | Silent regressions after every prompt or model change | 6 |
| Security and cost control | Contain untrusted input, protect data, cap spend | Prompt injection, data leaks, surprise invoices | 7 |
| Production architecture | Gateways, fallbacks, monitoring, safe rollout | Outages you cannot diagnose or roll back | 8 |
You do not need every layer on day one. You do need to know which ones you are skipping and what risk you accept by skipping them.
Your first call
Install the SDK and set two environment variables. This course never hard-codes model IDs, because model names, capabilities, and prices change often. Look up a current model ID in your provider's documentation or models endpoint and keep it in configuration.
pip install anthropic
export ANTHROPIC_API_KEY="..." # from your provider console; never commit it
export LLM_MODEL="<your-model-id>" # a current model ID from the provider's docs
Now make one call. The scenario is a common operations task: summarize an inbound support ticket for a queue view.
# first_call.py
import os
import anthropic
MODEL = os.environ["LLM_MODEL"]
client = anthropic.Anthropic() # reads ANTHROPIC_API_KEY from the environment
response = client.messages.create(
model=MODEL,
max_tokens=300, # hard cap on output tokens for this call
system="You summarize customer support tickets for an operations queue. "
"Reply with one sentence, under 25 words.",
messages=[
{
"role": "user",
"content": "Ticket: Hi, my card was charged twice for invoice 4417 "
"on the 3rd. I need one of the charges reversed ASAP.",
}
],
)
# The response holds a list of typed content blocks, not a single string.
for block in response.content:
if block.type == "text":
print(block.text)
print("stop_reason:", response.stop_reason)
print("tokens in/out:", response.usage.input_tokens, "/", response.usage.output_tokens)
model comes from configuration so you can change it without a deploy. max_tokens is required by the Anthropic API and caps output length, and with it cost and latency. system holds instructions for the whole conversation (Lesson 2.1), and messages is the conversation so far, as alternating user and assistant turns.
Reading the response
The SDK returns a typed object. Serialized, it looks like this (values are illustrative):
{
"id": "msg_...",
"type": "message",
"role": "assistant",
"model": "<your-model-id>",
"content": [
{"type": "text", "text": "Customer reports a duplicate charge on invoice 4417 and wants one reversed."}
],
"stop_reason": "end_turn",
"usage": {"input_tokens": 71, "output_tokens": 18}
}
Three fields deserve a habit of checking on every call:
contentis a list of blocks. Today it holds onetextblock. Later it will also holdtool_useblocks (Module 3) and possibly reasoning blocks on models that think before answering. Never assumecontent[0]is the text you want; filter bytype.stop_reasontells you why generation ended.end_turnmeans the model finished.max_tokensmeans you cut it off, so the output may be incomplete or invalid JSON.tool_usemeans the model wants you to run a tool. Arefusalvalue can appear when the model declines a request. Treat anything unexpected as a signal, not as success.usageis your bill for this call. Log it every time. It is the raw material for cost tracking, budgets, and capacity planning.
The same call with OpenAI and TypeScript
Providers differ in naming more than in concept. Here is the equivalent call with OpenAI's Responses API.
import os
from openai import OpenAI
MODEL = os.environ["LLM_MODEL"]
client = OpenAI() # reads OPENAI_API_KEY
response = client.responses.create(
model=MODEL,
instructions="You summarize customer support tickets for an operations queue. "
"Reply with one sentence, under 25 words.",
input="Ticket: Hi, my card was charged twice for invoice 4417 on the 3rd. "
"I need one of the charges reversed ASAP.",
max_output_tokens=300,
)
print(response.output_text) # convenience property that joins the text output
print(response.status) # "completed", or "incomplete" if cut off
print(response.usage.input_tokens, response.usage.output_tokens)
| Concept | Anthropic Messages API | OpenAI Responses API |
|---|---|---|
| Instructions | system= |
instructions=, or a developer/system message in input |
| Conversation | messages=[...] |
input= (a string or a list of items) |
| Output cap | max_tokens (required) |
max_output_tokens (optional) |
| Text output | Filter content blocks by type == "text" |
response.output_text |
| Why it stopped | stop_reason |
status plus incomplete_details |
And the Anthropic call in TypeScript. Run it as an ES module, for example with tsx, so top-level await works.
import Anthropic from "@anthropic-ai/sdk";
const MODEL = process.env.LLM_MODEL!;
const client = new Anthropic(); // reads ANTHROPIC_API_KEY
const response = await client.messages.create({
model: MODEL,
max_tokens: 300,
system: "You summarize customer support tickets for an operations queue. Reply with one sentence, under 25 words.",
messages: [
{ role: "user", content: "Ticket: Hi, my card was charged twice for invoice 4417 on the 3rd." },
],
});
for (const block of response.content) {
if (block.type === "text") console.log(block.text);
}
console.log(response.stop_reason, response.usage);
Tip: Pick one provider SDK for your core path and learn it deeply. Hide it behind a small interface of your own (you will start one below and finish it in Lesson 8.1). That interface is what lets you add a second provider for fallback or cost without touching feature code.
From one call to a wrapper
Scattering raw SDK calls across a codebase is the most common early mistake. Every call site then has to remember logging, truncation checks, and error handling, and most will not. Route every call through one module instead. Here is the first version; you will add cost tracking (1.3), retries (2.3), and tracing (6.3) later.
# llm.py: the single path every model call in your app goes through
from __future__ import annotations
import logging
import os
import time
from dataclasses import dataclass
import anthropic
MODEL = os.environ["LLM_MODEL"]
log = logging.getLogger("llm")
# Explicit timeout and retry settings beat silent defaults (Lesson 2.3).
client = anthropic.Anthropic(timeout=60.0, max_retries=2)
@dataclass
class LLMResult:
text: str
stop_reason: str | None
input_tokens: int
output_tokens: int
latency_s: float
model: str
def complete(
system: str,
messages: list[dict],
*,
max_tokens: int = 1024,
model: str = MODEL,
feature: str = "unknown", # tag every call with the product feature it serves
) -> LLMResult:
start = time.perf_counter()
response = client.messages.create(
model=model, max_tokens=max_tokens, system=system, messages=messages
)
latency = time.perf_counter() - start
result = LLMResult(
text="".join(b.text for b in response.content if b.type == "text"),
stop_reason=response.stop_reason,
input_tokens=response.usage.input_tokens,
output_tokens=response.usage.output_tokens,
latency_s=latency,
model=model,
)
log.info(
"llm_call feature=%s model=%s in=%d out=%d stop=%s latency=%.2fs",
feature, model, result.input_tokens, result.output_tokens,
result.stop_reason, latency,
)
if result.stop_reason == "max_tokens":
log.warning("feature=%s output truncated at max_tokens=%d", feature, max_tokens)
return result
Small as it is, this wrapper gives you a per-feature log line with tokens and latency, a truncation warning, and one place to change timeouts.
Demo code versus production code
| Concern | Typical demo | Production pattern (where you learn it) |
|---|---|---|
| Model choice | Whatever was in the tutorial | Chosen per task with an eval, stored in config (1.2) |
| Cost | Unknown until the invoice | Estimated up front, tracked per call (1.3, 7.3) |
| Prompt | Inline string | Versioned template with delimited inputs (2.1) |
| Failures | Unhandled exception | Timeouts, backoff, fallbacks (2.3, 8.1) |
| Output | Free text, parsed with hope | Schema-validated objects (3.1) |
| Knowledge | Whatever the model memorized | Retrieved, cited sources (Module 4) |
| Quality | "Looks good to me" | Golden dataset and CI regression tests (Module 6) |
| Security | Trusts all input | Threat model, least privilege, output handling (Module 7) |
What you will build
The course closes with a capstone that pulls every layer together: a grounded, tool-using assistant for an internal knowledge base. It answers questions with citations from your documents, calls a small set of business tools, such as looking up and creating support tickets, under budgets and approval rules, and ships with an evaluation suite that runs in CI. Each module adds one layer to that system, so the code you write in each "Try it" exercise is a building block, not a throwaway.
Try it
Set up your working environment and instrument your first calls (10 to 15 minutes).
- Create a project folder with a virtual environment and install
anthropic(oropenaiif that is your provider). Set your API key andLLM_MODELas environment variables. - Save
llm.pyfrom this lesson, adapting the client if you use OpenAI. Configure logging withlogging.basicConfig(level=logging.INFO). - Write a script that calls
complete()on five realistic inputs from your own work: support tickets, sales emails, or log lines. Use the same system prompt for all five. - Run the script twice. Compare the outputs and the logged token counts. Note which outputs changed between runs.
- Set
max_tokens=10and run once more. Confirm that you seestop_reason=max_tokensand the truncation warning. - Write down the average input and output tokens per call. You will use these numbers to estimate cost in Lesson 1.3.
Key takeaways
- An LLM API is a stateless, non-deterministic, metered function. Your code supplies all the context, all the testing, and all the cost discipline.
- Production AI features are a stack: model selection, a robust client, structured outputs and tools, retrieval, agents, evals, security, and production architecture.
- Read the whole response: filter
contentblocks by type, checkstop_reasonon every call, and logusageevery time. - Providers differ mostly in naming. Learn one SDK well and hide it behind your own interface.
- Keep model IDs in configuration, never in code. Models and prices change faster than your release cycle.
- Route every call through one wrapper so logging, limits, and error handling happen everywhere by default.
Enjoyed this preview?
Unlock all 24 lessons, 5 templates, and your certificate for $599.