Module 1 · Lesson 1 of 24The LLM Application StackFree preview

The LLM Application Stack

What sits between your users and a language model, how one API call works end to end, and the thin instrumented wrapper you will build on for the rest of the course.

18 min read

Watch the overview first, then read the lesson.
On this page
  1. Learning objectives
  2. The model is a stateless function
  3. The stack, layer by layer
  4. Your first call
  5. Reading the response
  6. The same call with OpenAI and TypeScript
  7. From one call to a wrapper
  8. Demo code versus production code
  9. What you will build
  10. Try it
  11. Key takeaways

Calling a large language model (LLM) API takes about five lines of code. You send some text, you get some text back, and the demo works on the first try. Then you ship it, and the real questions arrive. Why did the bill triple last Tuesday? Why does the support bot sometimes invent a refund policy? Why did a customer's pasted email convince your assistant to ignore its instructions? Why did a model upgrade break the invoice parser nobody touched?

None of those problems live inside the model. They live in the code around it. That code is what this course is about: the LLM application stack, the layers you build between your users and a model so the feature is accurate, safe, affordable, and maintainable. In this lesson you will map that stack, make a first well-instrumented API call in Python and TypeScript, and write the small wrapper module you will extend throughout the course.

Learning objectives

  • Describe the layers of a production LLM application and the failure each layer prevents
  • Explain why an LLM API behaves like a stateless, non-deterministic, metered function
  • Make a first call with the Anthropic Python SDK and read every part of the response
  • Translate the same call to the OpenAI SDK and to TypeScript
  • Write a thin wrapper that records tokens, latency, and stop reasons for every call

The model is a stateless function

Start with the right mental model. For engineering purposes, a hosted LLM API is a function:

f(model, instructions, conversation, tools, parameters)
    -> (content blocks, stop reason, token usage)

Three properties of that function shape everything you build:

  1. It is stateless. The model does not remember your last request. If you want a conversation, you send the whole conversation every time. If you want the model to know your refund policy, you put the policy, or the relevant part of it, in the request.
  2. It is non-deterministic. The same input can produce different outputs, and a new model version can change behavior you relied on. You test it with evaluations (Module 6), not with assertEqual on prose.
  3. It is metered. You pay per token in and per token out, and you wait roughly in proportion to the tokens out. Every design decision is also a cost and latency decision (Lesson 1.3).

Watch out: "Stateless" describes the model call. Some providers also offer stateful conveniences, such as stored conversations or hosted agents. For example, OpenAI's Responses API can store a response and continue from it by ID. Those features are useful, but they move state into someone else's system, which affects data retention, debugging, and portability. Know which one you are using.

The stack, layer by layer

Here is the stack you will build in this course. Every production LLM feature has some version of each layer, even if it is three lines of code today.

            +-----------------------------------------------+
 Users ---> | Product surface: web UI, API, Slack bot, job  |
            +-----------------------------------------------+
            | Orchestration                                 |
            |   - prompt and context assembly               |
            |   - retrieval (RAG)       <--- your documents |
            |   - tool calls / agents   <--- your APIs      |
            |   - output validation                         |
            +-----------------------------------------------+
            | Gateway: auth, routing, retries, quotas,      |
            | caching, redaction, cost accounting           |
            +-----------------------------------------------+
            | Model providers: hosted APIs or self-hosted   |
            +-----------------------------------------------+
  Cross-cutting: evals | observability | security | cost control
Layer Its job What breaks without it Module
Model selection Match each task to a model on capability, cost, latency, and data rules Overpaying for simple tasks, or failing hard ones 1
API client Messages, parameters, streaming, retries, timeouts Hung requests, cascading failures during provider incidents 2
Structured output and tools Turn text into validated data and actions Parsers that break on a stray comma; tools called with bad arguments 3
Retrieval (RAG) Ground answers in your current, private data Confident answers that are out of date or invented 4
Agents Multi-step tool use with budgets and stopping rules Runaway loops, runaway bills, unsafe actions 5
Evals and observability Measure quality and see what happened Silent regressions after every prompt or model change 6
Security and cost control Contain untrusted input, protect data, cap spend Prompt injection, data leaks, surprise invoices 7
Production architecture Gateways, fallbacks, monitoring, safe rollout Outages you cannot diagnose or roll back 8

You do not need every layer on day one. You do need to know which ones you are skipping and what risk you accept by skipping them.

Your first call

Install the SDK and set two environment variables. This course never hard-codes model IDs, because model names, capabilities, and prices change often. Look up a current model ID in your provider's documentation or models endpoint and keep it in configuration.

pip install anthropic
export ANTHROPIC_API_KEY="..."       # from your provider console; never commit it
export LLM_MODEL="<your-model-id>"   # a current model ID from the provider's docs

Now make one call. The scenario is a common operations task: summarize an inbound support ticket for a queue view.

# first_call.py
import os

import anthropic

MODEL = os.environ["LLM_MODEL"]
client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment

response = client.messages.create(
    model=MODEL,
    max_tokens=300,  # hard cap on output tokens for this call
    system="You summarize customer support tickets for an operations queue. "
           "Reply with one sentence, under 25 words.",
    messages=[
        {
            "role": "user",
            "content": "Ticket: Hi, my card was charged twice for invoice 4417 "
                       "on the 3rd. I need one of the charges reversed ASAP.",
        }
    ],
)

# The response holds a list of typed content blocks, not a single string.
for block in response.content:
    if block.type == "text":
        print(block.text)

print("stop_reason:", response.stop_reason)
print("tokens in/out:", response.usage.input_tokens, "/", response.usage.output_tokens)

model comes from configuration so you can change it without a deploy. max_tokens is required by the Anthropic API and caps output length, and with it cost and latency. system holds instructions for the whole conversation (Lesson 2.1), and messages is the conversation so far, as alternating user and assistant turns.

Reading the response

The SDK returns a typed object. Serialized, it looks like this (values are illustrative):

{
  "id": "msg_...",
  "type": "message",
  "role": "assistant",
  "model": "<your-model-id>",
  "content": [
    {"type": "text", "text": "Customer reports a duplicate charge on invoice 4417 and wants one reversed."}
  ],
  "stop_reason": "end_turn",
  "usage": {"input_tokens": 71, "output_tokens": 18}
}

Three fields deserve a habit of checking on every call:

  • content is a list of blocks. Today it holds one text block. Later it will also hold tool_use blocks (Module 3) and possibly reasoning blocks on models that think before answering. Never assume content[0] is the text you want; filter by type.
  • stop_reason tells you why generation ended. end_turn means the model finished. max_tokens means you cut it off, so the output may be incomplete or invalid JSON. tool_use means the model wants you to run a tool. A refusal value can appear when the model declines a request. Treat anything unexpected as a signal, not as success.
  • usage is your bill for this call. Log it every time. It is the raw material for cost tracking, budgets, and capacity planning.

The same call with OpenAI and TypeScript

Providers differ in naming more than in concept. Here is the equivalent call with OpenAI's Responses API.

import os

from openai import OpenAI

MODEL = os.environ["LLM_MODEL"]
client = OpenAI()  # reads OPENAI_API_KEY

response = client.responses.create(
    model=MODEL,
    instructions="You summarize customer support tickets for an operations queue. "
                 "Reply with one sentence, under 25 words.",
    input="Ticket: Hi, my card was charged twice for invoice 4417 on the 3rd. "
          "I need one of the charges reversed ASAP.",
    max_output_tokens=300,
)

print(response.output_text)  # convenience property that joins the text output
print(response.status)       # "completed", or "incomplete" if cut off
print(response.usage.input_tokens, response.usage.output_tokens)
Concept Anthropic Messages API OpenAI Responses API
Instructions system= instructions=, or a developer/system message in input
Conversation messages=[...] input= (a string or a list of items)
Output cap max_tokens (required) max_output_tokens (optional)
Text output Filter content blocks by type == "text" response.output_text
Why it stopped stop_reason status plus incomplete_details

And the Anthropic call in TypeScript. Run it as an ES module, for example with tsx, so top-level await works.

import Anthropic from "@anthropic-ai/sdk";

const MODEL = process.env.LLM_MODEL!;
const client = new Anthropic(); // reads ANTHROPIC_API_KEY

const response = await client.messages.create({
  model: MODEL,
  max_tokens: 300,
  system: "You summarize customer support tickets for an operations queue. Reply with one sentence, under 25 words.",
  messages: [
    { role: "user", content: "Ticket: Hi, my card was charged twice for invoice 4417 on the 3rd." },
  ],
});

for (const block of response.content) {
  if (block.type === "text") console.log(block.text);
}
console.log(response.stop_reason, response.usage);

Tip: Pick one provider SDK for your core path and learn it deeply. Hide it behind a small interface of your own (you will start one below and finish it in Lesson 8.1). That interface is what lets you add a second provider for fallback or cost without touching feature code.

From one call to a wrapper

Scattering raw SDK calls across a codebase is the most common early mistake. Every call site then has to remember logging, truncation checks, and error handling, and most will not. Route every call through one module instead. Here is the first version; you will add cost tracking (1.3), retries (2.3), and tracing (6.3) later.

# llm.py: the single path every model call in your app goes through
from __future__ import annotations

import logging
import os
import time
from dataclasses import dataclass

import anthropic

MODEL = os.environ["LLM_MODEL"]
log = logging.getLogger("llm")

# Explicit timeout and retry settings beat silent defaults (Lesson 2.3).
client = anthropic.Anthropic(timeout=60.0, max_retries=2)


@dataclass
class LLMResult:
    text: str
    stop_reason: str | None
    input_tokens: int
    output_tokens: int
    latency_s: float
    model: str


def complete(
    system: str,
    messages: list[dict],
    *,
    max_tokens: int = 1024,
    model: str = MODEL,
    feature: str = "unknown",  # tag every call with the product feature it serves
) -> LLMResult:
    start = time.perf_counter()
    response = client.messages.create(
        model=model, max_tokens=max_tokens, system=system, messages=messages
    )
    latency = time.perf_counter() - start

    result = LLMResult(
        text="".join(b.text for b in response.content if b.type == "text"),
        stop_reason=response.stop_reason,
        input_tokens=response.usage.input_tokens,
        output_tokens=response.usage.output_tokens,
        latency_s=latency,
        model=model,
    )
    log.info(
        "llm_call feature=%s model=%s in=%d out=%d stop=%s latency=%.2fs",
        feature, model, result.input_tokens, result.output_tokens,
        result.stop_reason, latency,
    )
    if result.stop_reason == "max_tokens":
        log.warning("feature=%s output truncated at max_tokens=%d", feature, max_tokens)
    return result

Small as it is, this wrapper gives you a per-feature log line with tokens and latency, a truncation warning, and one place to change timeouts.

Demo code versus production code

Concern Typical demo Production pattern (where you learn it)
Model choice Whatever was in the tutorial Chosen per task with an eval, stored in config (1.2)
Cost Unknown until the invoice Estimated up front, tracked per call (1.3, 7.3)
Prompt Inline string Versioned template with delimited inputs (2.1)
Failures Unhandled exception Timeouts, backoff, fallbacks (2.3, 8.1)
Output Free text, parsed with hope Schema-validated objects (3.1)
Knowledge Whatever the model memorized Retrieved, cited sources (Module 4)
Quality "Looks good to me" Golden dataset and CI regression tests (Module 6)
Security Trusts all input Threat model, least privilege, output handling (Module 7)

What you will build

The course closes with a capstone that pulls every layer together: a grounded, tool-using assistant for an internal knowledge base. It answers questions with citations from your documents, calls a small set of business tools, such as looking up and creating support tickets, under budgets and approval rules, and ships with an evaluation suite that runs in CI. Each module adds one layer to that system, so the code you write in each "Try it" exercise is a building block, not a throwaway.

Try it

Set up your working environment and instrument your first calls (10 to 15 minutes).

  1. Create a project folder with a virtual environment and install anthropic (or openai if that is your provider). Set your API key and LLM_MODEL as environment variables.
  2. Save llm.py from this lesson, adapting the client if you use OpenAI. Configure logging with logging.basicConfig(level=logging.INFO).
  3. Write a script that calls complete() on five realistic inputs from your own work: support tickets, sales emails, or log lines. Use the same system prompt for all five.
  4. Run the script twice. Compare the outputs and the logged token counts. Note which outputs changed between runs.
  5. Set max_tokens=10 and run once more. Confirm that you see stop_reason=max_tokens and the truncation warning.
  6. Write down the average input and output tokens per call. You will use these numbers to estimate cost in Lesson 1.3.

Key takeaways

  • An LLM API is a stateless, non-deterministic, metered function. Your code supplies all the context, all the testing, and all the cost discipline.
  • Production AI features are a stack: model selection, a robust client, structured outputs and tools, retrieval, agents, evals, security, and production architecture.
  • Read the whole response: filter content blocks by type, check stop_reason on every call, and log usage every time.
  • Providers differ mostly in naming. Learn one SDK well and hide it behind your own interface.
  • Keep model IDs in configuration, never in code. Models and prices change faster than your release cycle.
  • Route every call through one wrapper so logging, limits, and error handling happen everywhere by default.

Enjoyed this preview?

Unlock all 24 lessons, 5 templates, and your certificate for $599.

Get full access