Skip to content

Day 4 — How LLMs work: tokens, context, embeddings, attention, temperature

Why / क्यों

English: To build with LLMs you must know what happens inside: how text becomes tokens, how much the model can see, how meaning becomes numbers, and how it picks each next word.

Hindi: LLM के साथ काम करने के लिए अंदर की कहानी पता होनी चाहिए: text tokens कैसे बनता है, model कितना देख सकता है, मतलब numbers कैसे बनता है, और अगला शब्द कैसे चुना जाता है।

The full pipeline / पूरा रास्ता

flowchart LR
    A[Your text] --> B[Tokenize]
    B --> C[Embed: tokens to vectors]
    C --> D[Transformer layers: attention]
    D --> E[Next-token probabilities]
    E --> F[Sampling: temperature / top-p]
    F --> G[One new token]
    G -->|repeat until done| D

1. Tokens

LLMs read text in small pieces. Pricing and limits are counted in tokens. Roughly 1 English token ≈ 4 characters.

pip install tiktoken

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
for t in ["unbelievable", "Hello, how are you?", "नमस्ते, आप कैसे हो?"]:
    tokens = enc.encode(t)
    print(t, "->", len(tokens), "tokens:", [enc.decode([x]) for x in tokens])

Important

The same meaning in Hindi can take ~3x more tokens than English → higher cost and the context fills faster.

2. Context window

The context window is how many tokens the model can see at once — its short-term memory. System prompt, chat history, documents and the answer must all fit.

  • The LLM remembers nothing between calls; the full history is resent each time.
  • RAG exists because all your documents cannot fit — send only the relevant parts.
  • Lost in the middle: facts buried in the middle of long context are often missed.

3. Embeddings + cosine similarity

An embedding turns text into a list of numbers so similar meanings sit close together. This powers semantic search and RAG.

pip install numpy

import numpy as np

def cosine(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

# toy vectors: [animal-ness, cuteness, machine-ness]
dog   = np.array([0.9, 0.8, 0.1])
puppy = np.array([0.9, 0.95, 0.1])
car   = np.array([0.05, 0.1, 0.95])

print("dog vs puppy:", round(cosine(dog, puppy), 3))
print("dog vs car:  ", round(cosine(dog, car), 3))

Close to 1 = similar meaning, close to 0 = unrelated.

4. Transformer + attention

An LLM's real job is predicting the next token. Attention decides which earlier words matter most.

  • "The animal didn't cross the road because it was too tired" → it = animal.
  • "I went to the bank to deposit money" vs "We sat on the river bank" → meaning comes from nearby words.

Interview

"A Transformer uses self-attention to connect every token with every other token to understand context, and parallel processing lets it train fast on huge data."

5. Temperature + top-p

Setting Behaviour Use for
Temperature 0–0.3 Safe, consistent JSON, code, extraction, RAG
Temperature 0.7–1 Creative, varied Stories, marketing, brainstorming
Top-p 0.9 Picks only from top 90% probability words Usually tune this or temperature

Quiz / अपने आप को test करो

1. Why does Hindi cost more than English with the same meaning?

It splits into more tokens, and billing is per token.

2. What is a context window?

The maximum tokens the model can see at once — prompt, history, documents and answer together.

3. Why do we need RAG if models have big context windows?

All documents still cannot fit, cost grows, and models miss facts in long contexts; RAG sends only relevant chunks.

4. Cosine similarity of 0.99 vs 0.19 — what does it tell you?

0.99 = very similar meaning; 0.19 = mostly unrelated.

5. Which temperature for extracting invoice data as JSON?

Low (0–0.3) for consistent, accurate output.