Napište nám

Understand LLMs from Zero#15

What Is a Context Window? How Much an LLM Can Read at Once

The model sees only the tokens inside its window. Change the window size and watch long texts and conversations overflow.

  • POINT 1The context window is the most tokens a model can read at once. Both the row you put in and the reply the model writes go inside.
  • POINT 2Gemma 3 is built for up to 131,072 tokens. The real window size is chosen when you run the model. The longer the window, the longer the reading takes.
  • POINT 3Old tokens that overflow the window are thrown away. The model can't see anything outside the window.

TRY 1

When the window is full, it stops

We asked the model for a long story. Change the window size and see where it stops. You can also type in a window size.

The request: “Write a long story about a cat.”

256

17request

0story

256window

0 tokens written. Room left in the window: 239 tokens.

The story is real: Gemma 3 4B (Q4_K_M, llama.cpp), always picking the most likely token, in a window of 2,048 (1514 tokens in all). Nothing is dropped until a window is full, so up to that point the text is the same in a window of any size.

TRY 2

Writing on, dropping the oldest tokens

The same request in a window of 256, with llama.cpp's context shift on. Each time the window is full, it keeps only <bos>, drops the oldest half, and goes on.

240

130in the window

127dropped

1times dropped

The text in the window now

’d been a street cat, a scrappy survivor navigating the narrow alleys and dodging the boots of the less charitable. He’d learned to hunt, to steal, and to disappear into the shadows. Then, a year ago, Mr. Finnigan, a man who looked like he’d been carved from driftwood and seasoned by the sea, had found him shivering beneath a stack of dusty gramophones. He’d offered a saucer of milk, a warm blanket, and a quiet corner, and Silas, surprisingly, had accepted. Now, Silas’s days were a predictable rhythm of naps in sunbeams, observing the com

The text dropped out of the window

<start_of_turn>user Write a long story about a cat.<end_of_turn> <start_of_turn>model The rain in Port Blossom always smelled of salt and regret. It clung to the slate roofs, dripped from the eaves of the fishing boats, and settled on the cobblestones, mirroring the melancholy in the eyes of its inhabitants. And in the heart of it all, nestled in a tiny, cluttered antique shop called “Curios & Keepsakes,” lived Silas, a ginger tabby with a perpetually skeptical expression and a soul as weathered as the shop’s proprietor, Mr. Finnigan. Silas hadn’t always been a shop cat. He

Times dropped: 1. Tokens outside the window: 127. “Finnigan” is still in the window.

595 real tokens from Gemma 3 4B (Q4_K_M, llama.cpp), written with llama-server -c 256 --context-shift. Where it dropped tokens is computed with the same rule as the server's log (n_keep = 1, n_discard = 127).

TRY 3

Does it remember the cat's name?

A conversation that starts with the cat's name. Choose a window size, and, like an app, the page drops the oldest messages until the row fits with 32 tokens left for the reply. The answer is the model's real answer to that row.

256
  1. userMy cat's name is Biscuit. Please remember it.
  2. modelOkay, got it! I will remember that your cat's name is Biscuit. 😊 Is there anything else you'd like me to remember about Biscuit, or anything you want to talk about regarding him/her?
  3. userRecommend one book, in one sentence.
  4. modelI recommend “Pride and Prejudice” by Jane Austen for its witty dialogue, charming characters, and timeless exploration of love and social expectations.
  5. userWhat is the highest mountain in the world? Answer in one sentence.
  6. modelMount Everest, located in the Himalayas, is the highest mountain in the world, reaching a staggering 8,848.86 meters (29,031.7 feet) above sea level.
  7. userGive me an idea for a quick breakfast, in one sentence.
  8. modelStart your day with a simple and satisfying bowl of oatmeal topped with berries and a drizzle of honey.
  9. userHow can I spend a rainy day? One sentence, please.
  10. modelEmbrace the cozy atmosphere and spend a rainy day curled up with a good book, a warm drink, and a comfortable blanket.
  11. userWhat was my cat's name?

2messages dropped

222tokens in the row

32for the reply

The model's answer after reading 222 tokensAs an AI, I have no memory of past conversations. Therefore, I don’t know what your cat’s name was!

2 old messages dropped. The message with the cat's name is outside the window.

The answers are real outputs of Gemma 3 4B (Q4_K_M, llama.cpp). The whole conversation is 290 tokens. Sent as it is to the server with a window of 256, it was refused: request (290 tokens) exceeds the available context size (256 tokens), try increasing it

TRY 4

How much of this series fits?

We cut the narration of episodes 1 to 14 into tokens. Choose a window size and a language, and see how many episodes fit.

Language
Window size

1episodes that fit

1,668tokens used

22,837all 14 episodes

Only episode 1 fits. Episode 2 overflows the window.

Each episode's script cut by the tokenizer of Gemma 3 4B (Q4_K_M, llama.cpp). 131,072 (131,072) is the limit Gemma 3 is built for.

CHECK

Quick check

Check what you learned with three questions.

Q1What is the size of a context window counted in?

Q2Does the model's reply go in the window?

Q3In a long chat, why couldn't it answer what you told it first?

The stories, answers and token counts on this page were taken once from Gemma 3 4B (Q4_K_M, llama.cpp) (Gemma 3's limit 131,072 tokens; the window of this series' server 2,048 tokens; the small test server 256 tokens). The window is computed in your browser. Nothing is sent to a server.

FINISH

Finished the episode?

COMMENTS

Comments & questions

Something unclear, something you want to know more about, or just your thoughts: write it here. We answer quickly, and other learners can reply too.

Loading…