HomeGuide
Updated
AI Agent Context Window: How 100k Tokens Gets Used
An AI agent context window is the total token space shared by the prompt, the tool results and the model's answer. On this API that space is 100,000 tokens, and long-running agents tend to fill it faster than expected.

Key points
- The 100,000-token window covers prompt plus completion, so every tool result you keep reduces room for the next answer.
- In an agent loop the whole history is resent each step, so total input grows with the square of the step count, not linearly.
- Trim old tool output first, keep the system prompt and the latest turns, and measure with the usage field.
- A 20-step loop with 1,500 new tokens per step costs about $0.055 in total.
No card needed. After sign-up, claim $0.50 of free test credit with one click (once per person).
Get API keyThe short answer: one shared budget of 100,000 tokens
When people search for the ai agent context window, they usually want to know how much an agent can remember before it breaks. Here the number is 100,000 tokens, and it counts everything: the system prompt, the tool definitions, every earlier message, every tool result, and the completion the model is about to write. The completion can use whatever is left, and max_tokens is optional.
That makes the window a budget, not a storage space. A chatbot rarely comes near the limit. An agent that calls tools in a loop can reach it in an afternoon, because each step adds a model message and a tool result to the history, and the whole history is sent again on the next call. The model has no memory between requests. What looks like memory is you resending the transcript.
What actually fills the window in an agent
Most of the space is not the user's text. Track these five sources and you can predict where a run will run out.
- System prompt. Rules, persona and output format. Often 500 to 2,000 tokens and sent on every call.
- Tool definitions. Each function name, description and JSON schema counts. Ten tools with long descriptions can cost several thousand tokens before the first user message.
- Tool results. The usual culprit. A search result, a file dump or a database row set can be thousands of tokens, and it stays in the history unless you remove it.
- Assistant messages. Reasoning written out in text and the
tool_callsarguments the model produced. - The reply being generated. It needs space too. If the prompt is 99,000 tokens, the answer has about 1,000 left.
The API reports exact numbers in the usage block of each response, so you do not have to guess. Log prompt_tokens after every step and you will see the curve.
A worked example: 20 steps of a tool loop
Take an agent with a 3,000-token start (system prompt plus tool definitions). Each step adds about 1,500 tokens to the history: the model's tool call and the tool's result. It also writes about 300 tokens of output per step. The agent runs 20 steps.
The input at step k is 3,000 + 1,500 × (k − 1). Summed over 20 steps:
- 3,000 × 20 = 60,000 tokens for the fixed start, resent every time.
- 1,500 × (0 + 1 + ... + 19) = 1,500 × 190 = 285,000 tokens of accumulated history.
- Total input: 345,000 tokens. Total output: 20 × 300 = 6,000 tokens.
| Item | Tokens | Price per 1M | Cost |
|---|---|---|---|
| Input | 345,000 | $0.15 | $0.05175 |
| Output | 6,000 | $0.50 | $0.003 |
| Total | about $0.055 |
Two things stand out. First, the context at step 20 is 3,000 + 1,500 × 19 = 31,500 tokens, well inside the window, yet you paid for 345,000 input tokens, because the history was resent each time. Second, a single call with a completely full window (100,000 input tokens) costs 100,000 × $0.15 / 1M = $0.015 for the input side. Long contexts are affordable per call; the cost comes from repeating them over many steps. See the pricing page for the rates and top-up bonuses.
How many steps fit before the window is full
You can estimate this before writing any code. Keep a reserve for the answer, say 20,000 tokens, which leaves an 80,000-token budget for the prompt. With a 3,000-token start and 1,500 tokens added per step, the history reaches that budget after (80,000 − 3,000) / 1,500, or about 51 steps.
Now price a run of that length. Over 51 steps the input is 3,000 × 51 + 1,500 × (0 + 1 + ... + 50) = 153,000 + 1,500 × 1,275 = 153,000 + 1,912,500 = 2,065,500 tokens. Output at 300 tokens per step is 15,300 tokens. Cost: 2,065,500 × $0.15 / 1M = $0.30983, plus 15,300 × $0.50 / 1M = $0.00765, so about $0.32.
Compare that with the 20-step run at about $0.055. The run is 2.55 times longer but costs about 5.8 times more, because each step carries a longer history. This is the practical reason to trim: it caps the cost per step as well as protecting the window.
Keeping an agent inside the window
Drop old tool output first
Raw tool results are the biggest and least useful part of an old history. Once the model has acted on a file listing, it rarely needs the listing again. Replace old results with a one-line stub such as "result removed, 4,200 tokens", or delete the pair entirely.
Summarize the middle, keep both ends
Keep the system prompt and the last few turns verbatim. For everything between, ask the model for a short summary of decisions made and facts found, then store that summary as a single message. Run the summary call before you hit the limit, not after.
Move facts out of the transcript
If the agent learns something it must keep for hours, write it to a file or a small database through a tool, and read it back when needed. This is cheaper than carrying it in every request and survives restarts.
Keep tool schemas short
Trim descriptions to what the model needs to choose the tool. Pass only the tools relevant to the current phase of the task.
Code: a tool loop with a token budget
This loop trims the oldest messages when the rough size passes a budget of 80,000 tokens, leaving room for the answer. The estimate divides characters by four, a common approximation for English text; the real count comes back in usage. Trimming drops orphaned tool messages so a result never appears without its call.
import json, os
from openai import OpenAI
client = OpenAI(base_url="https://api.aivideogenapi.com/v1", api_key=os.environ["API_KEY"])
BUDGET = 80_000
tools = [{"type": "function", "function": {
"name": "get_scene_length",
"description": "Return the length of a scene in seconds",
"parameters": {"type": "object",
"properties": {"scene": {"type": "integer"}},
"required": ["scene"]}}}]
def rough_tokens(messages):
return sum(len(json.dumps(m)) for m in messages) // 4
def trim(messages):
system, rest = messages[0], messages[1:]
while rough_tokens([system] + rest) > BUDGET and len(rest) > 4:
rest.pop(0)
while rest and rest[0]["role"] == "tool":
rest.pop(0)
return [system] + rest
def run_tool(name, args):
return json.dumps({"seconds": 4 * args["scene"]}) # stub
messages = [
{"role": "system", "content": "You plan video scenes. Use tools for lengths."},
{"role": "user", "content": "Total the length of scenes 1 to 5."},
]
for step in range(20):
resp = client.chat.completions.create(
model="uncensored", messages=trim(messages), tools=tools)
msg = resp.choices[0].message
messages.append(msg.model_dump(exclude_none=True))
print(step, resp.usage.prompt_tokens)
if not msg.tool_calls:
print(msg.content)
break
for call in msg.tool_calls:
result = run_tool(call.function.name, json.loads(call.function.arguments))
messages.append({"role": "tool", "tool_call_id": call.id, "content": result})
Tool calling follows the OpenAI format: tools, tool_choice, and results returned as role: "tool". The docs list the parameters that pass through.
Limits that matter besides the window
The window is one constraint. Agents also hit request limits. A full key allows 8 concurrent requests and 300 requests per minute; a trial key allows 2 and 60. A tight loop with parallel sub-agents can exceed the concurrency cap quickly. The API answers with HTTP 429 and the error code concurrency or rate_limited, and on temporary overload with a Retry-After: 5 header. Wait that long and retry.
If you stream, remember that a streamed request holds its concurrency slot until the stream ends. A stalled client that never closes the stream keeps a slot busy.
The maximum body is 8 MB, which is far more than 100,000 tokens of text, so the window is the limit you will hit first. For an example of a text-only pipeline built on this API, see the video script guide. When you are ready to run it, create an API key; new users can claim $0.50 of trial credit with one click, valid for 7 days and with no payment needed.
More guides
Questions and answers
How big is the context window?
100,000 tokens, counting the prompt and the completion together. The completion can use whatever the prompt leaves free.
Does the model remember earlier requests?
No. Each request is independent, so your code must resend the messages it wants the model to see.
What happens when I go over the limit?
The request cannot fit, so trim or summarize the history before sending. Failed requests are not charged.
How do I count tokens?
Read prompt_tokens and completion_tokens from the usage block of each response, or use a rough estimate of four characters per token for English before sending.
Your key is one form away
Create an account, copy the key, change the base URL. That is the whole setup.