EN ▾
Get API key

HomeGuide

Updated

AI Agent Context Window: How 100k Tokens Gets Used

An AI agent context window is the total token space shared by the prompt, the tool results and the model's answer. On this API that space is 100,000 tokens, and long-running agents tend to fill it faster than expected.

Glass container filling with glowing blocks while an arrow loop circles it, picturing an agent's context window

Key points

  1. The 100,000-token window covers prompt plus completion, so every tool result you keep reduces room for the next answer.
  2. In an agent loop the whole history is resent each step, so total input grows with the square of the step count, not linearly.
  3. Trim old tool output first, keep the system prompt and the latest turns, and measure with the usage field.
  4. A 20-step loop with 1,500 new tokens per step costs about $0.055 in total.

No card needed. After sign-up, claim $0.50 of free test credit with one click (once per person).

Get API key

The short answer: one shared budget of 100,000 tokens

When people search for the ai agent context window, they usually want to know how much an agent can remember before it breaks. Here the number is 100,000 tokens, and it counts everything: the system prompt, the tool definitions, every earlier message, every tool result, and the completion the model is about to write. The completion can use whatever is left, and max_tokens is optional.

That makes the window a budget, not a storage space. A chatbot rarely comes near the limit. An agent that calls tools in a loop can reach it in an afternoon, because each step adds a model message and a tool result to the history, and the whole history is sent again on the next call. The model has no memory between requests. What looks like memory is you resending the transcript.

What actually fills the window in an agent

Most of the space is not the user's text. Track these five sources and you can predict where a run will run out.

  • System prompt. Rules, persona and output format. Often 500 to 2,000 tokens and sent on every call.
  • Tool definitions. Each function name, description and JSON schema counts. Ten tools with long descriptions can cost several thousand tokens before the first user message.
  • Tool results. The usual culprit. A search result, a file dump or a database row set can be thousands of tokens, and it stays in the history unless you remove it.
  • Assistant messages. Reasoning written out in text and the tool_calls arguments the model produced.
  • The reply being generated. It needs space too. If the prompt is 99,000 tokens, the answer has about 1,000 left.

The API reports exact numbers in the usage block of each response, so you do not have to guess. Log prompt_tokens after every step and you will see the curve.

A worked example: 20 steps of a tool loop

Take an agent with a 3,000-token start (system prompt plus tool definitions). Each step adds about 1,500 tokens to the history: the model's tool call and the tool's result. It also writes about 300 tokens of output per step. The agent runs 20 steps.

The input at step k is 3,000 + 1,500 × (k − 1). Summed over 20 steps:

  • 3,000 × 20 = 60,000 tokens for the fixed start, resent every time.
  • 1,500 × (0 + 1 + ... + 19) = 1,500 × 190 = 285,000 tokens of accumulated history.
  • Total input: 345,000 tokens. Total output: 20 × 300 = 6,000 tokens.
ItemTokensPrice per 1MCost
Input345,000$0.15$0.05175
Output6,000$0.50$0.003
Totalabout $0.055

Two things stand out. First, the context at step 20 is 3,000 + 1,500 × 19 = 31,500 tokens, well inside the window, yet you paid for 345,000 input tokens, because the history was resent each time. Second, a single call with a completely full window (100,000 input tokens) costs 100,000 × $0.15 / 1M = $0.015 for the input side. Long contexts are affordable per call; the cost comes from repeating them over many steps. See the pricing page for the rates and top-up bonuses.

How many steps fit before the window is full

You can estimate this before writing any code. Keep a reserve for the answer, say 20,000 tokens, which leaves an 80,000-token budget for the prompt. With a 3,000-token start and 1,500 tokens added per step, the history reaches that budget after (80,000 − 3,000) / 1,500, or about 51 steps.

Now price a run of that length. Over 51 steps the input is 3,000 × 51 + 1,500 × (0 + 1 + ... + 50) = 153,000 + 1,500 × 1,275 = 153,000 + 1,912,500 = 2,065,500 tokens. Output at 300 tokens per step is 15,300 tokens. Cost: 2,065,500 × $0.15 / 1M = $0.30983, plus 15,300 × $0.50 / 1M = $0.00765, so about $0.32.

Compare that with the 20-step run at about $0.055. The run is 2.55 times longer but costs about 5.8 times more, because each step carries a longer history. This is the practical reason to trim: it caps the cost per step as well as protecting the window.

Keeping an agent inside the window

Drop old tool output first

Raw tool results are the biggest and least useful part of an old history. Once the model has acted on a file listing, it rarely needs the listing again. Replace old results with a one-line stub such as "result removed, 4,200 tokens", or delete the pair entirely.

Summarize the middle, keep both ends

Keep the system prompt and the last few turns verbatim. For everything between, ask the model for a short summary of decisions made and facts found, then store that summary as a single message. Run the summary call before you hit the limit, not after.

Move facts out of the transcript

If the agent learns something it must keep for hours, write it to a file or a small database through a tool, and read it back when needed. This is cheaper than carrying it in every request and survives restarts.

Keep tool schemas short

Trim descriptions to what the model needs to choose the tool. Pass only the tools relevant to the current phase of the task.

Code: a tool loop with a token budget

This loop trims the oldest messages when the rough size passes a budget of 80,000 tokens, leaving room for the answer. The estimate divides characters by four, a common approximation for English text; the real count comes back in usage. Trimming drops orphaned tool messages so a result never appears without its call.

import json, os
from openai import OpenAI

client = OpenAI(base_url="https://api.aivideogenapi.com/v1", api_key=os.environ["API_KEY"])
BUDGET = 80_000

tools = [{"type": "function", "function": {
    "name": "get_scene_length",
    "description": "Return the length of a scene in seconds",
    "parameters": {"type": "object",
                   "properties": {"scene": {"type": "integer"}},
                   "required": ["scene"]}}}]

def rough_tokens(messages):
    return sum(len(json.dumps(m)) for m in messages) // 4

def trim(messages):
    system, rest = messages[0], messages[1:]
    while rough_tokens([system] + rest) > BUDGET and len(rest) > 4:
        rest.pop(0)
        while rest and rest[0]["role"] == "tool":
            rest.pop(0)
    return [system] + rest

def run_tool(name, args):
    return json.dumps({"seconds": 4 * args["scene"]})  # stub

messages = [
    {"role": "system", "content": "You plan video scenes. Use tools for lengths."},
    {"role": "user", "content": "Total the length of scenes 1 to 5."},
]

for step in range(20):
    resp = client.chat.completions.create(
        model="uncensored", messages=trim(messages), tools=tools)
    msg = resp.choices[0].message
    messages.append(msg.model_dump(exclude_none=True))
    print(step, resp.usage.prompt_tokens)
    if not msg.tool_calls:
        print(msg.content)
        break
    for call in msg.tool_calls:
        result = run_tool(call.function.name, json.loads(call.function.arguments))
        messages.append({"role": "tool", "tool_call_id": call.id, "content": result})

Tool calling follows the OpenAI format: tools, tool_choice, and results returned as role: "tool". The docs list the parameters that pass through.

Limits that matter besides the window

The window is one constraint. Agents also hit request limits. A full key allows 8 concurrent requests and 300 requests per minute; a trial key allows 2 and 60. A tight loop with parallel sub-agents can exceed the concurrency cap quickly. The API answers with HTTP 429 and the error code concurrency or rate_limited, and on temporary overload with a Retry-After: 5 header. Wait that long and retry.

If you stream, remember that a streamed request holds its concurrency slot until the stream ends. A stalled client that never closes the stream keeps a slot busy.

The maximum body is 8 MB, which is far more than 100,000 tokens of text, so the window is the limit you will hit first. For an example of a text-only pipeline built on this API, see the video script guide. When you are ready to run it, create an API key; new users can claim $0.50 of trial credit with one click, valid for 7 days and with no payment needed.

Questions and answers

How big is the context window?

100,000 tokens, counting the prompt and the completion together. The completion can use whatever the prompt leaves free.

Does the model remember earlier requests?

No. Each request is independent, so your code must resend the messages it wants the model to see.

What happens when I go over the limit?

The request cannot fit, so trim or summarize the history before sending. Failed requests are not charged.

How do I count tokens?

Read prompt_tokens and completion_tokens from the usage block of each response, or use a rough estimate of four characters per token for English before sending.

Your key is one form away

Create an account, copy the key, change the base URL. That is the whole setup.

Get API key