3 Under the Hood: LLM Mechanics & Deterministic Grounding
Once you have your agentic harness installed and running (as covered in Chapter 2), you are ready to dive straight into your research tasks. To get the most out of your agent—and steer complex projects with confidence—it helps to have an intuitive understanding of how the underlying engine operates.
You do not need a computer science degree to steer an agentic system effectively. You simply need an accurate mental model of how LLMs process information, how they predict words, and how tools ensure reliability.
This chapter pulls back the curtain on how Large Language Models (LLMs) process instructions, why step-by-step reasoning works, and how connecting models to deterministic tools anchors their output in reality.
3.1 How Large Language Models Process and Generate Text
The term “Artificial Intelligence” often creates the impression that an LLM possesses an internal mind, an ongoing stream of consciousness, or a persistent memory of past conversations. In reality, the mechanics of modern LLMs (such as Google Gemini, OpenAI GPT, or Anthropic Claude) are based on four fundamental principles:
3.1.1 Autoregressive Next-Token Prediction
At its core, a Large Language Model is an autoregressive statistical prediction engine.
- Tokens: Text is broken down into small units called tokens (a token is roughly 3–4 characters, or about 0.75 English words).
- Prediction: Given an active sequence of tokens, the model calculates a conditional probability distribution over all possible next tokens in its vocabulary (tens of thousands of candidate words and subwords) and selects the most likely candidate or samples among top probabilities: \[P(\text{Token}_t \mid \text{Token}_1, \text{Token}_2, \dots, \text{Token}_{t-1})\]
- Autoregression: Once a token is selected, it is appended to the sequence, and the model repeats the entire process to predict the token after that.
3.1.1.1 What Constitutes the “Context”?
When explaining how an LLM produces the most likely next word given a context, it is crucial to recognize that the context is not just the word or phrase immediately preceding the next one being generated.
Rather, the context is the entire accumulated sequence of preceding tokens currently active inside the model’s text window. In an agentic research workflow, this active context bundle consists of:
- Foundational Rules & System Prompts: Instructions from your
GEMINI.mdfile, disciplinary guidelines, and output formatting constraints. - Workspace Files & Tool Outputs: Code scripts, data tables, and terminal execution logs loaded into the conversation.
- Conversation History: All prior turns, questions, and responses from both the user and the agent.
- Current Turn & Intermediate Output: Every word, phrase, and intermediate derivation step generated so far in the current response.
Through the Transformer’s self-attention mechanism, the model evaluates the relationships across all of these tokens simultaneously. An instruction or variable definition given thousands of tokens earlier directly shapes which token is predicted as most likely at the current step, even when the word sitting immediately before the cursor is completely generic (such as "the" or "is").
How the Next Token is Generated from Full Accumulated Context:
┌──────────────────────────────────────────────────────────────────────────────────┐
│ ACTIVE CONTEXT WINDOW (The entire sequence: Token 1 through Token t-1) │
│ │
│ ┌────────────────┐ ┌────────────────┐ ┌────────────────┐ ┌────────────────┐ │
│ │ System Rules │ │ Files / Data │ │ User Prompt │ │ Response So Far│ │
│ │ (GEMINI.md) │ │ (Tool outputs) │ │ (Instruction) │ │ (Tokens 1..t-1)│ │
│ └───────┬────────┘ └───────┬────────┘ └───────┬────────┘ └───────┬────────┘ │
│ │ │ │ │ │
│ └─────────────┬─────┴─────────────┬─────┴───────────────────┘ │
│ │ Self-Attention │ │
│ ▼ (Attends simultaneously across ALL tokens) │
│ ┌───────────────────────────────────────────────┐ │
│ │ Probability Distribution over Vocabulary │ │
│ │ - "significant": 72% (Most likely) │ │
│ │ - "robust": 18% │ │
│ │ - "positive": 6% │ │
│ │ - ... │ │
│ └───────────────────────┬───────────────────────┘ │
└────────────────────────────────────┼─────────────────────────────────────────────┘
▼
Selects Next Token: Token t ("significant")
│
▼
[ Appended to Context Window for Token t+1 ]
The model does not plan out several sentences in advance or formulate a silent outline before speaking. It constructs meaning sequentially, one token at a time, guided by the statistical associations learned during training and the full context present in your prompt.
3.1.2 The Stateless Nature of LLMs
A crucial insight for researchers is that LLMs have no hidden internal memory between requests.
When you ask an agent a follow-up question in a chat window, the model does not “remember” what you discussed five minutes ago in some internal brain state. Instead, your harness (Antigravity, Claude Desktop, or VS Code) takes your entire past conversation history, packages it into a single prompt, and sends the whole package back to the model from scratch.
Every time you send a message, the LLM reads your instructions, your loaded rules (GEMINI.md), and the conversation history anew. If a piece of information is not present in that active bundle of text, it does not exist for the model.
3.1.3 Why “Chain of Thought” and Planning Work
Connecting back to these core principles—that an LLM generates text one token at a time, predicting the most likely next token conditioned on the entire sequence accumulated so far (not just the single word immediately before it)—we can see why asking for an immediate answer to a complex problem usually fails, and why “showing its work” works so well.
3.1.3.1 The Problem with Direct Answers
Suppose you give the model a multi-step task (like deriving an econometric formula, refactoring a complex script, or solving a logic puzzle) and ask for the final answer immediately.
At that moment, the only text in the model’s text window is your raw question. Because the model has no hidden internal scratchpad and cannot “think silently” before it writes, it must try to predict the final conclusion in a single leap. The gap between a raw question and a complex conclusion is too wide for a single prediction step, forcing the model to make a statistical guess.
3.1.3.2 How Chain of Thought Bridges the Gap
When an agent writes out a Chain of Thought (step-by-step reasoning before stating the final conclusion), it uses each step to prepare the ground for the next:
- Step 1 is easy: Moving from the Question to the first simple step (like stating a definition or identifying known variables) is a straightforward, high-probability prediction.
- Step 1 becomes context for Step 2: Once Step 1 is written, those words are added to the text window. The model now uses the Question plus Step 1 to predict Step 2.
- The final answer becomes obvious: By the time the model is ready to write the conclusion, all intermediate steps are sitting directly in its text window. Predicting the correct final answer is no longer a blind leap—it is the direct, logical next prediction based on the work already laid out.
Direct Answer (Blind Leap):
Context: [ Question ] ──────────────────────────────────────────► [ Final Answer ] (Often a guess)
(No intermediate reasoning in context)
Chain of Thought (Cumulative Context Expansion):
Turn 1: Context: [ Question ] ──────────────────────────────────► Predicts [ Step 1 ]
Turn 2: Context: [ Question + Step 1 ] ─────────────────────────► Predicts [ Step 2 ]
Turn 3: Context: [ Question + Step 1 + Step 2 ] ────────────────► Predicts [ Step 3 ]
Turn 4: Context: [ Question + Step 1 + Step 2 + Step 3 ] ───────► Predicts [ Final Answer ] (Grounded)
▲
└─ At every single step, the context is NOT just the immediately preceding word/step;
the model conditions on the ENTIRE sequence accumulated so far.
In short: the model cannot think without writing. Writing intermediate steps is how an LLM creates its own working memory on the fly.
3.1.3.3 Scaling from Prompts to Projects: Planning Mode
Planning Mode is this exact same principle scaled up to entire research projects:
- Instead of jumping straight into editing multiple
.tex,.qmd, or.Rfiles at once, the agent first writes an explicitimplementation_plan.mdfile to disk. - As the agent works through each task, it reads the plan into its text window.
- This keeps the full project roadmap directly in front of the model at each step, significantly reducing task drift, forgotten constraints, and skipped verification steps.
3.1.4 Probabilistic Association vs. Deterministic Logic
LLMs are brilliant at semantic synthesis, drafting prose, translating concepts, and finding patterns. However, they are probabilistic associative engines, not deterministic calculators or compilers.
| Task Type | How LLMs Behave | Recommended Strategy |
|---|---|---|
| Literature synthesis & drafting | Exceptional; connects disparate concepts fluently | Let LLM draft; verify key claims |
| Code generation (R, Python, Stata) | High fluency with standard libraries and idioms | Require agent to execute and test code |
| Arithmetic & numerical calculations | Prone to subtle calculation errors | Force agent to run code/calculator |
| LaTeX compilation & TikZ layout | Can hallucinate valid syntax that fails to compile | Compile with xelatex and check logs |
Because LLMs can confidently generate syntactically plausible but factually incorrect code or equations, an agentic system must pair the model with deterministic tools (compilers, interpreters, and test suites) to verify its work.
3.2 Grounding and Deterministic Tool Execution
A foundational risk in academic AI adoption is the illusion of self-verification.
If you ask an LLM: “Did you check that this LaTeX code compiles without error?”, it will almost always reply: “Yes, the code is fully valid and will compile cleanly.”
The model is not lying intentionally—it is predicting the most statistically plausible agreeable response. It cannot physically “run” the code inside its neural network.
┌─────────────────────────────────────────┐
│ THE ILLUSION OF SELF-QA │
│ Agent: "The LaTeX compiles cleanly!" │
│ (Pure prediction — no reality check) │
└────────────────────┬────────────────────┘
│
[ Reality: Syntax Error ]
│
┌────────────────────▼────────────────────┐
│ GROUNDED TOOL QA │
│ 1. Agent writes file │
│ 2. Agent executes: `xelatex file.tex` │
│ 3. Agent reads exit code (0 vs 1) │
│ 4. Agent fixes actual log errors │
└─────────────────────────────────────────┘
3.2.1 Tool Grounding as the Anchor of Truth
In an agentic harness, the AI is given tools to interact with your local operating system: 1. File System Tools: Read, write, and patch files in place. 2. Terminal Tools: Execute shell commands (xelatex, Rscript, stata-mp, pytest). 3. Inspection Tools: Search directories (grep, find), parse outputs, and inspect diffs.
By requiring the agent to compile documents and execute scripts as part of its workflow, the agent is grounded in deterministic reality. If a figure label overlaps in TikZ or an econometric matrix is singular in R, the execution output forces the agent to confront the truth and fix the error.
3.2.2 Model Drift and Verification Gates
Model checkpoints updated by AI providers can exhibit subtle shifts in style, capability, or adherence to edge cases.
Because you cannot guarantee that a model behaves identically from month to month, your research repository must rely on verification gates: * quality-gates.md (Scoring System): Automated checklists that deduct points for broken citations (-15) or equation overflows (-20). * Deterministic Scripts (passport.yaml): Anchoring numeric claims in a manuscript directly to the output log files produced by your R/Stata scripts.
3.3 Model Selection and Reasoning Depth
Not all tasks require the same computational power. Modern agentic systems allow you to select between different model tiers and adjust reasoning effort.
3.3.1 Model Tiers: Matching the Tool to the Task
| Model Tier | Representative Models | Strengths | Ideal Academic Tasks |
|---|---|---|---|
| Frontier / Deep Reasoning | Gemini 2.0/3.0 Pro, Claude 3.5/3.7 Sonnet, OpenAI o1/o3 | Deep logical reasoning, complex code generation, theorem proving | Deriving proofs, structural modeling, architectural design, peer-review critique |
| Fast / Lightweight | Gemini 2.0/3.0 Flash, Claude 3.5 Haiku, GPT-4o-mini | Extremely fast, low cost, large context throughput | Proofreading, typo scanning, bibliography validation, formatting checks |
3.3.2 Reasoning Effort (Thinking Tokens)
Frontier models introduce variable reasoning effort (sometimes called thinking tokens or System 2 deliberation).
- Standard Inference (System 1): The model immediately begins generating response tokens. Excellent for straightforward tasks like renaming variables, drafting an email, or writing standard LaTeX tables.
- High Reasoning Inference (System 2): Before outputting text, the model generates hidden reasoning tokens to explore alternatives, check corner cases, and verify mathematical consistency. Essential for complex econometric derivations or subtle code debugging.
Standard (Low Effort): User Prompt ──► [Immediate Generation] ──► Output
High Reasoning Effort: User Prompt ──► [Hidden Reasoning / Verification Loop] ──► Output
3.4 Summary: Mental Model for Model Mechanics
- LLMs are Probabilistic Predictors: They generate one token at a time, predicting the most likely candidate conditioned on the entire accumulated context (not just the single preceding word) via self-attention; they do not possess an internal subconscious mind.
- Writing Creates Working Memory: An LLM cannot think without writing. Generating intermediate reasoning tokens turns difficult problems into manageable steps.
- Tools Provide Deterministic Grounding: Never trust an LLM’s self-verification. Require it to execute compilers, linters, and statistical scripts.
- Match the Model Tier to the Complexity: Use lightweight fast models for proofreading and bibliography audits; deploy high-reasoning frontier models for proofs and structural modeling.