Need the same result? Read the existing artifact.
Structured documents, cleaned tables, and read-only tool results can be reused directly under the same conditions.
Try result replayUnderstand your intent. Connect tools to execution.
Natural language starts the task. Intent compilation turns a goal into plans, tool calls, and results.
Behavioral caching makes proven intermediate paths available for reuse.
How does a request become a task?
Start by understanding the goal, then move forward with feedback from execution.
- 01
Express intent
Describe the goal and provide the necessary local context.
- 02
Break down the task
Understand the request and turn it into executable steps.
- 03
Call tools
Operate applications and adjust the next step based on feedback.
- 04
Deliver results
Combine tool outputs to complete the task.
Behavioral cacheRepeated steps.
Repeated steps.
Less repeated inference.
The cache covers frequent, stable segments within a task. New requests and changing nodes remain with the model.
New parameters? Run the existing path again.
Insert new parameters and replay tool calls and actions. Writes, such as editing files or operating an interface, must be executed again.
Try action replayLearning from executionFrom each execution,
From each execution,
keep what can be reused.
We explore recording runtime traces and finding common paths that work repeatedly. After independent verification, results, actions, and environmental conditions are saved together.
- 01Observe executionRecord states and actions
- 02Find common pathsIdentify frequent, stable segments
- 03Verify, then reuseContinuously check, update, and retire
Environment guardsA path that worked before
A path that worked before
still needs checking.
Pages, APIs, and software keep changing. Check the environment before and during replay; on a mismatch, stop the old path and let the model replan.
See the fallback processIntent compilation · Behavioral cache
Want to go deeper?
Expand a topic to explore the principles, implementation, and limits of intent compilation and behavioral caching.
Why natural language can be a task interfaceHow semantic understanding, tool calls, and multi-step planning form an execution path.
Semantic understanding meets code and action generation
Early natural-language programming relied on hand-written rules and struggled with common sense and implicit assumptions. Large language models bring both an understanding of human meaning and the ability to generate formal code and actions.
Planning
Multi-step planning and control
Break down tasks, call external tools, and dynamically adjust the next step using environmental feedback.
MCP / Harness
Tool calls and runtime
MCP standardizes tool calls, while harness engineering supports model-driven actions in operating-system environments.
Token cost
Declining inference costs
Lower token costs make intent compilation viable in a wider range of use cases.
People express intent in natural language. A large model acts as the compiler's core, handling understanding, planning, and tool orchestration to change computer state. Traditional programming languages become downstream compilation outputs.
Handling uncertainty and inference costsType constraints, refined intent, and independent checks improve control.
Uncertainty and recurring inference costs
Uncertainty in the source language and compiler
Natural language contains ambiguity, unstated assumptions, and unclear references. Intent may evolve during execution. Model outputs are probabilistic, and long task chains can drift.
Inference costs on every execution
Inference predicts symbol sequences probabilistically using a parameterized model; computation applies fixed rules to known inputs. Reasoning from scratch repeatedly consumes tokens and compute on the same decisions.
Three engineering approaches
Type contract
Syntax and type contracts
Parse and validate outputs against a structural contract to manage uncertainty. Examples include Structured Outputs, TypeChat, Pydantic / Instructor, and BAML.
Specification
Refine ambiguous intent
Break requirements into reviewable specifications and intermediate documents. Examples include Spec Kit, Kiro, OpenSpec, and Tessl.
Verification
Independent verification and trace observation
External verifiers check candidates. Reasoning, tool calls, and results are recorded for replay analysis and regression testing.
The burden of specification does not disappear; it moves.
Models should fill in and refine ambiguous intent, while people confirm and audit key results. Lightweight domain-specific intermediate representations can feed deterministic compilers for low-level execution, avoiding heavy specifications and rigid formats.
KV caching, prompt caching, speculative decoding, and semantic caching already reduce inference costs. Behavioral caching focuses on task paths, replacing reusable reasoning with computation and storage.
Reusing intermediate paths with behavioral cachingDistinguish result reuse from action replay; cache only stable intermediate paths.
Reusable components along state-transition paths
A model searches an unknown state space from initial intent S₀ to final output Sₙ. Behavioral caching identifies heavily used paths and preserves frequent, stable A → B transitions.
Like JIT compilation of hot code, this does not require the entire task to stay unchanged. The start and end can retain flexible model understanding, with inference reserved for what actually changes.
Results as state · Cache node artifacts
Save structured Markdown, cleaned tables, or read-only tool results. Under the same conditions, read the intermediate artifact to move from node A to node B.
Paths as programs · Cache action sequences
When parameters change, or a task edits files or operates an interface, preserve the tool order, scripts, and action sequence. New parameters drive deterministic execution.
From predefined tools to runtime compilationFrom packaging tools in advance to discovering and compiling hot paths at runtime.
From anticipating needs to compiling at runtime
With AOT, developers anticipate common, frequent needs and package multi-step paths into APIs ahead of time. Our direction is to discover hot paths during intent compilation and let the system turn them into behavioral caches.
| Approach | What is stored, and when | Inference and limits after a hit |
|---|---|---|
| Prompt caching | Intermediate attention states for identical prefixes; reused at runtime | Token generation remains; useful for long prompts and repeated conversation prefixes |
| Semantic caching | Final text answers to similar questions; reused at runtime | No model call on a hit; limited to stateless Q&A without side effects |
| Skill / MCP | Instructions, common scripts, and tool communication protocols; authored at design time | The model still reads instructions and reasons about orchestration |
| Tool-result caching | Read-only return values or sandbox state trees; reused at runtime | Skips tool execution but still needs model orchestration; parameters or sandbox prefixes must match |
| Primitive services | Common multi-step operations and output files packaged as APIs; design-time AOT | Reduces inference within the subtask; examples include Firecrawl / Context7 |
| Trace-level behavioral caching | Parameterized actions, state artifacts, and environment guards; runtime JIT | Deterministic replay when guards pass; fallback on mismatch. For frequent, low-variance, multi-step long-tail tasks |
Engineering mechanisms behind behavioral cachingRecord traces, find common paths, compile control flow, and check execution conditions.
Four core engineering tasks for a behavioral cache layer
Observe & mine
Observability and automatic hot-path discovery
Record execution as structured state–action call trees. Process mining and common-subgraph extraction reveal frequently successful intermediate paths.
Read / Write
Separate result replay from action replay
Read-only segments can reuse previous results. Writes must replay actions in the real environment; a historical return value cannot replace a new write.
Control flow
Compile composition and control flow together
Compile tool combinations, loops, and conditional branches into deterministic code, keeping model calls only where semantic judgment is needed.
Lifecycle
Dynamic checks, eviction, and safe fallback
Block invalid paths as environments change and manage cache lifetime and capacity, preventing blind replay and unchecked cache growth.
Verifying, updating, and evicting cachesVerify before admission, update during use, and fall back or evict when invalid.
From admission checks to ongoing eviction
Trajectory
Parameterized traces
Record intermediate state results for A → B, or tool sequences, code skeletons, and dynamic parameter slots.
Guards
Environment guards
Record environmental assumptions such as page DOM, visual screen features, API versions, and hashes of data conventions.
Validation & Fallback
Reuse decisions and fallback
Compare the environment before and during replay. Execute deterministically when it matches; otherwise stop immediately and safely return to the model for replanning.
Verify & Evict
Admission verification and cache eviction
New traces enter the cache only after independent consistency tests on held-out data. TTL, LFU / LRU, and periodic probes then remove rarely used or invalid entries.
The limits of models and external toolsPreserve model exploration for new tasks while executing deterministic work efficiently.
External tools and human constraints also have limits: obscuring real environments and feedback may hinder model improvement. Engineering should preserve room for models to learn and adapt in real tasks.
Model progress and the role of external caches
Where data is abundant and feedback is nearly free, learning, data, and compute should improve model capability. But internalizing a task is not the same as executing it efficiently; deterministic work need not repeatedly use costly inference.
Engineering optimization does not assume model progress stops. Let models explore unknown paths, then automatically compile proven hot paths into caches. Stronger models can solve deeper long-tail chains and leave richer behavioral caches.
Engineering around intelligence is only beginning. Progress comes from solving problems in practice.
Explore the edge–cloud architectureThe long-term value of cross-app tasks and behavioral cachingBuild verification evidence, private context, and reusable experience around real tasks.
Development model and commercializationAddress complex, cross-app
Address complex, cross-app
long-tail needs.
Standardized software struggles with infrequent, personalized, cross-app needs. Intent compilation lets AI fulfill them through natural language, while shifting one-time development costs into recurring runtime inference costs.
By reusing state results between nodes and replaying action traces, behavioral caching reduces recurring inference costs and creates a structural marginal-cost advantage.
Intent-layer retentionVerification suitesPrivate contextBehavioral cache library