# Custom Interface
Source: https://simplellmfunc.cn/advanced/custom-interface

Implementing LLM_Interface for non-OpenAI providers

# Custom Interface

SimpleLLMFunc ships with `OpenAICompatible` and `OpenAIResponsesCompatible`. For providers that don't follow either protocol, implement `LLM_Interface` directly.

## The Abstract Base

```python theme={null}
from SimpleLLMFunc.interface import LLM_Interface, APIKeyPool

class LLM_Interface(ABC):
    def __init__(self, api_key_pool, model_name, base_url=None, context_window=200_000):
        self.api_key_pool = api_key_pool
        self.model_name = model_name
        self.base_url = base_url
        self.context_window = context_window

    @abstractmethod
    async def chat(self, *, trace_id, stream=False, messages, timeout=None, **kwargs):
        """Non-streaming call. Returns ChatCompletion-like object."""
        ...

    @abstractmethod
    async def chat_stream(self, *, trace_id, stream=True, messages, timeout=None, **kwargs):
        """Streaming call. Returns AsyncGenerator of ChatCompletionChunk-like objects."""
        ...
```

## Minimal Implementation

```python theme={null}
from SimpleLLMFunc.interface import LLM_Interface, APIKeyPool


class MyProviderInterface(LLM_Interface):
    def __init__(self, api_key_pool: APIKeyPool, model_name: str, base_url: str):
        super().__init__(api_key_pool, model_name, base_url)
        # Initialize your client here

    async def chat(self, *, trace_id, stream=False, messages, timeout=None, **kwargs):
        api_key = self.api_key_pool.get_key()
        # Make your API call
        response = await my_client.complete(
            model=self.model_name,
            messages=messages,
            api_key=api_key,
        )
        # Return in ChatCompletion-compatible shape
        return self._to_chat_completion(response)

    async def chat_stream(self, *, trace_id, stream=True, messages, timeout=None, **kwargs):
        api_key = self.api_key_pool.get_key()
        async for chunk in my_client.stream(
            model=self.model_name,
            messages=messages,
            api_key=api_key,
        ):
            yield self._to_chunk(chunk)
```

## Response Shape Requirements

The framework expects responses compatible with OpenAI's types:

### Non-streaming (`chat`)

Must return an object with:

* `.choices[0].message.content` — text response
* `.choices[0].message.tool_calls` — list of tool calls (optional)
* `.usage.prompt_tokens`, `.usage.completion_tokens` — token counts

### Streaming (`chat_stream`)

Must yield objects with:

* `.choices[0].delta.content` — text delta (may be None)
* `.choices[0].delta.tool_calls` — tool call deltas (optional)
* `.choices[0].finish_reason` — "stop", "tool\_calls", etc. (on final chunk)

## Tool Call Format

If your provider supports tool calling, tool calls must be in this shape:

```python theme={null}
{
    "id": "call_abc123",
    "type": "function",
    "function": {
        "name": "tool_name",
        "arguments": '{"param": "value"}'  # JSON string
    }
}
```

## Using Your Interface

```python theme={null}
llm = MyProviderInterface(
    api_key_pool=APIKeyPool(api_keys=["key1"], provider_id="my-provider"),
    model_name="my-model",
    base_url="https://api.myprovider.com/v1",
)

@llm_function(llm_interface=llm)
async def my_function(text: str) -> str:
    """Works with any LLM_Interface implementation."""
    pass
```

## OpenAIResponsesCompatible

For providers implementing OpenAI's Responses API (not the standard Chat Completions API):

```python theme={null}
from SimpleLLMFunc import OpenAIResponsesCompatible, APIKeyPool

llm = OpenAIResponsesCompatible(
    api_key_pool=APIKeyPool(api_keys=["sk-..."], provider_id="openai-responses"),
    model_name="gpt-4o",
    base_url="https://api.openai.com/v1",
)
```

The Responses adapter:

* Maps system prompts to Responses `instructions`
* Handles Responses-specific streaming format
* Supports `reasoning={...}` kwargs for reasoning effort control
* Keeps all wire-format differences in the adapter — your decorator code stays the same

→ [API Reference: Interfaces](/api/interfaces)


# Harness Patterns
Source: https://simplellmfunc.cn/advanced/harness-patterns

Production wrappers, custom state management, and supervisor architectures

# Harness Patterns

A "harness" is the code around your `@llm_chat` agent that manages state, context planning, and orchestration. The framework gives you the ReAct loop; you build the harness that makes it production-ready.

## Core Philosophy

> An agent is not a person. It is a method for constructing the right context for each reasoning step.

The harness engineer's job:

1. Ensure the model sees the **shortest complete context** at every step
2. Persist state so sessions can be **resumed deterministically**
3. Encode lessons into the **environment**, not into operator memory

## Pattern 1: TUI General Agent

The canonical pattern from `examples/tui_general_agent_example.py`:

```python theme={null}
repl = PyRepl(working_directory=workspace)
file_tools = FileToolset(workspace).toolset

@llm_chat(
    llm_interface=llm,
    toolkit=[*repl.toolset, *file_tools],
    stream=True,
    self_reference_key=MEMORY_KEY,
    temperature=1.0,
)
async def core_agent(message: str, history: HistoryList):
    """
    {system_prompt_here}
    {environment_block}
    """
    pass


@tui(custom_event_hook=[debug_hook])
async def agent(message: str, history=None, _abort_signal=None):
    prepared_message = inject_compaction_if_needed(message)
    prepared_history = prepare_history(history)
    template_params = build_template_params()

    async for output in core_agent(
        message=prepared_message,
        history=prepared_history,
        _template_params=template_params,
        _abort_signal=_abort_signal,
    ):
        yield output
```

The harness layer (`agent`) wraps the core agent to:

* Inject compaction instructions when context is large
* Build dynamic template parameters (environment detection, workspace info)
* Prepare history format
* Route abort signals
* Connect to the TUI

## Pattern 2: Context Window Management

```python theme={null}
COMPACTION_THRESHOLD = 0.2  # Compact when 20% through context window

def _should_request_compaction() -> bool:
    total_tokens = llm.input_token_count + llm.output_token_count
    context_window = llm.context_window
    return total_tokens > context_window * COMPACTION_THRESHOLD


def prepare_user_message(message: str) -> str:
    if not _should_request_compaction():
        return message

    compaction_instruction = (
        "After finishing this task, call runtime.selfref.context.compact(...) "
        "to checkpoint your context."
    )
    return f"{message}\n\n{compaction_instruction}"
```

This pattern keeps context fresh without manual intervention.

## Pattern 3: Dynamic Environment Block

Inject runtime context via template parameters:

```python theme={null}
def build_environment_block(workspace: Path) -> str:
    git_branch = run_command(["git", "branch", "--show-current"], workspace)
    git_status = run_command(["git", "status", "--porcelain"], workspace)

    return f"""
# Environment
- Workspace: {workspace}
- Git branch: {git_branch}
- Modified files: {len(git_status.splitlines())}
- Platform: {sys.platform}
- Python: {sys.version_info.major}.{sys.version_info.minor}
"""


TEMPLATE_PARAMS = {"environment_block": build_environment_block(workspace)}
```

## Pattern 4: Supervisor Agent

An outer agent that delegates to specialized inner agents:

```python theme={null}
@llm_chat(llm_interface=llm, toolkit=[delegate_to_coder, delegate_to_reviewer])
async def supervisor(task: str, history: list | None = None):
    """
    Route tasks to the appropriate specialist.
    Use delegate_to_coder for implementation work.
    Use delegate_to_reviewer for code review.
    Synthesize results before responding.
    """
    pass


@tool
async def delegate_to_coder(task: str, context: str) -> str:
    """Delegate a coding task to the implementation specialist."""
    result = []
    async for output in coder_agent(f"{task}\n\nContext: {context}", []):
        if is_response_yield(output):
            result.append(output.response)
    return "\n".join(result)
```

The supervisor pattern enables:

* Different models for different tasks (cheap model routes, expensive model implements)
* Isolated context per specialist (each starts fresh)
* Clear delegation boundaries

## Pattern 5: Session Persistence

Persist history externally for resume:

```python theme={null}
import json
from pathlib import Path

SESSION_DIR = Path("~/.myagent/sessions").expanduser()


def save_session(session_id: str, history: list, metadata: dict):
    SESSION_DIR.mkdir(parents=True, exist_ok=True)
    path = SESSION_DIR / f"{session_id}.json"
    path.write_text(json.dumps({
        "history": history,
        "metadata": metadata,
    }, ensure_ascii=False))


def load_session(session_id: str) -> tuple[list, dict] | None:
    path = SESSION_DIR / f"{session_id}.json"
    if not path.exists():
        return None
    data = json.loads(path.read_text())
    return data["history"], data["metadata"]
```

## Pattern 6: Custom Tool Runtime Override

Override which tools the agent sees based on context:

```python theme={null}
from SimpleLLMFunc.runtime.selfref import SELF_REFERENCE_TOOLKIT_OVERRIDE_TEMPLATE_PARAM


def build_runtime_toolkit(workspace: Path) -> list:
    """Build toolkit dynamically based on workspace state."""
    tools = [*repl.toolset]

    if (workspace / "package.json").exists():
        tools.extend(node_tools)
    elif (workspace / "pyproject.toml").exists():
        tools.extend(python_tools)

    tools.extend(FileToolset(workspace).toolset)
    return tools


template_params = {
    SELF_REFERENCE_TOOLKIT_OVERRIDE_TEMPLATE_PARAM: build_runtime_toolkit(workspace),
}
```

## The Key Insight

The harness is where **context engineering** happens:

* What the model sees = what you put in the template params + history + tool results
* When the model forgets = when you fail to compact or persist
* Why the model fails = usually missing context, not missing capability

Build harnesses that make the right context inevitable, not optional.


# Runtime Primitives
Source: https://simplellmfunc.cn/advanced/primitives

Extend the agent runtime with custom PrimitivePacks

# Runtime Primitives

Primitives are functions callable from inside `execute_code` (PyRepl) as `runtime.<namespace>.<name>(...)`. They let tools access agent-level capabilities without breaking the tool boundary.

## Concepts

| Concept                     | What It Is                                                                        |
| --------------------------- | --------------------------------------------------------------------------------- |
| **PrimitivePack**           | A named collection of related primitives + a backend                              |
| **RuntimePrimitiveBackend** | The stateful object a pack wraps (e.g., SelfReference)                            |
| **@primitive**              | Decorator that registers a function as a callable primitive                       |
| **PrimitiveSpec**           | The contract exposed to the model (name, description, parameters, best practices) |
| **PrimitiveCallContext**    | Runtime context passed to primitive handlers                                      |

## Built-in: SelfRef Pack

The framework ships with one built-in pack — `selfref` — providing 6 primitives:

```
runtime.selfref.context.inspect()
runtime.selfref.context.remember(text)
runtime.selfref.context.forget(experience_id)
runtime.selfref.context.compact(goal, instruction, ...)
runtime.selfref.fork.spawn(task, instruction, ...)
runtime.selfref.fork.gather_all(include_history=False)
```

## Creating a Custom Pack

### 1. Define a Backend

```python theme={null}
from SimpleLLMFunc.runtime import RuntimePrimitiveBackend


class MetricsBackend(RuntimePrimitiveBackend):
    def __init__(self):
        self.counters: dict[str, int] = {}

    def increment(self, name: str, amount: int = 1) -> int:
        self.counters[name] = self.counters.get(name, 0) + amount
        return self.counters[name]

    def get_all(self) -> dict[str, int]:
        return dict(self.counters)
```

### 2. Define Primitives

```python theme={null}
from SimpleLLMFunc.runtime import primitive, PrimitiveCallContext


@primitive
async def increment_counter(
    ctx: PrimitiveCallContext,
    name: str,
    amount: int = 1,
) -> dict:
    """
    Increment a named counter and return its new value.

    Args:
        name: Counter name.
        amount: How much to add. Default 1.

    Returns:
        Dict with counter name and new value.

    Best Practices:
        - Use descriptive counter names (e.g., "files_processed", "errors_found").
        - Use this to track progress across tool calls.
    """
    backend: MetricsBackend = ctx.backend
    new_value = backend.increment(name, amount)
    return {"name": name, "value": new_value}


@primitive
async def get_metrics(ctx: PrimitiveCallContext) -> dict:
    """
    Get all current counter values.

    Returns:
        Dict of all counter names to their current values.

    Best Practices:
        - Call this to report progress summaries.
        - Useful before context compaction to record final metrics.
    """
    backend: MetricsBackend = ctx.backend
    return backend.get_all()
```

<Warning>
  Primitive docstrings **must** include a `Best Practices` section. Registration fails without it.
</Warning>

### 3. Build the Pack

```python theme={null}
from SimpleLLMFunc.runtime import PrimitivePack

metrics_pack = PrimitivePack(
    namespace="metrics",
    backend=MetricsBackend(),
    primitives=[increment_counter, get_metrics],
    guide={
        "namespace": "metrics",
        "overview": "Track numeric counters across tool calls.",
        "best_practices": {
            "when_to_use": "To track progress, count occurrences, or measure work done.",
            "pattern": "Increment during work, get_metrics at milestones.",
        },
    },
)
```

### 4. Install in PyRepl

```python theme={null}
from SimpleLLMFunc.builtin import PyRepl

repl = PyRepl()
repl.install_primitive_pack(metrics_pack)
```

Now the model can call:

```python theme={null}
runtime.metrics.increment_counter("files_read")
runtime.metrics.get_metrics()
```

## Discoverability

The model discovers primitives through built-in meta-primitives:

```python theme={null}
runtime.list_primitives()              # All available primitive names
runtime.get_primitive_spec("metrics.increment_counter")  # Full contract
runtime.list_backends()                # Installed backend packs
```

The spec includes parameter descriptions, return type, and best practices — enough for the model to use the primitive correctly without prior training.

## PrimitiveCallContext

Every primitive handler receives a context object:

```python theme={null}
@dataclass
class PrimitiveCallContext:
    backend: RuntimePrimitiveBackend   # The pack's backend instance
    fork_context: Optional[ForkContext]  # Fork info (if inside a fork)
    event_emitter: ToolEventEmitter    # Emit custom events
```

Use `ctx.event_emitter` to stream progress from long-running primitives.

→ [API Reference: Runtime](/api/runtime)


# PyRepl
Source: https://simplellmfunc.cn/advanced/pyrepl

Persistent Python REPL with runtime primitives and streaming output

# PyRepl

PyRepl is a persistent IPython REPL running in a subprocess. It gives the model a continuous execution environment where variables persist across calls and runtime primitives are accessible without imports.

In 0.8.1, `PyRepl` remains the public facade while its internals are split into focused components: worker lifecycle (`pyrepl_worker_client.py`), execute/reset orchestration (`pyrepl_execution.py`), primitive host integration (`pyrepl_primitive_host.py`), tool factory/output formatting (`pyrepl_tools.py`), audit logging (`pyrepl_audit.py`), and input bridging (`pyrepl_input_bridge.py` / `pyrepl_input_mixin.py`). This is an internal architecture cleanup; user-facing usage stays the same.

## Core Properties

* **Persistent state** — Variables defined in one `execute_code` call are available in the next
* **Isolated process** — Runs in a separate subprocess (multiprocessing spawn). Crashes don't kill the main process
* **Runtime injection** — The `runtime` object is globally available. No imports needed
* **Streaming output** — stdout/stderr stream in real-time via custom events
* **Image artifacts** — images produced by `display(Image(...))` or image-rich last expressions are returned to the model as multimodal tool results
* **Timeout protection** — Default 600s per execution. Configurable

## Setup

```python theme={null}
from SimpleLLMFunc.builtin import PyRepl

repl = PyRepl(working_directory="/path/to/workspace")

# Use as toolkit in an agent
@llm_chat(llm_interface=llm, toolkit=repl.toolset, stream=True)
async def agent(message: str, history: list | None = None):
    """A code-executing agent."""
    pass
```

## Tools Provided

### execute\_code

Run arbitrary Python code in the persistent REPL:

```python theme={null}
result = await execute_code(code="""
import math
x = math.sqrt(144)
print(f"Result: {x}")
x
""")
# Returns: {"stdout": "Result: 12.0\n", "return_value": "12.0", "execution_time_ms": ...}
```

Return value is the last expression's repr (like IPython).

### Image Output

When executed code produces image output, `execute_code` returns a multimodal tool result instead of flattening the image to text. The model receives the normal execution summary plus the image content.

Supported patterns include explicit display calls:

```python theme={null}
from IPython.display import Image, display

display(Image(filename="chart.png"))
```

And image-rich last expressions:

```python theme={null}
from IPython.display import Image

Image(filename="chart.png")
```

For generated plots, saving the figure and returning an image payload is the clearest pattern:

```python theme={null}
import matplotlib.pyplot as plt
from SimpleLLMFunc.type import ImgPath

plt.plot([1, 2, 3])
path = "chart.png"
plt.savefig(path)
ImgPath(path)
```

Direct `PyRepl.execute(...)` calls expose captured images in the returned `artifacts` list. The `execute_code` tool converts those artifacts into `ImgPath` / `ImgUrl` multimodal returns for the agent loop.

### reset\_repl

Clear all user variables but keep runtime backends:

```python theme={null}
await reset_repl()
# After this: `x` is gone, but `runtime.selfref.context.inspect()` still works
```

## Runtime Namespace

Inside `execute_code`, the `runtime` object provides:

```python theme={null}
# Meta-primitives (always available)
runtime.list_primitives()
runtime.list_primitive_specs()
runtime.get_primitive_spec("selfref.context.compact")
runtime.list_backends()

# SelfRef primitives (when selfref is active)
runtime.selfref.context.inspect()
runtime.selfref.context.remember("important fact")
runtime.selfref.context.forget("exp_001")
runtime.selfref.context.compact(goal=..., instruction=..., ...)
runtime.selfref.fork.spawn(task=..., instruction=...)
runtime.selfref.fork.gather_all()

# Custom pack primitives (if installed)
runtime.metrics.increment_counter("files_read")
```

## Streaming Output

PyRepl emits custom events for real-time output:

| Event Name      | Data              | When              |
| --------------- | ----------------- | ----------------- |
| `kernel_stdout` | `{"text": "..."}` | Each stdout flush |
| `kernel_stderr` | `{"text": "..."}` | Each stderr flush |

Consume in your event handler:

```python theme={null}
from SimpleLLMFunc.hooks import is_event_yield, CustomEvent

async for output in agent("run some code", history):
    if is_event_yield(output):
        if isinstance(output.event, CustomEvent):
            if output.event.event_name == "kernel_stdout":
                print(output.event.data["text"], end="")
```

## Output Truncation

If a tool result exceeds \~20,000 tokens:

1. Full output is written to a temporary file
2. Truncated version (first \~4,096 tokens) + file path is returned to the model
3. The model can use `read_file` to access specific parts

Enable per-agent with `_too_long_to_file=True`.

## Working Directory

```python theme={null}
repl = PyRepl(working_directory="/path/to/project")
```

The REPL starts in this directory. `os.getcwd()` inside execute\_code returns this path.

## Installing Custom Primitive Packs

```python theme={null}
from SimpleLLMFunc.runtime import PrimitivePack

my_pack = PrimitivePack(namespace="mytools", backend=..., primitives=[...])
repl.install_primitive_pack(my_pack)
```

After installation, `runtime.mytools.*` is available in execute\_code.

## SelfRef Integration

When using `self_reference_key` on the agent, the framework automatically:

1. Creates a SelfReference backend
2. Builds the selfref primitive pack
3. Installs it in the PyRepl instance
4. Makes `runtime.selfref.*` available

```python theme={null}
@llm_chat(
    llm_interface=llm,
    toolkit=repl.toolset,
    self_reference_key="agent_main",
)
async def agent(message: str, history: list | None = None):
    """Agent with code execution + self-reference."""
    pass
```

## Practical Pattern: CodeAct Agent

The "CodeAct" pattern uses PyRepl as the primary action surface — the model writes Python to accomplish tasks instead of using separate tools for each operation:

```python theme={null}
repl = PyRepl(working_directory=workspace)
file_tools = FileToolset(workspace).toolset

@llm_chat(
    llm_interface=llm,
    toolkit=[*repl.toolset, *file_tools],
    self_reference_key="agent_main",
    stream=True,
)
async def coding_agent(message: str, history: list | None = None):
    """
    Solve coding tasks by writing and executing Python.
    Use execute_code for inspection, computation, and verification.
    Use file tools for reading and editing project files.
    Use runtime.selfref.context.compact(...) when context grows large.
    """
    pass
```

→ [API Reference: Builtins](/api/builtins)


# SelfRef Engineering
Source: https://simplellmfunc.cn/advanced/selfref-engineering

Advanced patterns for fork orchestration, compaction strategies, and multi-key memory

# SelfRef Engineering

This page covers advanced SelfRef patterns for production agent architectures.

For maintainers: 0.8.1 splits the former SelfReference god object into a facade plus `store`, `active_turn`, `mutations`, `memory_api`, `context_memory`, `agent_binding`, `fork_manager`, and `fork_utils`. Application code should still interact through `SelfReference`, `runtime.selfref.context.*`, and `runtime.selfref.fork.*`; the split is an internal maintainability boundary.

## Compaction Strategy

### When to Compact

Compact when:

* Token usage approaches context window limits
* A logical milestone is complete (task finished, phase transition)
* The working transcript contains mostly stale information (old tool outputs, superseded plans)

### What Makes a Good Compact Call

```python theme={null}
payload = runtime.selfref.context.compact(
    goal="Implement user authentication with OAuth",
    instruction="Continue with refresh token logic. Token exchange is working.",
    discoveries=[
        "OAuth provider requires PKCE challenge",
        "Access tokens expire in 1 hour",
        "Refresh endpoint: POST /oauth/token with grant_type=refresh_token",
    ],
    completed=[
        "Set up project structure",
        "Implemented PKCE challenge generation",
        "Built token exchange endpoint (tested, working)",
    ],
    current_status="Token exchange works. Next: refresh token flow.",
    likely_next_work="Implement refresh token rotation, add token storage, write integration tests",
    relevant_files_directories=[
        "src/auth/oauth.py",
        "src/auth/token_store.py",
        "tests/test_auth.py",
    ],
    remember=["OAuth provider requires PKCE — never skip it"],
)
```

Key rules:

* `discoveries` = facts learned that may be needed later
* `completed` = what's done (so the agent doesn't redo it)
* `current_status` = where we stopped
* `likely_next_work` = what to do next (gives the resuming agent direction)
* `remember` = short durable lessons that should persist as experiences (use sparingly)

### Compaction Lifecycle

1. `compact(...)` is called inside `execute_code`
2. The compaction is **queued** (not applied immediately)
3. After the current tool batch finishes, the runtime applies a compaction patch to the transcript
4. System prompt + experiences are preserved
5. Working transcript is replaced with the summary message
6. The agent's next turn starts with fresh, compact context

### Auto-Compaction Pattern

Inject compaction instructions when approaching token limits:

```python theme={null}
COMPACTION_THRESHOLD = 0.3  # 30% of context window

def should_compact(llm) -> bool:
    used = llm.input_token_count + llm.output_token_count
    return used > llm.context_window * COMPACTION_THRESHOLD

def prepare_message(message: str) -> str:
    if should_compact(llm):
        return message + "\n\nAfter completing this task, compact your context."
    return message
```

## Fork Orchestration

### Basic Fork Pattern

```python theme={null}
# Inside execute_code:

# Spawn children
handle_a = runtime.selfref.fork.spawn(
    task="Review src/auth/ for security issues",
    instruction="Check for: injection, auth bypass, secret leakage. Return a list of findings.",
)

handle_b = runtime.selfref.fork.spawn(
    task="Write unit tests for src/auth/oauth.py",
    instruction="Cover: token exchange, PKCE validation, error cases. Write to tests/test_oauth.py.",
)

print(f"Spawned: {handle_a['fork_id']}, {handle_b['fork_id']}")

# ... parent continues working on other things ...

# Gather when ready
results = runtime.selfref.fork.gather_all()
for fork_id, result in results.items():
    print(f"{fork_id}: {result['status']}")
    if result['status'] == 'completed':
        print(f"  Response: {result['response'][:200]}")
```

### Fork Design Rules

1. **Children inherit pre-fork context** — They see the parent's conversation up to the moment before the fork tool call. They do NOT see the parent's fork call itself.

2. **Children are independent** — They cannot read or modify the parent's context. Each child has its own isolated ReAct loop.

3. **Gather blocks** — `gather_all()` waits until all spawned children complete. Don't call it immediately after spawn unless you actually need results now.

4. **Keep tasks bounded** — Give each child a clear scope, acceptance criteria, and stop condition. Unbounded tasks waste tokens.

5. **Ask for summaries** — Tell children to return concise results + file paths, not full transcripts. Use `include_history=True` only when you need to inspect their reasoning.

### Fork vs. Sequential

Use forks when:

* Tasks are independent (no data dependencies between them)
* Tasks are substantial enough to justify the overhead of a child context
* Parallelism saves wall-clock time

Use sequential tool calls when:

* Tasks depend on each other's outputs
* Tasks are small (one tool call each)
* You need intermediate results to decide next steps

## Multi-Key Memory

SelfReference supports multiple memory keys for partitioned state:

```python theme={null}
from SimpleLLMFunc.builtin import SelfReference

selfref = SelfReference()

# Bind different histories to different keys
selfref.bind_history("coding", coding_history)
selfref.bind_history("research", research_history)
```

Use cases:

* Separate agent personas within one application
* Long-running project memory vs. ephemeral task context
* Shared reference context vs. per-user conversation

Each key has independent: history, experiences, summary state, and fork handles.

## Experience Management

### When to Remember

Good experiences:

* Durable lessons ("this API requires auth header X")
* User preferences ("prefers concise output")
* Project conventions ("always use pytest, not unittest")

Bad experiences:

* Transient state ("currently working on file X") — put this in compact summary instead
* Large data — experiences live in system prompt, keep them short
* Temporary context — use working messages or compact checkpoint instead

### Pruning Experiences

```python theme={null}
# Inspect current experiences
snapshot = runtime.selfref.context.inspect()
for exp in snapshot["experiences"]:
    print(f"  {exp['id']}: {exp['text']}")

# Remove stale ones
runtime.selfref.context.forget("exp_old_001")
```

## Production Pattern: Durable Agent

```python theme={null}
repl = PyRepl(working_directory=workspace)
file_tools = FileToolset(workspace).toolset

@llm_chat(
    llm_interface=llm,
    toolkit=[*repl.toolset, *file_tools],
    stream=True,
    self_reference_key="project_agent",
)
async def project_agent(message: str, history: list | None = None):
    """
    Long-running project agent with durable memory.

    Use runtime.selfref.context.remember(...) for durable project facts.
    Use runtime.selfref.context.compact(...) at milestones.
    Use runtime.selfref.fork.spawn(...) for parallelizable subtasks.
    """
    pass
```

This agent:

* Remembers across sessions (experiences persist in SelfReference backend)
* Compacts context at milestones (keeps context fresh)
* Delegates work to forks (parallelism for complex tasks)
* Has full file and code access (via toolsets)

→ [Context: SelfRef](/context/selfref) | [API Reference: Runtime](/api/runtime)


# Builtins
Source: https://simplellmfunc.cn/api/builtins

API reference for SelfReference, PyRepl, and FileToolset

# Builtins API Reference

## SelfReference

```python theme={null}
from SimpleLLMFunc.builtin import SelfReference
```

Durable backend for agent memory, context management, and fork orchestration.

### Constructor

```python theme={null}
SelfReference()
```

### Key Methods

| Method                       | Description                                 |
| ---------------------------- | ------------------------------------------- |
| `bind_history(key, history)` | Bind a conversation history to a memory key |
| `get_history(key)`           | Get the current history for a key           |
| `get_experiences(key)`       | Get stored experiences for a key            |

### Memory Key Isolation

Each key has independent state:

* History (conversation transcript)
* Experiences (durable facts)
* Summary state (compaction checkpoints)
* Fork handles (child agent state)

### Usage with @llm\_chat

```python theme={null}
# Automatic (recommended)
@llm_chat(llm_interface=llm, self_reference_key="agent_main", ...)
async def agent(...):
    ...

# Explicit backend
selfref = SelfReference()
selfref.bind_history("my_key", initial_history)

@llm_chat(llm_interface=llm, self_reference=selfref, self_reference_key="my_key", ...)
async def agent(...):
    ...
```

### Runtime Primitives (via PyRepl)

When SelfRef is active, these are available in `execute_code`:

```python theme={null}
# Context
runtime.selfref.context.inspect() -> dict
runtime.selfref.context.remember(text: str) -> dict  # {id, text}
runtime.selfref.context.forget(experience_id: str) -> bool
runtime.selfref.context.compact(
    goal: str,
    instruction: str,
    discoveries: list[str],
    completed: list[str],
    current_status: str,
    likely_next_work: str,
    relevant_files_directories: list[str],
    remember: list[str] = [],
) -> dict  # {assistant_message, ...}

# Fork
runtime.selfref.fork.spawn(
    task: str,
    instruction: str,
    ...
) -> dict  # {fork_id, status}
runtime.selfref.fork.gather_all(include_history: bool = False) -> dict  # {fork_id: ForkResult}
```

***

## PyRepl

```python theme={null}
from SimpleLLMFunc.builtin import PyRepl
```

Persistent IPython REPL in a subprocess with runtime primitive support.

### Constructor

```python theme={null}
PyRepl(working_directory: Optional[Path | str] = None)
```

| Parameter           | Default      | Description                            |
| ------------------- | ------------ | -------------------------------------- |
| `working_directory` | `None` (cwd) | Working directory for the REPL process |

### Properties

| Property   | Type         | Description                                  |
| ---------- | ------------ | -------------------------------------------- |
| `.toolset` | `List[Tool]` | The two tools: `execute_code` + `reset_repl` |

### Methods

| Method                         | Description                                |
| ------------------------------ | ------------------------------------------ |
| `install_primitive_pack(pack)` | Register a PrimitivePack in this REPL      |
| `get_runtime_backend(name)`    | Get an installed backend by pack namespace |
| `reset()`                      | Clear REPL variables (preserves backends)  |

### Tools Provided

#### execute\_code

```python theme={null}
async def execute_code(code: str, event_emitter=None) -> dict:
    """Run Python code in the persistent REPL."""
```

Returns:

```python theme={null}
{
    "stdout": str,          # Captured stdout
    "stderr": str,          # Captured stderr
    "return_value": str,    # Repr of last expression (or None)
    "artifacts": list,      # Captured image artifacts for direct PyRepl.execute calls
    "error": str | None,    # Error message if failed
    "execution_time_ms": float,
}
```

When used through the agent toolset, `execute_code` returns a natural-language summary. If code outputs images through `display(Image(...))`, an image-rich last expression, or `ImgPath(...)` / `ImgUrl(...)`, the tool result becomes multimodal so the model can inspect the image.

#### reset\_repl

```python theme={null}
async def reset_repl() -> str:
    """Clear REPL variables. Preserves runtime backends."""
```

### Example

```python theme={null}
from SimpleLLMFunc.builtin import PyRepl

repl = PyRepl(working_directory="/path/to/project")

@llm_chat(llm_interface=llm, toolkit=repl.toolset, stream=True)
async def code_agent(message: str, history: list | None = None):
    """Execute Python to solve problems."""
    pass
```

### Custom Events

| Event Name      | Data            | Description           |
| --------------- | --------------- | --------------------- |
| `kernel_stdout` | `{"text": str}` | Real-time stdout line |
| `kernel_stderr` | `{"text": str}` | Real-time stderr line |

### Execution Limits

| Setting            | Default                     |
| ------------------ | --------------------------- |
| Execution timeout  | 600 seconds                 |
| Input idle timeout | 300 seconds                 |
| Output truncation  | \~20,000 tokens → temp file |

***

## FileToolset

```python theme={null}
from SimpleLLMFunc.builtin import FileToolset
```

Workspace-scoped file operations with stale-write protection.

### Constructor

```python theme={null}
FileToolset(workspace: Path | str)
```

All paths are resolved relative to `workspace`. Path traversal above workspace is rejected.

### Properties

| Property   | Type         | Description      |
| ---------- | ------------ | ---------------- |
| `.toolset` | `List[Tool]` | The 5 file tools |

### Tools Provided

#### read\_file

```python theme={null}
async def read_file(path: str, start_line: int = None, end_line: int = None) -> str
```

Returns line-numbered content: `<lineno> | <content>`.

#### read\_image

```python theme={null}
async def read_image(path: str) -> ImgPath
```

Returns image as multimodal content (detail="high").

#### grep

```python theme={null}
async def grep(pattern: str, path_pattern: str) -> str
```

Regex search over workspace files. `path_pattern` scopes which files to search.

#### sed

```python theme={null}
async def sed(path: str, start_line: int, end_line: int, pattern_to_be_replaced: str, new_string: str) -> str
```

Regex find-and-replace within a line range. Protected by hash-based stale detection.

#### echo\_into

```python theme={null}
async def echo_into(path: str, content: str) -> str
```

Overwrite entire file. Protected by hash-based stale detection (file must be read first).

### Stale-Write Protection

FileToolset tracks file hashes on read. If a file was modified externally between read and write, `sed` and `echo_into` will reject the write and ask the agent to re-read.

### Example

```python theme={null}
from SimpleLLMFunc.builtin import FileToolset

files = FileToolset("/path/to/workspace")

@llm_chat(llm_interface=llm, toolkit=files.toolset, stream=True)
async def file_agent(message: str, history: list | None = None):
    """An agent that can read and edit files in the workspace."""
    pass
```


# Context Types
Source: https://simplellmfunc.cn/api/context-types

API reference for ContextState, CompileSource, all 10 ContextMutation types, and compiled outputs

# Context Types API Reference

## ContextState

```python theme={null}
from SimpleLLMFunc.base.types import ContextState
```

Runtime representation of conversation state.

```python theme={null}
@dataclass
class ContextState:
    messages: NormalizedMessageList
    data_from_selfref: Optional[DataFromSelfRef] = None
    pending_mutations: List[ContextMutation] = field(default_factory=list)
```

| Field               | Type                      | Description                               |
| ------------------- | ------------------------- | ----------------------------------------- |
| `messages`          | `NormalizedMessageList`   | Current conversation transcript           |
| `data_from_selfref` | `DataFromSelfRef \| None` | Durable self-reference state              |
| `pending_mutations` | `List[ContextMutation]`   | Changes waiting for next compile boundary |

***

## CompileSource

```python theme={null}
from SimpleLLMFunc.base.types import CompileSource
```

Input boundary for compilation.

```python theme={null}
@dataclass(frozen=True)
class CompileSource:
    data_from_agent_config: DataFromAgentConfig
    data_from_selfref: Optional[DataFromSelfRef]
    input_messages: NormalizedMessageList
```

***

## DataFromAgentConfig

```python theme={null}
from SimpleLLMFunc.base.types import DataFromAgentConfig
```

Static configuration from the decorator.

```python theme={null}
@dataclass(frozen=True)
class DataFromAgentConfig:
    base_system_prompt: str
    template_params: Optional[Dict[str, Any]] = None
    tool_prompt_specs: List[Dict[str, Any]] = field(default_factory=list)
    include_must_principles: bool = False
```

| Field                     | Description                                          |
| ------------------------- | ---------------------------------------------------- |
| `base_system_prompt`      | Docstring content (may contain `{placeholders}`)     |
| `template_params`         | Values for placeholder substitution                  |
| `tool_prompt_specs`       | Tool best-practice specs for system prompt injection |
| `include_must_principles` | Whether to append structured-call rules              |

***

## DataFromSelfRef

```python theme={null}
from SimpleLLMFunc.base.types import DataFromSelfRef
```

Durable state from the SelfReference backend.

```python theme={null}
@dataclass(frozen=True)
class DataFromSelfRef:
    base_system_prompt: str
    experiences: List[Dict[str, str]] = field(default_factory=list)
    summary: Optional[Dict[str, Any]] = None
    summary_message: Optional[Dict[str, Any]] = None
    working_messages: NormalizedMessageList = field(default_factory=list)
```

| Field                | Description                                              |
| -------------------- | -------------------------------------------------------- |
| `base_system_prompt` | System prompt with selfref markers                       |
| `experiences`        | `[{id: str, text: str}, ...]` — durable remembered facts |
| `summary`            | Compaction metadata                                      |
| `summary_message`    | The summary as a displayable message                     |
| `working_messages`   | Post-compaction working transcript                       |

***

## ContextMutation (Union)

```python theme={null}
from SimpleLLMFunc.base.types import ContextMutation
```

```python theme={null}
ContextMutation = Union[
    AssistantMessageMutation,
    ToolResultMutation,
    MultimodalToolResultMutation,
    UserMessageMutation,
    ContextReplaceMutation,
    ContextSummaryMutation,
    ExperienceRememberMutation,
    ExperienceForgetMutation,
    AssistantTruncatedMutation,
    ToolCancelledMutation,
]
```

### AssistantMessageMutation

```python theme={null}
@dataclass
class AssistantMessageMutation:
    role: Literal["assistant"] = "assistant"
    content: Optional[str] = None
    tool_calls: List[Dict[str, Any]] = field(default_factory=list)
    reasoning_details: List[Dict[str, Any]] = field(default_factory=list)
```

### ToolResultMutation

```python theme={null}
@dataclass
class ToolResultMutation:
    tool_call_id: str
    content: str
    role: Literal["tool"] = "tool"
```

### MultimodalToolResultMutation

```python theme={null}
@dataclass
class MultimodalToolResultMutation:
    tool_call_id: str
    tool_name: str
    arguments: str
    user_messages: List[Dict[str, Any]]
```

### UserMessageMutation

```python theme={null}
@dataclass
class UserMessageMutation:
    message: Dict[str, Any]
```

### ContextReplaceMutation

```python theme={null}
@dataclass
class ContextReplaceMutation:
    messages: List[Dict[str, Any]]
```

### ContextSummaryMutation

```python theme={null}
@dataclass
class ContextSummaryMutation:
    summary_message: Dict[str, Any]
    remember: List[Dict[str, str]] = field(default_factory=list)
```

### ExperienceRememberMutation

```python theme={null}
@dataclass
class ExperienceRememberMutation:
    text: str
```

### ExperienceForgetMutation

```python theme={null}
@dataclass
class ExperienceForgetMutation:
    experience_id: str
```

### AssistantTruncatedMutation

```python theme={null}
@dataclass
class AssistantTruncatedMutation:
    partial_content: str
    abort_reason: str = ""
```

### ToolCancelledMutation

```python theme={null}
@dataclass
class ToolCancelledMutation:
    tool_call_id: str
    tool_name: str
    abort_reason: str = ""
```

***

## Compiled Outputs

### ReducedTurnContext

```python theme={null}
from SimpleLLMFunc.base.types import ReducedTurnContext
```

Output of Stage 1 (mutation application):

```python theme={null}
@dataclass
class ReducedTurnContext:
    transcript: NormalizedMessageList
    selfref_snapshot: Optional[DataFromSelfRef] = None
```

### CompiledTurnContext

```python theme={null}
from SimpleLLMFunc.base.types import CompiledTurnContext
```

Output of the full pipeline (ready for LLM):

```python theme={null}
@dataclass
class CompiledTurnContext:
    transcript: NormalizedMessageList
    system_prompt: Optional[str]
    llm_messages: NormalizedMessageList
    selfref_snapshot: Optional[DataFromSelfRef] = None
```

| Field              | Description                                                |
| ------------------ | ---------------------------------------------------------- |
| `transcript`       | Messages after system prompt resolution                    |
| `system_prompt`    | The resolved system prompt text                            |
| `llm_messages`     | Final messages for the provider (with tool specs injected) |
| `selfref_snapshot` | Carried-forward selfref state                              |

### CompiledContext

```python theme={null}
from SimpleLLMFunc.base.types import CompiledContext
```

Intermediate compiled state (before LLM rendering):

```python theme={null}
@dataclass
class CompiledContext:
    messages: NormalizedMessageList
    data_from_selfref: Optional[DataFromSelfRef] = None
```

***

## Pipeline Functions

### compile\_invocation\_turn

```python theme={null}
from SimpleLLMFunc.base.compile_pipeline import compile_invocation_turn

def compile_invocation_turn(
    spec: InvocationSpec,
    transcript: NormalizedMessageList,
    pending_mutations: Optional[List[ContextMutation]] = None,
    selfref_snapshot: Optional[DataFromSelfRef] = None,
) -> CompiledTurnContext:
```

Single entry point for the full compile pipeline.

### reduce\_turn\_context

```python theme={null}
from SimpleLLMFunc.base.compile_pipeline import reduce_turn_context

def reduce_turn_context(
    transcript: NormalizedMessageList,
    pending_mutations: List[ContextMutation],
    selfref_snapshot: Optional[DataFromSelfRef] = None,
) -> ReducedTurnContext:
```

Stage 1 only: apply mutations, refresh selfref snapshot.

### convert\_to\_llm\_request

```python theme={null}
from SimpleLLMFunc.base.compile_pipeline import convert_to_llm_request

def convert_to_llm_request(
    reduced: ReducedTurnContext,
    prompt_contract: PromptContract,
) -> CompiledTurnContext:
```

Stage 2: resolve system prompt, render final messages.


# Decorators
Source: https://simplellmfunc.cn/api/decorators

API reference for @llm_function, @llm_chat, and @tool

# Decorators API Reference

## @llm\_function

```python theme={null}
from SimpleLLMFunc import llm_function

@llm_function(
    llm_interface: LLM_Interface,
    toolkit: Optional[List[Tool]] = None,
    max_tool_calls: Optional[int] = None,
    system_prompt_template: Optional[str] = None,
    user_prompt_template: Optional[str] = None,
    **llm_kwargs: Any,
)
```

Transforms an async function into an `LLMFunction` callable instance. `await fn(...)` returns the parsed typed result; `fn.stream(...)` yields `ReactOutput`.

For multimodal input, keep the normal Python-function style: explicitly type image parameters as `ImgUrl`, `ImgPath`, or lists/unions containing those types. `ImgUrl` accepts `http(s)` URLs and `data:` URLs; `ImgPath` is encoded as an image data URL at the provider boundary.

### Parameters

| Parameter                | Type                 | Default  | Description                                                        |
| ------------------------ | -------------------- | -------- | ------------------------------------------------------------------ |
| `llm_interface`          | `LLM_Interface`      | required | Model to call                                                      |
| `toolkit`                | `List[Tool] \| None` | `None`   | Tools available to the model                                       |
| `max_tool_calls`         | `int \| None`        | `None`   | Maximum tool calls before forcing final answer. `None` = unlimited |
| `system_prompt_template` | `str \| None`        | `None`   | Override the system prompt template                                |
| `user_prompt_template`   | `str \| None`        | `None`   | Override the user prompt template                                  |
| `**llm_kwargs`           | `Any`                | —        | Forwarded to LLM (temperature, max\_tokens, etc.)                  |

### Special Call-Time Parameters

| Parameter          | Type             | Description                                    |
| ------------------ | ---------------- | ---------------------------------------------- |
| `_template_params` | `Dict[str, Any]` | Template values for docstring `{placeholders}` |
| `_abort_signal`    | `AbortSignal`    | Cancellation signal                            |

### Example

```python theme={null}
from pydantic import BaseModel, Field
from SimpleLLMFunc import llm_function, OpenAICompatible

llm = OpenAICompatible.load_from_json_file("provider.json")["openrouter"]["gpt-4o"]

class Sentiment(BaseModel):
    label: str = Field(description="positive, negative, or neutral")
    score: float = Field(description="confidence 0.0–1.0")

@llm_function(llm_interface=llm)
async def classify(text: str) -> Sentiment:
    """Classify the sentiment of the text."""
    pass

# Usage
result = await classify("I love this!")
print(result)  # Sentiment(label="positive", score=0.95)
```

***

## @llm\_chat

```python theme={null}
from SimpleLLMFunc import llm_chat

@llm_chat(
    llm_interface: LLM_Interface,
    toolkit: Optional[List[Tool]] = None,
    max_tool_calls: Optional[int] = DEFAULT_MAX_TOOL_CALLS,
    stream: bool = False,
    strict_signature: bool = False,
    self_reference: Optional[SelfReference] = None,
    self_reference_key: Optional[str] = None,
    **llm_kwargs: Any,
)
```

Creates an `LLMChat` callable instance. Calling it yields `AsyncGenerator[ReactOutput, None]` and gives SelfRef a stable agent identity.

For multimodal chat input, prefer a single explicit `message: UserChatMessage` argument. `UserChatMessage` is OpenAI Chat Completions-compatible and can contain text and `image_url` content parts.

### Parameters

| Parameter            | Type                    | Default           | Description                            |
| -------------------- | ----------------------- | ----------------- | -------------------------------------- |
| `llm_interface`      | `LLM_Interface`         | required          | Model to call                          |
| `toolkit`            | `List[Tool] \| None`    | `None`            | Available tools                        |
| `max_tool_calls`     | `int \| None`           | framework default | Tool call limit per invocation         |
| `stream`             | `bool`                  | `False`           | Enable streaming chunks                |
| `strict_signature`   | `bool`                  | `False`           | Enforce strict parameter validation    |
| `self_reference`     | `SelfReference \| None` | `None`            | Explicit SelfReference backend         |
| `self_reference_key` | `str \| None`           | `None`            | Auto-create SelfReference for this key |
| `**llm_kwargs`       | `Any`                   | —                 | Forwarded to LLM                       |

### Special Call-Time Parameters

| Parameter           | Type             | Description                             |
| ------------------- | ---------------- | --------------------------------------- |
| `_template_params`  | `Dict[str, Any]` | Template values                         |
| `_abort_signal`     | `AbortSignal`    | Cancellation signal                     |
| `_too_long_to_file` | `bool`           | Truncate long tool results to temp file |

### History Parameter

The function must have a parameter named `history` or `chat_history` (type: `list | None`). The framework uses it to carry conversation state.

### Example

```python theme={null}
from SimpleLLMFunc import llm_chat, tool

@tool
async def search(query: str) -> str:
    """Search for information."""
    return f"Result for: {query}"

@llm_chat(llm_interface=llm, toolkit=[search], stream=True)
async def agent(message: str, history: list | None = None):
    """A helpful research assistant."""
    pass

# Usage
history = []
async for output in agent("What is SimpleLLMFunc?", history):
    if is_response_yield(output):
        print(output.response)
        history = output.messages
```

### Multimodal Chat Example

```python theme={null}
from SimpleLLMFunc import llm_chat
from SimpleLLMFunc.type import UserChatMessage, ImgUrl

@llm_chat(llm_interface=llm)
async def vision_agent(message: UserChatMessage, history: list | None = None):
    """Answer questions about user-provided images."""
    pass

async for output in vision_agent(
    UserChatMessage.multimodal(
        "What is in this image?",
        ImgUrl("https://example.com/cat.jpg", detail="high"),
    ),
    history=[],
):
    ...
```

***

## @tool

```python theme={null}
from SimpleLLMFunc import tool

@tool(
    name: Optional[str] = None,
    description: Optional[str] = None,
    best_practices: Optional[Sequence[str]] = None,
    prompt_injection_builder: Optional[Callable] = None,
    too_long_to_file: bool = False,
)
```

Registers an async function as a tool callable by the LLM. Can be used with or without arguments.

### Parameters

| Parameter                  | Type                    | Default        | Description                             |
| -------------------------- | ----------------------- | -------------- | --------------------------------------- |
| `name`                     | `str \| None`           | function name  | Override tool name                      |
| `description`              | `str \| None`           | from docstring | Override tool description               |
| `best_practices`           | `Sequence[str] \| None` | from docstring | Override best practices list            |
| `prompt_injection_builder` | `Callable \| None`      | `None`         | Dynamic system prompt injection         |
| `too_long_to_file`         | `bool`                  | `False`        | Write results > 20k tokens to temp file |

### Requirements

* Decorated function **must** be `async def`
* Docstring should include `Args`, `Returns`, and `Best Practices` sections
* Parameter types become the tool's JSON schema

### Example

```python theme={null}
from SimpleLLMFunc import tool

@tool
async def calculate(expression: str) -> float:
    """
    Evaluate a math expression.

    Args:
        expression: Python math expression (e.g., "2 ** 10").

    Returns:
        Numeric result.

    Best Practices:
        - Use Python syntax.
        - Keep expressions simple.
    """
    return float(eval(expression))

# Used without parentheses (simple form)
@tool
async def simple_tool(x: str) -> str:
    """Do something."""
    return x.upper()
```

***

## Tool Class

```python theme={null}
from SimpleLLMFunc import Tool

tool_instance = Tool(
    name: str,
    description: str,
    func: Callable[..., Awaitable[Any]],
    best_practices: Optional[Sequence[str]] = None,
    prompt_injection_builder: Optional[Callable] = None,
    too_long_to_file: bool = False,
)
```

For programmatic tool creation without the decorator.

### Methods

| Method                                    | Returns       | Description                       |
| ----------------------------------------- | ------------- | --------------------------------- |
| `.tool_spec()`                            | `Dict`        | Tool specification dict           |
| `.to_openai_tool()`                       | `Dict`        | OpenAI-format tool definition     |
| `.serialize_tools(tools)`                 | `List[Dict]`  | Class method: serialize tool list |
| `.build_system_prompt_injection(context)` | `str \| None` | Dynamic prompt injection          |

### Properties

| Property       | Type              | Description                   |
| -------------- | ----------------- | ----------------------------- |
| `.name`        | `str`             | Tool name                     |
| `.description` | `str`             | Tool description              |
| `.parameters`  | `List[Parameter]` | Extracted parameters          |
| `.func`        | `Callable`        | The underlying async function |


# Events
Source: https://simplellmfunc.cn/api/events

API reference for all 14 event types, ReactOutput, and stream utilities

# Events API Reference

## ReactOutput

```python theme={null}
from SimpleLLMFunc.hooks import ReactOutput, ResponseYield, EventYield
```

```python theme={null}
ReactOutput = ResponseYield | EventYield
```

### ResponseYield

```python theme={null}
@dataclass
class ResponseYield:
    response: Any                    # str, Pydantic model, or raw dict
    messages: NormalizedMessageList   # Updated conversation history
```

### EventYield

```python theme={null}
@dataclass
class EventYield:
    event: ReActEvent       # One of 14+ event types
    origin: EventOrigin     # Source identification
```

### EventOrigin

```python theme={null}
@dataclass
class EventOrigin:
    session_id: str
    agent_call_id: str
    event_seq: int
    parent_agent_call_id: str | None = None
    fork_id: str | None = None
    fork_depth: int = 0
    fork_seq: int | None = None
    selfref_instance_id: str | None = None
    source_memory_key: str | None = None
    memory_key: str | None = None
    tool_name: str | None = None
    tool_call_id: str | None = None
```

***

## Type Guards

```python theme={null}
from SimpleLLMFunc.hooks import is_response_yield, is_event_yield

is_response_yield(output: ReactOutput) -> bool
is_event_yield(output: ReactOutput) -> bool
```

***

## Stream Filters

```python theme={null}
from SimpleLLMFunc.hooks import responses_only, events_only, filter_events

# Only ResponseYield items
async for resp in responses_only(agent_stream):
    print(resp.response)

# Only EventYield items
async for evt in events_only(agent_stream):
    handle(evt.event)

# Only specific event types
async for evt in filter_events(agent_stream, LLMChunkArriveEvent):
    print(evt.event.accumulated_content)
```

***

## Event Types

### Loop Lifecycle Events

#### ReactStartEvent

```python theme={null}
@dataclass
class ReactStartEvent:
    event_type: ReActEventType = ReActEventType.REACT_START
```

Emitted once when the ReAct loop begins.

#### ReactIterationStartEvent

```python theme={null}
@dataclass
class ReactIterationStartEvent:
    iteration: int
    event_type: ReActEventType = ReActEventType.REACT_ITERATION_START
```

#### ReactIterationEndEvent

```python theme={null}
@dataclass
class ReactIterationEndEvent:
    iteration: int
    event_type: ReActEventType = ReActEventType.REACT_ITERATION_END
```

#### ReactEndEvent

```python theme={null}
@dataclass
class ReactEndEvent:
    final_messages: NormalizedMessageList
    response: Any
    event_type: ReActEventType = ReActEventType.REACT_END
```

Terminal event. Contains the final conversation state and response.

***

### LLM Call Events

#### LLMCallStartEvent

```python theme={null}
@dataclass
class LLMCallStartEvent:
    messages: NormalizedMessageList   # What the LLM will see
    event_type: ReActEventType = ReActEventType.LLM_CALL_START
```

#### LLMChunkArriveEvent

```python theme={null}
@dataclass
class LLMChunkArriveEvent:
    chunk: LLMStreamChunk             # Provider chunk object
    accumulated_content: str          # Content accumulated so far
    chunk_index: int
    event_type: ReActEventType = ReActEventType.LLM_CHUNK_ARRIVE
```

Emitted for each streaming text chunk. Only fires when `stream=True`.

#### LLMCallEndEvent

```python theme={null}
@dataclass
class LLMCallEndEvent:
    response: Any
    messages: NormalizedMessageList
    tool_calls: list[Any]
    execution_time: float
    content: str = ""
    usage: Optional[Dict[str, int]] = None
    event_type: ReActEventType = ReActEventType.LLM_CALL_END
```

#### LLMCallErrorEvent

```python theme={null}
@dataclass
class LLMCallErrorEvent:
    error: Exception
    event_type: ReActEventType = ReActEventType.LLM_CALL_ERROR
```

***

### Tool Execution Events

#### ToolCallsBatchStartEvent

```python theme={null}
@dataclass
class ToolCallsBatchStartEvent:
    tool_calls: List[Dict[str, Any]]  # All calls in this batch
    event_type: ReActEventType = ReActEventType.TOOL_CALLS_BATCH_START
```

#### ToolCallStartEvent

```python theme={null}
@dataclass
class ToolCallStartEvent:
    tool_name: str
    tool_call_id: str
    arguments: Dict[str, Any]
    event_type: ReActEventType = ReActEventType.TOOL_CALL_START
```

#### ToolCallArgumentsDeltaEvent

```python theme={null}
@dataclass
class ToolCallArgumentsDeltaEvent:
    tool_name: str
    tool_call_id: str
    argname: str
    argcontent_delta: str             # Streaming argument fragment
    event_type: ReActEventType = ReActEventType.TOOL_CALL_ARGUMENTS_DELTA
```

#### ToolCallEndEvent

```python theme={null}
@dataclass
class ToolCallEndEvent:
    tool_name: str
    tool_call_id: str
    arguments: Dict[str, Any]
    result: Any
    execution_time: float
    success: bool
    event_type: ReActEventType = ReActEventType.TOOL_CALL_END
```

#### ToolCallErrorEvent

```python theme={null}
@dataclass
class ToolCallErrorEvent:
    tool_name: str
    tool_call_id: str
    error: str
    event_type: ReActEventType = ReActEventType.TOOL_CALL_ERROR
```

#### ToolCallsBatchEndEvent

```python theme={null}
@dataclass
class ToolCallsBatchEndEvent:
    event_type: ReActEventType = ReActEventType.TOOL_CALLS_BATCH_END
```

***

### Custom Events

#### CustomEvent

```python theme={null}
@dataclass
class CustomEvent:
    event_name: str              # e.g., "kernel_stdout", "selfref_fork_spawn"
    data: Dict[str, Any]         # Event payload
    tool_call_id: Optional[str]  # Which tool emitted this
    event_type: ReActEventType = ReActEventType.CUSTOM
```

Emitted by tools via `ToolEventEmitter`. Common names:

| Event Name              | Source  | Data                                  |
| ----------------------- | ------- | ------------------------------------- |
| `kernel_stdout`         | PyRepl  | `{"text": "..."}`                     |
| `kernel_stderr`         | PyRepl  | `{"text": "..."}`                     |
| `selfref_fork_spawn`    | SelfRef | `{"fork_id": "..."}`                  |
| `selfref_fork_complete` | SelfRef | `{"fork_id": "...", "status": "..."}` |

***

## Event Emitters

### ToolEventEmitter

```python theme={null}
from SimpleLLMFunc.hooks import ToolEventEmitter

emitter = ToolEventEmitter()

# Inside a tool implementation:
await emitter.emit(CustomEvent(
    event_name="progress",
    data={"percent": 50},
    tool_call_id=current_tool_call_id,
))
```

### NoOpEventEmitter

```python theme={null}
from SimpleLLMFunc.hooks import NoOpEventEmitter
```

Silent emitter for testing or when events aren't needed.

***

## Event Observer

```python theme={null}
from SimpleLLMFunc.hooks import with_event_observer

@with_event_observer(my_observer_fn)
@llm_chat(...)
async def agent(...):
    ...
```

The observer function signature:

```python theme={null}
def my_observer(event: ReActEvent, origin: EventOrigin) -> None:
    ...
```

***

## AbortSignal

```python theme={null}
from SimpleLLMFunc.hooks import AbortSignal, ABORT_SIGNAL_PARAM

signal = AbortSignal()
signal.abort("reason")      # Trigger abort
signal.is_aborted           # bool
signal.reason               # str | None
```

`ABORT_SIGNAL_PARAM = "_abort_signal"` — the parameter name for passing abort signals.


# Interfaces
Source: https://simplellmfunc.cn/api/interfaces

API reference for LLM_Interface, OpenAICompatible, OpenAIResponsesCompatible, APIKeyPool

# Interfaces API Reference

## LLM\_Interface (ABC)

```python theme={null}
from SimpleLLMFunc.interface import LLM_Interface
```

Abstract base class for all model interfaces.

### Constructor

```python theme={null}
LLM_Interface(
    api_key_pool: APIKeyPool,
    model_name: str,
    base_url: Optional[str] = None,
    context_window: int = 200_000,
)
```

### Abstract Methods

| Method        | Signature                                                                                             | Description        |
| ------------- | ----------------------------------------------------------------------------------------------------- | ------------------ |
| `chat`        | `async def chat(*, trace_id, stream=False, messages, timeout=None, **kwargs)`                         | Non-streaming call |
| `chat_stream` | `async def chat_stream(*, trace_id, stream=True, messages, timeout=None, **kwargs) -> AsyncGenerator` | Streaming call     |

### Properties

| Property         | Type          | Description                   |
| ---------------- | ------------- | ----------------------------- |
| `model_name`     | `str`         | Model identifier              |
| `base_url`       | `str \| None` | API endpoint                  |
| `context_window` | `int`         | Context window size in tokens |
| `api_key_pool`   | `APIKeyPool`  | Key rotation pool             |

***

## OpenAICompatible

```python theme={null}
from SimpleLLMFunc import OpenAICompatible
```

Adapter for OpenAI Chat Completions API and compatible endpoints.

### Constructor

```python theme={null}
OpenAICompatible(
    api_key_pool: APIKeyPool,
    model_name: str,
    base_url: str,
    max_retries: int = 5,
    retry_delay: float = 1.0,
    rate_limit_capacity: int = 10,
    rate_limit_refill_rate: float = 1.0,
    context_window: int = 200_000,
)
```

### Class Methods

| Method                      | Returns                                  | Description                        |
| --------------------------- | ---------------------------------------- | ---------------------------------- |
| `load_from_json_file(path)` | `Dict[str, Dict[str, OpenAICompatible]]` | Load all models from provider.json |

### Instance Methods

| Method                    | Returns          | Description                |
| ------------------------- | ---------------- | -------------------------- |
| `get_rate_limit_status()` | `Dict[str, Any]` | Current rate limiter state |
| `reset_rate_limit()`      | `None`           | Reset the token bucket     |

### Example

```python theme={null}
from SimpleLLMFunc import OpenAICompatible, APIKeyPool

# From file
models = OpenAICompatible.load_from_json_file("provider.json")
llm = models["openrouter"]["openai/gpt-4o"]

# Direct
llm = OpenAICompatible(
    api_key_pool=APIKeyPool(api_keys=["sk-..."], provider_id="openai"),
    model_name="gpt-4o",
    base_url="https://api.openai.com/v1",
    rate_limit_capacity=20,
    rate_limit_refill_rate=5.0,
)
```

***

## OpenAIResponsesCompatible

```python theme={null}
from SimpleLLMFunc import OpenAIResponsesCompatible
```

Adapter for OpenAI Responses API. Same constructor and loading pattern as `OpenAICompatible`.

### Differences from OpenAICompatible

| Aspect            | OpenAICompatible            | OpenAIResponsesCompatible |
| ----------------- | --------------------------- | ------------------------- |
| System prompt     | `messages[0].role="system"` | `instructions` field      |
| Streaming format  | Chat Completion chunks      | Responses stream events   |
| Reasoning support | N/A                         | `reasoning={...}` kwargs  |
| Wire protocol     | Chat Completions API        | Responses API             |

### Example

```python theme={null}
from SimpleLLMFunc import OpenAIResponsesCompatible, APIKeyPool

llm = OpenAIResponsesCompatible(
    api_key_pool=APIKeyPool(api_keys=["sk-..."], provider_id="openai"),
    model_name="gpt-4o",
    base_url="https://api.openai.com/v1",
)

@llm_chat(llm_interface=llm, reasoning_effort="high")
async def reasoning_agent(message: str, history: list | None = None):
    """Reason carefully about the question."""
    pass
```

***

## APIKeyPool

```python theme={null}
from SimpleLLMFunc import APIKeyPool
```

Round-robin key rotation for load distribution.

### Constructor

```python theme={null}
APIKeyPool(
    api_keys: List[str],
    provider_id: str,
)
```

### Example

```python theme={null}
pool = APIKeyPool(
    api_keys=["sk-key-1", "sk-key-2", "sk-key-3"],
    provider_id="openrouter-gpt4o",
)
# Keys are rotated on each call automatically
```

***

## TokenBucket

```python theme={null}
from SimpleLLMFunc import TokenBucket
```

Token bucket rate limiter.

### Constructor

```python theme={null}
TokenBucket(
    capacity: int,
    refill_rate: float,
)
```

| Parameter     | Description                  |
| ------------- | ---------------------------- |
| `capacity`    | Maximum tokens in the bucket |
| `refill_rate` | Tokens added per second      |

Used internally by `OpenAICompatible`. Exposed for custom implementations.


# Runtime
Source: https://simplellmfunc.cn/api/runtime

API reference for PrimitivePack, PrimitiveRegistry, PrimitiveSpec, and related types

# Runtime API Reference

## PrimitivePack

```python theme={null}
from SimpleLLMFunc.runtime import PrimitivePack
```

A named collection of primitives with a shared backend.

```python theme={null}
@dataclass
class PrimitivePack:
    namespace: str                          # e.g., "selfref", "metrics"
    backend: RuntimePrimitiveBackend        # Stateful backend instance
    primitives: List[PrimitiveHandler]      # Registered primitive functions
    guide: Optional[Dict[str, Any]] = None  # Namespace documentation
```

### Example

```python theme={null}
from SimpleLLMFunc.runtime import PrimitivePack, RuntimePrimitiveBackend, primitive

class MyBackend(RuntimePrimitiveBackend):
    pass

@primitive
async def my_action(ctx: PrimitiveCallContext, arg: str) -> dict:
    """Do something.

    Args:
        arg: Input argument.

    Returns:
        Result dict.

    Best Practices:
        - Use for X.
    """
    return {"result": arg}

pack = PrimitivePack(
    namespace="myns",
    backend=MyBackend(),
    primitives=[my_action],
    guide={"namespace": "myns", "overview": "My custom primitives."},
)
```

***

## @primitive

```python theme={null}
from SimpleLLMFunc.runtime import primitive
```

Decorator that registers a function as a runtime primitive.

### Requirements

* First parameter must be `ctx: PrimitiveCallContext`
* Docstring must include `Best Practices` section
* Must be `async def`

```python theme={null}
@primitive
async def my_primitive(ctx: PrimitiveCallContext, param: str) -> dict:
    """Description.

    Args:
        param: Description.

    Returns:
        Result.

    Best Practices:
        - When to use this primitive.
    """
    return {"value": param}
```

***

## PrimitiveCallContext

```python theme={null}
from SimpleLLMFunc.runtime import PrimitiveCallContext
```

Runtime context passed to every primitive handler.

```python theme={null}
@dataclass
class PrimitiveCallContext:
    backend: RuntimePrimitiveBackend    # The pack's backend
    fork_context: Optional[ForkContext] # Fork info if inside a fork
    event_emitter: ToolEventEmitter     # For emitting custom events
```

***

## ForkContext

```python theme={null}
from SimpleLLMFunc.runtime import ForkContext
```

Information about the current fork execution context.

```python theme={null}
@dataclass
class ForkContext:
    parent_pack_name: str
    backend_name: str
    child_fork_id: str | None = None
    source_memory_key: str | None = None
    metadata: dict[str, Any] | None = None
```

***

## PrimitiveSpec

```python theme={null}
from SimpleLLMFunc.runtime import PrimitiveSpec
```

The contract exposed to the model for discoverability.

```python theme={null}
@dataclass
class PrimitiveSpec:
    name: str                                  # Full qualified name (e.g., "selfref.context.remember")
    description: str                           # What it does
    parameters: List[PrimitiveParameterSpec]   # Parameter descriptions
    returns: str                               # Return type description
    best_practices: List[str]                  # Usage guidance
```

***

## PrimitiveParameterSpec

```python theme={null}
@dataclass
class PrimitiveParameterSpec:
    name: str
    type: str
    description: str
    required: bool = True
    default: Optional[Any] = None
```

***

## PrimitiveRegistry

```python theme={null}
from SimpleLLMFunc.runtime import PrimitiveRegistry
```

Manages installed packs and routes primitive calls.

### Key Methods

| Method                 | Description                           |
| ---------------------- | ------------------------------------- |
| `install_pack(pack)`   | Register a PrimitivePack              |
| `call(name, **kwargs)` | Execute a primitive by qualified name |
| `list_primitives()`    | List all registered primitive names   |
| `get_spec(name)`       | Get PrimitiveSpec for a primitive     |
| `list_specs()`         | Get all specs                         |
| `list_backends()`      | List installed backend packs          |

***

## RuntimePrimitiveBackend

```python theme={null}
from SimpleLLMFunc.runtime import RuntimePrimitiveBackend
```

Base class for primitive pack backends. Subclass to create stateful backends.

```python theme={null}
class RuntimePrimitiveBackend:
    """Base class. Override to add state."""
    pass
```

***

## Built-in Meta-Primitives

Always available in PyRepl without any pack installation:

| Call                               | Returns                           |
| ---------------------------------- | --------------------------------- |
| `runtime.list_primitives()`        | `List[str]` — all primitive names |
| `runtime.list_primitive_specs()`   | `List[PrimitiveSpec]` — all specs |
| `runtime.get_primitive_spec(name)` | `PrimitiveSpec` — one spec        |
| `runtime.list_backends()`          | `List[str]` — installed packs     |

***

## SelfRef Pack Registration

```python theme={null}
from SimpleLLMFunc.runtime import build_self_reference_pack, register_self_reference_primitives

# Build the pack
pack = build_self_reference_pack(selfref_backend)

# Or register directly into a PyRepl
register_self_reference_primitives(repl, selfref_backend)
```

The selfref pack provides primitives under `runtime.selfref.context.*` and `runtime.selfref.fork.*`.


# Compile Pipeline
Source: https://simplellmfunc.cn/context/compile-pipeline

How invocation config, transcript, and runtime patches become the final LLM request

# Compile Pipeline

The compile pipeline transforms invocation configuration, a base transcript, and pending runtime patches into the final messages sent to the LLM provider. It's a two-stage process with a clean boundary between transcript patching and request rendering.

## The Two Stages

```
compile_invocation_turn(spec, transcript, pending_mutations, selfref_snapshot)
│
├─► Stage 1: reduce_turn_context(transcript, mutations, selfref_snapshot)
│       • Apply all pending runtime patches to the transcript
│       • Refresh selfref snapshot if markers detected
│       • Clone the result (no shared references)
│       → Returns: ReducedTurnContext
│
└─► Stage 2: convert_to_llm_request(reduced, prompt_contract)
        • Resolve the system prompt (selfref > explicit > transcript > docstring)
        • Place/replace system message in transcript
        • Render final messages (inject tool specs, must_principles)
        → Returns: CompiledTurnContext
```

## Stage 1: Reduce Turn Context

`reduce_turn_context` takes the current base transcript plus all pending runtime patches and produces a clean, patch-applied transcript.

```python theme={null}
def reduce_turn_context(
    transcript: NormalizedMessageList,
    pending_mutations: List[ContextMutation],
    selfref_snapshot: Optional[DataFromSelfRef] = None,
) -> ReducedTurnContext:
```

What happens:

1. **Apply runtime patches** — `apply_mutations(transcript, pending_mutations)` processes each mutation in order:
   * `AssistantMessageMutation` → appends assistant message
   * `ToolResultMutation` → appends tool result
   * `ContextReplaceMutation` → replaces entire list
   * `ContextSummaryMutation` → replaces with summary, stores experiences
   * `ExperienceRemember/Forget` → accumulated, committed before next non-experience mutation
   * etc.

2. **Refresh selfref snapshot** — If the transcript (after mutations) contains selfref markers (experiences, summaries), re-parse `DataFromSelfRef` from the transcript content. This ensures the snapshot reflects any compaction or experience mutations that just applied.

3. **Clone** — The result is a deep clone. No mutation of shared state.

Output:

```python theme={null}
@dataclass
class ReducedTurnContext:
    transcript: NormalizedMessageList      # Mutation-applied, cloned
    selfref_snapshot: Optional[DataFromSelfRef]  # Refreshed if needed
```

## Stage 2: Convert to LLM Request

`convert_to_llm_request` takes the reduced transcript plus the invocation's prompt contract and produces the final messages for the provider.

```python theme={null}
def convert_to_llm_request(
    reduced: ReducedTurnContext,
    prompt_contract: PromptContract,
) -> CompiledTurnContext:
```

What happens:

1. **Resolve system prompt** — Priority order:
   * If `selfref_snapshot` exists → render base prompt + experiences block
   * Else if `prompt_contract.system_prompt` is set → use it directly
   * Else if transcript has a system message → extract its content
   * Else if `prompt_contract.base_instruction` exists → use docstring fallback

2. **Place system message** — Either replace the existing system message in the transcript, or prepend a new one. If no system prompt was resolved, remove any existing system message.

3. **Render LLM messages** — `render_llm_input_messages()` finalizes the messages:
   * Prepends `<tool_best_practices>` block if tools are mounted
   * Appends `<must_principles>` block if required (tells model to use native tool calls)
   * Returns the final message list ready for the provider

Output:

```python theme={null}
@dataclass
class CompiledTurnContext:
    transcript: NormalizedMessageList       # The transcript after system prompt resolution
    system_prompt: Optional[str]           # The resolved system prompt text
    llm_messages: NormalizedMessageList    # Final messages for the provider
    selfref_snapshot: Optional[DataFromSelfRef]  # Carried forward
```

## Where This Runs in the ReAct Loop

```python theme={null}
# Simplified ReAct loop structure
while has_more_work:
    # 1. Collect mutations from hooks
    hook_mutations = collect_hook_mutations(state)
    
    # 2. Compile context (Stage 1 only — apply mutations)
    compiled_context = compile_context(state, hook_mutations + pending)
    
    # 3. Compile for LLM (Stage 1 + Stage 2 — full pipeline)
    turn = compile_invocation_turn(spec, compiled_context.messages, [], selfref)
    
    # 4. Send to LLM
    llm_result = execute_single_llm_phase(turn.llm_messages, ...)
    
    # 5. Execute tools if needed
    tool_result = schedule_tool_batch(llm_result.tool_calls, ...)
    
    # 6. Collect all new mutations for next iteration
    pending = llm_result.mutations + tool_result.mutations
```

Every iteration goes through the full pipeline. Runtime side effects do not "just append to the live list"; they produce patches that are applied at the boundary. This guarantees that even after 50 tool calls, the transcript state is consistent and auditable.

## The Single Entry Point

All compilation flows through one function:

```python theme={null}
def compile_invocation_turn(
    spec: InvocationSpec,
    transcript: NormalizedMessageList,
    pending_mutations: Optional[List[ContextMutation]] = None,
    selfref_snapshot: Optional[DataFromSelfRef] = None,
) -> CompiledTurnContext:
```

Both `@llm_function` and `@llm_chat` use this same entry point. There is no separate compilation path for different decorator modes.

## Why Two Stages?

The split enables:

* **Stage 1 alone** for internal state management (e.g., `compile_context` in the ReAct loop updates state without rendering for the LLM)
* **Stage 2** adds provider-specific rendering (tool injection, system prompt placement) only when actually calling the LLM
* **Testing** — you can test mutation application separately from prompt rendering
* **SelfRef refresh** — happens between stages, ensuring the snapshot is current before prompt resolution

## Practical Implications

For most users, this is invisible. You write a function, pass history, mount tools, and consume events.

But when you need to debug internals:

* **Debug transcript issues** → Check what runtime patches were produced and in what order
* **Understand system prompt behavior** → Know the priority order in Stage 2
* **Build framework extensions** → Use the compile boundary instead of mutating live messages directly

The important distinction: mutations are internal transcript patches. They are not the source of docstrings, template parameters, tool schemas, or the initial history.

<Card title="Next: SelfRef" icon="brain" href="/context/selfref">
  How durable context (experiences, compaction, forking) works through the SelfReference system.
</Card>


# Runtime Patches
Source: https://simplellmfunc.cn/context/mutations

The internal ContextMutation types used to patch the base transcript at compile boundaries

# Runtime Patches

`ContextMutation` is SimpleLLMFunc's internal typed patch protocol for transcript edits produced by runtime side effects.

Mutations do **not** define the whole context. The final LLM request is compiled from:

1. invocation configuration — docstring prompt, template parameters, tool guidance, output contract, SelfRef snapshot;
2. a base transcript/message list — history, current user input, previous assistant/tool messages;
3. runtime patches — the `ContextMutation` objects described on this page.

Understanding mutations is useful when debugging framework internals, but application code usually interacts with higher-level surfaces such as tools, event streams, `history`, and SelfRef primitives.

## The Mutation Union

```python theme={null}
ContextMutation = Union[
    AssistantMessageMutation,
    ToolResultMutation,
    MultimodalToolResultMutation,
    UserMessageMutation,
    ContextReplaceMutation,
    ContextSummaryMutation,
    ExperienceRememberMutation,
    ExperienceForgetMutation,
    AssistantTruncatedMutation,
    ToolCancelledMutation,
]
```

## Standard Flow Patches

These are produced during normal ReAct loop operation.

### AssistantMessageMutation

Emitted when the LLM produces a response (text, tool calls, or both).

```python theme={null}
@dataclass
class AssistantMessageMutation:
    role: Literal["assistant"] = "assistant"
    content: Optional[str] = None
    tool_calls: List[Dict[str, Any]] = field(default_factory=list)
    reasoning_details: List[Dict[str, Any]] = field(default_factory=list)
```

**When:** After every successful LLM call.\
**Effect:** Appends an assistant message to the transcript.\
**Producer:** `execute_single_llm_phase()` in the ReAct loop.

### ToolResultMutation

Emitted when a tool finishes execution with a text result.

```python theme={null}
@dataclass
class ToolResultMutation:
    tool_call_id: str
    content: str
    role: Literal["tool"] = "tool"
```

**When:** After each tool in a batch completes.\
**Effect:** Appends a tool result message linked to its call by `tool_call_id`.\
**Producer:** `schedule_tool_batch()` → tool execution.

### MultimodalToolResultMutation

Emitted when a tool returns images or mixed content that can't be represented as a simple string.

```python theme={null}
@dataclass
class MultimodalToolResultMutation:
    tool_call_id: str
    tool_name: str
    arguments: str
    user_messages: List[Dict[str, Any]]
```

**When:** Tool returns `ImgPath`, `ImgUrl`, or a tuple containing multimodal content.\
**Effect:** Replaces the tool call with an assistant summary message + user messages containing the multimodal content. This restructuring is required because most providers only support images in user messages.\
**Producer:** `schedule_tool_batch()` with multimodal tool returns.

### UserMessageMutation

Emitted when a new user message needs to be injected mid-loop.

```python theme={null}
@dataclass
class UserMessageMutation:
    message: Dict[str, Any]
```

**When:** Rare in normal flow. Used by input routing (e.g., TUI tool-input) or custom hooks.\
**Effect:** Appends a user message to the transcript.\
**Producer:** Custom hooks, `AgentInputRouter`.

## Context Reset Patches

These replace or restructure the working transcript.

### ContextReplaceMutation

Hard reset — replaces the entire message list.

```python theme={null}
@dataclass
class ContextReplaceMutation:
    messages: List[Dict[str, Any]]
```

**When:** External history override, fork context injection.\
**Effect:** Discards the current transcript entirely, replaces with `messages`.\
**Producer:** SelfRef fork spawn (to set child context), custom hooks.

### ContextSummaryMutation

Context compaction — replaces transcript with a summary and optionally stores new experiences.

```python theme={null}
@dataclass
class ContextSummaryMutation:
    summary_message: Dict[str, Any]
    remember: List[Dict[str, str]] = field(default_factory=list)
```

**When:** `runtime.selfref.context.compact(...)` is called from a tool.\
**Effect:**

1. Preserves the system prompt message
2. Replaces the working transcript with `summary_message`
3. Stores any items in `remember` as durable experiences

**Producer:** SelfRef `compact` primitive.

## Experience Patches

These update durable experiences stored in the system context (via SelfRef).

### ExperienceRememberMutation

Store a new durable experience.

```python theme={null}
@dataclass
class ExperienceRememberMutation:
    text: str
```

**When:** `runtime.selfref.context.remember(...)` is called.\
**Effect:** Adds the text as a new experience entry in the system context's experience section.\
**Producer:** SelfRef `remember` primitive.

### ExperienceForgetMutation

Remove a stored experience by ID.

```python theme={null}
@dataclass
class ExperienceForgetMutation:
    experience_id: str
```

**When:** `runtime.selfref.context.forget(...)` is called.\
**Effect:** Removes the experience with the matching ID from the system context.\
**Producer:** SelfRef `forget` primitive.

## Abort Patches

These handle interruption and cancellation.

### AssistantTruncatedMutation

Records a partially-received assistant response that was interrupted.

```python theme={null}
@dataclass
class AssistantTruncatedMutation:
    partial_content: str
    abort_reason: str = ""
```

**When:** `AbortSignal` fires while the LLM is streaming.\
**Effect:** Appends the partial content as an assistant message with truncation metadata.\
**Producer:** `execute_single_llm_phase()` on abort.

### ToolCancelledMutation

Records a tool call that was aborted before completion.

```python theme={null}
@dataclass
class ToolCancelledMutation:
    tool_call_id: str
    tool_name: str
    abort_reason: str = ""
```

**When:** `AbortSignal` fires while tools are executing.\
**Effect:** Appends a cancelled-tool result message so the transcript remains structurally valid (every tool call has a corresponding result).\
**Producer:** `schedule_tool_batch()` on abort.

## Mutation Application Order

Mutations are applied sequentially in the order they were produced. Within a single ReAct iteration:

1. **Hook mutations** (from SelfRef session `collect_context_mutations`) — applied first
2. **LLM mutations** (AssistantMessage) — from the LLM call
3. **Tool mutations** (ToolResult, MultimodalToolResult, Experience\*) — from tool execution
4. **Abort mutations** (AssistantTruncated, ToolCancelled) — if interruption occurred

Experience mutations (Remember/Forget) are accumulated and committed as a batch before the next non-experience mutation is applied. This ensures experiences are stored before any context replacement that might reference them.

## The Guarantee

Runtime-produced transcript edits go through this patch protocol. This means LLM calls, tool execution, SelfRef primitives, and abort handling do not directly mutate the live transcript.

This does **not** mean every LLM-visible input comes from a mutation. The initial transcript, docstring/system prompt, template parameters, tool guidance, and SelfRef snapshot are separate compile inputs.

<Card title="Next: Compile Pipeline" icon="microchip" href="/context/compile-pipeline">
  How invocation config, transcript, and runtime patches are transformed into the final LLM request.
</Card>


# Context Overview
Source: https://simplellmfunc.cn/context/overview

The mental model: what IS context in SimpleLLMFunc and how data flows through the system

# Context Overview

This page explains how SimpleLLMFunc compiles context — the information the LLM sees at each reasoning step. The final provider-facing message list is built from invocation configuration, a base transcript, and internal runtime patches.

## The Core Data Structures

### ContextState

The runtime representation of a conversation's current state:

```python theme={null}
@dataclass
class ContextState:
    messages: NormalizedMessageList           # The conversation transcript
    data_from_selfref: Optional[DataFromSelfRef]  # Durable self-reference state
    pending_mutations: List[ContextMutation]  # Changes waiting to be applied
```

* `messages` is the base transcript for the current runtime state — assistant messages, tool results, user messages
* `data_from_selfref` carries durable state from the SelfReference backend (experiences, summaries)
* `pending_mutations` accumulates internal runtime patches during a ReAct iteration, applied to the transcript at the next compile boundary

### CompileSource

The input boundary — everything needed to produce an LLM request:

```python theme={null}
@dataclass(frozen=True)
class CompileSource:
    data_from_agent_config: DataFromAgentConfig  # Static: system prompt, tool specs
    data_from_selfref: Optional[DataFromSelfRef] # Durable: experiences, summaries
    input_messages: NormalizedMessageList         # Dynamic: conversation messages
```

### DataFromAgentConfig

Static configuration from the decorator:

```python theme={null}
@dataclass(frozen=True)
class DataFromAgentConfig:
    base_system_prompt: str                    # From docstring (possibly templated)
    template_params: Optional[Dict[str, Any]]  # Runtime template values
    tool_prompt_specs: List[Dict[str, Any]]    # Tool best-practice injections
    include_must_principles: bool              # Whether to add structured-call rules
```

### DataFromSelfRef

Durable state from the SelfReference backend:

```python theme={null}
@dataclass(frozen=True)
class DataFromSelfRef:
    base_system_prompt: str                    # System prompt with selfref markers
    experiences: List[Dict[str, str]]          # Durable remembered experiences
    summary: Optional[Dict[str, Any]]          # Compaction summary metadata
    summary_message: Optional[Dict[str, Any]]  # The summary as a message
    working_messages: NormalizedMessageList     # Post-compaction working transcript
```

## Context Inputs

SimpleLLMFunc does not build LLM context from mutations alone. Each LLM request is compiled from three inputs:

1. **Invocation configuration** — the `InvocationSpec` / prompt contract built from the decorated function, docstring, template parameters, tool guidance, return mode, and SelfRef snapshot.
2. **Base transcript** — the message list created from `history` / `chat_history`, the current user input, and the previous finalized messages.
3. **Runtime patches** — `ContextMutation` objects produced internally by LLM calls, tools, SelfRef primitives, compaction, and abort/cancel handling.

```text theme={null}
provider messages = render(invocation config, patch(base transcript, runtime patches))
```

## Data Flow: From Your Code to the LLM

```
┌─ Your Code ──────────────────────────────────────────────┐
│                                                           │
│  @llm_chat(llm_interface=llm, toolkit=[...])             │
│  async def agent(message: str, history: list):            │
│      """System prompt here."""                            │
│      pass                                                 │
│                                                           │
│  async for output in agent("hello", history):             │
│      ...                                                  │
│                                                           │
└───────────────────────────────────────────────────────────┘
                          │
                          ▼
┌─ Decorator Layer ────────────────────────────────────────┐
│                                                           │
│  Build InvocationSpec:                                    │
│    - Parse docstring → base_system_prompt                 │
│    - Collect tool specs → tool_prompt_specs               │
│    - Resolve template_params                              │
│    - Build initial transcript from history + user message │
│                                                           │
└───────────────────────────────────────────────────────────┘
                          │
                          ▼
┌─ ReAct Loop ─────────────────────────────────────────────┐
│                                                           │
│  while has_more_work:                                     │
│                                                           │
│    1. Collect hook mutations (selfref, etc.)              │
│    2. compile_context(state, pending_mutations)           │
│         → apply_mutations → CompiledContext               │
│    3. compile_invocation_turn(spec, messages, [], selfref)│
│         → reduce + convert → CompiledTurnContext          │
│    4. execute_single_llm_phase(compiled.llm_messages)     │
│         → yields events + produces mutations              │
│    5. If tool calls: schedule_tool_batch(...)             │
│         → executes tools → produces mutations             │
│    6. Merge all mutations → loop back to step 1           │
│                                                           │
└───────────────────────────────────────────────────────────┘
                          │
                          ▼
┌─ Interface Layer ────────────────────────────────────────┐
│                                                           │
│  OpenAICompatible.chat_stream(messages=[...])             │
│  → HTTP request to provider                               │
│  → Stream response back                                   │
│                                                           │
└───────────────────────────────────────────────────────────┘
```

## The Key Invariants

1. **Runtime side effects do not edit the live transcript directly** — LLM calls, tools, SelfRef primitives, and abort handling produce `ContextMutation` patches instead of mutating `ContextState.messages` in place.

2. **Mutations are transcript patches, not the whole context** — docstrings, template params, tool guidance, SelfRef snapshots, and initial history come from invocation configuration and the base transcript.

3. **Mutations accumulate, then apply atomically** — During a ReAct iteration, runtime patches collect in `pending_mutations`. They're applied together at the compile boundary.

4. **Compile produces a snapshot** — The compiled output is a clone. The original state is never mutated in place.

5. **SelfRef state propagates** — `DataFromSelfRef` is carried forward through compilations. If selfref markers are detected in the transcript after mutation application, the snapshot is refreshed.

6. **System prompt resolution has priority order**:
   * SelfRef snapshot (if present) → renders base prompt + experiences
   * PromptContract.system\_prompt (if set explicitly)
   * Latest system message in transcript
   * PromptContract.base\_instruction (docstring fallback)

## Next Steps

<CardGroup>
  <Card title="Runtime Patches" icon="arrows-rotate" href="/context/mutations">
    The internal mutation types — when runtime effects produce transcript patches.
  </Card>

  <Card title="Compile Pipeline" icon="microchip" href="/context/compile-pipeline">
    How invocation config, transcript, and patches become the final LLM request.
  </Card>
</CardGroup>


# SelfRef
Source: https://simplellmfunc.cn/context/selfref

Durable context management: experiences, compaction, and forking through the SelfReference system

# SelfRef

SelfRef is SimpleLLMFunc's system for **durable, self-modifying context** — letting agents remember across turns, compress their history, and delegate work to child agents.

## Two Components, Sharp Separation

### SelfReference (durable backend)

The stateful storage layer. Lives across invocations. Holds:

* **History** per memory key — the full conversation transcript
* **Experiences** — durable remembered facts/lessons
* **Summaries** — compaction checkpoints
* **Fork state** — child agent handles and results

Since 0.8.1, `SelfReference` is implemented as a small public facade over focused internal components:

| Module              | Responsibility                                                                        |
| ------------------- | ------------------------------------------------------------------------------------- |
| `state.py`          | Public facade, lifecycle, compatibility exports                                       |
| `store.py`          | Durable history/source storage                                                        |
| `active_turn.py`    | Active memory key, fork context, runtime toolkit, template params, active ReAct state |
| `mutations.py`      | Pending compaction/context/destructive mutation queues                                |
| `memory_api.py`     | `self_reference.memory[...]` proxy and handle API                                     |
| `context_memory.py` | Context snapshots, experience CRUD, compaction commit, direct memory editing          |
| `agent_binding.py`  | Bound recursive agent callable state                                                  |
| `fork_manager.py`   | Fork/spawn/gather lifecycle and result materialization                                |
| `fork_utils.py`     | Fork helper functions and compatibility constants                                     |

This split does not change the public `SelfReference` API; it makes the durable backend easier to maintain and reason about.

```python theme={null}
from SimpleLLMFunc.builtin import SelfReference

selfref = SelfReference()
selfref.bind_history("agent_main", initial_history)
```

### SelfRefSession (invocation-scoped plugin)

The per-call lifecycle adapter. Created fresh for each `@llm_chat` invocation. Implements ReAct hooks:

* `collect_context_mutations()` — provides selfref-originated internal transcript patches before each compile
* `finalize()` — persists final state back to the SelfReference backend after the turn ends

The session bridges the gap between the stateless ReAct loop and the stateful backend.

## How They Connect

```
@llm_chat(self_reference_key="agent_main")
                    │
                    ▼
        ┌─ SelfRefSession (per invocation) ─────────────┐
        │                                                │
        │  Before each compile:                          │
        │    → collect_context_mutations()               │
        │    → may emit Experience/Summary patches       │
        │                                                │
        │  After turn ends:                              │
        │    → finalize()                                │
        │    → persist updated history to backend        │
        │                                                │
        └────────────────────────────────────────────────┘
                    │
                    ▼
        ┌─ SelfReference (durable backend) ─────────────┐
        │                                                │
        │  Memory["agent_main"]:                         │
        │    - history: [...]                            │
        │    - experiences: [{id, text}, ...]            │
        │    - summary: {...}                            │
        │                                                │
        └────────────────────────────────────────────────┘
```

## DataFromSelfRef: The Snapshot

When SelfRef is active, the compile pipeline receives a `DataFromSelfRef` snapshot:

```python theme={null}
@dataclass(frozen=True)
class DataFromSelfRef:
    base_system_prompt: str          # System prompt (may include experience markers)
    experiences: List[Dict[str, str]]  # [{id: "...", text: "..."}, ...]
    summary: Optional[Dict[str, Any]]  # Compaction metadata
    summary_message: Optional[Dict[str, Any]]  # Summary as a message
    working_messages: NormalizedMessageList  # Post-compaction working transcript
```

This snapshot determines:

* The system prompt (base + rendered experiences)
* What messages the LLM sees (working\_messages after compaction)
* What experiences are active

## The 6 Runtime Primitives

When a `@llm_chat` agent has SelfRef enabled and uses `PyRepl`, these primitives are available inside `execute_code`:

### Context Primitives

| Primitive                                                                                                                                               | What It Does                                                                                     |
| ------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| `runtime.selfref.context.inspect()`                                                                                                                     | Returns read-only snapshot: active key, experiences, summary, messages, has\_pending\_compaction |
| `runtime.selfref.context.remember(text)`                                                                                                                | Stores a durable experience through an internal experience patch                                 |
| `runtime.selfref.context.forget(experience_id)`                                                                                                         | Removes an experience through an internal experience patch                                       |
| `runtime.selfref.context.compact(goal, instruction, discoveries, completed, current_status, likely_next_work, relevant_files_directories, remember=[])` | Queues context compaction → later applies an internal summary patch                              |

### Fork Primitives

| Primitive                                                | What It Does                                                                     |
| -------------------------------------------------------- | -------------------------------------------------------------------------------- |
| `runtime.selfref.fork.spawn(task, instruction, ...)`     | Spawns a child agent with pre-fork context snapshot. Returns `{fork_id, status}` |
| `runtime.selfref.fork.gather_all(include_history=False)` | Waits for all spawned children. Returns `dict[fork_id → ForkResult]`             |

## Experience Lifecycle

Experiences are durable facts stored in the system prompt:

```
System Prompt:
  You are a coding agent...

  <experiences>
    <exp id="exp_001">User prefers dark mode terminal output</exp>
    <exp id="exp_002">Always run tests before committing</exp>
  </experiences>
```

* `remember("...")` → records an experience through the runtime patch boundary
* `forget("exp_001")` → removes it
* Experiences survive compaction — they're stored in the system prompt, not the working transcript
* They're rendered by `render_system_prompt_with_experiences()` during compile Stage 2

## Compaction Lifecycle

When context grows too large, the agent can compact:

```python theme={null}
# Inside execute_code:
payload = runtime.selfref.context.compact(
    goal="Build the authentication module",
    instruction="Continue implementing OAuth flow",
    discoveries=["API requires PKCE", "Token refresh interval is 1h"],
    completed=["Set up project structure", "Added OAuth dependency"],
    current_status="Implementing token exchange endpoint",
    likely_next_work="Add refresh token logic, then write tests",
    relevant_files_directories=["src/auth/", "tests/test_auth.py"],
    remember=["OAuth endpoint requires PKCE challenge"]
)
```

What happens:

1. Compaction is **queued** (not applied immediately)
2. After the current tool batch completes, the runtime applies a summary patch
3. The system prompt is preserved
4. Working transcript is replaced with the summary message
5. Items in `remember` become durable experiences
6. SelfReference backend stores the new state

## Fork Lifecycle

Forks let an agent delegate work to child agents that inherit context:

```python theme={null}
# Spawn (inside execute_code)
handle = runtime.selfref.fork.spawn(
    task="Review the auth module for security issues",
    instruction="Check for: SQL injection, XSS, auth bypass. Return findings as a list.",
)
print(handle["fork_id"])  # "fork_abc123"

# ... do other work ...

# Gather (when you need results)
results = runtime.selfref.fork.gather_all()
for fork_id, result in results.items():
    print(f"{fork_id}: {result['status']} - {result['response']}")
```

Key rules:

* Children inherit the **pre-fork** context snapshot (not the parent's in-flight state)
* Children cannot modify the parent's context
* `gather_all()` blocks until all children complete
* Results contain `status`, `response`, and optionally `history`

## Activation

SelfRef is activated on `@llm_chat` via `self_reference_key`:

```python theme={null}
@llm_chat(
    llm_interface=llm,
    toolkit=[*repl.toolset, *file_tools],
    self_reference_key="agent_main",
)
async def agent(message: str, history: list):
    """Your agent prompt here."""
    pass
```

The framework automatically:

1. Creates/retrieves the `SelfReference` backend for this key
2. Wraps each invocation in a `SelfRefSession`
3. Injects selfref primitives into the PyRepl runtime
4. Handles finalization (persist updated history) after each turn

<CardGroup>
  <Card title="Building Guide: llm_chat" icon="comments" href="/guide/llm-chat">
    How to use @llm\_chat with SelfRef in practice.
  </Card>

  <Card title="Advanced: SelfRef Engineering" icon="gear" href="/advanced/selfref-engineering">
    Advanced patterns: multi-key memory, compaction strategies, fork orchestration.
  </Card>
</CardGroup>


# Contributing
Source: https://simplellmfunc.cn/contributing

How to contribute to SimpleLLMFunc

# Contributing

## Development Setup

```bash theme={null}
git clone https://github.com/NiJingzhe/SimpleLLMFunc.git
cd SimpleLLMFunc
poetry install
```

## Running Tests

```bash theme={null}
# All tests
poetry run pytest

# Specific test file
poetry run pytest tests/test_base/test_react_core_scheduler.py

# With verbose output
poetry run pytest -v
```

## Project Structure

```
SimpleLLMFunc/
├── SimpleLLMFunc/          # Source code
│   ├── llm_decorator/     # L1: Decorator layer
│   ├── base/              # L2+L3: Compile boundary + ReAct runtime
│   ├── runtime/           # Runtime primitives + selfref
│   ├── builtin/           # PyRepl, FileToolset, SelfReference
│   ├── hooks/             # Events, stream, abort
│   ├── interface/         # LLM adapters
│   ├── tool/              # @tool decorator
│   └── utils/             # TUI, stdio
├── tests/                 # Mirror of source structure
├── examples/              # Runnable examples
├── mintlify_docs/         # This documentation
└── spec/                  # Architecture specs
```

## Development Workflow

1. Read relevant source and tests before making changes
2. Write or update tests first (TDD)
3. Make the smallest coherent change
4. Run targeted tests, then broader tests
5. Update docs if user-facing behavior changed

## Key Rules

* All decorated functions must be `async def`
* Runtime side effects must flow through the internal patch boundary (never mutate live messages directly)
* Runtime primitive docstrings must include `Best Practices`
* Tests mirror source structure: `SimpleLLMFunc/base/react_loop.py` → `tests/test_base/test_react_loop.py`

## Code Style

* Python 3.12+
* PEP 8 formatting
* Type annotations on public APIs
* snake\_case functions, PascalCase classes, UPPER\_SNAKE\_CASE constants

## Documentation

* English docs in `mintlify_docs/` (root)
* Chinese translation follows (separate PR)
* Keep examples runnable and imports current

## Pull Requests

1. Fork the repository
2. Create a feature branch from `dev`
3. Make your changes with tests
4. Open a PR against `dev`
5. Describe what changed and why

## Issues

* Use GitHub Issues for bugs and feature requests
* Include: what you expected, what happened, minimal reproduction
* Label appropriately: `bug`, `feature`, `docs`, `refactor`


# Examples
Source: https://simplellmfunc.cn/cookbook/examples

Categorized runnable examples from the SimpleLLMFunc repository

# Examples

All examples are in the [`examples/`](https://github.com/NiJingzhe/SimpleLLMFunc/tree/master/examples) directory. Each is self-contained and runnable.

## Quick Start

| Example                      | What It Shows                                       |
| ---------------------------- | --------------------------------------------------- |
| `basic_usage.py`             | Minimal `@llm_function` call with typed output      |
| `template_params_example.py` | Dynamic docstring templates with `_template_params` |

## Tool Usage

| Example                      | What It Shows                           |
| ---------------------------- | --------------------------------------- |
| `tool_usage_example.py`      | `@tool` decorator, multi-tool agent     |
| `multimodal_tool_example.py` | Tools returning `ImgPath`/`ImgUrl`      |
| `agent_as_tool_example.py`   | Nesting one agent as a tool for another |

## Chat & Streaming

| Example                   | What It Shows                                        |
| ------------------------- | ---------------------------------------------------- |
| `streaming_example.py`    | Consuming `LLMChunkArriveEvent` for real-time output |
| `event_stream_example.py` | Full event stream consumption pattern                |
| `abort_example.py`        | `AbortSignal` for cancellation                       |

## PyRepl & Runtime

| Example                               | What It Shows                                                                        |
| ------------------------------------- | ------------------------------------------------------------------------------------ |
| `pyrepl_example.py`                   | Persistent REPL: variables, streaming, reset                                         |
| `pyrepl_seaborn_multimodal_images.py` | PyRepl image artifact capture and multimodal tool return with multiple seaborn plots |
| `selfref_example.py`                  | `runtime.selfref.context.*` primitives                                               |

## Full Agent

| Example                        | What It Shows                                                                  |
| ------------------------------ | ------------------------------------------------------------------------------ |
| `tui_general_agent_example.py` | Complete coding agent with TUI, PyRepl, FileToolset, SelfRef, fork, compaction |

This is the reference implementation for building a production agent. See [Harness Patterns](/advanced/harness-patterns) for an architectural walkthrough.

## Running Examples

```bash theme={null}
# Clone the repo
git clone https://github.com/NiJingzhe/SimpleLLMFunc.git
cd SimpleLLMFunc

# Install dependencies
poetry install

# Copy and configure provider.json
cp examples/provider_template.json examples/provider.json
# Edit examples/provider.json with your API keys

# Run any example
poetry run python examples/basic_usage.py
poetry run python examples/tui_general_agent_example.py
```

## Building Your Own

Start from one of these patterns:

<CardGroup>
  <Card title="One-shot Function" icon="bolt" href="/quickstart">
    `@llm_function` — typed input/output, no state
  </Card>

  <Card title="Chat Agent" icon="comments" href="/first-agent">
    `@llm_chat` + tools — multi-turn with ReAct loop
  </Card>

  <Card title="Full Agent" icon="robot" href="/advanced/harness-patterns">
    PyRepl + SelfRef + FileToolset + TUI — production-grade
  </Card>
</CardGroup>


# TUI
Source: https://simplellmfunc.cn/cookbook/tui

Build terminal chat interfaces with the @tui decorator

# TUI

The `@tui` decorator wraps a `@llm_chat` agent into a full Textual terminal UI with streaming output, tool visualization, and input handling.

## Basic Usage

```python theme={null}
from SimpleLLMFunc import llm_chat, OpenAICompatible
from SimpleLLMFunc.utils.tui import tui

models = OpenAICompatible.load_from_json_file("provider.json")
llm = models["openrouter"]["openai/gpt-4o"]


@tui
@llm_chat(llm_interface=llm, toolkit=[...], stream=True)
async def agent(message: str, history: list | None = None, _abort_signal=None):
    """A helpful assistant."""
    pass


if __name__ == "__main__":
    agent()
```

Run it and you get a full chat interface in the terminal.

## Features

* **Streaming display** — Model responses appear character-by-character
* **Tool call cards** — Each tool call gets a visual card showing name, arguments, and result
* **Specialized cards** — `execute_code`, `read_file`, `grep`, `sed`, `echo_into` have custom rendering
* **Abort** — `Ctrl+C` cancels the current response
* **Copy** — `Ctrl+Y` copies the full transcript to clipboard
* **Multi-column forks** — When using SelfRef forks, child agents get their own columns

## Slash Commands

| Command                | Action                                    |
| ---------------------- | ----------------------------------------- |
| `/exit`, `/quit`, `/q` | Quit the TUI                              |
| `/copy`, `/copyall`    | Copy transcript to clipboard              |
| `/chat <msg>`          | Send a message (bypasses tool-input mode) |

## Custom Event Hooks

Handle custom events (e.g., PyRepl streaming output) with hooks:

```python theme={null}
from SimpleLLMFunc.hooks.events import CustomEvent
from SimpleLLMFunc.utils.tui import ToolRenderSnapshot


def my_event_hook(event: CustomEvent, snapshot: ToolRenderSnapshot):
    """Process custom events for display."""
    if event.event_name == "kernel_stdout":
        # Handle PyRepl stdout streaming
        pass
    return None


@tui(custom_event_hook=[my_event_hook])
@llm_chat(llm_interface=llm, toolkit=repl.toolset, stream=True)
async def agent(message: str, history: list | None = None, _abort_signal=None):
    """My agent."""
    pass
```

## Decorator Parameters

| Parameter           | Type             | Description                    |
| ------------------- | ---------------- | ------------------------------ |
| `custom_event_hook` | `List[Callable]` | Custom event handler functions |
| `title`             | `str`            | Window title                   |

## ToolRenderSnapshot

Passed to event hooks with context about the current tool card:

```python theme={null}
@dataclass
class ToolRenderSnapshot:
    tool_name: str
    tool_call_id: str
    status: str          # "running", "completed", "error"
    arguments: dict
```

## Keyboard Shortcuts

| Key      | Action                 |
| -------- | ---------------------- |
| `Ctrl+C` | Abort current response |
| `Ctrl+Q` | Quit                   |
| `Ctrl+Y` | Copy transcript        |
| `Enter`  | Send message           |

## When to Use @tui vs Custom UI

Use `@tui` when:

* You want a working chat interface quickly
* The built-in tool cards are sufficient
* You don't need custom widgets or layouts

Build custom when:

* You need confirmation modals for dangerous actions
* You want a status bar with custom metrics
* You need non-chat UI elements (file trees, progress bars)
* You're building a product, not a tool

For custom UIs, consume the `ReactOutput` stream directly and render with Textual or any other framework.


# Build Your First Agent
Source: https://simplellmfunc.cn/first-agent

A tool-using agent with @llm_chat in 10 minutes

# Build Your First Agent

This guide builds a simple agent that can use tools to answer questions. You'll see the full loop: user message → LLM reasoning → tool call → tool result → final response.

## Define a Tool

Tools are async functions decorated with `@tool`:

```python theme={null}
from SimpleLLMFunc import tool


@tool
async def search_docs(query: str) -> str:
    """
    Search the documentation for relevant information.

    Args:
        query: The search query.

    Returns:
        Matching documentation excerpts.

    Best Practices:
        - Use specific, focused queries.
        - Prefer exact terms over vague descriptions.
    """
    # In reality, this would search a vector store or API
    if "install" in query.lower():
        return "Install with: pip install SimpleLLMFunc"
    return f"No results found for: {query}"
```

Key points:

* Tools MUST be `async def`
* The docstring becomes the tool's description for the LLM
* `Best Practices` section is injected into the system prompt as usage guidance
* Return type annotation tells the framework what to expect

## Define the Agent

```python theme={null}
from SimpleLLMFunc import OpenAICompatible, llm_chat

models = OpenAICompatible.load_from_json_file("provider.json")
llm = models["openrouter"]["openai/gpt-4o"]


@llm_chat(llm_interface=llm, toolkit=[search_docs], stream=True)
async def assistant(message: str, history: list | None = None):
    """
    You are a helpful documentation assistant.
    Answer questions using the search tool when you need specific information.
    Be concise and direct.
    """
    pass
```

Key points:

* `@llm_chat` creates a multi-turn agent (vs `@llm_function` for single calls)
* `toolkit=[...]` gives the agent access to tools
* `stream=True` enables streaming responses
* `history` parameter name is special — the framework manages conversation state through it
* The docstring is the agent's system prompt

## Run and Consume Events

`@llm_chat` returns an async generator of `ReactOutput` — either response chunks or lifecycle events:

```python theme={null}
import asyncio
from SimpleLLMFunc.hooks import is_response_yield, is_event_yield


async def main():
    history = []

    async for output in assistant("How do I install SimpleLLMFunc?", history):
        if is_response_yield(output):
            # Final response text
            print(output.response, end="")
            history = output.messages  # Updated history for next turn
        elif is_event_yield(output):
            # Lifecycle events (tool calls, LLM chunks, etc.)
            event = output.event
            # You can log, display, or react to events here

    print()  # newline after streaming


asyncio.run(main())
```

## What Happened Under the Hood

```
1. User message "How do I install SimpleLLMFunc?" → compiled into LLM request
2. LLM decides to call search_docs(query="install SimpleLLMFunc")
3. Framework executes tool, gets result
4. Result is applied to the transcript through an internal runtime patch
5. Context re-compiled, sent back to LLM
6. LLM generates final response using tool result
7. Response streamed back as ReactOutput events
```

This is the **ReAct loop** — reason, act, observe, repeat. The framework handles all of it. You defined a function signature and a tool.

## Add More Tools

```python theme={null}
@tool
async def calculate(expression: str) -> float:
    """
    Evaluate a mathematical expression.

    Args:
        expression: A Python math expression (e.g., "2 ** 10").

    Returns:
        The numeric result.

    Best Practices:
        - Use Python syntax for math operations.
        - Keep expressions simple and readable.
    """
    return float(eval(expression))  # simplified for demo


@llm_chat(llm_interface=llm, toolkit=[search_docs, calculate], stream=True)
async def assistant(message: str, history: list | None = None):
    """You are a helpful assistant that can search docs and do math."""
    pass
```

## What's Next

<CardGroup>
  <Card title="Multi-Turn Chat" icon="comments" href="/first-chat">
    Manage longer conversations with history and streaming.
  </Card>

  <Card title="Why This Design?" icon="lightbulb" href="/philosophy/llm-is-function">
    Understand why LLM calls are functions, not chains.
  </Card>
</CardGroup>


# Multi-Turn Chat
Source: https://simplellmfunc.cn/first-chat

Build a streaming multi-turn chat with history management

# Multi-Turn Chat

This guide shows how to build a stateful conversation with streaming output, history management, and proper turn lifecycle.

## The History Pattern

SimpleLLMFunc agents are **stateless by default** — they don't store conversation history internally. You pass history in, you get updated history back. This makes state management explicit and testable.

```python theme={null}
import asyncio
from SimpleLLMFunc import OpenAICompatible, llm_chat
from SimpleLLMFunc.hooks import is_response_yield, is_event_yield

models = OpenAICompatible.load_from_json_file("provider.json")
llm = models["openrouter"]["openai/gpt-4o"]


@llm_chat(llm_interface=llm, stream=True)
async def chat(message: str, history: list | None = None):
    """
    You are a concise, helpful assistant.
    Remember context from earlier in the conversation.
    """
    pass
```

The parameter named `history` (or `chat_history`) is special — the framework uses it to carry conversation state between turns.

## Streaming Responses

```python theme={null}
async def main():
    history = []

    # Turn 1
    print("User: What is Python?")
    print("Assistant: ", end="")
    async for output in chat("What is Python?", history):
        if is_response_yield(output):
            print(output.response, end="")
            history = output.messages
    print("\n")

    # Turn 2 — the agent remembers turn 1
    print("User: What are its main features?")
    print("Assistant: ", end="")
    async for output in chat("What are its main features?", history):
        if is_response_yield(output):
            print(output.response, end="")
            history = output.messages
    print()


asyncio.run(main())
```

Each `output.messages` contains the full updated conversation. Pass it back for the next turn.

## Event-Aware Consumption

For richer UX, handle individual events:

```python theme={null}
from SimpleLLMFunc.hooks import (
    LLMChunkArriveEvent,
    ToolCallStartEvent,
    ToolCallEndEvent,
    ReactEndEvent,
)


async def run_turn(message: str, history: list) -> list:
    async for output in chat(message, history):
        if is_event_yield(output):
            event = output.event
            if isinstance(event, LLMChunkArriveEvent):
                print(event.accumulated_content, end="", flush=True)
            elif isinstance(event, ToolCallStartEvent):
                print(f"\n[calling {event.tool_name}...]")
            elif isinstance(event, ToolCallEndEvent):
                print(f"[done: {event.tool_name}]")
            elif isinstance(event, ReactEndEvent):
                return event.final_messages
    return history
```

## Non-Streaming Mode

For simpler use cases where you just want the final response:

```python theme={null}
@llm_chat(llm_interface=llm, stream=False)
async def simple_chat(message: str, history: list | None = None):
    """A helpful assistant."""
    pass


async def main():
    history = []
    async for output in simple_chat("Hello!", history):
        if is_response_yield(output):
            print(output.response)
            history = output.messages


asyncio.run(main())
```

## Why History is External

This is a deliberate design choice:

* **Testable** — You can snapshot and replay any conversation state
* **Flexible storage** — Store in memory, Redis, disk, database — your choice
* **Fork-friendly** — Branch a conversation by copying the history list
* **No hidden state** — The agent has no memory you don't control

For agents that need **durable, self-modifying context** (experiences, compaction, forking), see [SelfRef](/context/selfref).

## What's Next

<CardGroup>
  <Card title="Design Philosophy" icon="lightbulb" href="/philosophy/llm-is-function">
    Understand why the framework is designed this way.
  </Card>

  <Card title="Context Model" icon="database" href="/context/overview">
    Learn how context is structured, mutated, and compiled.
  </Card>
</CardGroup>


# Abort
Source: https://simplellmfunc.cn/guide/abort

Cancel running LLM calls and tool executions with AbortSignal

# Abort

`AbortSignal` provides cooperative cancellation for running agent invocations. It cleanly terminates LLM streaming and tool execution.

## Basic Usage

```python theme={null}
from SimpleLLMFunc.hooks import AbortSignal

signal = AbortSignal()


@llm_chat(llm_interface=llm, toolkit=[...], stream=True)
async def agent(message: str, history: list | None = None):
    """An agent that can be cancelled."""
    pass


async def run():
    async for output in agent("do something long", history, _abort_signal=signal):
        if is_response_yield(output):
            history = output.messages


# From another coroutine or callback:
signal.abort("user cancelled")
```

## What Happens on Abort

### During LLM Streaming

1. The streaming connection is terminated
2. Whatever content was received so far is preserved
3. The runtime produces an internal truncation patch with:
   * `partial_content` — what was received before abort
   * `abort_reason` — the reason you provided
4. The ReAct loop terminates via the finalize path

### During Tool Execution

1. Running tools are cancelled (asyncio task cancellation)
2. For each cancelled tool, the runtime produces an internal cancellation patch with:
   * `tool_call_id` — which call was cancelled
   * `tool_name` — the tool that was running
   * `abort_reason` — your reason
3. This ensures the transcript stays structurally valid (every tool call has a result)

### The Guarantee

Abort always produces a valid final state. The transcript is never left with dangling tool calls or incomplete messages. Internal transcript patches ensure structural consistency.

## Abort in @llm\_function

Works the same way:

```python theme={null}
@llm_function(llm_interface=llm, toolkit=[research_tool])
async def research(question: str) -> Report:
    """Research the question."""
    pass

signal = AbortSignal()

async for output in research.stream("...", _abort_signal=signal):
    ...
```

## TUI Integration

The built-in `@tui` decorator handles abort automatically — `Ctrl+C` triggers the signal:

```python theme={null}
from SimpleLLMFunc.utils.tui import tui

@tui
@llm_chat(llm_interface=llm, toolkit=[...], stream=True)
async def agent(message: str, history: list | None = None, _abort_signal=None):
    """My agent."""
    pass
```

## Checking Abort State

```python theme={null}
signal = AbortSignal()

# Check if aborted
if signal.is_aborted:
    print(f"Aborted: {signal.reason}")
```

## Practical Pattern: Timeout

```python theme={null}
import asyncio

signal = AbortSignal()

async def with_timeout():
    task = asyncio.create_task(consume(agent("...", history, _abort_signal=signal)))

    try:
        await asyncio.wait_for(task, timeout=60.0)
    except asyncio.TimeoutError:
        signal.abort("timeout after 60s")
        await task  # Let it clean up
```

→ [API Reference: Events](/api/events)


# Event Stream
Source: https://simplellmfunc.cn/guide/events

ReactOutput, event types, and patterns for consuming the agent event stream

# Event Stream

`@llm_chat` calls return an `AsyncGenerator[ReactOutput, None]`; `@llm_function` exposes the same event stream through `fn.stream(...)`. Each yielded item is either a **response** (final output) or an **event** (lifecycle signal).

## ReactOutput

```python theme={null}
ReactOutput = ResponseYield | EventYield
```

Use type guards to distinguish:

```python theme={null}
from SimpleLLMFunc.hooks import is_response_yield, is_event_yield

async for output in agent("hello", history):
    if is_response_yield(output):
        # Final response — text or typed result
        print(output.response)
        history = output.messages
    elif is_event_yield(output):
        # Lifecycle event
        handle_event(output.event)
```

## ResponseYield

Contains the final agent output:

```python theme={null}
@dataclass
class ResponseYield:
    response: Any              # The result (str, Pydantic model, raw dict)
    messages: NormalizedMessageList  # Updated conversation history
```

## EventYield

Contains a lifecycle event:

```python theme={null}
@dataclass
class EventYield:
    event: ReActEvent          # One of 14 event types
    origin: EventOrigin        # Source identification (main agent or fork)
```

## The 14 Event Types

### Loop Lifecycle

| Event                      | When                 | Key Fields                   |
| -------------------------- | -------------------- | ---------------------------- |
| `ReactStartEvent`          | ReAct loop begins    | —                            |
| `ReactIterationStartEvent` | New iteration starts | `iteration`                  |
| `ReactIterationEndEvent`   | Iteration completes  | `iteration`                  |
| `ReactEndEvent`            | Loop terminates      | `final_messages`, `response` |

### LLM Call

| Event                 | When                        | Key Fields                                 |
| --------------------- | --------------------------- | ------------------------------------------ |
| `LLMCallStartEvent`   | Before calling the provider | `messages` (what the LLM sees)             |
| `LLMChunkArriveEvent` | Each streaming chunk        | `chunk` (text delta)                       |
| `LLMCallEndEvent`     | LLM response complete       | `usage`, `content`, `response`, `messages` |
| `LLMCallErrorEvent`   | LLM call failed             | `error`                                    |

### Tool Execution

| Event                         | When                    | Key Fields                                         |
| ----------------------------- | ----------------------- | -------------------------------------------------- |
| `ToolCallsBatchStartEvent`    | Tool batch begins       | `tool_calls`                                       |
| `ToolCallStartEvent`          | Single tool starts      | `tool_name`, `tool_call_id`, `arguments`           |
| `ToolCallArgumentsDeltaEvent` | Streaming tool args     | `delta`                                            |
| `ToolCallEndEvent`            | Tool completes          | `tool_name`, `result`, `execution_time`, `success` |
| `ToolCallErrorEvent`          | Tool failed             | `tool_name`, `error`                               |
| `ToolCallsBatchEndEvent`      | All tools in batch done | —                                                  |

### Custom Events

| Event         | When                   | Key Fields                           |
| ------------- | ---------------------- | ------------------------------------ |
| `CustomEvent` | Tool emits custom data | `event_name`, `data`, `tool_call_id` |

Custom events are emitted by tools via `ToolEventEmitter` — used for streaming tool output (e.g., PyRepl stdout, shell output).

## Convenience Filters

```python theme={null}
from SimpleLLMFunc.hooks import responses_only, events_only, filter_events

# Only final responses
async for resp in responses_only(agent("hello", history)):
    print(resp.response)

# Only events
async for evt in events_only(agent("hello", history)):
    handle(evt.event)

# Only specific event types
from SimpleLLMFunc.hooks import LLMChunkArriveEvent

async for evt in filter_events(agent("hello", history), LLMChunkArriveEvent):
    print(evt.event.accumulated_content, end="")
```

## EventOrigin (Fork Routing)

When using SelfRef forks, events come from multiple agents. `EventOrigin` identifies the source:

```python theme={null}
@dataclass
class EventOrigin:
    session_id: str
    agent_call_id: str
    event_seq: int
    parent_agent_call_id: str | None = None
    fork_id: str | None = None
    fork_depth: int = 0
    fork_seq: int | None = None
    selfref_instance_id: str | None = None
    source_memory_key: str | None = None
    memory_key: str | None = None
    tool_name: str | None = None
    tool_call_id: str | None = None
```

Use this to route events to different UI panels (e.g., main agent vs. child agents).

## Common Patterns

### Streaming Text to Terminal

```python theme={null}
async for output in agent("hello", history):
    if is_event_yield(output):
        if isinstance(output.event, LLMChunkArriveEvent):
            print(output.event.accumulated_content, end="", flush=True)
    elif is_response_yield(output):
        print()  # newline
        history = output.messages
```

### Progress Tracking

```python theme={null}
async for output in agent("do complex task", history):
    if is_event_yield(output):
        event = output.event
        if isinstance(event, ToolCallStartEvent):
            print(f"  → calling {event.tool_name}...")
        elif isinstance(event, ToolCallEndEvent):
            print(f"  ✓ {event.tool_name} ({event.execution_time}ms)")
        elif isinstance(event, ReactIterationStartEvent):
            print(f"[iteration {event.iteration}]")
```

### Event Observer Decorator

For cross-cutting event handling without modifying consumption logic:

```python theme={null}
from SimpleLLMFunc.hooks import with_event_observer

def log_events(event: ReActEvent, origin: EventOrigin):
    if isinstance(event, LLMCallEndEvent):
        print(f"Tokens: {event.usage}")

@with_event_observer(log_events)
@llm_chat(llm_interface=llm, toolkit=[...])
async def agent(message: str, history: list | None = None):
    """My agent."""
    pass
```

→ [API Reference: Events](/api/events)


# @llm_chat
Source: https://simplellmfunc.cn/guide/llm-chat

Multi-turn agents with history, tools, streaming, and SelfRef

# @llm\_chat

`@llm_chat` creates a multi-turn conversational agent. It manages history, executes a ReAct loop with tools, streams responses, and optionally integrates with SelfRef for durable context.

## Basic Usage

```python theme={null}
from SimpleLLMFunc import OpenAICompatible, llm_chat

models = OpenAICompatible.load_from_json_file("provider.json")
llm = models["openrouter"]["openai/gpt-4o"]


@llm_chat(llm_interface=llm, stream=True)
async def assistant(message: str, history: list | None = None):
    """
    You are a helpful, concise assistant.
    Answer directly without unnecessary preamble.
    """
    pass
```

## History Management

The `history` parameter (or `chat_history`) is special. The framework:

1. Takes your provided history as the conversation transcript
2. Appends the current user message
3. Runs the ReAct loop
4. Returns updated history in `output.messages`

```python theme={null}
history = []

async for output in assistant("What is Python?", history):
    if is_response_yield(output):
        print(output.response)
        history = output.messages  # Save for next turn

# Next turn — agent remembers the conversation
async for output in assistant("What are its main features?", history):
    ...
```

History is **external** — you control storage, persistence, and branching.

## Streaming

With `stream=True`, you receive chunks as events:

```python theme={null}
from SimpleLLMFunc.hooks import is_event_yield, LLMChunkArriveEvent

async for output in assistant("Tell me about Python", history):
    if is_event_yield(output):
        if isinstance(output.event, LLMChunkArriveEvent):
            print(output.event.accumulated_content, end="", flush=True)
    elif is_response_yield(output):
        history = output.messages
```

With `stream=False`, the model response arrives as a single `ResponseYield`.

## Multimodal User Messages

For multimodal chat input, use exactly one canonical user-message object: `UserChatMessage`. This keeps `@llm_chat` as an Agent abstraction over one user turn and avoids multiple competing image-parameter styles.

```python theme={null}
from SimpleLLMFunc import llm_chat
from SimpleLLMFunc.type import ImgPath, ImgUrl, UserChatMessage

@llm_chat(llm_interface=llm, stream=True)
async def vision_agent(message: UserChatMessage, history: list | None = None):
    """Answer questions about user-provided images."""
    pass

async for output in vision_agent(
    UserChatMessage.multimodal(
        "Compare these images and list visible differences.",
        ImgUrl("https://example.com/reference.jpg", detail="high"),
        ImgPath("./candidate.png", detail="high"),
    ),
    history=[],
):
    ...
```

`UserChatMessage.multimodal(...)` accepts text plus any number of `ImgUrl` / `ImgPath` values. It normalizes them to an OpenAI-compatible user message with `text` and `image_url` content parts. Future modalities should extend `UserChatMessage`, not introduce another chat input convention.

## Tools

```python theme={null}
@llm_chat(llm_interface=llm, toolkit=[search, calculate], stream=True, max_tool_calls=10)
async def agent(message: str, history: list | None = None):
    """
    A research assistant. Use search for facts, calculate for math.
    Always cite your sources.
    """
    pass
```

The ReAct loop handles tool calling automatically:

1. LLM decides to call a tool → `ToolCallStartEvent`
2. Framework executes the tool → `ToolCallEndEvent`
3. The runtime records the tool result as an internal transcript patch
4. Context is recompiled → LLM sees the patched transcript → decides next action

`max_tool_calls` limits total tool calls per invocation. Default is framework-defined. `None` means unlimited.

## SelfRef Integration

For agents that need durable memory, context compaction, or sub-agent forking:

```python theme={null}
from SimpleLLMFunc.builtin import PyRepl

repl = PyRepl()


@llm_chat(
    llm_interface=llm,
    toolkit=[*repl.toolset],
    stream=True,
    self_reference_key="agent_main",
)
async def coding_agent(message: str, history: list | None = None):
    """
    A coding agent with persistent memory.
    Use runtime.selfref.context.remember(...) to store durable lessons.
    Use runtime.selfref.context.compact(...) when context grows large.
    """
    pass
```

With `self_reference_key`, the framework:

* Binds a `SelfReference` backend to this key
* Creates a `SelfRefSession` per invocation
* Makes selfref primitives available in PyRepl
* Persists updated history after each turn

See [SelfRef](/context/selfref) for the full context model.

## Template Parameters

Inject runtime values into the system prompt:

```python theme={null}
@llm_chat(llm_interface=llm, toolkit=[...])
async def agent(message: str, history: list | None = None):
    """
    You are an assistant for {project_name}.

    Workspace: {workspace_path}
    Git branch: {git_branch}
    """
    pass

async for output in agent(
    "Fix the bug",
    history,
    _template_params={
        "project_name": "MyApp",
        "workspace_path": "/src",
        "git_branch": "main",
    },
):
    ...
```

## Return Mode

| Mode               | Behavior                                                    |
| ------------------ | ----------------------------------------------------------- |
| `"text"` (default) | `output.response` is the final text string                  |
| `"raw"`            | `output.response` is the raw message dict from the provider |

## System Prompt Construction

For `@llm_chat`, the final system prompt is built from multiple sources:

1. **Docstring** → base system prompt (with template params applied)
2. **Tool best practices** → `<tool_best_practices>` block prepended
3. **Must principles** → `<must_principles>` block appended (use native tool calls)
4. **SelfRef experiences** → rendered into the system prompt if active
5. **Latest system message in history** → overrides docstring if present

Write your docstring as **stable assistant policy** — identity, behavioral rules, and long-lived constraints. Put per-turn context in arguments or template params.

## Concurrent Sessions

Multiple independent conversations can run simultaneously:

```python theme={null}
# Each call gets its own history — no shared state
task1 = assistant("Question A", history_a)
task2 = assistant("Question B", history_b)

# Run concurrently
results = await asyncio.gather(
    consume_stream(task1),
    consume_stream(task2),
)
```

## Parameters Reference

| Parameter            | Type                    | Default           | Description                       |
| -------------------- | ----------------------- | ----------------- | --------------------------------- |
| `llm_interface`      | `LLM_Interface`         | required          | The model to call                 |
| `toolkit`            | `List[Tool]`            | `None`            | Available tools                   |
| `max_tool_calls`     | `int \| None`           | framework default | Tool call limit                   |
| `stream`             | `bool`                  | `False`           | Enable streaming                  |
| `self_reference`     | `SelfReference \| None` | `None`            | Explicit selfref backend          |
| `self_reference_key` | `str \| None`           | `None`            | Auto-create selfref for this key  |
| `**llm_kwargs`       | `Any`                   | —                 | Passed to LLM (temperature, etc.) |

Special call-time parameters:

* `_template_params: Dict[str, Any]` — template values
* `_abort_signal: AbortSignal` — cancellation
* `_too_long_to_file: bool` — truncate long tool results to file

→ [API Reference: Decorators](/api/decorators)


# @llm_function
Source: https://simplellmfunc.cn/guide/llm-function

Stateless typed LLM calls — one function, one result

# @llm\_function

`@llm_function` turns a Python function signature into a complete LLM call. One input, one typed output. No history, no tools (or tools with auto-loop), no state management.

## Basic Usage

```python theme={null}
from pydantic import BaseModel, Field
from SimpleLLMFunc import OpenAICompatible, llm_function

models = OpenAICompatible.load_from_json_file("provider.json")
llm = models["openrouter"]["openai/gpt-4o"]


class Translation(BaseModel):
    text: str = Field(description="The translated text")
    confidence: float = Field(description="0.0-1.0 translation confidence")


@llm_function(llm_interface=llm)
async def translate(text: str, target_language: str) -> Translation:
    """
    Translate the text to the target language.
    Preserve tone and intent. If uncertain about a phrase, lower confidence.
    """
    pass


result = await translate("Hello world", "French")
# Translation(text="Bonjour le monde", confidence=0.95)
```

## How the System Prompt Is Built

Your docstring is the starting point, but the framework augments it:

1. **Your docstring** → task policy, quality bar, constraints
2. **Parameter types** → automatically described for the model
3. **Return type schema** → output format instructions (XML-based structured extraction)
4. **Tool best practices** → prepended if tools are mounted

You write what the model should DO. The framework handles what the model should OUTPUT.

## Return Types

### Pydantic Models (recommended for structured output)

```python theme={null}
@llm_function(llm_interface=llm)
async def analyze(review: str) -> ProductReview:
    """Analyze the product review..."""
    pass
```

The framework generates an XML schema from the Pydantic model and instructs the model to produce matching output. Parsing is automatic.

### Plain str (for free-form text)

```python theme={null}
@llm_function(llm_interface=llm)
async def write_poem(topic: str) -> str:
    """Write a short poem about the topic."""
    pass
```

### Lists and nested types

```python theme={null}
@llm_function(llm_interface=llm)
async def extract_entities(text: str) -> list[Entity]:
    """Extract all named entities."""
    pass
```

### Primitive types

```python theme={null}
@llm_function(llm_interface=llm)
async def score(text: str) -> float:
    """Rate quality from 0.0 to 1.0."""
    pass
```

## Template Parameters

Inject runtime values into the docstring:

```python theme={null}
@llm_function(llm_interface=llm)
async def answer(question: str, context: str) -> str:
    """
    Answer the question using the provided context.

    Domain: {domain}
    Style: {style}
    """
    pass

result = await answer(
    "What is X?",
    "X is a framework...",
    _template_params={"domain": "software engineering", "style": "concise"},
)
```

## Multimodal Image Parameters

`@llm_function` keeps the normal Python-function shape for image input: declare explicit image parameters in the function signature.

```python theme={null}
from SimpleLLMFunc import llm_function
from SimpleLLMFunc.type import ImgPath, ImgUrl, Text

@llm_function(llm_interface=llm)
async def compare_images(
    instruction: Text,
    reference: ImgUrl,
    candidate: ImgPath,
) -> str:
    """Compare the two images according to the instruction."""
    pass

result = await compare_images(
    Text("Explain the visual differences."),
    ImgUrl("https://example.com/reference.jpg", detail="high"),
    ImgPath("./candidate.png", detail="high"),
)
```

Use `ImgUrl` for web URLs or `data:` URLs. Use `ImgPath` for local files; the framework encodes the image as a data URL before sending it to the model. Multiple images should be expressed as explicit parameters or typed lists such as `list[ImgUrl]` / `list[ImgPath]`.

## With Tools

`@llm_function` can use tools via a ReAct loop:

```python theme={null}
@llm_function(llm_interface=llm, toolkit=[search_tool], max_tool_calls=5)
async def research(question: str) -> ResearchReport:
    """Research the question using search. Cite your sources."""
    pass
```

The function still returns a single typed result, but internally the model may call tools multiple times before producing the final answer.

`max_tool_calls` limits how many tool calls the model can make. `None` means unlimited.

## Event Stream Mode

`@llm_function` returns an `LLMFunction` callable instance. Normal calls use `await fn(...)`; use `fn.stream(...)` for `ReactOutput`:

```python theme={null}
from SimpleLLMFunc.hooks import is_response_yield, is_event_yield

async for output in translate.stream("hello", "French"):
    if is_response_yield(output):
        result = output.response  # The typed Translation object
    elif is_event_yield(output):
        # LLM events, tool events, etc.
        pass
```

For simple usage, collect the final response:

```python theme={null}
from SimpleLLMFunc.hooks import responses_only

async for response in responses_only(translate.stream("hello", "French")):
    result = response.response
```

## AbortSignal

Cancel a running call:

```python theme={null}
from SimpleLLMFunc.hooks import AbortSignal

signal = AbortSignal()

async def run():
    async for output in translate.stream("...", "French", _abort_signal=signal):
        ...

# From another task:
signal.abort("timeout")
```

## Parameters Reference

| Parameter                | Type            | Default  | Description                                |
| ------------------------ | --------------- | -------- | ------------------------------------------ |
| `llm_interface`          | `LLM_Interface` | required | The model to call                          |
| `toolkit`                | `List[Tool]`    | `None`   | Tools the model can use                    |
| `max_tool_calls`         | `int \| None`   | `None`   | Max tool calls before forcing final answer |
| `system_prompt_template` | `str \| None`   | `None`   | Override system prompt template            |
| `user_prompt_template`   | `str \| None`   | `None`   | Override user prompt template              |
| `**llm_kwargs`           | `Any`           | —        | Passed to the LLM (temperature, etc.)      |

Special call-time parameters (prefixed with `_`):

* `_template_params: Dict[str, Any]` — template values for docstring formatting
* `_abort_signal: AbortSignal` — cancellation signal

→ [API Reference: Decorators](/api/decorators)


# Tools
Source: https://simplellmfunc.cn/guide/tools

Define tools with @tool, compose toolsets, and handle multimodal returns

# Tools

Tools are async functions that the LLM can call. They're the bridge between model reasoning and real-world actions.

## Basic @tool

```python theme={null}
from SimpleLLMFunc import tool


@tool
async def get_weather(city: str, unit: str = "celsius") -> str:
    """
    Get current weather for a city.

    Args:
        city: City name (e.g., "Tokyo", "New York").
        unit: Temperature unit — "celsius" or "fahrenheit".

    Returns:
        Current weather description.

    Best Practices:
        - Use full city names, not abbreviations.
        - Default to celsius unless the user specifies otherwise.
    """
    # Your implementation here
    return f"Sunny, 22°C in {city}"
```

## Key Rules

1. **Must be async** — `@tool` enforces `async def`. No sync functions.
2. **Docstring is the spec** — The model sees your docstring as the tool description.
3. **Best Practices matter** — The `Best Practices` section is injected into the system prompt as `<tool_best_practices>`. Use it to guide the model on when/how to use the tool.
4. **Type annotations define the schema** — Parameter types become the tool's JSON schema.

## Docstring Structure

```python theme={null}
@tool
async def my_tool(param: str) -> str:
    """
    One-line description of what the tool does.

    Longer explanation if needed. This becomes the tool description
    the model sees.

    Args:
        param: Description of this parameter.

    Returns:
        What the tool returns.

    Best Practices:
        - When to use this tool vs. alternatives.
        - Common pitfalls to avoid.
        - Expected input formats.
    """
```

## Composing Toolsets

Pass multiple tools to an agent:

```python theme={null}
@llm_chat(llm_interface=llm, toolkit=[get_weather, search_docs, calculate])
async def agent(message: str, history: list | None = None):
    """Use available tools to answer questions."""
    pass
```

## Built-in Toolsets

### FileToolset

Workspace-scoped file operations with stale-write protection:

```python theme={null}
from SimpleLLMFunc.builtin import FileToolset

file_tools = FileToolset("/path/to/workspace")

@llm_chat(llm_interface=llm, toolkit=file_tools.toolset)
async def file_agent(message: str, history: list | None = None):
    """A file-aware assistant."""
    pass
```

FileToolset provides 5 tools: `read_file`, `read_image`, `grep`, `sed`, `echo_into`.

### PyRepl

Persistent Python REPL with runtime primitives:

```python theme={null}
from SimpleLLMFunc.builtin import PyRepl

repl = PyRepl(working_directory="/path/to/workspace")

@llm_chat(llm_interface=llm, toolkit=repl.toolset)
async def code_agent(message: str, history: list | None = None):
    """Execute Python code to solve problems."""
    pass
```

PyRepl provides 2 tools: `execute_code`, `reset_repl`.

## Multimodal Returns

Tools can return images and mixed content:

```python theme={null}
from SimpleLLMFunc.type import ImgPath, ImgUrl


@tool
async def generate_chart(data: str) -> ImgPath:
    """Generate a chart from the data."""
    # ... create chart, save to file ...
    return ImgPath(path="/tmp/chart.png")


@tool
async def fetch_screenshot(url: str) -> ImgUrl:
    """Take a screenshot of a webpage."""
    return ImgUrl(url="https://screenshot-api.example.com/capture?url=" + url)
```

Multimodal returns are restructured by the runtime through an internal transcript patch — the image is placed in a user message (as required by most providers).

## Tuple Returns (mixed content)

Return both text and images:

```python theme={null}
@tool
async def analyze_image(image_path: str) -> tuple[str, ImgPath]:
    """Analyze an image and return description + annotated version."""
    description = "A landscape photo..."
    annotated = ImgPath(path="/tmp/annotated.png")
    return description, annotated
```

For multiple images, return a non-empty list or pair text with a list:

```python theme={null}
@tool
async def compare_images(left: str, right: str) -> tuple[str, list[ImgPath | ImgUrl]]:
    """Return a comparison note and both images."""
    return "Compare these two images", [ImgPath(left), ImgPath(right)]
```

## Long Output Handling

For tools that produce very long output:

```python theme={null}
@tool(too_long_to_file=True)
async def read_large_file(path: str) -> str:
    """Read a potentially large file."""
    with open(path) as f:
        return f.read()
```

With `too_long_to_file=True`, if the result exceeds \~20,000 tokens, the framework:

1. Writes the full output to a temporary file
2. Sends a truncated version + file path to the model
3. The model can then read specific sections if needed

## Dynamic Tool Creation

Create tools programmatically:

```python theme={null}
from SimpleLLMFunc import Tool


def make_api_tool(endpoint: str, description: str) -> Tool:
    async def call_api(params: str) -> str:
        # ... call endpoint ...
        return "result"

    return Tool(
        name=f"call_{endpoint}",
        description=description,
        func=call_api,
        best_practices=["Use structured JSON for params"],
    )
```

## System Prompt Injection

Each tool's `Best Practices` section is collected and injected as a `<tool_best_practices>` block at the top of the system prompt. This is automatic — you don't need to reference tools in your docstring.

Additionally, tools can provide dynamic prompt injection via `prompt_injection_builder`:

```python theme={null}
@tool(prompt_injection_builder=lambda ctx: f"Current workspace: {ctx.get('workspace', '/')}")
async def list_files(path: str) -> str:
    """List files in a directory."""
    ...
```

→ [API Reference: Decorators](/api/decorators) | [API Reference: Builtins](/api/builtins)


# SimpleLLMFunc
Source: https://simplellmfunc.cn/index

Build typed LLM agents with Python functions

# SimpleLLMFunc

**LLM calls as typed Python functions. Context compiled from prompts, history, and runtime patches. Prompts as code.**

SimpleLLMFunc is a framework for building LLM-powered agents where every LLM interaction is a normal Python function call — typed, testable, and composable. No chains, no graphs, no YAML. Just functions.

***

## Choose Your Path

<CardGroup>
  <Card title="Start Building" icon="rocket" href="/quickstart">
    Get a working agent in 5 minutes. Install, configure, run.
  </Card>

  <Card title="Understand the Model" icon="brain" href="/philosophy/llm-is-function">
    Learn why SimpleLLMFunc works the way it does. Three short essays on design philosophy.
  </Card>

  <Card title="API Reference" icon="book" href="/api/decorators">
    Jump straight to signatures, types, and parameters.
  </Card>
</CardGroup>

***

## What Makes This Different

### LLM is Function

```python theme={null}
@llm_function(llm_interface=llm)
async def analyze(text: str) -> SentimentReport:
    """Classify sentiment. Return structured report."""
    pass

result = await analyze("Great product, terrible shipping.")
print(result.sentiment)  # "mixed"
```

No SDK boilerplate. No message list construction. The decorator handles context compilation, type parsing, and structured output extraction. You write a function signature and a docstring.

### Context-Centric

Every LLM call is compiled into a provider-facing message list. The compiler combines invocation configuration (docstrings, template values, tool guidance), the base transcript/history, and internal runtime patches. Those patches are represented as **mutations** so LLM calls, tools, SelfRef, and abort handling do not directly edit the live transcript.

### Prompt as Code

Your docstring IS the system prompt. It lives next to the code that uses it, versions with git, and benefits from IDE tooling. No separate prompt files, no template engines, no drift between what you wrote and what the model sees.

***

## The Stack at a Glance

```
┌─────────────────────────────────────────┐
│         Your Agent Code                  │
│   @llm_function / @llm_chat / @tool     │
├─────────────────────────────────────────┤
│         Compile Boundary                 │
│   Config + Transcript + Patches → LLM   │
├─────────────────────────────────────────┤
│         ReAct Runtime                    │
│   Event stream, tool scheduling, abort  │
├─────────────────────────────────────────┤
│         Interface Layer                  │
│   OpenAICompatible / ResponsesCompatible│
└─────────────────────────────────────────┘
```

<Card title="Ready to build?" icon="code" href="/quickstart">
  Start with a 5-minute quickstart that gets you from zero to a working LLM function call.
</Card>


# Configuration
Source: https://simplellmfunc.cn/infra/config

provider.json, environment variables, and logging setup

# Configuration

## provider.json

The primary model configuration file. Defines available providers and models:

```json theme={null}
{
  "openrouter": [
    {
      "model_name": "openai/gpt-4o",
      "api_keys": ["sk-key-1", "sk-key-2"],
      "base_url": "https://openrouter.ai/api/v1",
      "api_params": {"reasoning_effort": "high"},
      "max_retries": 5,
      "retry_delay": 1.0,
      "rate_limit_capacity": 20,
      "rate_limit_refill_rate": 3.0
    },
    {
      "model_name": "anthropic/claude-3.5-sonnet",
      "api_keys": ["sk-key-1"],
      "base_url": "https://openrouter.ai/api/v1",
      "max_retries": 3,
      "retry_delay": 2.0,
      "rate_limit_capacity": 10,
      "rate_limit_refill_rate": 2.0
    }
  ],
  "local": [
    {
      "model_name": "llama-3.1-70b",
      "api_keys": ["not-needed"],
      "base_url": "http://localhost:8000/v1",
      "max_retries": 2,
      "retry_delay": 0.5,
      "rate_limit_capacity": 50,
      "rate_limit_refill_rate": 10.0
    }
  ]
}
```

### Structure

* **Top level** = provider ID → array of model configs
* **Lookup** = `providers[provider_id][model_name]`
* **Multiple keys** = load-balanced across keys in `api_keys`

### Fields

| Field                    | Type      | Required | Description                                                                                                       |
| ------------------------ | --------- | -------- | ----------------------------------------------------------------------------------------------------------------- |
| `model_name`             | string    | yes      | Model identifier for the provider                                                                                 |
| `api_keys`               | string\[] | yes      | One or more API keys (rotated)                                                                                    |
| `base_url`               | string    | yes      | API endpoint base URL                                                                                             |
| `api_params`             | object    | no       | Extra default API kwargs for this model (for example `reasoning_effort`). Call-time kwargs override these values. |
| `max_retries`            | int       | no       | Retry count on transient failures. Default: 5                                                                     |
| `retry_delay`            | float     | no       | Seconds between retries. Default: 1.0                                                                             |
| `rate_limit_capacity`    | int       | no       | Token bucket capacity. Default: 10                                                                                |
| `rate_limit_refill_rate` | float     | no       | Tokens per second refill. Default: 1.0                                                                            |

### Loading

```python theme={null}
from SimpleLLMFunc import OpenAICompatible

models = OpenAICompatible.load_from_json_file("provider.json")
llm = models["openrouter"]["openai/gpt-4o"]
```

Or for Responses API:

```python theme={null}
from SimpleLLMFunc import OpenAIResponsesCompatible

models = OpenAIResponsesCompatible.load_from_json_file("provider.json")
llm = models["openrouter"]["openai/gpt-4o"]
```

### Best Practices

* Keep one `provider.json` per project (version control it minus the keys)
* Put multiple keys in `api_keys` for hot models (automatic rotation)
* Use `api_params` for stable per-model defaults such as `reasoning_effort`; pass call-time kwargs when you need a one-off override
* Tune `rate_limit_capacity` and `rate_limit_refill_rate` per model based on your tier
* Use `max_retries=2` for local models (fast failure), `max_retries=5` for cloud (transient errors)

## Environment Variables (.env)

The framework reads `.env` for logging and observability:

```bash theme={null}
# Logging
LOG_LEVEL=WARNING          # DEBUG, INFO, WARNING, ERROR, CRITICAL
LOG_DIR=logs               # Directory for log files

# Langfuse (optional)
LANGFUSE_PUBLIC_KEY=your_public_key
LANGFUSE_SECRET_KEY=your_secret_key
LANGFUSE_BASE_URL=https://cloud.langfuse.com
LANGFUSE_EXPORT_ALL_SPANS=true
LANGFUSE_ENABLED=true
```

### Precedence

Runtime environment → `.env` file → framework defaults

### Recommended Defaults

* `LOG_LEVEL=WARNING` — reduces framework noise during normal use
* `LOG_DIR=logs` — keeps logs out of your project root
* Langfuse disabled by default — enable only when you need trace collection

## Direct Construction (No Files)

For scripts and one-offs, skip provider.json entirely:

```python theme={null}
from SimpleLLMFunc import APIKeyPool, OpenAICompatible

llm = OpenAICompatible(
    api_key_pool=APIKeyPool(
        api_keys=["sk-your-key"],
        provider_id="openai",
    ),
    model_name="gpt-4o",
    base_url="https://api.openai.com/v1",
)
```

This is ideal for shell scripts, demos, and quick experiments.


# LLM Interface
Source: https://simplellmfunc.cn/infra/llm-interface

OpenAICompatible, OpenAIResponsesCompatible, key pools, and rate limiting

# LLM Interface

The interface layer handles model communication, key rotation, and rate limiting.

## OpenAICompatible

Works with any provider that implements the OpenAI Chat Completions API:

```python theme={null}
from SimpleLLMFunc import OpenAICompatible

# From provider.json
models = OpenAICompatible.load_from_json_file("provider.json")
llm = models["openrouter"]["openai/gpt-4o"]

# Direct construction
from SimpleLLMFunc import APIKeyPool

llm = OpenAICompatible(
    api_key_pool=APIKeyPool(api_keys=["sk-key"], provider_id="openai"),
    model_name="gpt-4o",
    base_url="https://api.openai.com/v1",
    max_retries=3,
    retry_delay=1.0,
    rate_limit_capacity=20,
    rate_limit_refill_rate=3.0,
    context_window=128_000,
)
```

Compatible with: OpenAI, OpenRouter, Together, Groq, local vLLM, Ollama, etc.

## OpenAIResponsesCompatible

For providers implementing OpenAI's Responses API:

```python theme={null}
from SimpleLLMFunc import OpenAIResponsesCompatible

llm = OpenAIResponsesCompatible(
    api_key_pool=APIKeyPool(api_keys=["sk-key"], provider_id="openai"),
    model_name="gpt-4o",
    base_url="https://api.openai.com/v1",
)
```

Differences from OpenAICompatible:

* Maps system prompts to `instructions` field
* Handles Responses-specific streaming events
* Supports `reasoning={...}` kwargs for reasoning effort
* Different wire format for tool calls

From your decorator code, both adapters look the same. The wire-format differences are handled internally.

## APIKeyPool

Manages multiple keys with round-robin rotation:

```python theme={null}
from SimpleLLMFunc import APIKeyPool

pool = APIKeyPool(
    api_keys=["sk-key-1", "sk-key-2", "sk-key-3"],
    provider_id="openrouter-gpt4",
)
```

When a key hits rate limits, the pool rotates to the next. Put your highest-rate keys first.

## Rate Limiting

Built-in token bucket rate limiter:

```python theme={null}
# Configured via constructor
llm = OpenAICompatible(
    ...,
    rate_limit_capacity=20,       # Max concurrent "tokens" in the bucket
    rate_limit_refill_rate=3.0,   # Tokens added per second
)

# Check status
status = llm.get_rate_limit_status()
# {"available": 15, "capacity": 20, "refill_rate": 3.0}

# Reset after rate limit errors
llm.reset_rate_limit()
```

The rate limiter is per-instance. Multiple `OpenAICompatible` instances for the same model can have different rate limits.

## Passing LLM kwargs

Extra parameters are forwarded to the provider:

```python theme={null}
@llm_chat(
    llm_interface=llm,
    temperature=0.7,
    max_tokens=4096,
    top_p=0.9,
)
async def agent(message: str, history: list | None = None):
    """My agent."""
    pass
```

For OpenAIResponsesCompatible, you can pass reasoning effort:

```python theme={null}
@llm_chat(
    llm_interface=llm,
    reasoning_effort="high",
)
async def reasoning_agent(message: str, history: list | None = None):
    """An agent that reasons deeply."""
    pass
```

## Context Window

Set `context_window` to enable framework features that depend on knowing the model's capacity:

```python theme={null}
llm = OpenAICompatible(
    ...,
    context_window=128_000,  # GPT-4o's context window
)
```

Used by: auto-compaction threshold calculations, token usage tracking.

Default: 200,000 tokens.

→ [API Reference: Interfaces](/api/interfaces)


# Observability
Source: https://simplellmfunc.cn/infra/observability

Langfuse integration for tracing LLM calls and tool executions

# Observability

SimpleLLMFunc integrates with [Langfuse](https://langfuse.com) for tracing LLM generations, tool calls, and agent sessions.

## Setup

### 1. Install (included with SimpleLLMFunc)

Langfuse client is bundled. No extra install needed.

### 2. Configure Environment

```bash theme={null}
# .env
LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_BASE_URL=https://cloud.langfuse.com
LANGFUSE_ENABLED=true
LANGFUSE_EXPORT_ALL_SPANS=true
```

### 3. Use Normally

Once configured, tracing is automatic. Every decorated `@llm_function` and `@llm_chat` invocation creates:

* A **trace** for the full invocation
* **Generations** for each LLM call
* **Spans** for tool executions

## What Gets Traced

| Component            | Langfuse Entity | Contains                          |
| -------------------- | --------------- | --------------------------------- |
| `@llm_function` call | Trace           | Full invocation lifecycle         |
| `@llm_chat` call     | Trace           | Full ReAct loop                   |
| Each LLM API call    | Generation      | Messages in/out, tokens, latency  |
| Each tool execution  | Span            | Tool name, args, result, duration |
| ReAct iterations     | Span            | Iteration boundaries              |

## Trace IDs

Each invocation gets a unique `trace_id` propagated through all nested operations:

```python theme={null}
from SimpleLLMFunc.logger import get_current_trace_id

# Inside a tool or custom code
trace_id = get_current_trace_id()
```

## Custom Trace Names

The trace name defaults to the decorated function name. Override with:

```python theme={null}
@llm_function(llm_interface=llm)
async def my_func(text: str) -> str:
    """Process text."""
    pass

# The trace will be named "my_func" in Langfuse
```

## Flushing

Observations are batched and sent asynchronously. Force flush at shutdown:

```python theme={null}
from SimpleLLMFunc.observability import flush_all_observations

await flush_all_observations()
```

## Disabling

Set `LANGFUSE_ENABLED=false` or simply don't configure the keys. The framework gracefully degrades — no errors, just no traces.

## Log Levels

Control framework logging independently:

```bash theme={null}
LOG_LEVEL=WARNING   # Recommended: suppress noisy internal logs
LOG_DIR=logs        # File output location
```

Available levels: `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`.

Use `DEBUG` only when investigating framework behavior. `WARNING` is the recommended default for production.


# Context-Centric
Source: https://simplellmfunc.cn/philosophy/context-centric

Why LLM context is compiled from invocation config, transcript, and runtime patches

# Context-Centric

SimpleLLMFunc does not treat provider messages as something every component edits directly. Each LLM call is compiled into a provider-facing message list from three inputs:

1. **Invocation configuration** — docstring prompt, template parameters, tool guidance, output contract, and SelfRef snapshot.
2. **Base transcript** — the history/current messages for this invocation.
3. **Runtime transcript patches** — internal typed edits produced by LLM calls, tool execution, SelfRef primitives, abort handling, and compaction.

This is the precise meaning of the context model:

```text theme={null}
provider messages = render(invocation config, patch(base transcript, runtime patches))
```

## The Problem

In many agent systems, the same mutable message list is edited from everywhere:

* tools append results directly;
* orchestration pushes assistant messages into the list;
* memory systems inject or rewrite context in arbitrary places;
* summarization overwrites history in place;
* abort/cancel handling leaves partially valid messages behind.

The result is unpredictable context. It becomes difficult to answer: "what exactly will the model see next?"

## The Runtime Patch Model

SimpleLLMFunc uses `ContextMutation` as an **internal transcript patch protocol**.

A mutation is not the whole context and not the user's main API. It is a typed incremental edit to the base transcript, produced by runtime side effects and applied at a compile boundary.

```text theme={null}
runtime side effect → ContextMutation → [accumulate] → apply to transcript → render LLM request
```

Examples:

| Runtime fact                           | Internal patch                 |
| -------------------------------------- | ------------------------------ |
| LLM produced a final assistant message | `AssistantMessageMutation`     |
| Tool returned text                     | `ToolResultMutation`           |
| Tool returned multimodal content       | `MultimodalToolResultMutation` |
| SelfRef compacted working context      | `ContextSummaryMutation`       |
| User aborted streaming LLM output      | `AssistantTruncatedMutation`   |
| Tool execution was cancelled           | `ToolCancelledMutation`        |

The rule is:

> Runtime side effects do not directly mutate the live transcript. They produce typed patches, and the compile boundary applies those patches in order.

This keeps tool scheduling, LLM output handling, SelfRef, compaction, and abort behavior from racing over the same mutable list.

## What Mutations Are Not

Mutations are deliberately narrower than "context".

They are **not**:

* the source of the docstring/system prompt;
* the source of template parameters;
* the source of tool schemas or tool best-practice prompts;
* the source of the initial `history` passed to `@llm_chat`;
* the public event stream consumed by UIs;
* the primary user-facing way to customize an agent.

Those belong to invocation configuration, transcript input, tools, SelfRef primitives, or event consumers.

## Why This Still Matters

### Safe concurrency

Parallel tools can finish in any order. Instead of each tool directly appending to a shared message list, tools produce result patches. The runtime collects them and applies them at the next compile boundary.

### Controlled compaction

Context compaction is a transcript patch. It can replace stale working messages with a summary while preserving system prompt and durable experiences.

### Valid abort/cancel state

Abort handling must leave a structurally valid transcript. `AssistantTruncatedMutation` and `ToolCancelledMutation` centralize this repair logic instead of scattering it across the loop.

### Testable internals

Patch application can be tested separately from prompt rendering and provider transport.

## The Compile Boundary

The compile boundary has two jobs:

1. **Patch the base transcript** with pending runtime mutations.
2. **Render the provider request** from the patched transcript plus invocation configuration.

```text theme={null}
┌─────────────────────────────────────────────┐
│  Inputs                                      │
│                                             │
│  Invocation config                           │
│    - docstring / prompt contract             │
│    - template params                         │
│    - tool guidance                           │
│    - SelfRef snapshot                        │
│                                             │
│  Base transcript / history                   │
│  Runtime patches (ContextMutation[])         │
├──────────── COMPILE BOUNDARY ────────────────┤
│                                             │
│  apply_mutations(transcript, patches)        │
│  → patched transcript                        │
│                                             │
│  convert_to_llm_request(...)                 │
│  → provider-facing messages[]                │
└─────────────────────────────────────────────┘
```

See [Compile Pipeline](/context/compile-pipeline) for the technical walkthrough.

## Practical Implication

For most users, mutations are invisible. You write functions, provide history, mount tools, consume events, and optionally use SelfRef primitives.

If you are debugging framework internals, mutations explain how runtime effects become transcript edits. If you are building application code, your main surfaces are still:

* `@llm_function` / `@llm_chat` signatures and docstrings;
* `history` / `chat_history`;
* tools and event streams;
* SelfRef primitives such as `remember(...)` and `compact(...)`.

<Card title="Deep dive: Runtime Patches" icon="list" href="/context/mutations">
  See the internal mutation types and when the runtime produces them.
</Card>


# LLM is Function
Source: https://simplellmfunc.cn/philosophy/llm-is-function

Why treating LLM calls as typed Python functions changes everything

# LLM is Function

The central insight of SimpleLLMFunc: an LLM call should be indistinguishable from a Python function call.

## The Problem with Existing Approaches

Most LLM frameworks introduce new abstractions between you and the model:

* **Chain frameworks** make you think in terms of linked steps, each passing data to the next through a framework-defined protocol
* **Graph frameworks** make you define nodes and edges, routing data through a visual DAG
* **Agent frameworks** hide the LLM behind an "agent" abstraction that manages its own state

All of these put a layer of framework-specific concepts between your intent and the model. You learn the framework's vocabulary instead of expressing your task directly.

## The Function Model

In SimpleLLMFunc, an LLM call IS a function call:

```python theme={null}
@llm_function(llm_interface=llm)
async def extract_entities(text: str) -> list[Entity]:
    """Extract named entities from the text. Include person, org, and location types."""
    pass
```

This is a complete, runnable program. The function:

* Has a **name** — `extract_entities`
* Has **typed parameters** — `text: str`
* Has a **typed return value** — `list[Entity]`
* Has a **docstring** that describes the behavior — this IS the prompt
* Is **awaitable** — `await extract_entities("...")`
* Is **composable** — call it from other functions, pass it around, test it

There is no "chain", no "node", no "agent object". There's a function.

## What You Gain

### Type Safety at the Boundary

The return type annotation is a contract. The framework ensures the LLM output is parsed into your declared type — or raises a clear error. No manual JSON parsing, no "sometimes the model returns a string instead of an object".

```python theme={null}
class Analysis(BaseModel):
    sentiment: Literal["positive", "negative", "neutral"]
    confidence: float = Field(ge=0.0, le=1.0)
    reasoning: str

result: Analysis = await analyze(text)  # Type-checked, guaranteed structure
```

### Composability

Functions compose naturally. Build complex pipelines with plain Python:

```python theme={null}
async def full_pipeline(document: str) -> Report:
    entities = await extract_entities(document)
    summary = await summarize(document)
    sentiment = await analyze_sentiment(document)
    return Report(entities=entities, summary=summary, sentiment=sentiment)
```

No framework DSL needed. No chain definitions. Just functions calling functions.

### Testability

Mock it like any other function:

```python theme={null}
async def test_pipeline():
    with mock.patch("my_module.extract_entities") as mock_extract:
        mock_extract.return_value = [Entity(name="Alice", type="person")]
        result = await full_pipeline("Alice went to Paris.")
        assert result.entities[0].name == "Alice"
```

### IDE Support

Your IDE already knows how to work with async functions, type annotations, and docstrings. Autocomplete, jump-to-definition, inline docs — all free.

## The Extension: LLM as Agent

`@llm_chat` extends the function model to multi-turn agents. The agent is still just a function — it takes input, returns output. The difference is that it can use tools and maintain conversation state:

```python theme={null}
@llm_chat(llm_interface=llm, toolkit=[search, calculate], stream=True)
async def research_agent(question: str, history: list | None = None):
    """Research the question using available tools. Cite your sources."""
    pass
```

Same function interface. Same composability. But now the function can reason across multiple steps internally (the ReAct loop), use tools, and stream results.

## The Key Insight

The framework's job is to make the gap between "I want to call an LLM" and "I called a Python function" as close to zero as possible. Everything else — context management, tool execution, type parsing — is implementation detail that you can ignore until you need to customize it.

When you DO need to customize, the [Context Model](/context/overview) gives you full control. But the default is: just write a function.


# Prompt as Code
Source: https://simplellmfunc.cn/philosophy/prompt-as-code

Why docstrings are system prompts and prompts should never be separated from code

# Prompt as Code

The third design principle: **your docstring IS the system prompt**. Prompts live in code, version with code, and are maintained like code.

## The Problem with Separated Prompts

Common patterns in LLM applications:

* Prompts in YAML/JSON files, loaded at runtime
* Prompt templates in a database, edited through a UI
* Prompts as string constants defined far from where they're used
* Prompts managed by a "prompt engineering" tool separate from the codebase

All of these create the same problem: **prompt drift**. The prompt and the code that uses it evolve independently. The code expects certain output formats, but the prompt was edited without updating the parser. The prompt references parameters that no longer exist. Nobody knows which version of the prompt is running in production.

## Docstring as System Prompt

In SimpleLLMFunc, the function's docstring IS the prompt:

```python theme={null}
@llm_function(llm_interface=llm)
async def classify(text: str, categories: list[str]) -> Classification:
    """
    Classify the text into one of the provided categories.

    Rules:
    - Choose exactly one category.
    - If none fit well, choose the closest match and set confidence below 0.5.
    - Explain your reasoning in one sentence.
    """
    pass
```

The docstring is:

* **Co-located** — Right next to the function signature it describes
* **Versioned** — Changes tracked in git like any other code
* **Type-aware** — The framework adds parameter types and return schema automatically
* **Refactorable** — Rename a parameter, IDE updates the docstring reference
* **Reviewable** — PRs show prompt changes in the same diff as code changes

## What the Framework Adds

Your docstring is the starting point, not the entire system prompt. The framework augments it with:

1. **Parameter type descriptions** — Generated from your type annotations
2. **Return type schema** — Structured output format instructions
3. **Tool best practices** — Injected from tool docstrings when tools are mounted
4. **Output constraints** — XML/JSON formatting rules based on return type

You write the **task policy** — what the model should do, what quality bar to meet, what constraints to respect. The framework handles the structural plumbing.

## Template Parameters for Dynamic Prompts

When you need runtime-variable content in the prompt:

```python theme={null}
@llm_chat(llm_interface=llm, toolkit=[...])
async def agent(message: str, history: list | None = None):
    """
    You are an assistant for the {project_name} project.

    Project context:
    {project_description}

    Available commands: {command_list}
    """
    pass

# At call time:
async for output in agent(
    "Help me with this bug",
    history,
    _template_params={
        "project_name": "MyApp",
        "project_description": "A web API for...",
        "command_list": "build, test, deploy",
    },
):
    ...
```

The template lives in the docstring. The values come from your code. Both are traceable, testable, and version-controlled.

## For llm\_chat: Latest System Message Wins

In multi-turn agents, you can update the system prompt mid-conversation by placing a `system` message in history. The framework uses the **latest** system message — or falls back to the docstring if none exists in history.

This enables:

* Dynamic system prompts that evolve based on conversation state
* SelfRef context injection (experiences, summaries) without docstring mutation
* Per-turn context customization while keeping the base prompt in code

## The Implication for Prompt Engineering

Prompt engineering becomes software engineering:

* **Test prompts** — Assert on function outputs with known inputs
* **Review prompts** — Prompt changes go through code review
* **Version prompts** — Git blame tells you when and why a prompt changed
* **Profile prompts** — Measure latency and token usage per function
* **Compose prompts** — Build complex behaviors from smaller, tested prompt-functions

There's no separate "prompt management" concern. The prompt is the function. The function is the prompt.


# Quickstart
Source: https://simplellmfunc.cn/quickstart

From zero to a working LLM function call in 5 minutes

# Quickstart

## Install

```bash theme={null}
pip install SimpleLLMFunc
```

Or with Poetry:

```bash theme={null}
poetry add SimpleLLMFunc
```

## Configure a Model

Create `provider.json` in your project root:

```json theme={null}
{
  "openrouter": [
    {
      "model_name": "openai/gpt-4o",
      "api_keys": ["sk-your-key"],
      "base_url": "https://openrouter.ai/api/v1",
      "max_retries": 3,
      "retry_delay": 1.0,
      "rate_limit_capacity": 20,
      "rate_limit_refill_rate": 3.0
    }
  ]
}
```

This works with any OpenAI-compatible endpoint — OpenRouter, Together, local vLLM, etc.

## Your First LLM Function

```python theme={null}
import asyncio
from pydantic import BaseModel, Field
from SimpleLLMFunc import OpenAICompatible, llm_function


class Summary(BaseModel):
    headline: str = Field(description="One-sentence summary")
    key_points: list[str] = Field(description="3-5 bullet points")
    tone: str = Field(description="formal, casual, or technical")


models = OpenAICompatible.load_from_json_file("provider.json")
llm = models["openrouter"]["openai/gpt-4o"]


@llm_function(llm_interface=llm)
async def summarize(text: str) -> Summary:
    """
    Summarize the input text concisely.
    Focus on the most actionable information.
    """
    pass


async def main():
    result = await summarize("SimpleLLMFunc treats every LLM call as a typed Python function...")
    print(result.headline)
    print(result.key_points)


asyncio.run(main())
```

That's it. The decorator:

1. Builds a system prompt from the docstring + parameter types + return type schema
2. Sends the function arguments as the user message
3. Parses the LLM response into your Pydantic model

No message lists, no JSON mode flags, no output parsers.

## Direct Construction (No provider.json)

For quick scripts, construct the model directly:

```python theme={null}
from SimpleLLMFunc import APIKeyPool, OpenAICompatible

llm = OpenAICompatible(
    api_key_pool=APIKeyPool(
        api_keys=["sk-your-key"],
        provider_id="openai",
    ),
    model_name="gpt-4o",
    base_url="https://api.openai.com/v1",
)
```

## What's Next

<CardGroup>
  <Card title="Build a Tool-Using Agent" icon="wrench" href="/first-agent">
    Add tools and see the ReAct loop in action.
  </Card>

  <Card title="Build a Multi-Turn Chat" icon="comments" href="/first-chat">
    History management, streaming, and stateful conversations.
  </Card>
</CardGroup>