Advanced LLM

Defaults hide the wiring; advanced mode labels every cable before anyone reaches for scissors.

1. Motivation

Most applications can stay with hb.LLM(preset="chat"). The features here matter when transport, reproducibility, SDK interop, image handling, or concurrency must be explicit. The same resolution order still applies: preset, model, provider, then gateway.

2. Choose a Gateway

The default openai gateway uses the OpenAI Python SDK against an OpenAI-compatible endpoint.

Gateway	Use it when
`openai`	An OpenAI-compatible endpoint is the simplest transport
`anthropic`	Native Claude Messages payloads are required
`portkey`	Portkey routing, policy, or observability owns transport
`bifrost`	A Bifrost-compatible gateway owns provider routing
`litellm`	LiteLLM already standardizes provider access
`mock`	Tests need deterministic offline behavior

import heavenbase as hb

llm = hb.LLM(model="ds-flash", provider="deepseek", gateway="portkey")

Gateway limits fail explicitly. Native Anthropic has no embeddings or image generation, Portkey with OpenRouter embeddings remains blocked, and Bifrost image support depends on its upstream implementation. If an optional non-default gateway cannot be imported, HeavenBase falls back to openai.

3. Register a Runtime Shape

Add a provider in configuration when an existing gateway already speaks its API. Register LLMGateway only for a genuinely different runtime argument shape.

def materialize(spec, gateway, override):
    return {
        "gateway": gateway,
        "model": spec.model_id,
        "args": {**spec.args, **override},
    }


hb.LLMGateway("company-proxy", materialize).register()

Declare the same identifier under heavenbase.llm.gateways so LLMEngine.resolve(...) can validate it. LLMEngine resolves policy, LLMSpec carries the result, and LLMGateway materializes transport.

4. Control Response Caching

HeavenBase stores normalized chat, embedding, and image responses in a dedicated llm-cache workspace. Caching is enabled by default and uses a deterministic policy.

llm = hb.LLM(preset="chat", cache=False)
text = llm.chat("Reply with exactly: hb-ok")

Per-call overrides accept cache=False, cache=True, or a cache mapping. Executable tool loops skip text cache automatically. Deterministic text and image writes require stable generation arguments. Stochastic requests without a fixed seed bypass cache reads and writes.

hb cfg set heavenbase.llm.cache.namespaces.text.ttl_seconds 86400
hb cfg set heavenbase.llm.cache.namespaces.embedding.enabled true

5. Reuse and Export Clients

LLMSpec.hash_key() identifies the complete resolved request defaults. client_key() includes only gateway client construction fields, allowing compatible LLM instances to reuse in-memory SDK clients.

llm = hb.LLM(model="ds-flash", provider="deepseek")

spec_key = llm.spec.hash_key()
client_key = llm.spec.client_key()

OpenAI-compatible gateways can export raw SDK clients:

client = llm.to_client()
aclient = llm.to_aclient()

response = client.chat.completions.create(
    messages=[{"role": "user", "content": "Hello"}],
    **llm.to_args(),
)

to_client(), to_aclient(), and to_args() support openai, portkey, and bifrost. Native Anthropic uses to_anthropic_client() and to_anthropic_aclient().

6. Generate and Normalize Images

image = hb.LLM(preset="imagen").imagen("A clean product render of a ceramic mug")
image.save("mug.png")

Use imagen-local when Ollama serves z-image-turbo. Reference images accept the same normalized inputs as multimodal chat:

reference = hb.LLMImage.from_any("./style-reference.png")
image = hb.LLM(preset="imagen").imagen("Apply this style to an HB mark", images=reference)

LLMImage normalizes paths, bytes, base64, URLs, provider items, and generated results.

raw = image.to_bytes()
data_url = image.to_data_url()
pil_image = image.to_pil()

URL-backed values fetch lazily. The timeout is configured by heavenbase.llm.image_url_timeout.

7. Repair Tool Calls

LLMToolCallRepair can strip Markdown fences, balance JSON brackets, fill missing required schema fields, and compact OpenAI-style tool arguments.

hb cfg set heavenbase.llm.tool_call_repair.enabled true
hb cfg set heavenbase.llm.tool_call_repair.strict false

Pass tool_call_repair={...} to hb.LLM(...) or repair_tool_calls=True to one chat call. Strict mode raises when arguments cannot be parsed; non-strict mode returns the original string.

8. Use a Custom Provider

The custom preset is for an unbundled OpenAI-compatible endpoint:

llm = hb.LLM(
    preset="custom",
    base_url="http://localhost:9999/v1",
    model="third-party-model",
    api_key="optional-key",
)

Both base_url and a concrete model are required. Use a custom gateway instead when the payload family itself is different.

9. Run Asynchronously

import asyncio
import heavenbase as hb


async def main():
    llm = hb.LLM(preset="chat")
    text = await llm.achat("hello")
    async for chunk in llm.astream("Count to three."):
        print(chunk, end="")
    vector = await llm.aembed("semantic text")


asyncio.run(main())

Every primary synchronous operation has an async counterpart. Async executable tools require achat; synchronous chat rejects async callables.

Summary

Gateways own transport shape after model and provider resolution.
Caching is visible, deterministic, and independently configurable by namespace.
SDK exports support interop without duplicating resolution.
Images and async calls use the same resolved client model.

Further Exploration

Related resources:

LLM Overview — Presets, models, providers, and gateways
First LLM — Provider setup and client export examples
LLM Providers — Provider-specific configuration and route checks

Introduction

Quickstart

Core Concepts

Developer

Utilities Toolbox

LLM

Agent Runtime

Community

1. Motivation

2. Choose a Gateway

3. Register a Runtime Shape

4. Control Response Caching

5. Reuse and Export Clients

6. Generate and Normalize Images

7. Repair Tool Calls

8. Use a Custom Provider

9. Run Asynchronously

Summary

Further Exploration

​1. Motivation

​2. Choose a Gateway

​3. Register a Runtime Shape

​4. Control Response Caching

​5. Reuse and Export Clients

​6. Generate and Normalize Images

​7. Repair Tool Calls

​8. Use a Custom Provider

​9. Run Asynchronously

​Summary

​Further Exploration

1. Motivation

2. Choose a Gateway

3. Register a Runtime Shape

4. Control Response Caching

5. Reuse and Export Clients

6. Generate and Normalize Images

7. Repair Tool Calls

8. Use a Custom Provider

9. Run Asynchronously

Summary

Further Exploration