Local models on the web

The Browser clients guide called a model that lived on a server. This guide removes the server: interact_with_gemma loads a Gemma-family model into the page and runs it on the browser's GPU through WebGPU. Nothing leaves the device, and the reply streams onto an AsyncNode exactly as a remote backend's would.

interact_with_gemma is an ordinary A11 action. It has the same ports as every other backend — an interactions input, a unary config input, and text_output / new_interactions outputs — so the code that drives it is the same code you already write for interact_with_llm. The only new idea is that its handler runs a model locally instead of calling an API.

Before you start

You need npm install a11@npm:@curiositystack/a11, a browser with WebGPU enabled, and a URL to a hosted Gemma model asset (.task / .litertlm) that MediaPipe can load. The model file is large; serve it from a location with permissive CORS. No API key and no server are involved.

1. The action contract

Import the backend and the SDK names you need. INTERACT_WITH_GEMMA_SCHEMA already describes the ports; you register it like any other schema.

import {
    Action,
    ActionRegistry,
    INTERACT_WITH_GEMMA_SCHEMA,
    interactWithGemma,
    makeTextMessageInteraction,
    parseInteraction,
    isOk,
    StatusCode,
    type Status,
} from '@curiositystack/a11';

const need = <T>(value: T | Status): T => {
    if (!isOk(value)) throw new Error(`${StatusCode[value.code]}: ${value.message}`);
    return value as T;
};

2. Register and run the action locally

There is no session and no transport. Create the action from its schema, bind the handler, and run() it. run() starts the handler on the same node map, so the ports you open next are the very ones the handler reads and writes.

const registry = new ActionRegistry();
need(registry.register('interact_with_gemma', INTERACT_WITH_GEMMA_SCHEMA, interactWithGemma));

const action = need(Action.create(INTERACT_WITH_GEMMA_SCHEMA, {
    handler: interactWithGemma,
    registry,
}));
need(action.run());

3. Feed the conversation and the model URL

The interactions port takes the whole conversation; the unary config port carries the browser-specific knobs, most importantly model_asset_path — the URL of the Gemma model to download and run. makeTextMessageInteraction builds a portable text turn.

const user = need(makeTextMessageInteraction('Explain WebGPU in one sentence.'));

const interactions = need(await action.getInput('interactions'));
need(await interactions.put(user, {final: true}));

const config = need(await action.getInput('config'));
need(await config.putFinal({
    model_asset_path:
        'https://huggingface.co/litert-community/gemma-4-E2B-it-litert-lm/resolve/main/gemma-4-E2B-it-web.litertlm?download=true',
}));

That HuggingFace URL is also the SDK default, so omitting model_asset_path loads the same model. The asset is fetched with redirects followed (the resolve URL 302s to a CDN), then handed to the runtime as bytes. The downloaded bytes are stored in the browser's Cache Storage, so a page reload serves the model from disk instead of downloading it again. The first turn triggers the download and WebGPU compilation; later turns reuse the loaded model.

4. Stream the reply as it arrives

text_output is an AsyncNode. Reading it in a loop lets tokens appear the moment the model produces them — next() returns each streamed piece and null once the turn is complete.

const output = need(await action.getOutput('text_output', false));
let reply = '';
while (true) {
    const token = need(await output.next({timeoutMs: 120_000}));
    if (token === null) break;
    reply += token;
    render(reply); // append to your chat bubble
}

The completed assistant turn also lands, structured, on new_interactions. Keep it and prepend it to the next interactions write to continue the conversation:

const newInteractions = need(await action.getOutput('new_interactions', false));
const assistant = need(parseInteraction(need(await newInteractions.next())));
need(await action.wait(5_000));
history = [...history, user, assistant];

5. Swap the runtime (optional)

By default the handler dynamically imports Google's MediaPipe LlmInference task and runs it on WebGPU. You can replace that runtime — to cache the loaded model across turns, report download progress, or plug in a different engine — with setGemmaEngineFactory:

import {setGemmaEngineFactory, type GemmaEngine} from '@curiositystack/a11';

setGemmaEngineFactory(async (config) => {
    const engine: GemmaEngine = {
        async generate(prompt, onToken) {
            // stream pieces via onToken(delta); resolve with the full text,
            // or return a Status on failure — never throw.
            return '…';
        },
    };
    return engine;
});

A factory returns a StatusOr<GemmaEngine>: on failure return a status built with a helper such as unavailableError(...) rather than throwing, and the runtime aborts the output ports with it.

6. Display failures

Every A11 call returns a StatusOr<T>: success values pass isOk, failures carry a code, message, and details. Convert them at the UI boundary — a missing model URL, an absent WebGPU adapter, or a load error all arrive as ordinary statuses, not exceptions:

try {
    await runTurn(text);
} catch (error) {
    errorRegion.textContent = error instanceof Error ? error.message : String(error);
}

Try it

Paste a hosted Gemma model URL, then chat. The first message downloads and compiles the model in your browser (this can take a while); after that, generation is local and the reply streams token by token into the bubble. The download is cached, so reloading the page skips it. A WebGPU-capable browser is required.

Send