Nodes
3
ingest · infer · emit
Prompts,
local.
An on-device LLM pipeline. Stream CSV prompts in, run them through TinyLlama on the host CPU, output answers to a CLI. No cloud. No API key. The model never leaves the box.
Model
TinyLlama
on-device · CPU-first
Egress
0
network not required
Output
CLI / CSV
streamable
An LLM
that never phones home.
The mainstream pattern for using a language model in a pipeline is to post a prompt to a hosted API. That gives you, in exchange for fluency, a third party reading every prompt and every answer.
This pipeline does not do that. The model file lives next to the workflow. A signed Protor extension loads it into the process. Prompts arrive over CSV. Answers go out over CLI or another CSV. The network adapter is not even initialised.
For a bank scoring a customer-letter draft, a hospital summarising a clinical note, or a city extracting structure from a permit, this changes the threat model entirely. The model is small. The privacy is total.
Three nodes,
one .viow file.
From E2E Tests/LocalLLM/localllm-tinyllama-csv-to-cli.viow. Smallest workflow on the whole site.
ingestcsv.Subscriber
→
inferlocalllm.Processor
→
emitetl.CLIProcessor
A sibling workflow localllm-csv-to-cli.viow uses a configurable model loader for swap-in alternatives — Phi, Qwen, Mistral, anything ONNX-compatible.
Why local LLMs
are getting good.
A short list of why this is suddenly practical on commodity hardware.
| Trend | What it gives you |
|---|---|
| Small models (TinyLlama, Phi-3-mini, Qwen-2.5-1.5B) | 1–4 B parameters that run on a CPU at usable token rates. |
| Quantisation (Q4 / Q5 / GGUF, ONNX INT8) | Model fits in 1–4 GB RAM. Workstation-class is enough. |
| Domain fine-tunes | A small model that is good at one thing matches a big model that is OK at everything. |
| Deterministic decoding (seed + temperature 0) | Same prompt + same model → same answer. Replayable evidence. |
Why this matters.
- Sovereign-grade language AI. An LLM that can be deployed inside a regulator, a defence integrator, a hospital — with no policy waiver required for cross-border data flow.
- Same workflow shape as everything else. CSV in, CSV out. Drop the LLM node next to any other Protor pipeline.
- Replayable. With deterministic decoding the same prompt produces the same answer — every audit, every six months, every time.