arcodangeandClaude Fable 5
5aea83749d
Operator direction 2026-07-15: the orchestration layer itself (builder sessions, cold verifiers) must run on Mistral or hermes+Ornith/MLX too. The protocol already carries everything in files+issues; new model-fleet section defines the evidence-gated ladder — verifier roles migrate first (cross-family refutation is stronger verification), scoped builders benched on unchanged acceptance gates, Claude default until the bench says otherwise. D2 row records the direction; spike = erp#63. Co-Authored-By: Claude Fable 5 <[email protected]>
74 lines
8.2 KiB
Markdown
74 lines
8.2 KiB
Markdown
[vibe](../../README.md) > [PRD](../README.md) > [AI back-office](README.md) > **Model fleet**
|
||
|
||
# Model fleet — four tiers, one routing policy
|
||
|
||
> **Status:** In design
|
||
> **Last Updated:** 2026-07-11
|
||
> **Up:** [AI back-office hub](README.md)
|
||
> **Related:** [Agent architecture](agent-architecture.md) · [Task inventory](task-inventory.md) · [POC plan](poc-plan.md)
|
||
|
||
## The four tiers
|
||
|
||
| Tier | Where | Availability | Assigned work | Data policy | Marginal cost |
|
||
| --- | --- | --- | --- | --- | --- |
|
||
| **Pi SLM** | k3s cluster (pi1–3, arm64), llama.cpp/Ollama server, quantized 1–4B | **24/7** (survives cloud + laptop outages) | T01 triage, T11 reminders, event detection, queue enrichment | everything stays in the lab | ~0 € (electricity) |
|
||
| **M4 local** | MacBook Pro M4 — the hermes runtime (local **Ornith** reasoning model, `127.0.0.1:18080`) · Ollama/MLX 7–30B class | **when awake** — opportunistic, never time-critical | T02/T16 sensitive extraction, T09 cash report, T17 vault capture/retrieval, second extractor, drafting | on-device; bank/contract/vault content never leaves | 0 € |
|
||
| **Mistral (EU cloud)** | La Plateforme API (Mistral Large/Medium class + OCR) | on-demand | second/independent extractor, OCR for scans, FR fiscal wording, volume overflow | EU residency; acceptable for business documents | cents/doc |
|
||
| **Claude (frontier)** | Claude Code + skills (interactive), Agent SDK / API (headless) | on-demand | business validation vs fiscal profile, manifest assembly, orchestration, escalations, T12 research, **building the atoms themselves** | prefer minimized/structured payloads; full docs only when the human says so | subscription + API cents |
|
||
|
||
Model *candidates* per tier (evaluate at POC time — the named models will age faster than this PRD): Pi → Qwen3 1.7B/4B, Gemma 3 1B/4B class GGUF Q4; M4 → already resident: **Ornith served by hermes**; candidates Qwen3 14B/30B-A3B, Mistral Small 3.x, Gemma 3 27B class (RAM-dependent); Mistral → current Large/Medium + dedicated OCR; Claude → current Opus-class frontier model. [POC-5](poc-plan.md#poc-5--model-routing-bench) produces the actual accuracy/latency/cost table; the registry's `model_policy` fields hold the outcome, not this page.
|
||
|
||
The [second brain](agent-architecture.md#knowledge-layer--the-second-brain) already declares its own routing doctrine — *Claude by default · Mistral for well-defined tasks · Ornith/hermes local for the confidential* — this fleet generalizes a policy the vault has been living by, it does not invent one.
|
||
|
||
## Routing policy
|
||
|
||
Route by **(sensitivity, complexity, stakes, availability)** — in that order:
|
||
|
||
1. **Sensitivity floor**: bank statements, contracts, anything with credentials → local tiers (M4/Pi) or EU cloud at most; escalation to Claude sends *extracted fields*, not raw documents, unless the human explicitly widens it.
|
||
2. **Complexity ceiling per tier**: Pi handles closed-set classification and template rendering only; M4/Mistral handle structured extraction and drafting; ambiguity, multi-document reasoning, and anything touching the fiscal profile go to Claude.
|
||
3. **Stakes gate**: any output that feeds a `write-*` or `outbound` atom must come from a tier that passed that atom's eval at the required accuracy — regardless of what cheaper tier "could" do it.
|
||
4. **Availability fallback**: each atom's `model_policy` lists an ordered fallback chain; the router degrades along it and *flags the degradation in the journal* (a result produced by a fallback tier is marked as such).
|
||
|
||
**Escalation rules** (mechanical, not vibes): confidence below the atom's threshold → next tier up; dual-extraction disagreement on critical fields → Claude; Claude uncertain → human review queue. Every escalation is journaled with its reason — escalation *rates* are a fleet health metric.
|
||
|
||
## Structured output enforcement
|
||
|
||
The format guarantee never rests on the model:
|
||
|
||
| Tier | Mechanism |
|
||
| --- | --- |
|
||
| Pi (llama.cpp) | GBNF grammar / JSON-schema constrained decoding — a 1–4B model *cannot* emit malformed JSON |
|
||
| M4 (Ollama/MLX) | JSON-schema `format` constrained decoding |
|
||
| Mistral | JSON mode / function-calling schemas |
|
||
| Claude | tool-use schemas (forced tool choice) |
|
||
|
||
…and regardless of tier, every payload passes the same deterministic validators downstream ([agent architecture](agent-architecture.md#atom-contract)). Constrained decoding guarantees *shape*; validators guarantee *truth conditions* (arithmetic, checksums, plausibility).
|
||
|
||
## Harness portability
|
||
|
||
The **atoms** above are multi-model by construction. The **harness** — the orchestration layer that executes backlog issues (builder sessions), verifies them (cold readers, locate-tests) and files PRs — ran its first full loop on Claude (erp#38 → PR erp#62, 2026-07-15). **Operator direction (2026-07-15): that layer must not have Anthropic as a hard dependency either — the same loop should run on Mistral, or on hermes + Ornith / MLX-served local models.**
|
||
|
||
The protocol was built for this: everything an executing agent needs lives in **files and issues** — AGENTS.md orientation, per-issue Execution footers (first command, done-means), acceptance tests defined in the issue itself, deterministic validators around every output. Nothing load-bearing lives in a vendor's context window. Any runtime that can read the forge and run a shell can attempt an issue; the acceptance gates don't care who built the thing.
|
||
|
||
Migration ladder (evidence-gated like everything else — erp#63):
|
||
|
||
1. **Verifier roles first** — locate-tests, cold-reader audits, refutation passes: bounded, read-only, cheap. Re-run the already-passed erp#38 locate-test on **Ornith** (hermes, `127.0.0.1:18080`) and on an **MLX-served candidate** (Qwen3 14B/30B-A3B, Mistral Small 3.x class); require verdict parity with the Claude run. Once parity holds, **prefer cross-family verification**: a different model family refuting the builder is stronger evidence than the builder's family agreeing with itself.
|
||
2. **Scoped builders benched** — one small, tightly-footered issue (a #56-class fix) attempted end-to-end by a Mistral-driven runtime against the unchanged acceptance gates; measure completion rate and intervention count.
|
||
3. **Claude stays the default** for multi-file builders, ambiguity and fiscal reasoning **until the bench says otherwise** — a routing-policy fact (complexity ceiling above), not an allegiance.
|
||
|
||
## Degraded modes
|
||
|
||
| Outage | Keeps working | Queues | Lost until recovery |
|
||
| --- | --- | --- | --- |
|
||
| **Cloud down** (Anthropic + Mistral) | Pi triage, reminders, digests; M4 extraction when awake | writes awaiting business validation | escalations, T12 research |
|
||
| **Laptop asleep/away** | everything cloud + Pi | M4-assigned sensitive extraction (or reroute to Mistral if policy allows) | nothing time-critical (by design) |
|
||
| **Cluster down** | cloud tiers driven manually from the M4 | sentinel triage, reminders | 24/7 watching — operator falls back to the manual runbooks |
|
||
| **ERP down** | triage, extraction, drafting | all `write-*` and read-verify stages | recording; restore runbook applies |
|
||
| **Source or channel down** (Zoho, a bank API, Telegram) | every other lane, all tiers | the affected lane parks; item age stays visible once the channel returns | that feed/channel — its manual fallback applies ([C9](challenges.md#c9--provider--api-dependency): IMAP for mail, CSV export for banks, direct check-in replacing the digest) |
|
||
|
||
The quarterly game-day ([QA strategy](qa-strategy.md#ops-qa)) exercises one of these on purpose.
|
||
|
||
## Cost envelope
|
||
|
||
At current volumes (~30 relevant mails, ~5–10 supplier invoices, 1 client invoice, 4 recos, ≤1 fiscal event per month), cloud inference is **single-digit euros per month** — the 30 €/month budget in the [success criteria](README.md#success-criteria) is generous headroom, with an alert at 20 €. The honest framing: at Arcodange's scale, the local tiers are **not** a cost play — they buy **resilience** (24/7 sentinel through cloud outages), **privacy** (bank/contract content stays home), and **institutional learning** (operating SLMs is itself lab capital). The expensive resource is frontier-tier *authoring* of atoms (Claude Code sessions), covered by the existing subscription and amortized as each atom graduates to cheaper tiers.
|