We fine-tune and ship small local language models under real hardware and safety constraints —
and publish what breaks along the way.
BRIEF · AUG 13 2026RED KIT, THE COMPANY · RED (commercial) · BLUE (open source)ALL-IN ≈ $5.5K · MARGINAL RESEARCH SPEND ≈ $85
01 · What we're building
Red — an offline emergency first-aid and survival model for iPhone (12 GB), distilled from
frontier teachers onto a 4B base, filtered through a safety pipeline we validate by audit. No connectivity,
sub-second first tokens, App Store path kept legally clean from day one (redkit.ai secured).
Blue — the open-source research line: a 35B-A3B mixture-of-experts fine-tuned for rare-knowledge retrieval
over a self-hosted Wikipedia index (53M embedded chunks); candidate open release of weights plus training
data plus methodology.
02 · Findings so far (11 logged; the six that matter)
Extreme quantization destroys rare facts specifically. A 1.58-bit ternary 27B keeps general capability but drops to 4.3% on rare-fact recall vs
9–11% for its 4–8-bit relatives — a 2.2–2.5× gap hiding inside a "~5–14% retention loss" marketing number.
MoE knowledge density beats dense. A 35B-total/3B-active MoE holds the most retrievable rare knowledge in our 9-model bake-off
(16.5% SimpleQA-V vs 10.8% best dense) while inferring 3–5× faster.
Benchmark generations lie about knowledge. The newer 27B posts higher MMLU than its predecessor and scores lower on rare facts. Base-model
selection for knowledge products requires direct rare-fact evals; MMLU-class proxies actively mislead.
Honest-abstention protocols invert frontier-vs-local comparisons. Given explicit permission to abstain, a frontier model scores 12.6% raw — below our local MoE —
because it declines 71% of questions (and is right 43% when it answers). Public leaderboard
numbers are substantially attempt-pressure artifacts.
"Faithful hallucination of priority" is a distinct safety failure mode. Single-chunk-grounded teachers produce answers faithful to the source but wrong in clinical priority
(folk remedy first, cool-water omitted). Faithfulness checkers pass them; only a priority-aware judge catches them —
28%→16.7% failure reduction after the fix. Faithfulness and safety-priority are orthogonal filters.
Model capability tiers are not totally ordered. Under one validated rubric, teacher A leads on procedure ordering (2.2% fails), teacher B — at
2× the price — leads on faithfulness but matches the budget teacher on ordering. Attribute-level eval, not tier
pricing, decides distillation spend.
03 · The model card — 9-model closed-book bake-off
Model
Rare facts¹
PopQA
Abstains¹
Right when answering²
Qwen3.6-35B-A3B MoE · Blue base
16.5%
28.3%
18%
47%
Claude Sonnet 5 · frontier reference
12.6%
37.8%
71%
66%
Qwen3.5-27B
10.8%
26.5%
11%
31%
Qwen3.6-27B · newer gen, 8-bit
9.0%
25.8%
43%
44%
Gemma 4 31B · Q8
7.4%
18.1%
23%
45%
Qwen3.5-9B
5.3%
17.7%
40%
32%
Qwen3.5-4B · Red base
4.5%
17.0%
2%
17%
Ternary Bonsai 27B · 1.58-bit
4.3%
17.2%
1%
18%
Gemma 4 E2B QAT · Red fallback
3.3%
12.7%
41%
23%
Red 1 · the goal — 4B + distilled emergency corpus, on-device
0% refusals on legitimate emergencies · 100% safe-completion on harmful asks ·
≥95% primary-steps-first (judged) · <1s first token on iPhone 17 Pro, offline
¹ SimpleQA-Verified, n=1000, closed book, explicit permission-to-abstain protocol. ² PopQA accuracy when the model attempts an answer.
Red 1's exam is different by design: a 350-scenario emergency vignette bench (frozen pre-training, decontaminated) — rare-fact recall is Blue's mission, not Red's.
04 · Hardware reality
MacBook M3 Max · 48 GB
Eval bench + quantization lab. Shared with a day job — available evenings only; every long job fights a 6:30am clock-out.
RTX 4080 Super · 16 GB
Only trainable GPU. Caps local fine-tuning at ~4B QLoRA class; lives behind Windows/WSL whose VM lifecycle kills unattended jobs (documented, workaround pending).
Rented burst compute
RunPod per-campaign: community tier proved unreliable (phantom stock, no logs); secure datacenter tier works — $30–55 per embedding/training campaign.
Deploy target
iPhone 17 Pro, 12 GB — the real constraint shaping everything: model size, quantization tolerance (see finding 1), and thermal budget.
No always-on box
Nothing runs 24/7. Every experiment is orchestrated remotely across borrowed windows of uptime.
05 · The ledger — full cost recognition
Item
Cost
Nature
MacBook M3 Max 48 GB
$2,500
capex · one-time, shared with day job
RTX 4080 Super rig
$2,500
capex · one-time, training + embedding
Network (WiFi antenna)
$100
capex
Brand — redkit.ai
$160
2-year registration
AI engineering (Claude subscription)
$200/mo
opex · orchestration, research, code
API teachers + judges (all 11 findings)
≈ $30
marginal · Batch API, audited pipeline
GPU rentals (embedding fleet)
≈ $55
marginal · burst, auto-terminated
The headline: a 9-model closed-book bake-off with a frontier reference row, a 53M-chunk
embedded Wikipedia corpus, a validated teacher-distillation pipeline (judge audited at 91% agreement), and 11
publishable findings — on ≈ $5.1K of one-time consumer hardware and ≈ $85 of marginal compute.
The capital is reusable; the marginal cost of a finding is coffee money. Constraint isn't the obstacle;
it's the research program.
Prepared Aug 13, 2026 · findings from the lab's research log (claim → evidence → caveats format) ·
numbers reproducible from frozen eval files and committed scorecards.