<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en"><generator uri="https://jekyllrb.com/" version="4.4.1">Jekyll</generator><link href="https://rjb7731.github.io/feed.xml" rel="self" type="application/atom+xml"/><link href="https://rjb7731.github.io/" rel="alternate" type="text/html" hreflang="en"/><updated>2026-11-16T13:25:51+00:00</updated><id>https://rjb7731.github.io/feed.xml</id><title type="html">blank</title><subtitle>Robotics &amp; ML engineer. Building action-conditioned world models for robot navigation. </subtitle><entry><title type="html">Teaching a Tiny Model to Invent Kitchen Gadgets on a Laptop</title><link href="https://rjb7731.github.io/blog/2026/inventing-new-gadgets-rl/" rel="alternate" type="text/html" title="Teaching a Tiny Model to Invent Kitchen Gadgets on a Laptop"/><published>2026-11-15T00:00:00+00:00</published><updated>2026-11-15T00:00:00+00:00</updated><id>https://rjb7731.github.io/blog/2026/inventing-new-gadgets-rl</id><content type="html" xml:base="https://rjb7731.github.io/blog/2026/inventing-new-gadgets-rl/"><![CDATA[<style>#markdown-content h2,.post-content h2{margin-top:3.25rem;margin-bottom:1rem}#markdown-content h3,.post-content h3{margin-top:2.25rem;margin-bottom:.75rem}#markdown-content table,.post-content table{margin-top:1rem;margin-bottom:1.75rem}#markdown-content ul,.post-content ul{margin-bottom:1.5rem}#markdown-content blockquote,.post-content blockquote{margin:1.5rem 0}#markdown-content details summary,.post-content details summary{cursor:pointer;color:var(--global-theme-color,#2a78d6)}</style> <p>Code coming soon :)</p> <div class="row align-items-center mt-3"> <div class="col-md-6 mb-3 mb-md-0"> <img src="/assets/img/rotating_drum_cooker.jpg" alt="Rotating Drum Cooker, generated from the model's claim" class="img-fluid rounded z-depth-1" style="width:100%;"/> </div> <div class="col-md-6"> <p class="mb-1"><b>What it was given:</b> Kitchen Appliance for Processing Foodstuff and Method of Operating Same.</p> <p class="mb-1"><b>What it wrote:</b> <i>... the assembly comprising a removable drum that is rotatable within the pot to process the foodstuff contained within the pot....</i></p> <p class="mb-2"><b>What Gemini thought:</b> <i>Proven and practical, though nearly identical to existing rotating air fryers and drum roasters.</i></p> <p class="mb-2"><b>Use cases:</b> <i>Rinsing rice, boiling and draining pasta or potatoes, frying chips so they cook evenly, roasting coffee beans</i></p> <p class="mt-2 mb-0 text-muted"><small>The picture is an image model's take on the claim, a lid that wouldn't quite close and all.</small></p> </div> </div> <div class="pb-4"></div> <p>This kitchen gadget doesn’t exist. Its claim was written by a 0.6 billion parameter model running on my MacBook, after I taught it to write patent claims and then used reinforcement learning to make it write better ones.</p> <p>After reinforcement learning, the quality of what it writes, as judged by Gemini, roughly doubled. Along the way it tried to cheat twice, and so far it never really became Edison in terms of inventions. This is the story of how it got there, what failed first, and why I now think the reward matters more than the model.</p> <h3 id="a-personal-invention-favourite">A personal invention favourite</h3> <div class="row justify-content-center"> <div class="col-sm-12 col-md-10 mt-3 mt-md-0"> <figure> <picture> <source class="responsive-img-srcset" srcset="/assets/img/cheese_cutter-480.webp 480w,/assets/img/cheese_cutter-800.webp 800w,/assets/img/cheese_cutter-1400.webp 1400w," type="image/webp" sizes="95vw"/> <img src="/assets/img/cheese_cutter.jpg" class="img-fluid rounded z-depth-1" width="100%" height="auto" title="Cheese cutter" loading="lazy" onerror="this.onerror=null; $('.responsive-img-srcset').remove();"/> </picture> </figure> </div> </div> <div class="caption text-center"> Figure 1: an invention I have coined the 'Dairy Guillotine' </div> <h2 id="where-this-started">Where this started</h2> <p>Most of my recent projects have had the same shape: a small model, a well-built environment, and a reward. The parrot learned to read and passed the Year 1 phonics check. A VQ-VAE learned to draft letters and was pushed by a checker. Another model learned to stop asking questions people had already answered. All of them ran on my laptop, where a rollout costs next to nothing.</p> <p>The idea of novelty is something I’ve been thinking about a lot with models that autonomously produce things with code, language or images. LLMs give the average or safe bet from the internet rather than anything novel unless pushed further either by users or training methods. Reinforcement learning is playing an increasingly strong part in training pipelines to push sampling away from the peak of the curve of possible solutions to those areas which are much more inventive.</p> <p>So I asked myself a slightly cheeky question. If rollouts are this cheap with a model of 270M to 600M parameters, could you <em>discover</em> something, rather than just get good at a task we already know the answer to?</p> <p>Patents felt like the right place to have a go. They are humanity’s dated record of new ideas, so novelty can be checked against the past. Better still, you can hide the most recent years from the model and see whether it anticipates them. Research on invention also keeps finding that most inventions are new combinations of existing parts, which sounds like exactly the kind of search a small model could do.</p> <p>I did toy with something Manchester-flavoured but settled on kitchen gadgets. Everyone knows what a kettle is, and I can judge an idea just by looking at it.</p> <h2 id="why-bother">Why bother?</h2> <p>The more of these projects I build, the more convinced I am that a model is only ever as capable as its environment and the reward that environment hands out. With a sharp enough reward, a very small model can be genuinely useful. AlphaZero didn’t need the entire internet, just the rules of Go and a clear signal of who won.</p> <p>Invention is a much harder test of that idea, because nobody can write down the rules of a good invention. That makes it a good place to find out where the idea breaks.</p> <p>The rules I set myself:</p> <ul> <li><strong>Tiny and local.</strong> One 0.6B model, fine-tuned and trained with RL on my MacBook using MLX.</li> <li><strong>Novelty against the real record, not vibes.</strong> Every idea is checked against thousands of real patents. Patents filed from 2016 onwards are hidden from training.</li> <li><strong>A big model judges, it never invents.</strong> Gemini scores ideas, but every idea comes from the small model.</li> <li><strong>Count the failures.</strong> A claim that doesn’t make sense scores as a fail, not as a missing value.</li> </ul> <h2 id="the-setup-ten-years-of-kitchen-patents">The setup: ten years of kitchen patents</h2> <p>I used HUPD, the Harvard USPTO Patent Dataset, which sits on Hugging Face with one archive per filing year and no login. I streamed each year through and kept only kitchen equipment, patent class A47J.</p> <p>That gave 8,910 applications filed between 2004 and 2018, split by filing date:</p> <table> <thead> <tr> <th>Split</th> <th>Filed</th> <th>Applications</th> <th>Used for</th> </tr> </thead> <tbody> <tr> <td>Train</td> <td>2004 to 2014</td> <td>6,096</td> <td>Everything the model learns from</td> </tr> <tr> <td>Validation</td> <td>2015</td> <td>956</td> <td>Checking while building</td> </tr> <tr> <td>Future</td> <td>2016 to 2018</td> <td>1,858</td> <td>Never seen; the novelty band and the backtest</td> </tr> </tbody> </table> <p>A few HUPD entries are filed under kitchen equipment but plainly aren’t: hair dye, diamond-coated particles, “a method for operating a computer”. Filtering those out left 7,400.</p> <p>The part of a patent I care about is <strong>claim 1</strong>, the legal definition of the invention. It always has the same shape: <em>“A [gadget] comprising: a [part]; a [part]; and a [part] configured to …”</em>. If you can write a sensible, new claim 1, you have described a new invention.</p> <p><strong>How do you score novelty?</strong> The trick I am happiest with is letting real inventions set the bar. Each patent becomes a vector using a small sentence model. I then measure how far each real 2016 to 2018 patent sits from its closest earlier patent. Most land between 0.23 and 0.52. That range becomes the <strong>novelty band</strong>: an idea only scores if it sits as far from the past as real new inventions do. Too close means it already exists. Too far usually means nonsense.</p> <h2 id="what-worked---a-tiny-pretrained-drafter">What worked - a tiny pretrained drafter</h2> <p>So I tried a few ways before this went back to the original plan, a small pretrained language model. Qwen3-0.6B-Base has read plenty of patents during pretraining, so it already knows English, legal English, and roughly what a kettle is. I fine-tuned it with LoRA in MLX on the 6,096 training claims: a patent title in, its claim 1 out. It took 2,000 steps on my MacBook.</p> <p>The difference was immediate. Of the first 300 claims it wrote for titles it had seen during training:</p> <ul> <li><strong>271 (90%) were valid claims</strong>: they started properly, had a claim verb, finished, and named at least three parts</li> <li><strong>none were copies</strong>: none overlapped more than 70% in wording with the closest existing claim</li> <li><strong>the median novelty was 0.41</strong>, right in the middle of the band where real later patents sit</li> </ul> <p>Here is one of them, as written:</p> <blockquote> <p><em>A home brewed coffee machine, comprising: a coffee pod holder; an outlet; a stand designed for slidable movement between a coffee watering position and a stand upright position; a plurality of baskets designed to support a set of coffee pods; said baskets comprising: a plurality of compartments; a filter basket; and a ground coffee basket; …</em></p> </blockquote> <p>It reads like a real claim, nested “said … comprising” and all.</p> <h2 id="the-judge-and-a-verifier-that-learns-its-taste">The judge, and a verifier that learns its taste</h2> <p>To push the drafter towards <em>better</em> claims, I needed a score for “better”. I used Gemini 3.8 Flash as the judge. It scores each claim from 1 to 10 on three things: is it <strong>sensible</strong> (would it physically work?), <strong>useful</strong> (does it add a real benefit?) and <strong>novel</strong> (is it meaningfully different from the closest existing patent, which it is shown)? I told it to be strict, and it was. The first 180 claims averaged 3.97, and the best reached about 7.</p> <p>Calling Gemini on every RL sample would be slow and would put the network inside the training loop. So I distilled its judgement into a small local <strong>verifier</strong>. I labelled 1,505 claims, mostly the drafter’s own at a mix of temperatures, plus deliberately bad ones so the verifier would learn what bad looks like:</p> <table> <thead> <tr> <th>Claims</th> <th>Count</th> <th>Gemini’s mean score</th> </tr> </thead> <tbody> <tr> <td>The drafter’s own</td> <td>1,205</td> <td>3.00</td> </tr> <tr> <td>Old recombination ideas</td> <td>120</td> <td>2.60</td> </tr> <tr> <td>Claims with their parts shuffled</td> <td>180</td> <td>1.49</td> </tr> </tbody> </table> <p>I also had Gemini judge 100 of them a second time. It agreed with itself at 0.80 on the drafter’s claims (Spearman rank correlation), which is the ceiling: no verifier can match Gemini better than Gemini matches itself.</p> <p>Then I tried different ways of turning a claim into numbers for the verifier:</p> <table> <thead> <tr> <th>Verifier built on</th> <th>Agreement with Gemini on the drafter’s claims</th> </tr> </thead> <tbody> <tr> <td>MiniLM (22M, frozen)</td> <td>0.41</td> </tr> <tr> <td>MPNet (110M, frozen)</td> <td>0.42</td> </tr> <tr> <td>BGE-base (110M, frozen)</td> <td>0.46</td> </tr> <tr> <td><strong>The drafter’s own hidden states (0.6B)</strong></td> <td><strong>0.53</strong></td> </tr> <tr> <td>Gemini against itself (the ceiling)</td> <td>0.80</td> </tr> </tbody> </table> <p>The drafter itself made the best verifier, because it has read every kitchen claim in training. A learning curve showed agreement flat from half the labels onwards, so more labels of the same kind wouldn’t have helped.</p> <p>Novelty was the hardest score to predict, because the verifier never sees the closest existing patent. So the verifier only predicts <strong>sensible × useful</strong>, and the novelty band handles novelty.</p> <h2 id="reinforcement-learning-and-two-attempts-to-cheat">Reinforcement learning, and two attempts to cheat</h2> <p>The RL is GRPO, the same family of method I used for my previous work on phonics and what powers models like Deepseek. Sadly US frontier labs are not open about the recipes they use. Each step:</p> <ol> <li>Pick two training titles. For each, the drafter writes a group of 8 claims.</li> <li>Score every claim: <strong>valid × not a copy × inside the novelty band × verifier score / 10</strong>.</li> <li>Compare each claim with its siblings. Better than the group average gets reinforced, worse gets discouraged.</li> <li>Add a KL penalty that keeps the model close to the original drafter, so it can’t drift into text only the verifier likes.</li> </ol> <details class="mb-4"> <summary>The reward in more detail</summary> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>reward = valid × not_copy × band(novelty) × verifier / 10

valid       starts "A/An …", has a claim verb (comprising, including, having …), ends with a
            full stop, 12 to 300 words, names at least 3 parts, and is ONE claim
not_copy    word overlap with the closest existing claim below 70%
band        1 inside the novelty band (0.23 to 0.52), falling off smoothly outside it
verifier    ridge regression on the drafter's own hidden states (mean + last token),
            predicting Gemini's sensible × useful score, 1 to 10

advantage   (reward − group mean) / group std, groups of 8 claims per title
loss        − advantage × mean log-prob of the claim's tokens + 0.1 × KL(policy ‖ drafter)
trained     2.9M LoRA parameters, learning rate 3e-6, ~25 to 40 s per step on a MacBook
</code></pre></div> </div> </details> <p>Only the 2.9M LoRA parameters train, at about 25 to 40 seconds a step. I built a little live dashboard in the browser to watch the reward, KL, claim length and fresh samples as it trained.</p> <p>My first run drifted from the drafter too quickly, with KL at 0.025 after only 9 steps. I restarted with a smaller learning rate and a stronger KL penalty, and KL settled between 0.04 and 0.06.</p> <p>Then came the fun part. Goodhart’s law says that when a measure becomes a target, it stops being a good measure. The model found that out twice.</p> <p><strong>Exploit one: just keep writing.</strong> The best claims started ending like this: <em>“… in response to a user input. A food cooking assembly wherein, in a first operating mode, the drive assembly …”</em>. It had learned to tack a second claim onto the first, restating it with more detail, which nudged the verifier up. I made “more than one claim” invalid. Valid claims dipped to about 75%, then the model learned to stop after one.</p> <p><strong>Exploit two: say less.</strong> With that closed, claim length fell from about 230 tokens to 145, and claims like this appeared:</p> <blockquote> <p><em>A cooking basket holder comprising: a housing defining an interior holding chamber; and a removable component mounted on the housing …</em></p> </blockquote> <p>That describes half the objects in a kitchen. It’s always valid, easily lands in the novelty band, and the verifier gave it 4.3, one of its higher scores. Closing loopholes with rules becomes whack-a-mole, so I stopped at 100 steps and measured whether any of this had actually helped.</p> <div class="row justify-content-center"> <div class="col-sm-12 col-md-10 mt-3 mt-md-0"> <figure> <picture> <source class="responsive-img-srcset" srcset="/assets/img/patent_rl_reward_curve-480.webp 480w,/assets/img/patent_rl_reward_curve-800.webp 800w,/assets/img/patent_rl_reward_curve-1400.webp 1400w," type="image/webp" sizes="95vw"/> <img src="/assets/img/patent_rl_reward_curve.png" class="img-fluid rounded z-depth-1" width="100%" height="auto" title="Learning curve" loading="lazy" onerror="this.onerror=null; $('.responsive-img-srcset').remove();"/> </picture> </figure> </div> </div> <div class="caption text-center"> Figure 1: the reward over training steps and points with exploits. </div> <h2 id="results">Results</h2> <p>RL roughly doubled the Gemini-judged quality per attempt, mostly by teaching the model to write one clean, finished claim. I gave the original drafter and two RL checkpoints the same 100 titles and had Gemini score every valid claim. Invalid attempts count as the minimum score of 1.</p> <table> <thead> <tr> <th>Model</th> <th>Valid claims (of 100)</th> <th>Gemini mean, valid claims</th> <th>Gemini mean, per attempt</th> </tr> </thead> <tbody> <tr> <td>Fine-tuned drafter</td> <td>36</td> <td>2.72</td> <td>~1.6</td> </tr> <tr> <td>After 50 RL steps</td> <td>53</td> <td>3.27</td> <td>~2.2</td> </tr> <tr> <td>After 100 RL steps</td> <td>97</td> <td>3.26</td> <td><strong>~3.2</strong></td> </tr> </tbody> </table> <p>A few things stand out:</p> <ul> <li><strong>Most of the gain is reliability.</strong> Under the “one claim only” rule, the original drafter fails 64% of the time, because it habitually runs on into extra claims. After 100 steps it almost never does.</li> <li><strong>Valid claims improved by about half a point</strong> (2.72 to 3.27). With only 36 valid drafter claims, that’s suggestive rather than proven.</li> <li><strong>Quality per claim stopped improving by step 50.</strong> Steps 50 to 100 bought reliability, not better ideas.</li> <li><strong>The vague-claim exploit didn’t fool Gemini.</strong> Step 100 scored no worse than step 50.</li> </ul> <p>The honest weak spot is novelty. Gemini’s reasons keep saying things like “standard”, “entirely unoriginal” and “essentially identical to existing espresso makers”. RL made the model better at writing sound, conventional claims. It did not make it more inventive. That does sense if you think about it, the verifier rewards sensible and useful, and the novelty band only measures distance from the past, not inventiveness.</p> <p>I haven’t run the backtest yet (giving both models titles of 2016+ patents and checking how close they get to the real claims). That’s next.</p> <h2 id="the-inventions">The inventions</h2> <p>To pick ideas worth drawing, I took the top 25 claims from the RL model by Gemini’s score, at most two per kind of gadget. Gemini then wrote each one a product name, a two-sentence description and an image prompt, using only the parts in the claim. Here are the top three, in Gemini’s order. (These write-up scores run more generous than the strict scores above, so treat them as a ranking, not a quality measure.)</p> <h3 id="1-variable-chamber-brewer-73">1. Variable Chamber Brewer (7.3)</h3> <p>A brewing device whose brew basket has an adjustable piston, so you can change the volume of the brewing chamber.</p> <div class="row justify-content-center"> <div class="col-sm-12 col-md-10 mt-3 mt-md-0"> <figure> <picture> <source class="responsive-img-srcset" srcset="/assets/img/variable_chamber_brewer-480.webp 480w,/assets/img/variable_chamber_brewer-800.webp 800w,/assets/img/variable_chamber_brewer-1400.webp 1400w," type="image/webp" sizes="95vw"/> <img src="/assets/img/variable_chamber_brewer.jpg" class="img-fluid rounded z-depth-1" width="100%" height="auto" title="Variable Chamber Brewer" loading="lazy" onerror="this.onerror=null; $('.responsive-img-srcset').remove();"/> </picture> </figure> </div> </div> <div class="caption text-center"> Figure 2: the Variable Chamber Brewer, as an image model drew it from the claim. </div> <h3 id="2-smart-lid-cooker-71">2. Smart-Lid Cooker (7.1)</h3> <p>A cooker whose lid carries the control panel and a temperature probe that reaches down into the cooking chamber.</p> <div class="row justify-content-center"> <div class="col-sm-12 col-md-10 mt-3 mt-md-0"> <figure> <picture> <source class="responsive-img-srcset" srcset="/assets/img/smart_lid_cooker-480.webp 480w,/assets/img/smart_lid_cooker-800.webp 800w,/assets/img/smart_lid_cooker-1400.webp 1400w," type="image/webp" sizes="95vw"/> <img src="/assets/img/smart_lid_cooker.jpg" class="img-fluid rounded z-depth-1" width="100%" height="auto" title="Smart-Lid Cooker" loading="lazy" onerror="this.onerror=null; $('.responsive-img-srcset').remove();"/> </picture> </figure> </div> </div> <div class="caption text-center"> Figure 3: the Smart-Lid Cooker, as an image model drew it from the claim. </div> <h3 id="3-rotating-drum-cooker-67">3. Rotating Drum Cooker (6.7)</h3> <p>A pot on a raised frame with a removable perforated drum inside, a lid, and a drain.</p> <p>(Image at the start)</p> <h2 id="what-it-was-given-and-what-it-learned">What it was given, and what it learned</h2> <p>Like the parrot, this system starts with priors, and here they are much bigger. It’s worth being clear about where the line sits.</p> <table> <thead> <tr> <th> </th> <th>Learned from experience</th> <th>Given to it</th> </tr> </thead> <tbody> <tr> <td>Writing claims</td> <td>The shape and vocabulary of kitchen claims, from 6,096 training claims</td> <td>Qwen3-0.6B’s pretraining: English, and plenty of patents</td> </tr> <tr> <td>What counts as good</td> <td>Which claims Gemini prefers, through the verifier and RL</td> <td>Gemini’s judgement, and the three questions I asked it</td> </tr> <tr> <td>Novelty</td> <td>How far its ideas sit from the real record</td> <td>MiniLM’s notion of similarity, and a band set by real 2016+ patents</td> </tr> <tr> <td>Staying honest</td> <td>To write one finished claim, and not to copy</td> <td>The validity and copy checks I wrote</td> </tr> </tbody> </table> <p>The biggest given is Gemini’s taste. Everything RL learned, it learned through a verifier that agrees with Gemini at 0.53 out of a possible 0.80. The model can only ever be as good as that signal, and the two exploits show exactly what happens in the gap.</p> <h2 id="whats-next">What’s next</h2> <p>The lesson I keep relearning is that capability follows the reward. The drafter got better exactly where the reward was sharp (writing valid, finished claims) and found loopholes where it was blurry. So the next step probably means a better verifier:</p> <ul> <li><strong>Ask yes/no questions, not for a score.</strong> “Rate usefulness 1 to 10” is fuzzy. “Does every part act on another part?”, “Does the claim say what the new feature does?” and “Is the mechanism concrete or generic?” are easier to answer consistently, easier for a small model to learn, and much harder to game. Both exploits would have failed one of them.</li> <li><strong>Compare pairs.</strong> GRPO only needs to know which of 8 sibling claims is best, so ask Gemini “which of these two is better?” rather than for an absolute score.</li> <li><strong>Label what the policy actually writes.</strong> The exploits live in the claims the verifier likes most. Judging those each round and feeding them back is how loopholes close.</li> <li><strong>Train a proper reward head</strong> on the drafter itself, and use a few of them together, so a claim that fools one model but not the others gets marked down.</li> <li><strong>Use a real-world signal.</strong> HUPD records whether each application was granted or rejected: thousands of real examiner verdicts, sitting unused.</li> <li><strong>Narrow the world.</strong> Beverage makers alone are about 30% of the data. A coffee-machine-only drafter with a coffee-specific rubric is a much smaller search space.</li> <li><strong>Reward inventiveness directly,</strong> then run the backtest on 2016 to 2018 titles.</li> </ul> <p>The first attempts couldn’t write a sentence. Now a 0.6B model on a laptop writes patent claims that are 97% valid, are new by the patent record’s own standard, and are judged twice as good as when it started. It isn’t an inventor yet, but I now know exactly what it would take…a better reward.</p> <h2 id="prior-work">Prior work</h2> <table> <thead> <tr> <th>Work</th> <th>Why it matters here</th> </tr> </thead> <tbody> <tr> <td>Suzgun et al. (2022), the Harvard USPTO Patent Dataset (HUPD)</td> <td>The data: patent applications with claims, classifications, filing dates and examiner decisions</td> </tr> <tr> <td>Fleming (2001); Youn et al. (2015), “Invention as a combinatorial process”</td> <td>Invention as new combinations of existing parts, the idea behind attempt two</td> </tr> <tr> <td>Romera-Paredes et al. (2023), FunSearch</td> <td>A language model plus an automatic evaluator, searching for genuinely new results</td> </tr> <tr> <td>Shao et al. (2024), DeepSeekMath</td> <td>Introduced GRPO, the group-relative RL used here</td> </tr> <tr> <td>Christiano et al. (2017); Ouyang et al. (2022)</td> <td>Learning a reward model from human (here, AI) judgements and optimising against it</td> </tr> <tr> <td>Gao et al. (2023), “Scaling Laws for Reward Model Overoptimization”</td> <td>Why optimising against a learned verifier ends in exploits like the two above</td> </tr> <tr> <td>van den Oord et al. (2017), VQ-VAE</td> <td>The discrete autoencoder behind attempt one</td> </tr> <tr> <td>Reimers and Gurevych (2019), Sentence-BERT</td> <td>The sentence models (MiniLM, MPNet) used for novelty and the first verifiers</td> </tr> <tr> <td>Hu et al. (2021), LoRA; Apple’s MLX</td> <td>How a 0.6B model gets fine-tuned and trained with RL on a MacBook</td> </tr> <tr> <td>Sutton (2019), “The Bitter Lesson”; Silver and Sutton (2025), “Welcome to the Era of Experience”</td> <td>Learning from experience and rewards rather than hand-built rules</td> </tr> <tr> <td>Thaler v Comptroller-General (UK Supreme Court, 2023)</td> <td>An AI can’t be named as an inventor in the UK, so these are ideas for people, not patent filings</td> </tr> </tbody> </table>]]></content><author><name></name></author><category term="reinforcement-learning"/><category term="small-models"/><category term="patents"/><category term="learning-from-experience"/><summary type="html"><![CDATA[How a 0.6B model on my MacBook learned to write new patent claims for kitchen gadgets, tried to cheat twice, and taught me that the reward matters more than the model]]></summary></entry><entry><title type="html">The Generalisation Myth in Enterprise AI, and Why Your Models Don’t Need to Be Massive</title><link href="https://rjb7731.github.io/blog/2026/the-myth-of-enterprise-ai/" rel="alternate" type="text/html" title="The Generalisation Myth in Enterprise AI, and Why Your Models Don’t Need to Be Massive"/><published>2026-11-08T00:00:00+00:00</published><updated>2026-11-08T00:00:00+00:00</updated><id>https://rjb7731.github.io/blog/2026/the-myth-of-enterprise-ai</id><content type="html" xml:base="https://rjb7731.github.io/blog/2026/the-myth-of-enterprise-ai/"><![CDATA[<style>#markdown-content h2,.post-content h2{margin-top:3.25rem;margin-bottom:1rem}#markdown-content h3,.post-content h3{margin-top:2.25rem;margin-bottom:.75rem}#markdown-content table,.post-content table{margin-top:1rem;margin-bottom:1.75rem}#markdown-content ul,.post-content ul{margin-bottom:1.5rem}#markdown-content blockquote,.post-content blockquote{margin:1.5rem 0}#markdown-content details summary,.post-content details summary{cursor:pointer;color:var(--global-theme-color,#2a78d6)}</style> <p>The prevailing assumption around modern language models is that in-context learning alone will seamlessly adapt them to any novel domain. People feed proprietary documents into a prompt and expect the system to function like a seasoned expert right out of the box.</p> <p>The trouble is that deployed models are fundamentally static. When you interact with them in production, their weights are frozen. They do not retain memories, they do not update their parameters, and they do not learn from their mistakes within your operating environment.</p> <h2 id="the-onboarding-problem">The onboarding problem</h2> <p>A great analogy here is onboarding a new hire. You would never expect an employee to operate at full capacity on day one. They need a genuine learning curve to settle in, absorb the company culture, make mistakes, and steadily improve over time.</p> <p>Handing an LLM a context window is much more like handing someone who has never played chess a copy of the official rulebook. Simply reading the rules does not turn someone into a grandmaster capable of beating seasoned players or even a half decent player! Real competence requires trial and error, feedback, and the experience of winning and losing.</p> <h2 id="the-notebook-versus-the-grandmaster">The notebook versus the grandmaster</h2> <p>To get around the fact that models are frozen, leading labs and frameworks often market external memory layers. Whether it is agent skills, persistent memory files, or dynamic scratchpads, the setup is essentially the same. The model jots notes down into an external file, and then it is trained to look up those notes later to keep its context fresh.</p> <p>Think about how absurd that would look over a chessboard.</p> <p>Instead of actually learning from your past games and updating your neural circuits, you sit at the board with a spiral notebook next to you. Every time your opponent makes a move, you pause, flip through your notes to find what you did wrong three games ago, and try to apply it.</p> <p>That is an incredibly brittle way to play. It burns time, it clutters your attention, and the moment you fail to look up the right page, your entire game falls apart. A grandmaster does not play world-class chess by constantly checking a notebook mid-match. They play world-class chess because thousands of hours of trial, error, and feedback have ingrained deep pattern recognition directly into their brain.</p> <p>External memory files are helpful for static references, like looking up an API key or checking documentation. But for actual operational competence, procedural skill cannot remain an external lookup. It has to be baked into the system itself.</p> <h2 id="engineered-breakthroughs-vs-spontaneous-magic">Engineered breakthroughs vs spontaneous magic</h2> <p>Part of this confusion stems from the impressive breakthroughs we see from frontier labs. When models pull off remarkable feats in fields like fluid dynamics, 3D modeling, or higher mathematics, it is easy to assume the model taught itself through sheer generalised intelligence. From there, it is tempting to believe it will adapt to your business processes with that same effortless magic.</p> <p>In reality, those breakthroughs are not spontaneous. They are the direct result of months of deliberate engineering. Dedicated teams design custom reinforcement learning environments, build synthetic training pipelines, and actively update the model weights to instill those exact skills. The models advance because engineers specifically trained them to do so.</p> <blockquote> <p>It is worth stating cleanly, models do not learn while you are using them. In-context learning can only take you so far, and building AI that adds serious enterprise value requires deliberate training and feedback systems, not passive prompting.</p> </blockquote> <h2 id="you-do-not-need-a-trillion-parameters">You do not need a trillion parameters</h2> <p>The upside to this reality is liberating. True, practical intelligence tailored to your domain does not require a monolithic frontier model. You do not need a system trained on fluid dynamics, 3D assets, advanced Python, and protein folding just to process your data, triage communication, or execute workflows in your environment.</p> <p>When you focus on deliberate adaptation for the problem right in front of you, the actual model you need is far smaller, faster, and more efficient than most people think.</p> <h2 id="key-takeaways">Key takeaways</h2> <ul> <li><strong>In-context learning is transient.</strong> Feeding documents to an API prompt gives transient reference material, not institutional memory or persistent parameter updates.</li> <li><strong>Mastery comes from environments, not rulebooks.</strong> High-end model capabilities are built by engineers designing feedback loops and reward systems, not by spontaneous general intelligence.</li> <li><strong>Smaller models win on focused tasks.</strong> Once you stop treating AI as a magical generalist and start designing deliberate adaptation pipelines, compact, task-focused models outperform unwieldy giants at a fraction of the compute cost.</li> </ul>]]></content><author><name></name></author><category term="machine-learning"/><category term="llms"/><category term="enterprise-ai"/><category term="reinforcement-learning"/><summary type="html"><![CDATA[Why in-context learning is not adaptation, why production models remain frozen, and why domain intelligence requires far fewer parameters than you think]]></summary></entry><entry><title type="html">Teaching a Tiny Model to Read Like a Child, and Sit the Year 1 Phonics Check</title><link href="https://rjb7731.github.io/blog/2026/learning-from-scratch/" rel="alternate" type="text/html" title="Teaching a Tiny Model to Read Like a Child, and Sit the Year 1 Phonics Check"/><published>2026-10-30T00:00:00+00:00</published><updated>2026-10-30T00:00:00+00:00</updated><id>https://rjb7731.github.io/blog/2026/learning-from-scratch</id><content type="html" xml:base="https://rjb7731.github.io/blog/2026/learning-from-scratch/"><![CDATA[<style>#markdown-content h2,.post-content h2{margin-top:3.25rem;margin-bottom:1rem}#markdown-content h3,.post-content h3{margin-top:2.25rem;margin-bottom:.75rem}#markdown-content table,.post-content table{margin-top:1rem;margin-bottom:1.75rem}#markdown-content ul,.post-content ul{margin-bottom:1.5rem}#markdown-content blockquote,.post-content blockquote{margin:1.5rem 0}#markdown-content details summary,.post-content details summary{cursor:pointer;color:var(--global-theme-color,#2a78d6)}</style> <p>Code coming soon :)</p> <div class="row align-items-center mt-3"> <div class="col-md-6 mb-3 mb-md-0"> <img src="/assets/img/how_is_it_going.png" alt="The page the model reads: pictures with the words how is it going printed underneath" class="img-fluid rounded z-depth-1" style="width:100%; image-rendering: pixelated;"/> </div> <div class="col-md-6"> <p class="mb-1"><b>What it sees:</b> a page with the words "how is it going" printed underneath, in pixels.</p> <p class="mb-2"><b>What it says:</b> /haʊ ɪs ɪt gɔɪŋ/</p> <audio controls="" preload="none" style="width:100%"> <source src="/assets/audio/how_is_it_going.wav" type="audio/wav"/> </audio> <p class="mt-2 mb-1"><small>It has never seen these words. It sounds them out with the phonics it learned and what it can see with its eyes, so "is" comes out with an s, and it spots the "oi" inside "going", just like a child who has just learned "oi" as in coin.</small></p> <p class="mb-0 text-muted"><small>It does sound a bit creepy, but you would too if you didn't have vocal cords ha! I am reminded a little bit of Rocky from Project Hail Mary "Amaze! Amaze! Amaze!"</small></p> </div> </div> <div class="pb-4"></div> <h2 id="where-this-started">Where this started</h2> <p>I remember being in primary school and I must have been around the age of five, the class I was in we were all collectively reading a banner of the alphabet that was hung up on the wall with the teachers direction. We went through it from a to z, but saying each letter in the simplistic phonetic way to help us learn to read. We made the sounds: /æ/ for apple, /b/ for ball, /k/ for cat. Years later that memory got me thinking. We teach children that way as its the building blocks for being able to read. Could a small model learn to do the same thing, actually speaking the sound it is being taught, and then use those sounds to read words it has never seen before? And how much teaching would it need?</p> <p>It turns out this idea is older than I am:</p> <blockquote> <p>“Instead of trying to produce a programme to simulate the adult mind, why not rather try to produce one which simulates the child’s? If this were then subjected to an appropriate course of education one would obtain the adult brain.” - Alan Turing, “Computing Machinery and Intelligence” (1950)</p> </blockquote> <p>With that idea in mind, and inspired by David Silver’s talk on a recent podcast, I set out to create a machine that can read and speak. I ended up calling the little model <strong>the parrot</strong>, putting it through a simplified version of a UK primary school reading education. It learns words by listening, learns to talk by copying, learns phonics by imitating a teacher reading aloud, and then teaches itself new words by reading picture books. The final exam is the same one every Year 1 child in England sits <strong>the Phonics Screening Check</strong>. Spoiler but my model passes it :)</p> <h2 id="why-bother">Why bother?</h2> <p>The tools we use every day get their skill from enormous amounts of human-labelled data. Whisper learned speech from hundreds of thousands of hours of transcribed audio, and GPT learned language from a huge share of the written internet. A five-year-old has none of that. They learn by listening, babbling, copying the people around them, and being rewarded when it works.</p> <p>I have been getting more and more into reinforcement learning, and this felt like a great battleground for it. Rich Sutton and David Silver argue in “Welcome to the Era of Experience” that agents should learn from their own interaction with a world, rewarded by consequences rather than by copying human judgements. Silver has since started Ineffable Intelligence around this idea, here in the UK, which I think is really exciting. Sutton’s “Bitter Lesson” makes a related point on this too, the methods that win in the long run are the general ones that learn, not the clever ones we hand-build.</p> <p>The rules I set myself:</p> <ul> <li><strong>No answer keys.</strong> The model is never told what a letter sounds like. It hears the teacher, tries to copy, and judges itself with its own ear.</li> <li><strong>Rewards from consequences.</strong> Sounding like the teacher, pointing at the right picture, being understood by a listener.</li> <li><strong>Tiny.</strong> A few megabytes, trained on my MacBook Pro for quick experimentation.</li> <li><strong>Judged like a child.</strong> Not by a phonetic checklist, but by whether our listener hears the right word.</li> </ul> <h2 id="what-is-phonics">What is phonics?</h2> <p>By the time children start school they can already talk. They know thousands of words by ear. Phonics teaches the bridge from print to those sounds: which letters, or groups of letters (graphemes), make which sounds (phonemes). “sh” says /ʃ/, “igh” says /aɪ/, and in “cake” the e at the end changes the a. Once you know the code, you can sound out “f-r-o-g”, blend it, and recognise a word you already know by ear.</p> <p>That last step is the clever part. Another David called David Share called it the self-teaching hypothesis (1995), the idea that every time a child successfully sounds out a word, they have taught themselves that spelling without anyone marking it. Sounding out acts as a built-in teacher. That idea sits at the heart of this project.</p> <p>In England, every Year 1 child (age 5 to 6) sits the <strong>Phonics Screening Check</strong> each June. It is 40 words read aloud to a teacher:</p> <table> <thead> <tr> <th> </th> <th>Section 1</th> <th>Section 2</th> </tr> </thead> <tbody> <tr> <td>Alien words</td> <td>12</td> <td>8</td> </tr> <tr> <td>Real words</td> <td>8</td> <td>12</td> </tr> <tr> <td>Code tested</td> <td>Single letters, ch, sh, th, ng, ck, ar, ee, oi, oo, or…</td> <td>Adds ph, wh, a-e, i-e, ay, ou, ie, ea, aw, ew, ir, ur…, longer clusters (str, spl) and two syllables</td> </tr> </tbody> </table> <p>The alien words are made up (“blarsh”, “quoag”) and shown with a picture of an alien so children cannot rely on words they have just memorised. The pass mark is 32 out of 40. If the model can pass that, it can genuinely decode. We have a target to work towards.</p> <h2 id="the-setup-a-little-world-of-picture-books">The setup: a little world of picture books</h2> <p>First things first, the model needed a world to grow up in. Rather than a dataset, I built a simulation, a bit like AlphaZero’s board where the world is designed, but the skills have to be learned. Building this type of simulations has become a real interest of mine, if we have perfect simulators there is a lot that even small systems can learn just from the environment.</p> <ul> <li><strong>Picture books.</strong> Every page is a 240×320 image with 3 to 5 emoji pictures and a caption like “spot the frog” in a simple pixel font. There are 120 picture words, chosen from the UK Letters and Sounds phases: 72 the model sees in print, 28 it only ever hears until the test (frog, pie, dolphin…), and 20 tricky words where sounding out gives the wrong answer (glove, bread, house, bear, rainbow…).</li> <li><strong>Teacher voices.</strong> Every English voice on my Mac reads every word and line aloud using the built-in <code class="language-plaintext highlighter-rouge">say</code> command, with three voices kept back so we can test on voices it has never heard. A second teacher, espeak, can read anything at all, including alien words, which is what phonics lessons need.</li> <li><strong>The supervisor.</strong> A separate listener network that learns the words from the teacher voices only. It never hears the model during training, then acts as the examiner to check whether it understands what the model says.</li> </ul> <p>The model only ever receives pixels and audio. Sounds never arrive as labels.</p> <div class="row align-items-center mt-3"> <div class="col-md-6 mb-3 mb-md-0"> <img src="/assets/img/spot_the_frog.png" alt="An example page: emoji pictures with the caption spot the frog" class="img-fluid rounded z-depth-1" style="width:100%; image-rendering: pixelated;"/> </div> <div class="col-md-6"> <p class="mb-2"><b>A teacher reading the caption</b></p> <audio controls="" preload="none" style="width:100%"> <source src="/assets/audio/teacher_spot_the_frog.wav" type="audio/wav"/> </audio> <p class="mt-2 mb-0"><small>One of the Mac voices it learns from. During listening it only hears this and has to point at the frog; the print means nothing to it yet.</small></p> </div> </div> <div class="caption text-center"> Figure 1: a page from the model's picture books. Pictures on top, the caption underneath, all one 240×320 image. </div> <h2 id="the-model">The model</h2> <p>The model is two small networks: a <strong>brain</strong> of about 2M parameters and a <strong>voice</strong> of about 2.2M, under 10 MB together.</p> <div class="row justify-content-center"> <div class="col-sm-10 col-md-6 mt-3 mt-md-0"> <figure> <picture> <source class="responsive-img-srcset" srcset="/assets/img/architecture-480.webp 480w,/assets/img/architecture-800.webp 800w,/assets/img/architecture-1400.webp 1400w," type="image/webp" sizes="95vw"/> <img src="/assets/img/architecture.png" class="img-fluid rounded z-depth-1" width="100%" height="auto" title="How the parrot reads and learns" loading="lazy" onerror="this.onerror=null; $('.responsive-img-srcset').remove();"/> </picture> </figure> </div> </div> <div class="caption text-center"> Figure 2: how the model reads, top to bottom. The pink path is the sounding-out loop: it commits to real sounds, says them in its own voice, and hears itself. The dotted arrows are the two reinforcement learning rewards that train the reading head. Not shown: a sight-word route straight from the print to the picture, and the small helpers used for self-teaching. </div> <details class="mb-4"> <summary>The layers in more detail</summary> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>THE BRAIN (d = 128, ~2M parameters)
  pictures ─► Scene encoder: 5-layer CNN → 220 patch tokens → 2-layer transformer
  print ────► Print encoder: CNN keeping 4-px columns → 80 column tokens → 2-layer transformer
                ├─► Reading head: per-column classifier over 44 classes (42 sounds, pause, blank), CTC
                │     → a sound sequence /f r ɒ g/ → THE VOICE → spectrogram
                │     → Ear: two strided 1-D convs (40 ms steps) → 2-layer transformer → attention pool
                │     → query · picture patches → points at a picture
                └─► Sight-word route: attention pool over the print columns → query · patches → points too
  Namer: picture features → its spoken name (GRU)
  Echo:  ear frames → which of its own sounds made them (CTC), learned from babbling

THE VOICE (d = 192, ~2.2M parameters)
  sounds + whose voice → 3-layer transformer encoder → duration predictor (alignment learned by
  monotonic alignment search) → repeat each sound for its frames → 3 conv blocks + 2-layer transformer
  → 64-band log-mel spectrogram, 10 ms frames → Griffin-Lim → audio (only for our ears)
</code></pre></div> </div> </details> <p>A few design choices matter more than the rest:</p> <ul> <li><strong>The bottleneck is discrete.</strong> The reading head has to commit to a sequence of real sounds, which go out through the voice and back in through the ear. Rewards for finding the right picture cannot sneak through as some private code, so what it says always stays real speech that a listener can mark. This is the sounding-out loop.</li> <li><strong>It hears itself with the ear it learned words with.</strong> Reading only works if what it says sounds like a word it already knows by ear, exactly like a child blending “f-r-o-g” and recognising it.</li> <li><strong>Two routes to meaning.</strong> Sounding out, and recognising familiar words on sight, following the dual-route model of human reading (Coltheart et al.). The sight route cannot reshape the eyes, so memorising words cannot corrupt the decoding skill.</li> <li><strong>The echo is its own inverse model.</strong> It learns which of its sounds made what it heard, purely from its own babbling, never from teacher labels. That is what makes phonics without an answer key possible.</li> </ul> <h2 id="growing-up">Growing up</h2> <p>The model grows up in stages, a bit like a child, each one learning from a different kind of experience. The whole thing runs end to end on my MacBook in a few hours.</p> <p><strong>1. Listening (the early years).</strong> Teacher voices talk about the pictures on the page: “spot the frog”. The ear learns to point at the right picture, using a contrastive objective where the query from the audio has to match the right picture patch out of 220. It also learns to name pictures and repeat what it heard. No printed words yet.</p> <p><strong>2. Finding its voice.</strong> The neural voice learns to talk by mimicking recordings. Given the sounds that were said, it draws the spectrogram and learns from the difference (L1 loss), working out where each sound starts and stops on its own. Its own voice starts as Jamie, a natural-sounding British Mac voice, and is then polished two ways. A small critic network learns to tell its spectrograms apart from real ones, teaching the voice to fool the critic (a GAN-style loss that removes the smeared, averaged sound), and it practises saying words until the supervisor understands them.</p> <p><strong>3. Babbling.</strong> Like an infant, it makes random strings of sounds and listens to itself. From pairs of “what I said” and “what I heard” it learns the echo, an inverse model of its own mouth. No teacher involved.</p> <p><strong>4. Phonics by imitation.</strong> Now for the alphabet frieze. The teacher reads a letter card, then words, then whole lines aloud. The model reads the same print aloud, trying 4 different readings. Each attempt is spoken, and the reward is how close it sounds to the teacher, measured by dynamic time warping over MFCC features so it matches sound rather than symbols. Better-than-average attempts are reinforced and worse ones discouraged through a group-relative REINFORCE update, similar in spirit to GRPO. On top of that, when the echo is confident it can transcribe what the teacher said into its own sounds, that becomes a target to copy. It is learning phonics with no answer key, judged only by its own ear.</p> <p><strong>5. Reading practice.</strong> It reads picture books aloud to itself in its own voice, hears itself, and points at the picture it thinks the words describe. The reward is pointing at the right picture (REINFORCE again). This is where self-teaching kicks in: when it points at the bread, its namer knows the bread is called /bred/, and that becomes a lesson for how that print should be read. Phonics practice carries on alongside so it does not forget the code.</p> <h2 id="what-broke-along-the-way">What broke along the way</h2> <p>Most of what I learned came from things failing, and nearly every fix replaced something I had hand-built with something learned. The bitter lesson, in miniature. Claude Opus 5.5 was a big help too.</p> <ul> <li><strong>Pure trial and error stalled.</strong> My first go at phonics was pure RL: try sounds, get scored. It got 1.6% of letter cards right. Adding babbling first, so it knows what its own mouth sounds like before it tries to copy anyone, took letter cards to 72%, and alien-word decoding later reached 97%. Infants babble for months before they copy words, and it turns out there is a good reason.</li> <li><strong>My hand-built voice box sounded awful.</strong> I built a little source-filter synthesiser (voicing, hiss, pitch, mouth resonances) and had the model learn to drive it by trial and error. The supervisor understood about half its words, and it sounded like a broken robot. A learned neural voice copying real recordings was far clearer straight away.</li> <li><strong>Switching voices wiped out phonics.</strong> When I retrained phonics through the new voice, decoding dropped from 97% to 3%. Its ear retuned to the new voice and lost the code. The fix was to learn phonics hearing itself through a clear synthetic voice, then switch to its own voice for reading practice while continuing phonics drills.</li> <li><strong>Words ran together.</strong> Early readings came out as one long word (/ðəhen/ for “the hen”). Learning pauses between words in its own speech fixed it.</li> <li><strong>A learned vocoder did not help yet.</strong> I trained a small Vocos-style network to turn spectrograms into audio instead of Griffin-Lim. It sounded great on real voices, but no better on the model’s, because the blur came from the voice itself. That is what the critic fixed.</li> </ul> <h2 id="results">Results</h2> <h3 id="the-first-model-48-words-phase-2-to-3">The first model: 48 words, Phase 2 to 3</h3> <p>My first model learned a smaller world: 48 picture words and the early phonics code. Every test was on first sight, with no learning during the test.</p> <table> <thead> <tr> <th>Test</th> <th>Result</th> </tr> </thead> <tbody> <tr> <td>Alien words it had never seen (blarsh, quoag), read exactly right</td> <td>91.3%</td> </tr> <tr> <td>A listener picks the right alien word from near-misses one sound away</td> <td>98.0%</td> </tr> <tr> <td>Words it had only ever heard, read aloud correctly the first time it saw them in print</td> <td>93.8%</td> </tr> <tr> <td>The supervisor understands its reading aloud (practised / tricky / never printed)</td> <td>96.5% / 98.0% / 85.3%</td> </tr> </tbody> </table> <p>Two checks convinced me it is really reading. With the caption removed, it points at the right picture 25.8% of the time, roughly chance with 3 to 5 pictures, showing it is not guessing from the artwork. Also, the sight route on its own reads familiar words at 91% but new words at 16%. Familiar words move to recognition on sight, while new words can only be sounded out. That two-route split is what reading research describes in people, and it appeared naturally.</p> <h3 id="my-favourite-moment-it-taught-itself-the-tricky-words">My favourite moment: it taught itself the tricky words</h3> <table> <thead> <tr> <th>Page</th> <th>After listening</th> <th>After phonics</th> <th>After reading practice</th> </tr> </thead> <tbody> <tr> <td>spot the bread</td> <td>babble</td> <td>/bræd/, wrong picture</td> <td>/bred/, right picture</td> </tr> <tr> <td>spot the glove</td> <td>babble</td> <td>/glɒvə/, wrong picture</td> <td>/glʌv/, right picture</td> </tr> </tbody> </table> <p>After phonics it read these the way a beginner does, strictly by the rules, and got them wrong. After reading practice it read them right. Nobody told it they were exceptions. /bræd/ matched no word it knew by ear, /bred/ did, and the bread was on the page. That is Share’s self-teaching loop happening inside a model.</p> <div class="row mt-3"> <div class="col-md-4 mb-3"> <div class="p-3 rounded z-depth-1 h-100"> <p class="mb-1"><b>1. After listening</b></p> <p class="mb-2"><code>babble</code></p> <audio controls="" preload="none" style="width:100%"> <source src="/assets/audio/glove_after_listening.wav" type="audio/wav"/> </audio> </div> </div> <div class="col-md-4 mb-3"> <div class="p-3 rounded z-depth-1 h-100"> <p class="mb-1"><b>2. After phonics</b></p> <p class="mb-2"><code>by the rules: /gləʊv/</code> <small>(magic e, like "clove")</small></p> <audio controls="" preload="none" style="width:100%"> <source src="/assets/audio/glove_after_phonics.wav" type="audio/wav"/> </audio> </div> </div> <div class="col-md-4 mb-3"> <div class="p-3 rounded z-depth-1 h-100"> <p class="mb-1"><b>3. After reading practice</b></p> <p class="mb-2"><code>self-taught: /glʌv/</code></p> <audio controls="" preload="none" style="width:100%"> <source src="/assets/audio/glove_after_reading_practice.wav" type="audio/wav"/> </audio> </div> </div> </div> <div class="caption text-center"> The same page, "spot the glove", read by the model at each stage of growing up. </div> <h3 id="the-phonics-check">The phonics check</h3> <p>I sat an earlier version of model down for the test before teaching it the rest of the code. It scored <strong>26.8/40</strong> on average over 10 papers: 18.4/20 on section 1, which uses the code it had learned, but 8.4/20 on section 2, which it had never been taught. Not a pass.</p> <p>So I ‘grew’ a new model in a bigger world which had all 120 words and the full Phase 2 to 5 code the check covers. Each paper is freshly generated to the official structure so it cannot be memorised, and it was marked like a teacher would do it. It only scores if it chose the right sounds <strong>and</strong> said them clearly enough to be heard as the right word.</p> <table> <thead> <tr> <th> </th> <th>Section 1</th> <th>Section 2</th> <th>Total</th> <th>Passed</th> </tr> </thead> <tbody> <tr> <td>First model (Phase 2-3)</td> <td>18.4/20</td> <td>8.4/20</td> <td>26.8/40</td> <td>0 of 10 papers</td> </tr> <tr> <td>After phonics</td> <td>19.8/20</td> <td>16.8/20</td> <td><strong>36.6/40</strong></td> <td><strong>10 of 10 papers</strong></td> </tr> <tr> <td>After reading practice</td> <td>19.6/20</td> <td>17.0/20</td> <td><strong>36.6/40</strong></td> <td><strong>10 of 10 papers</strong></td> </tr> </tbody> </table> <p>It passed every one of the 10 papers. Its lowest paper was 34/40 and its best 38/40, which is about as high as it can realistically go. Each paper has two real words with an alternative pronunciation (like “snow” or “head”) that you can only get right if you already know the word, and the model has never heard them. Its voice never marked it down, every word it chose the right sounds for was heard as the right word. The spellings it still slips on are y, the split digraph i-e and igh, which are the same ones that trip up a lot of children.</p> <h3 id="how-much-teaching-did-it-need">How much teaching did it need?</h3> <p>This is the number I care about most for what comes next. During training I kept a snapshot every 500 steps and sat the check on each one, plotting the score against how many times the model had heard the teacher read aloud.</p> <div class="row justify-content-center"> <div class="col-sm-12 col-md-10 mt-3 mt-md-0"> <figure> <picture> <source class="responsive-img-srcset" srcset="/assets/img/learning_curve-480.webp 480w,/assets/img/learning_curve-800.webp 800w,/assets/img/learning_curve-1400.webp 1400w," type="image/webp" sizes="95vw"/> <img src="/assets/img/learning_curve.png" class="img-fluid rounded z-depth-1" width="100%" height="auto" title="Learning curve" loading="lazy" onerror="this.onerror=null; $('.responsive-img-srcset').remove();"/> </picture> </figure> </div> </div> <div class="caption text-center"> Figure 3: the phonics check score against how many times the model had heard the teacher read aloud. It passes the 32/40 mark after 60,000 readings. </div> <p>It passed after hearing the teacher read aloud <strong>60,000 times</strong>. For the first 24,000 the score sits at zero: those lessons are mostly letter cards, learning single sounds, and the check needs whole words. Then it climbs steeply, passes at 60,000, and plateaus at around 36/40 by 84,000. The remaining 60,000 readings in the phonics stage added about one mark, and reading practice afterwards added a little more on top (37.3/40 on the curve’s papers), mostly helping it point at pictures and read the tricky words rather than the check itself. 60,000 is the baseline I now want to drive down.</p> <h2 id="what-it-was-given-and-what-it-learned">What it was given, and what it learned</h2> <p>Every learning system starts with priors. AlphaZero was handed the rules of Go, and this model lives in a world I designed around English. Let’s look at where that line actually is.</p> <table> <thead> <tr> <th> </th> <th>Learned from experience</th> <th>Given to it</th> </tr> </thead> <tbody> <tr> <td>Hearing</td> <td>What words sound like, in any voice</td> <td>None</td> </tr> <tr> <td>Phonics</td> <td>Which letters make which sounds, by imitating the teacher</td> <td>None</td> </tr> <tr> <td>Reading</td> <td>Sounding out, sight words, the tricky words</td> <td>None</td> </tr> <tr> <td>Speaking</td> <td>What each of its sounds sounds like, how long it lasts, its own voice</td> <td>42 sound “slots” to choose between: it learned what each one sounds like, but not that English has 42. The voice also learned to talk from teacher recordings labelled with the sounds in them</td> </tr> <tr> <td>Hearing itself</td> <td>None</td> <td>During phonics it heard itself through a clear synthetic voice, then switched to its own</td> </tr> <tr> <td>Sound for us</td> <td>None</td> <td>Griffin-Lim, a fixed method that turns its spectrograms into audio for our ears. The model itself only ever hears spectrograms</td> </tr> </tbody> </table> <p>The biggest given is the sound inventory. A baby discovers the sound categories of its language by listening in its first year, before any school. The model skips that. Babies are probably born expecting speech to be made of reusable units, but which units, and how many, is learned. Letting the model discover its own units is the obvious next frontier.</p> <p>Of course it is small: 120 words, synthetic voices, one font. Nothing here shows it would cope with a real children’s book yet.</p> <h2 id="whats-next">What’s next</h2> <p>The part I am most excited about is efficiency. A child arrives at school already talking, and learns the phonics code in a year or so of short daily lessons. The model needed to hear the teacher read aloud 60,000 times to pass, and was given 144,000 in total. Now that the learning curve measures this, the aim is to drive that number down in human-like ways:</p> <ul> <li><strong>Teach in order, like the alphabet frieze.</strong> s, a, t, p, i, n first, instead of everything mixed from day one.</li> <li><strong>Practise what it gets wrong.</strong> Take the time to focus on trouble spots, the way a teacher spends longer on “ay” than on “s”. This is where the RL and self-play angle really comes in.</li> <li><strong>One creature, one life.</strong> Today the brain and voice are separate networks trained in separate stages. Joining them, so it learns to speak through its own ear and reading and speaking improve each other, is much closer to how a child develops.</li> <li><strong>Discover its own sounds</strong> from babbling and listening, instead of being handed 42 slots.</li> <li><strong>Read a whole page,</strong> moving its eyes across lines, and eventually a real decodable reading book.</li> </ul> <p>Turing suggested building a child machine and educating it. Seventy-five years later, a few megabytes on a laptop can sit the same reading check as a six-year-old. And it passes, 36.6 out of 40.</p> <h2 id="prior-work">Prior work</h2> <p>Links still to add before posting.</p> <table> <thead> <tr> <th>Work</th> <th>Why it matters here</th> </tr> </thead> <tbody> <tr> <td>Silver and Sutton (2025), “Welcome to the Era of Experience”</td> <td>The framing: agents learning from their own experience, rewarded by consequences</td> </tr> <tr> <td>Silver, Singh, Precup and Sutton (2021), “Reward is Enough”</td> <td>Rewards from the world (being understood, the right picture) instead of answer keys</td> </tr> <tr> <td>Sutton (2019), “The Bitter Lesson”</td> <td>Why the hand-built voice box lost to a learned one</td> </tr> <tr> <td>Share (1995), the self-teaching hypothesis</td> <td>The core idea: sounding out acts as a built-in teacher</td> </tr> <tr> <td>Coltheart et al. (2001), the dual-route cascaded model</td> <td>Two routes, sounding out vs sight words, which the results reproduce</td> </tr> <tr> <td>Seidenberg and McClelland (1989), Plaut et al. (1996)</td> <td>Earlier neural reading models, which learned spelling-to-sound from labels</td> </tr> <tr> <td>Kim et al. (2020), Glow-TTS; Siuzdak (2023), Vocos</td> <td>The alignment trick behind the voice, and the vocoder design</td> </tr> <tr> <td>Letters and Sounds (DfE, 2007), Year 1 Phonics Screening Check</td> <td>The UK phonics phases the words follow, and the alien-word test</td> </tr> <tr> <td>Turing (1950), “Computing Machinery and Intelligence”</td> <td>The child machine: build a child’s mind and educate it</td> </tr> <tr> <td>Ziegler, Perry and Zorzi (2014), Pritchard et al. (2018)</td> <td>The closest predecessors: computational models of self-teaching, working on letters and sound symbols rather than pixels and audio</td> </tr> <tr> <td>Dehaene, “Reading in the Brain” (2009)</td> <td>Neuronal recycling: reading repurposes visual cortex and wires it to spoken language</td> </tr> </tbody> </table>]]></content><author><name></name></author><category term="reinforcement-learning"/><category term="world-models"/><category term="learning-from-experience"/><summary type="html"><![CDATA[How I trained a tiny model to teach itself to pass the Year 1 phonics check with no labels, no answer key]]></summary></entry><entry><title type="html">Beyond Static Embeddings: Training a 6,000-Parameter RL Agent to Actively Hunt Down Historical Planning Maps</title><link href="https://rjb7731.github.io/blog/2026/osmapping-rl/" rel="alternate" type="text/html" title="Beyond Static Embeddings: Training a 6,000-Parameter RL Agent to Actively Hunt Down Historical Planning Maps"/><published>2026-10-15T19:10:00+01:00</published><updated>2026-10-15T19:10:00+01:00</updated><id>https://rjb7731.github.io/blog/2026/osmapping-rl</id><content type="html" xml:base="https://rjb7731.github.io/blog/2026/osmapping-rl/"><![CDATA[<p><strong>Created on October 15, 2026</strong> <code class="language-plaintext highlighter-rouge">2026</code> · <code class="language-plaintext highlighter-rouge">reinforcement-learning</code> · <code class="language-plaintext highlighter-rouge">markov-decision-processes</code> · <code class="language-plaintext highlighter-rouge">computer-vision</code> · <code class="language-plaintext highlighter-rouge">sovereign-ai</code></p> <p><em>Why throw a huge LLM or brute-force grid search at a spatial business problem when you can frame it as a Markov Decision Process (MDP) and solve it in milliseconds with a tiny policy network in pure NumPy?</em></p> <div class="row"> <div class="col-sm mt-3 mt-md-0"> <figure> <picture> <source class="responsive-img-srcset" srcset="/assets/img/agent_search-480.webp 480w,/assets/img/agent_search-800.webp 800w,/assets/img/agent_search-1400.webp 1400w," type="image/webp" sizes="95vw"/> <img src="/assets/img/agent_search.gif" class="img-fluid rounded z-depth-1" width="100%" height="auto" title="Agent Search in Action" loading="lazy" onerror="this.onerror=null; $('.responsive-img-srcset').remove();"/> </picture> </figure> </div> </div> <div class="caption"> Figure 1: Our 6,000-parameter Decision Encoder actively exploring real Ordnance Survey geometry in Trafford. Starting from a rough street-level geocode (blue cross), it zooms out, spots a candidate peak across the neighbourhood, flies over, zooms in to verify, and commits to the exact Land Registry plot (green). </div> <div class="row"> <div class="col-sm mt-3 mt-md-0"> <figure> <picture> <source class="responsive-img-srcset" srcset="/assets/img/agent_search_v0-480.webp 480w,/assets/img/agent_search_v0-800.webp 800w,/assets/img/agent_search_v0-1400.webp 1400w," type="image/webp" sizes="95vw"/> <img src="/assets/img/agent_search_v0.gif" class="img-fluid rounded z-depth-1" width="100%" height="auto" title="Agent Search in Action" loading="lazy" onerror="this.onerror=null; $('.responsive-img-srcset').remove();"/> </picture> </figure> </div> </div> <p>Back in April, I wrote about training a 49-million-parameter contrastive ResNet50 model to geolocate planning documents across Greater Manchester. That approach treated map localization like a giant one-shot classification problem: chop Greater Manchester into 3,478 reference tiles, embed the document, run a matrix multiplication, and hope the nearest tile embedding is the right one.</p> <p>It worked well as a proof of concept, but as I dug deeper into real-world council archives and started following Sergey Levine’s Deep Reinforcement Learning course (CS285) at UC Berkeley, I realized that <strong>one-shot static perception is the wrong abstraction for how real localization works</strong>.</p> <p>When a human caseworker—or an intelligent system—tries to locate a messy, historical planning document on a map, they don’t take a single glance at the entire city and guess. They <strong>investigate</strong>:</p> <ol> <li>They start at a rough clue (an address, a postcode, or a street name).</li> <li>They look around at a medium zoom. If the street layout doesn’t match, they <strong>zoom out</strong> to get their bearings.</li> <li>If the title block says <code class="language-plaintext highlighter-rouge">Scale 1:1250</code>but nothing lines up, they suspect the OCR or title block is wrong and <strong>test a <code class="language-plaintext highlighter-rouge">1:2500</code> scale hypothesis</strong>.</li> <li>In repetitive suburban estates where three cul-de-sacs look almost identical, they <strong>jump between rival candidates</strong> at fine resolution to compare them side by side.</li> <li>Most importantly, if the scan is too degraded or ambiguous, they <strong>know when to escalate to a human</strong> rather than corrupting the database with a bad guess.</li> </ol> <p>Instead of jumping on the LLM wrapper bandwagon, I wanted to see if we could turn this exact operational workflow into a <strong>Markov Decision Process (MDP)</strong> and train a tiny neural network via <strong>Imitation Learning + Reinforcement Learning (REINFORCE)</strong> to actively explore the map.</p> <hr/> <h2 id="step-1-framing-business-economics-as-an-mdp">Step 1: Framing Business Economics as an MDP</h2> <p>In reinforcement learning, an agent interacts with an environment over discrete time steps: observing a state $s_t$, picking an action $a_t$ from a menu, and receiving a reward $r_t$.</p> <p>To test this properly without relying on clean synthetic grids, I built an environment on top of <strong>real Ordnance Survey OpenMap Local buildings, Open Roads, and HM Land Registry INSPIRE title polygons</strong> across Trafford and Greater Manchester.</p> <h3 id="the-harder-reality-hand-drawn-scans--demolished-buildings">The Harder Reality: Hand-Drawn Scans &amp; Demolished Buildings</h3> <p>Lets be honest modern digital plans are easy. The real headache for councils is historical plans they are documents traced by hand decades ago against older maps, where terraced houses were merged into single blocks, pen lines wobble, and <strong>10% to 30% of the buildings on the map have since been demolished or newly built</strong>.</p> <p>On top of visual degradation, the starting address prior is often unreliable:</p> <ul> <li><strong>UPRN (45%):</strong> Near the property, though 12% of the time it geocodes to the wrong house up to 200m away.</li> <li><strong>Postcode (25%):</strong> Anywhere within a 100m to 350m radius of the site.</li> <li><strong>Street (15%):</strong> Dropped at a random midpoint along the named road.</li> <li><strong>None (15%):</strong> Dropped at a completely random coordinate in a 2.4km $\times$ 2.4km town.</li> </ul> <h3 id="the-action-space-17-discrete-decisions">The Action Space (17 Discrete Decisions)</h3> <p>At each step (up to a budget of 16 looks), the agent views a local crop of the map through a fast Normalized Cross-Correlation (NCC) template matcher and chooses from 17 actions:</p> <ul> <li><strong>Navigate:</strong> <code class="language-plaintext highlighter-rouge">pan_N</code>, <code class="language-plaintext highlighter-rouge">pan_E</code>, <code class="language-plaintext highlighter-rouge">pan_S</code>, <code class="language-plaintext highlighter-rouge">pan_W</code>, <code class="language-plaintext highlighter-rouge">zoom_in</code>, <code class="language-plaintext highlighter-rouge">zoom_out</code> (switching between 960m coarse, 480m medium, and 240m fine viewports), or <code class="language-plaintext highlighter-rouge">goto_peak</code> (center on the highest correlation peak in the current view).</li> <li><strong>Investigate Rivals:</strong> The environment keeps a short-term memory of the top 3 distinct match locations seen so far. Picking <code class="language-plaintext highlighter-rouge">check_0</code>, <code class="language-plaintext highlighter-rouge">check_1</code>, or <code class="language-plaintext highlighter-rouge">check_2</code> jumps straight to that candidate at the finest zoom level to verify it.</li> <li><strong>Gather Clues &amp; Test Hypotheses:</strong> <code class="language-plaintext highlighter-rouge">read_street</code> runs OCR on the plan to unlock street jump candidates (<code class="language-plaintext highlighter-rouge">jump_0..2</code>), while <code class="language-plaintext highlighter-rouge">switch_scale</code> flips the agent’s scale hypothesis between <code class="language-plaintext highlighter-rouge">1:1250</code> and <code class="language-plaintext highlighter-rouge">1:2500</code> (since 25% of plans have misread or missing scale text).</li> <li><strong>Terminal Actions (<code class="language-plaintext highlighter-rouge">commit</code> vs. <code class="language-plaintext highlighter-rouge">escalate</code>):</strong> Committing runs a sub-pixel registration and checks if the plan’s red boundary polygon overlaps the true Land Registry plot ($\text{IoU} \ge 0.5$). Escalating stops the search and flags the document for human review.</li> </ul> <hr/> <h2 id="step-2-the-brain-a-6000-parameter-decision-encoder-in-pure-numpy">Step 2: The Brain (A 6,000-Parameter Decision Encoder in Pure NumPy)</h2> <p>Does the RL policy need to be a massive Vision Transformer? <strong>No.</strong></p> <p>The template matcher already turns image comparisons into spatial correlation surfaces. What the agent actually needs is a <strong>sequential decision policy</strong> that looks at the <em>summary statistics</em> of the search so far and decides what to do next.</p> <p>I built a <code class="language-plaintext highlighter-rouge">DecisionEncoder</code> in pure NumPy—a two-layer, 64-unit <code class="language-plaintext highlighter-rouge">tanh</code> Multi-Layer Perceptron (MLP) with a companion Value Network baseline:</p> <ul> <li>At each step, it takes a 48-dimensional global situation vector $G$ (current zoom, remaining step budget, current peak score, peak distinctiveness over the runner-up, distance from the geocode, scale state, and the score lead of the top remembered candidates) concatenated with a 21-dimensional per-action feature vector $A[a]$.</li> <li>Because it operates on structured state-action features <code class="language-plaintext highlighter-rouge">[situation; option]</code>, the entire policy network is roughly <strong>6,000 parameters</strong>, runs in microseconds on a CPU, and requires zero deep learning frameworks.</li> </ul> <hr/> <h2 id="step-3-sft-then-rl-why-rl-from-scratch-fails">Step 3: SFT Then RL (Why RL From Scratch Fails)</h2> <p>To train the policy, I used the two-stage pipeline standard in modern RL:</p> <ol> <li><strong>Supervised Fine-Tuning / Behavior Cloning (SFT):</strong> First, I wrote two rule-based heuristic agents (<code class="language-plaintext highlighter-rouge">coarse_to_fine</code> and a more cautious <code class="language-plaintext highlighter-rouge">smart_expert</code> that double-checks rival peaks) and recorded 2,400 episodes (~20,500 expert decisions) on a 2.4km $\times$ 2.4km training area in Trafford. The MLP is trained via cross-entropy to imitate the expert (<code class="language-plaintext highlighter-rouge">96.8%</code> action accuracy).</li> <li><strong>REINFORCE with a Learned Value Baseline:</strong> Starting from the cloned weights, the agent plays its own rollouts. For every trajectory $\tau$, we compute the reward-to-go $R_t = \sum_{t’=t}^{T} r_{t’}$ and subtract the Value Network’s expected return $V_\phi(s_t)$ to get the <strong>Advantage</strong>:</li> </ol> \[A_t = R_t - V_\phi(s_t)\] <p>We then update the policy weights along the policy gradient, combined with an entropy bonus $\mathcal{H}(\pi_\theta)$ to encourage exploration and annealed potential-based distance shaping:</p> \[\nabla_\theta J(\theta) \approx \frac{1}{N} \sum_{i, t} \nabla_\theta \log \pi_\theta(a_{i,t} \mid s_{i,t}) \cdot A_{i,t} + \beta \nabla_\theta \mathcal{H}(\pi_\theta(\cdot \mid s_{i,t}))\] <h3 id="the-always-escalate-trap-when-training-rl-from-scratch">The “Always Escalate” Trap When Training RL From Scratch</h3> <p>What happens if you skip Behavior Cloning and train REINFORCE from random weights (<code class="language-plaintext highlighter-rouge">--scratch</code>)?</p> <p>In our experiments, a randomly initialized agent wanders around the map and makes random commits, losing <code class="language-plaintext highlighter-rouge">-1.0</code> eighty percent of the time (for an average starting return of <code class="language-plaintext highlighter-rouge">-0.66</code>). Within 50 iterations, <strong>RL-from-scratch discovers a clever, degenerate local optimum</strong>: if it simply presses <code class="language-plaintext highlighter-rouge">escalate</code> on Step 1 of every single episode, it collects a guaranteed <code class="language-plaintext highlighter-rouge">-0.35</code> reward with <code class="language-plaintext highlighter-rouge">0%</code> wrong commits! Once it collapses into 100% instant escalation, it never experiences a <code class="language-plaintext highlighter-rouge">+1.0</code> commit again.</p> <p>Warm-starting with Behavior Cloning (<code class="language-plaintext highlighter-rouge">SFT</code>) places the policy into the region of state space where it actually finds maps, and <strong>REINFORCE (<code class="language-plaintext highlighter-rouge">SFT+RL</code>)</strong> then refines its decision boundaries far beyond the expert it copied.</p> <div class="row"> <div class="col-sm mt-3 mt-md-0"> <figure> <picture> <source class="responsive-img-srcset" srcset="/assets/img/agent_search_v2-480.webp 480w,/assets/img/agent_search_v2-800.webp 800w,/assets/img/agent_search_v2-1400.webp 1400w," type="image/webp" sizes="95vw"/> <img src="/assets/img/agent_search_v2.png" class="img-fluid rounded z-depth-1" width="100%" height="auto" title="Agent Search in Action" loading="lazy" onerror="this.onerror=null; $('.responsive-img-srcset').remove();"/> </picture> </figure> </div> </div> <div class="caption"> Figure 2: Step-by-step filmstrip of the SFT+RL agent locating a degraded hand-drawn plan (`hand` style, where buildings have changed since the plan was drawn). Each panel shows the agent's local viewport crop, the matcher peak (orange crosshair), and the Decision Encoder's live action probability distribution._ </div> <hr/> <h2 id="results-evaluating-on-unseen-trafford-map-geometry">Results: Evaluating on Unseen Trafford Map Geometry</h2> <p>I trained the system on 1,200 plans in one 2.4km $\times$ 2.4km area of Trafford (<code class="language-plaintext highlighter-rouge">BNG 374000, 393000</code>) and evaluated every method on the exact same 400 unseen test episodes in a separate 2.4km $\times$ 2.4km town (<code class="language-plaintext highlighter-rouge">BNG 377000, 391000</code>).</p> <table> <thead> <tr> <th>Method</th> <th>Correct (Recall)</th> <th>Wrong (Bad Commit)</th> <th>Escalate</th> <th>Avg Looks</th> <th>Precision</th> <th>Total Reward</th> </tr> </thead> <tbody> <tr> <td><strong>Geocode Commit</strong> (no search)</td> <td>32.8%</td> <td>17.0%</td> <td>50.2%</td> <td>1.0</td> <td>65.8%</td> <td><code class="language-plaintext highlighter-rouge">-0.018</code></td> </tr> <tr> <td><strong>Exhaustive Grid</strong> (1 scale)</td> <td>29.5%</td> <td>1.3%</td> <td>69.3%</td> <td>26.0</td> <td>95.9%</td> <td><code class="language-plaintext highlighter-rouge">-0.460</code></td> </tr> <tr> <td><strong>Exhaustive Grid</strong> (both scales)</td> <td>30.0%</td> <td>1.8%</td> <td>68.2%</td> <td>51.0</td> <td>94.5%</td> <td><code class="language-plaintext highlighter-rouge">-0.956</code></td> </tr> <tr> <td><strong>Coarse-to-Fine</strong> (heuristic)</td> <td>65.7%</td> <td>5.2%</td> <td>29.0%</td> <td>6.7</td> <td>92.6%</td> <td><code class="language-plaintext highlighter-rouge">+0.355</code></td> </tr> <tr> <td><strong>Smart Expert</strong> (heuristic)</td> <td>58.5%</td> <td>2.5%</td> <td>39.0%</td> <td>8.3</td> <td>95.9%</td> <td><code class="language-plaintext highlighter-rouge">+0.267</code></td> </tr> <tr> <td><strong>Pure SFT</strong> (Behavior Cloning)</td> <td>59.0%</td> <td>2.5%</td> <td>38.5%</td> <td>8.2</td> <td>95.9%</td> <td><code class="language-plaintext highlighter-rouge">+0.265</code></td> </tr> <tr> <td><strong>SFT + RL (REINFORCE)</strong></td> <td><strong>76.2%</strong></td> <td><strong>7.0%</strong></td> <td><strong>16.8%</strong></td> <td><strong>5.2</strong></td> <td><strong>91.6%</strong></td> <td><strong><code class="language-plaintext highlighter-rouge">+0.548</code></strong></td> </tr> </tbody> </table> <p><em>Paired 95% bootstrap confidence interval of <strong>SFT + RL</strong> over the best hand-written baseline (<code class="language-plaintext highlighter-rouge">coarse_to_fine</code>): **Reward <code class="language-plaintext highlighter-rouge">+0.193</code> <code class="language-plaintext highlighter-rouge">[+0.126, +0.254]**</code>, **Correct <code class="language-plaintext highlighter-rouge">+10.5%</code> <code class="language-plaintext highlighter-rouge">[+6.5%, +14.2%]**</code>, while taking **1.5 fewer looks** per plan.</em></p> <h3 id="where-rl-crushes-rules-hand-drawn--historical-plans-hand-style">Where RL Crushes Rules: Hand-Drawn &amp; Historical Plans (<code class="language-plaintext highlighter-rouge">hand</code> style)</h3> <p>When we break down the results by plan style and starting address quality, we see why learned sequential policies beat hand-tuned thresholds:</p> <table> <thead> <tr> <th>Subset</th> <th>Coarse-to-Fine</th> <th>Smart Expert</th> <th>Pure SFT (Cloned)</th> <th><strong>SFT + RL (Ours)</strong></th> </tr> </thead> <tbody> <tr> <td><strong>Hand-Drawn + Demolished Buildings (<code class="language-plaintext highlighter-rouge">hand</code>)</strong></td> <td>66% <em>(7.0 steps)</em></td> <td>60% <em>(8.4 steps)</em></td> <td>56% <em>(8.8 steps)</em></td> <td><strong>78% <em>(5.3 steps, 92% prec)</em></strong></td> </tr> <tr> <td><strong>Modern Colour (<code class="language-plaintext highlighter-rouge">os_colour</code>)</strong></td> <td>65% <em>(6.6 steps)</em></td> <td>55% <em>(8.6 steps)</em></td> <td>59% <em>(8.0 steps)</em></td> <td><strong>78% <em>(4.9 steps, 89% prec)</em></strong></td> </tr> <tr> <td><strong>Modern Monochrome (<code class="language-plaintext highlighter-rouge">os_mono</code>)</strong></td> <td>66% <em>(6.6 steps)</em></td> <td>60% <em>(8.1 steps)</em></td> <td>60% <em>(7.9 steps)</em></td> <td><strong>75% <em>(5.3 steps, 92% prec)</em></strong></td> </tr> <tr> <td><strong>Exact / Near Address (<code class="language-plaintext highlighter-rouge">uprn</code>)</strong></td> <td>81% <em>(4.6 steps)</em></td> <td>70% <em>(6.4 steps)</em></td> <td>70% <em>(6.3 steps)</em></td> <td><strong>91% <em>(3.1 steps, 94% prec)</em></strong></td> </tr> <tr> <td><strong>Postcode Only (100–350m off)</strong></td> <td>64% <em>(7.0 steps)</em></td> <td>55% <em>(8.8 steps)</em></td> <td>57% <em>(8.4 steps)</em></td> <td><strong>83% <em>(5.5 steps, 91% prec)</em></strong></td> </tr> <tr> <td><strong>Standard <code class="language-plaintext highlighter-rouge">1:1250</code> Plans</strong></td> <td>71% <em>(6.2 steps)</em></td> <td>65% <em>(7.5 steps)</em></td> <td>66% <em>(7.4 steps)</em></td> <td><strong>81% <em>(5.0 steps, 96% prec)</em></strong></td> </tr> </tbody> </table> <p>Look at what happened between <strong>Pure SFT (<code class="language-plaintext highlighter-rouge">59.0%</code> overall, <code class="language-plaintext highlighter-rouge">56%</code> on hand-drawn)</strong> and <strong>SFT + RL (<code class="language-plaintext highlighter-rouge">76.2%</code> overall, <code class="language-plaintext highlighter-rouge">78%</code> on hand-drawn)</strong>:</p> <ul> <li>The hand-written <code class="language-plaintext highlighter-rouge">smart_expert</code> was tuned to be cautious (<code class="language-plaintext highlighter-rouge">95.9%</code> precision), which caused it to second-guess itself on degraded hand-drawn plans—burning <strong>8.4 steps</strong> checking rival peaks and escalating 40% of the time.</li> <li>Pure Behavior Cloning (<code class="language-plaintext highlighter-rouge">sft_decision_encoder</code>) faithfully copied that hesitation (<code class="language-plaintext highlighter-rouge">56%</code> correct in <code class="language-plaintext highlighter-rouge">8.8</code> steps).</li> <li>Once unleashed with <strong>REINFORCE</strong>, the policy discovered that it didn’t need a rigid correlation threshold. By combining the current peak score, local distinctiveness (<code class="language-plaintext highlighter-rouge">score - second</code>), and the lead over remembered candidates in <code class="language-plaintext highlighter-rouge">obs()</code>, <strong>SFT + RL boosted recall on hand-drawn historical plans by +22% over the cloned expert (<code class="language-plaintext highlighter-rouge">56%</code> $\rightarrow$ <code class="language-plaintext highlighter-rouge">78%</code>) while cutting search time from 8.8 steps down to 5.3 steps!</strong></li> </ul> <hr/> <h2 id="the-takeaway-small-mdps-over-giant-models">The Takeaway: Small MDPs Over Giant Models</h2> <p>There is a temptation in AI engineering right now to treat every multi-step business problem as a prompt-engineering exercise for a massive vision-language model.</p> <p>But when you strip a problem down to its actual structure—<strong>what is the state, what are the actions, and what are the business costs of time, errors, and human escalation</strong>—you can often solve it with a 6,000-parameter network that trains on a laptop CPU in 15 minutes and executes in microseconds.</p> <p>By pairing fast classical signal processing (Normalized Cross-Correlation) with a lightweight RL decision policy (<code class="language-plaintext highlighter-rouge">SFT + REINFORCE</code>), our agent doesn’t just recognize maps—it <strong>actively hunts them down</strong>, adapts to hand-drawn historical drift, verifies its own hypotheses, and knows when to ask a human for help.</p>]]></content><author><name></name></author><category term="machine-learning"/><category term="computer-vision"/><category term="mapping"/><category term="sovereign-ai"/><category term="reinforcement-learning"/><summary type="html"><![CDATA[Matching planning documents to the entirety of Greater Manchester using tiny models and reinforcemment learning]]></summary></entry><entry><title type="html">Sovereign AI for Greater Manchester: Geolocating 20 Years of Planning Docs with Edge-Ready Models</title><link href="https://rjb7731.github.io/blog/2026/sovereign-ai-manchester/" rel="alternate" type="text/html" title="Sovereign AI for Greater Manchester: Geolocating 20 Years of Planning Docs with Edge-Ready Models"/><published>2026-04-11T13:00:00+01:00</published><updated>2026-04-11T13:00:00+01:00</updated><id>https://rjb7731.github.io/blog/2026/sovereign-ai-manchester</id><content type="html" xml:base="https://rjb7731.github.io/blog/2026/sovereign-ai-manchester/"><![CDATA[<p><strong>Training a single 49-million-parameter model to match planning documents to Ordnance Survey Maps for the entirety of Greater Manchester!</strong></p> <blockquote> <h3 id="-live-interactive-demo">🚀 Live Interactive Demo</h3> <p>I have deployed the model to a public Hugging Face Space. You can test the 3,478-tile scale test in real-time here:<br/> <strong><a href="https://huggingface.co/spaces/ryanbrady773/greater-manchester-map-matcher">Greater Manchester Map Matcher Live Dashboard</a></strong></p> </blockquote> <p>Most plans within the past 20 years rely on data taken from Ordnance Survey (OS) mapping. Before this, people used to hand-draw buildings, roads, and trees with pencils and pens—hatching lines, hand-signing certain sections, and smudging ink. The great thing with ‘new’ mapping is that people generally no longer go through this effort and use these official tiles instead.</p> <p>This does raise an interesting question: can we train a system on these actual maps for the purpose of being able to place plans exactly where they belong on an official map? Any such system would need to match the visual patterns of roads, buildings, and rivers as a form of spatial fingerprinting. It feels like this is something that should be possible with relatively small compute, without the need to rely on huge multi-billion parameter models and squashing their capabilities into this quite niche task.</p> <p>Richard Sutton’s <strong>“The Bitter Lesson”</strong> comes to mind here; a lot of advanced prompting strategies and “agent” frameworks that many AI Engineers put together are trying to encode complexity into a process which should just be a job for deep learning to learn by itself.</p> <p>First thing I like to think about when starting any AI project (and this one is no different), where is the data going to come from to prove this out? One exciting area is the possibility of synthesising new forms of data for the purposes of training intelligent systems that generalise. Inspired by early DeepMind experiments such as AlphaGo (which learned through self-play) and their work on Atari games combined with reinforcement learning, I thought it would be interesting to gamify the process of matching planning documents back to their OS Map location. The plans should reflect what we would see in actual council plans, which might be poor scans or have distractors such as stamps and large white spaces on the page.</p> <p>If I could train a model to recognize spatial fingerprints associated with roads, rivers, and buildings, it should be able to locate a planning document simply by looking at the map. This would mean no text reading, no ground control points, and no complex georeferencing transforms. Just visual pattern matching which is a field where deep learning models have excelled since the “AlexNet moment” way back in 2012.</p> <hr/> <h2 id="step-1-building-the-environment">Step 1: Building the Environment</h2> <p>I started with OS Open Map Local data, which is free vector data covering all of Great Britain, including roads, buildings, water, woodland, and railways. Using this, I rendered a grid of clean reference tiles covering Greater Manchester: <strong>3,478 tiles</strong>, each representing a 1km × 1km area. Each tile captures the raw spatial structure of its location: road layout, building footprints, and water features.</p> <p>I built a synthetic document generator that produces realistic planning documents from the OS vector data. Each document includes:</p> <ul> <li>The map rendered in color or monochrome OS styles.</li> <li>Title blocks (“HM Land Registry”, “Planning Application”, council names).</li> <li>Scale bars, north arrows, copyright notices.</li> <li>Road name labels at appropriate angles.</li> <li>Building numbers.</li> <li>Subject property highlighting with red outlines and hatching.</li> <li>Randomized page layouts (portrait/landscape, varying margins).</li> </ul> <div class="row"> <div class="col-sm mt-3 mt-md-0"> <figure> <picture> <source class="responsive-img-srcset" srcset="/assets/img/synth_plan-480.webp 480w,/assets/img/synth_plan-800.webp 800w,/assets/img/synth_plan-1400.webp 1400w," type="image/webp" sizes="95vw"/> <img src="/assets/img/synth_plan.png" class="img-fluid rounded z-depth-1" width="100%" height="auto" title="Synthetic Document Generation" loading="lazy" onerror="this.onerror=null; $('.responsive-img-srcset').remove();"/> </picture> </figure> </div> </div> <div class="caption"> Figure 1: Examples of synthetic planning documents generated with perfect ground truth. </div> <p>The key point, though, is that I know exactly where each document is located, giving me perfect ground truth for training!</p> <hr/> <h2 id="step-2-contrastive-learning">Step 2: Contrastive Learning</h2> <p>The model architecture is conceptually simple: a dual encoder trained with contrastive loss, similar to CLIP but for maps. Two ResNet50 encoders (pretrained on ImageNet) map images into a shared 256-dimensional embedding space:</p> <ol> <li><strong>The query encoder</strong> processes planning documents.</li> <li><strong>The tile encoder</strong> processes reference map tiles.</li> </ol> <p>During training, matching pairs (a planning document and its corresponding tile) are pushed together in embedding space, while non-matching pairs are pushed apart. The model learns to ignore rendering styles, title blocks, labels, and annotations, focusing instead on the spatial features that identify a location.</p> <p>I utilized <strong>ResNet50</strong> as it meant I did not have to go through the pain and effort of teaching the model to recognize shapes and patterns from scratch. At inference time, encoding a new planning document and finding the nearest tile embedding is a single matrix multiplication so essentially instant. All my results and training have been conducted on a single MacBook Pro.</p> <hr/> <h2 id="results-urmston-proof-of-concept">Results: Urmston (Proof of Concept)</h2> <p>On a 4km × 4km area with 225 tiles:</p> <table> <thead> <tr> <th style="text-align: left">Metric</th> <th style="text-align: left">Value</th> </tr> </thead> <tbody> <tr> <td style="text-align: left"><strong>Top-1 Accuracy</strong></td> <td style="text-align: left">65%</td> </tr> <tr> <td style="text-align: left"><strong>Top-5 Accuracy</strong></td> <td style="text-align: left"><strong>98.6%</strong></td> </tr> <tr> <td style="text-align: left"><strong>Random Chance</strong></td> <td style="text-align: left">0.4%</td> </tr> <tr> <td style="text-align: left"><strong>Parameters</strong></td> <td style="text-align: left">49M (ResNet50 backbone)</td> </tr> <tr> <td style="text-align: left"><strong>Training Time</strong></td> <td style="text-align: left">~50 minutes on a MacBook Pro</td> </tr> </tbody> </table> <p>The correct tile is in the model’s top 5 guesses 98.6% of the time.</p> <hr/> <h2 id="results-greater-manchester-scale-test">Results: Greater Manchester (Scale Test)</h2> <p>Scaling to all of Greater Manchester—<strong>3,478 tiles</strong> covering a 30km × 30km area—with only 1,993 training documents (less than one per tile on average):</p> <table> <thead> <tr> <th style="text-align: left">Metric</th> <th style="text-align: left">Value</th> </tr> </thead> <tbody> <tr> <td style="text-align: left"><strong>Top-1 Accuracy</strong></td> <td style="text-align: left">51.4%</td> </tr> <tr> <td style="text-align: left"><strong>Top-5 Accuracy</strong></td> <td style="text-align: left"><strong>79.8%</strong></td> </tr> <tr> <td style="text-align: left"><strong>Random Chance</strong></td> <td style="text-align: left">0.03%</td> </tr> <tr> <td style="text-align: left"><strong>Training Documents</strong></td> <td style="text-align: left">1,993</td> </tr> <tr> <td style="text-align: left"><strong>Total Tiles</strong></td> <td style="text-align: left">3,478</td> </tr> </tbody> </table> <div class="row"> <div class="col-sm mt-3 mt-md-0"> <figure> <picture> <source class="responsive-img-srcset" srcset="/assets/img/geolocation_1-480.webp 480w,/assets/img/geolocation_1-800.webp 800w,/assets/img/geolocation_1-1400.webp 1400w," type="image/webp" sizes="95vw"/> <img src="/assets/img/geolocation_1.png" class="img-fluid rounded z-depth-1" width="100%" height="auto" title="Greater Manchester Scale Test" loading="lazy" onerror="this.onerror=null; $('.responsive-img-srcset').remove();"/> </picture> </figure> </div> </div> <div class="caption"> Figure 2: The model identifying correct tiles across a major metropolitan area. </div> <p>The model correctly identifies the exact 1km square more than half the time, and the correct location is in the top 5 guesses four-fifths of the time across a major metropolitan area. This is achieved with completely different rendering styles between the planning documents and reference tiles.</p> <p>For context, this is achieved with less than one training example per tile. The model is genuinely learning spatial features, not memorizing. Its learning to ignore the noise contained in planning documents and focus on the map portion of the page for visual clues.</p> <div class="row"> <div class="col-sm mt-3 mt-md-0"> <figure> <picture> <source class="responsive-img-srcset" srcset="/assets/img/geolocation_2-480.webp 480w,/assets/img/geolocation_2-800.webp 800w,/assets/img/geolocation_2-1400.webp 1400w," type="image/webp" sizes="95vw"/> <img src="/assets/img/geolocation_2.png" class="img-fluid rounded z-depth-1" width="100%" height="auto" title="Greater Manchester Scale Test 2" loading="lazy" onerror="this.onerror=null; $('.responsive-img-srcset').remove();"/> </picture> </figure> </div> </div> <div class="caption"> Figure 3: The model identifying correct tiles across a major metropolitan area. </div> <hr/> <h2 id="the-takeaway">The Takeaway</h2> <p>The big takeaway here is the synthetic data pipeline. The ability to generate unlimited, perfectly-labeled planning documents from OS vector data is a powerful idea. The approach is simple enough that a single developer can build it on a laptop, and the pipeline can be retrained for any council area using freely available OS data.</p> <p>For councils with backlogs of modern planning documents, a tool that takes a scanned plan and says “this is located here” could be built, trained, and deployed locally for effectively zero cost. No API subscriptions, no cloud dependencies, and no per-document charges.</p> <p>As the model is a lean 49 million parameters, it doesn’t require a high-end GPU for deployment either. In fact, the entire inference engine and the database of Manchester’s spatial fingerprints could run on a Raspberry Pi. This makes ‘Sovereign AI’ a reality for local government: high-performance geolocation that is offline, private, and runs on cheap &amp; small hardware.</p>]]></content><author><name></name></author><category term="machine-learning"/><category term="computer-vision"/><category term="mapping"/><category term="sovereign-ai"/><summary type="html"><![CDATA[Matching planning documents to the entirety of Greater Manchester using contrastive learning and synthetic data.]]></summary></entry><entry><title type="html">Building a GAIA-1 Style World Model for Indoor Robot Navigation</title><link href="https://rjb7731.github.io/blog/2026/gaia-world-model/" rel="alternate" type="text/html" title="Building a GAIA-1 Style World Model for Indoor Robot Navigation"/><published>2026-04-01T00:00:00+01:00</published><updated>2026-04-01T00:00:00+01:00</updated><id>https://rjb7731.github.io/blog/2026/gaia-world-model</id><content type="html" xml:base="https://rjb7731.github.io/blog/2026/gaia-world-model/"><![CDATA[<h2 id="what-is-a-world-model">What is a World Model?</h2> <p>A world model is a neural network that learns to predict <em>what the world will look like next</em> given a current observation and an action. Instead of directly learning a policy (what action to take), you first learn the dynamics of the environment and then you can plan, imagine counterfactuals, and reason about consequences.</p> <p>This is the core idea behind <a href="https://wayve.ai/thinking/scaling-gaia-1/">GAIA-1</a> from Wayve, which does this for autonomous driving at scale. I wanted to build something similar for indoor robot navigation. Wayve as a company I think we should be really proud of in the UK, originally starting in Cambridge they are now leading the way with autonomous vehicles and their cars are on our roads in the UK today. Home robotics is an area ripe with research and investment and is one area we are seeing huge advancements.</p> <p>The best way to achieve my aim here and learn, as always, is to get hands on and try and solve a problem after all “What I cannot create, I do not understand” - Feynman.</p> <h2 id="the-setup">The Setup</h2> <p>First things first I set about buying and building a raspberry PI controlled car. This would work as my data collection engine and through tele-operation, I would collect information about my home. It collects data in RGB frames at ~10fps paired with motor commands (left/right wheel speeds). Each trajectory is a few hundred frames of the robot navigating through rooms.</p> <p>Did some tinkering and got the rig together, I then was able to SSH into the car and upload some .py files for it to take commands and send visuals from its camera back using pyzmq.</p> <div class="row justify-content-center"> <div class="col-sm-8 col-md-6 mt-3 mt-md-0"> <figure> <picture> <source class="responsive-img-srcset" srcset="/assets/img/pi_car-480.webp 480w,/assets/img/pi_car-800.webp 800w,/assets/img/pi_car-1400.webp 1400w," type="image/webp" sizes="95vw"/> <img src="/assets/img/pi_car.jpeg" class="img-fluid rounded z-depth-1" width="100%" height="auto" title="Pi Car" loading="lazy" onerror="this.onerror=null; $('.responsive-img-srcset').remove();"/> </picture> </figure> </div> </div> <div class="caption text-center"> Figure 1: PI car after finishing the build. </div> <p>The pipeline has two stages:</p> <p><strong>Stage 1 — DINO VQ-VAE</strong>: Compress each 128×128 image frame into 64 discrete tokens using a Vector Quantised VAE with a DINO distillation loss. This gives semantically meaningful tokens — similar scene regions get similar token IDs. These tokens become useful as my Transformer that I want to train can use these tokens as a representation of say a couch or a rug as a means to view how it changes with the respective robot actions. We know through GPT models how powerful these models can use language tokens to help users and perform coding tasks etc. Instead of just minimizing pixel-wise error (which leads to blurry averages), the DINO distillation forces the VQ-VAE to group pixels into semantic clusters. It learns that a ‘rug’ is a single entity, making the Transformer’s job of moving the rug with actions much easier.</p> <p><strong>Stage 2 — GAIA Transformer</strong>: A causal GPT-style transformer trained on sequences of <code class="language-plaintext highlighter-rouge">[frame_tokens, action, frame_tokens, action, ...]</code>. Given N context frames and an action, it predicts the next frame’s tokens autoregressively. The transformer here is trained on the trajectories of these compressed latent representations of my home and learns how to recreate these latents frame by frame so when I feed the latent back into the decoder portion of my above mentioned VQ-VAE it should be able to build the images back up!</p> <p>You could say I am creating a mini ‘world’ model of my own home here….we have Wayve at home!</p> <div class="row"> <div class="col-sm mt-3 mt-md-0"> <figure> <picture> <source class="responsive-img-srcset" srcset="/assets/img/gaiai_showcase_2-480.webp 480w,/assets/img/gaiai_showcase_2-800.webp 800w,/assets/img/gaiai_showcase_2-1400.webp 1400w," type="image/webp" sizes="95vw"/> <img src="/assets/img/gaiai_showcase_2.png" class="img-fluid rounded z-depth-1" width="100%" height="auto" title="Example rollouts" loading="lazy" onerror="this.onerror=null; $('.responsive-img-srcset').remove();"/> </picture> </figure> </div> </div> <div class="caption text-center"> Figure 2: Rollout trajectories examples, notice the rugs and cabinets changing with turns. </div> <h2 id="results">Results</h2> <p>The model learned real spatial structure from trajectory data alone there is no 3D supervision, no depth sensors. Applying TURN_LEFT causes the scene to pan right; applying TURN_RIGHT enough steps eventually reveals a completely different part of the room.</p> <p>Most impressively, the model maintains “object permanence.” If it pans away from the couch and then turns back, the couch reappears in the correct relative position. It has internalized the room’s geometry.</p> <p>Below is model generated.</p> <div class="row justify-content-center"> <div class="col-sm-10 col-md-8 mt-3 mt-md-0"> <video width="100%" height="auto" autoplay="" loop="" muted="" playsinline="" class="img-fluid rounded z-depth-1"> <source src="/assets/video/gaia_rollout.mp4" type="video/mp4"/> Your browser does not support the video tag. </video> </div> </div> <div class="caption text-center"> Figure 3: Imagined rollout. The model "dreams" a 360-degree turn based on consecutive actions. </div> <h2 id="whats-next">What’s Next</h2> <ul> <li>Better action conditioning (continuous motor values instead of 5 discrete actions)</li> <li>Longer rollouts with less drift</li> <li>Using the world model for planning and policy learning</li> </ul>]]></content><author><name></name></author><category term="robotics"/><category term="world-models"/><category term="deep-learning"/><summary type="html"><![CDATA[How I trained an action-conditioned video world model from scratch on a home robot dataset.]]></summary></entry></feed>