Hackathon · Localhost: On-Device Agent Hackathon · London 2026
OffBabel
Practise speaking or fingerspelling with a tutor that never goes online.
An offline tutor on a Reachy Mini robot that helps people practise speaking Spanish or English, or try fingerspelling in British Sign Language.
Cosine and EXO Labs Track Winner - On-Device Agent Hackathon
OffBabel won two of the event's tracks, Cosine and EXO Labs.
- Event
- Localhost: On-Device Agent Hackathon, London
- Result
- Cosine Track Winner and EXO Labs Track Winner
- Stack
- Python, FastAPI, React, Exo, faster-whisper, MediaPipe
- Platform
- Reachy Mini and one Mac, offline
- Team
- Bill
Why it matters
Speaking is hard to practise on your own: it needs someone to answer you and to tell you when you got something wrong. OffBabel puts that partner on a small robot you simply talk to, in Spanish or English, with no screen needed. The same tutor helps people practise British Sign Language fingerspelling in front of a camera. The tutor runs on EXO and everything else runs beside it on the same Mac, with the internet switched off, so a learner's mistakes never leave the room.
How it works
Talk to Reachy in Spanish or English, and the tutor answers out loud.
Laptop mic
Hands-free, no button to hold
16 kHz · 30 ms blocks
Listener
Waits for a pause, then transcribes
faster-whisper small · int8
"Estoy de Barcelona."
Tutor
Replies, translates and corrects
Gemma 4 E2B · 4-bit
- reply
- translation
- correction
Second check
Reads the sentence again if nothing was corrected
same model
estoy de Barcelona → soy de Barcelona
Reachy Mini
Says the reply out loud
macOS voice, played on the robot
Speaking
Reachy's camera
Watches your hands
MJPEG over SSH · webcam backup
Sign reader
Turns hand shape into a letter
MediaPipe · 126 numbers · KNN
- hands found
- 12 frames agree
Got it
Memory
Keeps every hit and miss
SQLite · Leitner, 5 boxes
missed 1x
Progress
What to review, on screen or as a PDF
streak · mastery · review sheet
1 due to review
Decisions we made
Local models behind a cloud-style API
The tutor talks to Exo through the same OpenAI client a cloud model would use, so moving between Exo, Ollama and the cloud is a one-line change of address. Speech recognition, sign reading and the memory of mistakes run on the Mac too, and if the model cannot be reached, a scripted conversation keeps the lesson going without inventing corrections.
Trade-off. Exo accepts the request for strict JSON but ignores it, so every reply goes through a parser that strips stray text and repairs broken JSON before the screen can use it.
A second check for mistakes
In one answer the tutor writes its reply, an English translation and any correction. Juggling all three, the small model often let a mistake slip, so when no correction comes back, a separate checker reads the learner's last sentence again. It is told to let fillers, false starts and likely transcription slips pass, and to flag only real grammar or word choice mistakes.
Trade-off. Every turn without a correction costs one more call to the model, and giving the learner the benefit of the doubt means some real mistakes still go unmarked.
Letters from hand shape, not pixels
MediaPipe finds 21 points on each hand. We move both hands to the wrist, scale them to the size of the hand so distance from the camera does not matter, and pass the 126 numbers to a nearest-neighbour classifier. A letter only counts when 12 frames in a row agree, each with enough confidence.
Trade-off. A new letter needs only fresh recordings, with no neural network to train, but we recorded just four, A, B, G and W, with 60 examples each. It reads those four, not the whole alphabet.
Demo
Gallery
The full engineering is on GitHub.