// MIT Media Lab · HARD MODE Hardware × AI Hackathon · IFM Prize
8 Milliwatts
A wearable system that teaches hands-on skills by projecting guidance onto the real workspace, instead of into a pair of glasses.
Overview
AR guidance usually lives in a headset that only the wearer can see. We asked what happens if the guidance lands on the real world instead: a helmet that watches your hands, understands where you are in a task, and projects the next step directly onto the surface you are working on.
We built it in 48 hours at HARD MODE, MIT Media Lab’s hardware × AI hackathon, and demonstrated it on piano: the helmet lights up the next key to play and moves on when you play it. The project won the IFM Prize.
At a glance
Problem
Guidance you can’t shareGlasses-based AR keeps instructions private to one wearer and pulls attention away from the hands.
Approach
Project onto the taskCameras read the workspace, a task state machine tracks progress, and a projector and laser mark the next action in place.
Outcome
IFM Prize winnerA working piano-learning demo, built and shown at a 200-person hackathon at the MIT Media Lab.
My part
Real-world agent for task guidancePerception-to-action pipelineAprilTag spatial trackingTask state machines
System
01
From perception to action
The system works as a loop. It sees the workspace, works out where the learner is in the task, and acts by placing a cue exactly where the next action happens. Voice commands let the learner steer without taking their hands off the keys.
Seehelmet + Mentra Glass cameras
→
LocateAprilTags map the keyboard
→
Decidetask state machine
→
Actprojector + laser cue
→
Listen“next” · “repeat” · “stop”
Knowing where things are
AprilTag markers on the keyboard give the cameras a fixed reference. From them the system computes the transform from camera space to projector space, so a cue lands on the right key, not just near it.
Knowing what comes next
A session state machine holds the lesson: which note is due, whether it was played, and when to move on. A central orchestrator routes voice commands into that state, so “repeat” and “next” work mid-lesson.
Agent design
02
An agent you can always interrupt
The helmet acts on its own: when it sees the right key played, it moves to the next step. That raised the core interaction question: how does a learner stay in charge of a system that advances by itself, without reaching for a screen?
Visible state
The cue is the statusA lit key means the agent is waiting for that note; when the note is played, the cue moves on. Feedback arrives at every step, right where the hands are.
Override
Voice beats autonomy“Next” skips ahead, “repeat” replays the step, and “stop” ends the session, all routed into the task state mid-lesson.
Hands free
No screen in the loopControl never needs a tap: hands stay on the keys and eyes stay on the workspace, while two cameras keep track of the scene.
Interaction
03
Guidance that stays in the room
Because the cue is light on a real surface, everyone nearby sees the same thing as the learner, so a teacher can follow along and point to the same key. The learner keeps their eyes on their hands rather than on a screen, and talks to the system instead of tapping it.
A
Look
the next key lights up
B
Play
the system checks and moves on
C
Say
“repeat” or “next” to steer
Piano was the demo, but the same loop applies to any hands-on task where the instruction can be placed in the workspace, such as technician assembly or training on factory machinery.
Build
04
48 hours on the floor of the Media Lab
Outcome
IFM Prize winner at HARD MODE.
HARD MODE brought about 200 participants to the MIT Media Lab for 48 hours to build intelligent hardware. Our team took the IFM Prize for a working wearable that turns computer vision and a task model into guidance you can see on the real world.