MagSense

MagSense is a multimodal sensing system that turns any tabletop into an intelligent, privacy-preserving input surface — using nothing but a passive magnetic pen and a small USB device.

This was my final project for EECS 498-010: ML for MWIoT.

Prior Works & Motivation

The project is motivated by two prior works:

1. MagDesk (Huang et al.) — provides spatial localization via a passive magnetic stylus and an array of magnetometers, but cannot distinguish slight hovering from touch.

2. SAWSense (Iravantchi et al.) — recognizes surface-bound events through propagating acoustic waves sensed by a Voice Pickup Unit (VPU), but provides no spatial localization.

Each system alone captures only half of the pen-interaction input. MagSense combines the two modalities on a single embedded device: the magnetometer array localizes the magnetic pen in 2D, while a VPU-based classifier determines whether the pen is idle, tapping, or drawing. Together, MagSense streams both the position and interaction state of the user’s magnetic pen without any camera, active electronics in the pen, or surface instrumentation.

PCB

A 4-layer PCB was designed using KiCAD.

Core Components

• ESP32-S3-WROOM-1

• 5 × MMC5983MA magnetometers (linear array, 10 mm spacing)

• AP7361C LDO

• USB-C direct male plug

• Header pin connectors to VPU (Voice Pickup Unit) --> on a separate breakout

3D Printed Enclosure

To improve coupling between the VPU and the tabletop, I added a spring between the VPU breakout and the enclosure lid to maintain consistent contact. In practice, this still wasn’t enough. Lack of coupling introduced airborne noise to the successive ML pipeline, so improving this mechanical interface is a key area for future work.

ML Pipeline

MagSense runs two parallel sensing pipelines that are fused into a single interaction stream.

1) Pen State Classification (ML Model)

The VPU captures structure-borne vibrations from the table at 48 kHz. Then, a lightweight ML pipeline processes the audio @ 30 Hz using overlapping windows:

• Convert audio → cMFCC features

• Feed into Random Forest classifier

• Output:

  • Nothing

  • Tap (touch)

  • Drawing

2) Magnetic Pen Localization (Mathematical Model)

The magnetometer array measures 3-axis magnetic fields at 250 Hz. Then, a physics-based dipole model is used to invert magnetic field readings into (x, y) position:

• Baseline subtraction removes ambient field

• Least-squares fitting estimates pen position

• Exponential smoothing stabilizes output

3) Fusion: Stroke Capture & Digit Classification

The two branches are fused at the host:

1. When pen_state is “drawing,” and a magnet is confidently localized, the (x,y) position is appended to the current stroke segment.

2. A 300 ms grace period rides through brief pen-state flickers, so strokes are not fragmented by classifier noise.

3. Pen lift closes the current segment. After 1.0 s with no new stroke, all segments are rendered into a 28×28 MNIST-style grayscale image (a 1-second grace period is given to account for the numbers that require lifting the pen in the middle of writing, like 4 and 5).

4. A scikit-learn MLP classifier (trained on the full MNIST dataset) outputs digit classification of 0–9 plus per-class probabilities.

Evaluation

I evaluated the full pipeline by logging each prediction alongside a ground-truth label. From 3 different participants, I measured end-to-end system performance across ~100 samples.

In summary, MagSense achieved 88% overall accuracy in number prediction. Stronger classes were 3, 7, 8, and 9, whereas weaker classes were 4 and 6. We attribute these inaccuracies to unstable pen-state classification that led to disconnected strokes being recorded, and imperfect finetuning of per-user settings (e.g. longer pen-lift grace period for writing numbers like 4 for slower writers).

Limitations and Future Work

Limitation 1 — Magnetic tracking range and geometry

• Magnetic field strength follows an inverse-cube relationship, meaning detection range drops sharply. The stable drawing area is limited to 4~8cm from the board.

• 1 of 5 sensors was excluded due to instability and constant glitching.

• 10mm inter-sensor distance constrains better triangulation.

Limitation 2 — VPU acoustic sensing

• VPU needs a firm mechanical coupling to the tabletop surface. Failure to do so led to significant airborne noises.

• Inconsistent, unstable pen state detection due to bad coupling and small training data.

Future Work

• More magnetometers, wider spacing: scaling to 10+ sensors with 15–30 mm spacing would simultaneously extend the usable area and improve triangulation accuracy.

• Improved VPU mechanical coupling: a new mechanism that ensures good coupling between the VPU and the surface, regardless of which laptop the dongle is plugged into, would stabilize the acoustic input to the pen-state classification model.

• Note writing applications: with improved magnetic tracking and VPU coupling, the MagSense system should be able to support basic handwriting and sketch tracking in a bigger play area.