Model-Based Gesture Detection for a Wine Tasting Wristband

Food scientists can read how much someone likes what they taste from their body: skin temperature, skin conductance, heart rate, the voice. But these measurements only mean something inside a short window, two to three seconds, starting at the exact moment the sample enters the mouth. A few seconds late and you are measuring chewing instead.

In every study I reviewed, that moment came from an experimenter in a lab booth who handed over the sample and knew when it happened. Nobody drinks wine in a booth. Take the experimenter away and the method has nothing to line its window up against.

My approach: let the device play the experimenter. It tells you what to do next (swirl, sniff, sip), watches your wrist until it sees you do it, and only then moves on. The instant it confirms the gesture becomes the start of the measurement window. If it never sees the gesture, it waits and asks again instead of logging a wrong sample.

I split the hardware by what each part has to do on the table. The wristband only carries what has to be on the wrist: the IMU, a thermistor pressed against the skin, a capacitive touch pad to start a session, and a vibration motor for prompts. It has no processor. All of its signals run back to the hub over one CAT6 cable, which gives me eight conductors in a single flexible tether.

The hub holds everything that works better sitting still: a Raspberry Pi Pico 2 W, the microphone, the pulse sensor, and the NFC reader. The lid has a circular pocket for the glass, with the NFC reader directly underneath. Setting the tagged glass down in the pocket is what identifies the wine. I designed it that way on purpose: reading the wine is a placement, not a hand motion, so it can never be confused with the gestures the classifier is watching for.

The pulse sensor is the one part left outside the case, so a taster can hold it against a fingertip without opening anything. Its cable runs in through the ventilation holes in the enclosure wall.

The Pico samples four channels at once, at rates that differ by more than a thousand times: audio at 16 kHz, pulse at 100 Hz, motion at 50 Hz, and skin temperature at 10 Hz. Audio sets the hardest limit. At 16 kHz a late sample is not just late, it is a hole in the waveform.

So I gave audio a core of its own. The second core does nothing but capture sound in a tight timing loop. The first core runs everything else as cooperative tasks: the IMU, pulse, and temperature readers, the USB link to the PC, and a small Wi-Fi dashboard. Under full load the slower channels held 87 to 98 percent of their target rates. Audio reached about half its nominal rate, so each recording is stamped with the rate actually measured rather than the one requested.

Every channel leaves the device already in physical units, so the PC never has to guess what a raw count meant.

Pulse to beats per minute. The pulse sensor is an optical sensor (a photoplethysmograph): its output is a wavy ADC signal whose peaks are heartbeats, riding on a slowly drifting baseline. Two moving averages pull them apart. A fast one (α = 0.3) follows the waveform, a slow one (α = 0.01) follows the drift, and their difference is the pulse alone. A beat counts when that difference crosses a threshold set at half the recent peak height, with a floor above the noise I measured with no finger on the sensor. Each heartbeat also has a smaller second bump about half a second later. I measured it at 460 to 550 ms, so the detector ignores any crossing within 500 ms of the last beat. The reported rate is the median of the last five beat-to-beat intervals, and an interval roughly double the median (a missed beat) is kept out of it.

Thermistor to degrees. The thermistor sits in a voltage divider. The firmware converts the ADC reading to resistance and then to temperature with the β-form of the Steinhart–Hart equation (β = 3950). The cable adds brief contact-resistance spikes, so a five-sample median filter removes them at the cost of about 0.2 s of lag.

Motion to features. The IMU stream is cut into one-second windows that overlap by half. A ten-second still period at the start of each session measures the gyroscope's zero offset and the direction of gravity. From each window the model gets 24 features: how often the gyroscope crosses zero (a swirl is periodic), peak acceleration (a sip has a lift), wrist pitch and roll (a sniff tilts less than a sip), and rotation measured around the gravity axis, which describes a swirl however the band happens to sit.

A session goes: touch the pad, sit still for a resting baseline, set the glass on the hub, then swirl, sniff, sip, and answer three spoken questions about the wine. Each motion step follows the same loop. The device buzzes and speaks the instruction, the classifier watches one-second windows, and the step only counts when two windows in a row agree. That confirmation is logged to the millisecond as the start time for every other channel.

Two different detectors can confirm a step: a random forest on the PC and a set of hand-tuned rules on the Pico. The PC also runs speech recognition (Whisper), a neural voice for the prompts, and a local language model for the conversation, with nothing sent to the cloud. The step instructions themselves are fixed text, not generated, because the part of the system that must never drift should not depend on the part that can.

I trained on 1,026 labeled windows from 122 gesture instances. The recording firmware paced itself with haptic cues, so each label boundary is the exact moment the device asked, not a person's reaction time. Besides the three gestures and rest, I recorded a set of deliberate confounders: reaching for the glass, setting it down, and talking with my hands. Cross-validation folds were split by gesture instance, so overlapping windows from the same gesture never end up on both sides.

A full session recorded from the live dashboard, with the spoken prompts audible. The dashboard shows the current step, live motion readings, pulse, skin temperature, and the wine read from the tag.

All gesture data comes from one wearer, so these numbers describe the pipeline, not how it would do across many people. The pulse and skin-temperature calibrations are single-point corrections against my own readings and need a proper calibration before the values mean anything absolute. Predicting whether someone likes a wine was out of scope; the device logs the data for that as a byproduct, and a real preference study would need many more tasters.