Designing an AI-led experience across voice, touch, and body

About three minutes on the exhibition floor: the evaluator speaks, a visitor answers out loud, moves, and receives a score. Sound on, the voice is half the interface.

My thesis began with telephone fraud. I spent months on it, mapping who gets targeted, why shame keeps victims silent, and where an intervention could sit. Then my attention shifted. The version of the problem that kept pulling at me was no longer a stranger on the phone but a synthesized voice that sounded like someone you love.

Each detector eventually loses, which left me with a question underneath the whole effort.

That question moved the thesis from a real-world problem to a philosophical one: voice scam, to authentication, to humanness. Seven years of design education and practice had trained me to look for the practical solution first. For a master's thesis I wanted the opposite, the bigger and slower question that shapes how designers work with emerging technology.

A quick interview would have forced people to answer an abstract question on demand. I wanted three kinds of evidence: what people notice in their own lives, what a broader community intuitively associates with humanity, and how different disciplines frame the human-machine boundary.

Participants documented moments when they felt deeply human, with context, action, and feeling. The mundane answers became the most revealing: morning coffee, guacamole, being overwhelmed, small rituals, procrastination, relationships.

I resisted forcing the research into a neat definition too early. Across diaries, interviews, survey responses, and media research, recurring clusters formed around five qualities, and each one had to become something a person actually does in the room.

A screen full of survey questions would have contradicted all of it. If embodiment mattered, participants needed to move. If emotional complexity mattered, fixed-choice responses were not enough. If unpredictability mattered, the system needed to observe behavior rather than ask people whether they were unpredictable.

Before building the prototype, I physically mapped the sequence of prompts, inputs, measurements, feedback, and final reflection. I was designing the rhythm of attention, not just individual screens: listen, respond, move, receive feedback, continue.

Not every human response should use the same input. The screen is the stimulus and touch is the answer when reaction should happen before explanation. Voice carries expression when the response is a value judgment. The body becomes the input when the quality being tested is embodiment, and once movement began, voice mattered more than text, because looking back at instructions would break the interaction.

I moved away from the bright, friendly, often feminine assistant archetype. The premise was closer to an airport checkpoint: the system was evaluating you. I worked through Google Cloud's voice catalog, listening to hundreds of them, before choosing a lower-pitched male voice that balanced authority, warmth, dry humor, and audibility in a noisy room. Casting was only half of it: the same voice at default settings still sounded like an assistant, so I slowed the delivery and dropped the pitch until it read as an evaluator.

The original plan sent raw voice to Hume AI, a vocal emotion model. In exhibition conditions it was unreliable, so the working system transcribes speech on device and interprets the transcript semantically instead.

This was not an equivalent replacement. Vocal emotion asks how the person sounded; semantic emotion asks what the person expressed. I kept that limitation visible in the framing rather than pretending the system could measure something it no longer could. I changed the signal, not the interaction intent.

This was not a lab prototype. An academic committee and hundreds of visitors encountered it while other people were talking, waiting, watching, and moving around them. No formal onboarding, ambient noise, a social cost to speaking in public, shifting attention, and spectators who made the interaction partly performative.

So the exhibition became a usability study in the wild. I stood to the side and watched, and these are the things I could not have learned in a quiet room.

The goal was never an accurate measurement of humanity. I considered the experience successful when participants stopped treating the machine as an authority and started debating the assumptions behind its judgment: why did it think that, could I fool it, would another person get the same result, and why do we need to prove that we are human at all.

This is where those observations landed. The exhibited build used a tablet, a cast voice, capacitive touch, and head tracking, and the browser has none of that, so the port trades fidelity for access: the emotion reading is coarser, a pulse stands in for capacitive sensing, a cursor for head tracking, and your device speaks with whatever voice it has.

What it does carry is the script, the images, the scoring, and every interface change the exhibition argued for. Sound on. Five steps, about three minutes.