How Voice and Vision AI Can Understand the Complete Customer Journey in a Store

The journey as the customer lives it

Most store visits in assisted retail follow the same shape. The customer enters, often with someone. They orient, then move to a display or counter. They wait, or they are approached. A conversation happens, or it does not. A decision follows, and the customer either bills, leaves with a promise to return, or simply leaves. Each stage leaves a different kind of trace, and only some of those traces are audible.

In plain terms: the conversation is the record of the interaction, and the camera is the record of what surrounded it. Voice carries intent, objection, offer and outcome. Vision carries footfall, wait, group, path and the walk-outs that nobody spoke to. The join happens on the advisor's tap to record, by time and zone, and never by who the person is.

Which signal sees which stage

One visit, told both ways

A furniture store in Pune, Saturday, 4.10 pm. The camera pipeline records: two people enter together; they spend six minutes in the sofa zone; nobody approaches for the first four; at 4.16 an advisor arrives; the group moves to the desk at 4.21; they leave at 4.29 without passing billing.

The consented conversation, started when the advisor tapped at 4.16, records: the customer asks about a three-seater in fabric; the advisor quotes the festive scheme correctly; the second person asks about delivery to a fourth-floor flat without a lift; the advisor says it will be checked; the customer says they will come back Sunday and shares a number.

Read separately, the camera reports an unconverted group visit with a four-minute wait, and the conversation reports a delivery objection with a lead captured. Read together, the visit says something more useful: the wait was long, the objection was logistical not price, the decider was the second person, and the store has until Sunday to confirm fourth-floor delivery. That is the record the store manager needs, and neither system alone produces it.

What multimodal customer intelligence adds by stage

• Arrival and group: conversion by group size, so the playbook can add a step for addressing the second person.
• Browsing: which display the customer stood at before the conversation, so the advisor's opening can start from what they were looking at.
• Waiting: wait time as a variable next to adherence and outcome, so staffing decisions use evidence from the floor.
• Engagement: the conversation scored as today, with the wait and the group as context columns.
• Exit: unattended walk-outs counted as a share of entries, the one number the voice half can never produce, because nobody tapped.

Confidence and failure modes

A join is a hypothesis with a score. The strongest case is one visual track in the desk zone at the moment of the tap. Real floors are messier. Two shoppers cross in front of a camera and one track splits into two, so a six-minute dwell becomes a two and a four. Two advisors stand near one group at a busy counter, and it is not certain whose tap belongs to which conversation. A customer walks out of frame to the parking area and back, and arrives as a new entry.

The right response is not to hide this. Each joined visit carries a confidence, weak joins are reported separately, and the store-week aggregates lean on strong joins. A retailer reading a multimodal report should expect to see how many visits were joined with confidence, how many were vision-only because nobody tapped, and how many were voice-only because the camera did not cover the zone.

Privacy at each stage

The camera half runs under signage, processes video on an edge device in the store, and emits events (an entry, a dwell, a walk-out), not frames or faces. No facial recognition, no re-identification across visits. The voice half starts on the advisor's tap after the customer agrees, and audio is deleted after transcription. The join uses time and zone, so no identity is needed to connect the two. Under India's DPDP Act, that is the difference between processing anonymous events and processing personal data, and the design keeps the second to the conversation alone.

Borentis today runs the voice half, live as Borentis Floor. ShopperDNA, which would add the camera events and the join described here, is on the roadmap.