Beyond Visible Speakers: Context-Aware Active Speaker Detection with Speaker Registration

ROAM-ASD-C with missing modalities

Vision hidden: for each face, random 3-second intervals covering 30% of its track have the face crop and the mouth crop replaced by black, so the model receives no image of that person during those intervals.

Each video has three levels, top to bottom:

1. Original scene — the real frame, every tracked face boxed and numbered; a green dot marks who is speaking according to the ground truth.
2. Prediction comparison — ROAM-ASD (left) vs ROAM-ASD-C (right), same instant: green box = predicted speaking, red = silent, orange ring = prediction disagrees with the ground truth.
3. Per-face scores — one panel per numbered face: speaking score (0–1, y-axis) over time in the clip (seconds, x-axis), drawn on the ground-truth band.

WASD — four faces, vision hidden in 3 s gaps

Frame accuracy on this segment: ROAM-ASD 92.3 → ROAM-ASD-C 95.1. Watch face #3's dark interval near the end: the ROAM-ASD curve collapses, the ROAM-ASD-C curve stays on the ground-truth band — the other three faces are visible and tell the block who is speaking.

UniTalk — six faces, vision hidden in 3 s gaps

wh3-1DUTb3Q, 268–276 s. Only face #6 speaks. Frame accuracy over the six faces: ROAM-ASD 79.9 → ROAM-ASD-C 97.8; inside the gaps 53.1 → 95.8. Whenever a silent face is hidden, ROAM-ASD scores it as speaking because it hears speech and cannot see the face; ROAM-ASD-C sees the other five faces, finds the speaker among them, and keeps the hidden face silent.

AVA-ActiveSpeaker — three faces, vision hidden in 3 s gaps

5milLu-6bWI, 1440.5–1448.5 s. Frame accuracy over the three faces: ROAM-ASD 90.8 → ROAM-ASD-C 97.2; inside the gaps 79.5 → 97.7. Face #3 is hidden while face #1 talks in the first two seconds: ROAM-ASD assigns the speech to the hidden face, ROAM-ASD-C does not.