Beyond Visible Speakers: Context-Aware Active Speaker Detection with Speaker Registration

ROAM-ASD-C in crowded scenes

When several people are visible, the context block decides who among them is speaking by comparing the candidates.

Each video has three levels, top to bottom:

  1. Original scene — the real frame, every tracked face boxed and numbered; a green dot marks who is speaking according to the ground truth.
  2. Prediction comparison — ROAM-ASD (left) vs ROAM-ASD-C (right), same instant: green box = predicted speaking, red = silent, orange ring = prediction disagrees with the ground truth.
  3. Per-face scores — one panel per numbered face: speaking score (0–1, y-axis) over time in the clip (seconds, x-axis), drawn on the ground-truth band.
ground-truth speakingpredicted speakingpredicted silentdisagrees with ground truthROAM-ASD-CROAM-ASD

AVA-ActiveSpeaker — HKjR70GCRPE, 1150.5–1156.5 s, 4 faces (full scene)

Four faces on screen for the whole window (numbered #1–#4 in all three levels). Frame accuracy over the four faces: ROAM-ASD 93.3% → ROAM-ASD-C 99.5%. The per-face model lights up a silent face while another person talks (orange ring on the left); the context model, comparing the four, keeps it silent.

UniTalk — ABqo634gI5s, 219–225 s, 4 faces (full scene)

Four singers, all voicing at once, four numbered faces in all three levels. Frame accuracy over the four faces: ROAM-ASD 76.8% → ROAM-ASD-C 90.7%. ROAM-ASD scores face #4 as silent for most of the clip although he is singing; ROAM-ASD-C, seeing the other three, keeps him on.