Real-Time Face & Emotion Inference in the Browser
REPORT_ABSTRACT
Face & Emotion Tracker has no offline pipeline: every frame is captured from getUserMedia and run through face-api.js live, on the visitor's own machine, the moment the page loads. TinyFaceDetector locates faces, a 68-point landmark model traces the facial contour (iBUG 300-W layout), and a 7-class expression classifier scores the dominant emotion. This report details each stage of the model stack and the render loop that stitches it together.
01. THE THREE-STAGE MODEL STACK
face-api.js runs on TensorFlow.js and chains three lightweight models per frame. TinyFaceDetector is a compact single-shot detector that returns face bounding boxes with confidence scores; the 68-point landmark model then regresses facial keypoints from each box; finally, a 7-class expression classifier (neutral, happy, sad, angry, fearful, disgusted, surprised) emits softmax scores over the cropped face region. Each stage feeds the next, keeping per-frame compute proportional to the number of detected faces.
02. 68-POINT LANDMARK REGRESSION
The landmark model follows the iBUG 300-W annotation layout: 17 points along the jawline, 5 per eyebrow, 4 per eye, 9 on the nose, and 20 around the mouth. Rather than a fixed template, the overlay is drawn per frame from these coordinates onto a canvas sized to the native video resolution, so the mesh tracks head movement, scale changes, and partial occlusion in real time.
03. EMOTION SCORING AND RENDERING
For each detected face, the expression classifier produces a probability distribution over seven emotions. The dominant class is picked and rendered as a label anchored to the face mesh. Because the classifier consumes the same GPU tensors as the detection stages, there is no expensive host-device roundtrip — the entire chain stays inside WebGL whenever the backend supports it.
04. THE FRAME LOOP
The capture loop is the standard getUserMedia + requestAnimationFrame pattern, with two refinements: the canvas is sized to the native video stream resolution (never the element's CSS size) so the mesh is pixel-accurate, and inference is decoupled from paint via a rolling 20-frame FPS counter that reports the true end-to-end cost of detection + landmarking + classification.
05. RUNTIME HARDENING AND FALLBACKS
Model weights are pinned in public/models/, removing any external CDN dependency at runtime. The TensorFlow.js WebGL backend is preferred for speed with an automatic CPU fallback for machines without a usable GPU. If camera access is denied or getUserMedia is unavailable, the UI degrades gracefully with an inline message instead of a silent crash.