The live picture was chained to the recognition pipeline

Reported from the first Windows install: the camera feed lags. It did,
and not because of the network, the proxy or the webview.

The MJPEG stream served _annotated_jpeg - the frame the pipeline had
most recently FINISHED with, encoded after detection, quality scoring,
tracking and identification had all run on it. On a modest shop PC that
is a few frames a second, and every picture was already as old as that
processing. It looked like lag because it was lag. On the fast machine
it was developed on the pipeline kept up with the stream's own 10 fps
cap, which is why nobody here ever saw it.

Two more things compounded it. Every processed frame was JPEG-encoded
whether or not a viewer existed - CPU spent on precisely the machine
short of it. And ffmpeg ran its RTSP demuxer with default buffering,
which holds a comfortable queue of frames before handing over the first:
half a second to two seconds a live view can never recover.

Now the picture and the boxes are decoupled. latest_jpeg_since takes the
capture thread's freshest frame at the camera's own rate and draws the
boxes from the last processed frame over it - encoded on demand, per
request, so a camera nobody watches costs no encode at all. The stream
sends a frame only when the camera has a newer one, capped at 15 fps;
nothing is sent twice. Boxes older than a second are not drawn, so a
stalled pipeline cannot leave one floating over an empty spot.
_publish_annotated becomes _remember_tracks: a handful of tuples under
the lock, no copy, no encode. ffmpeg gets nobuffer / low_delay /
max_delay.

Measured on cam2's sub-stream, same machine, ten seconds each:

  before   99 frames sent,  98 distinct    9.8 new pictures/s
  after   141 frames sent, 141 distinct   14.0 new pictures/s

against a 15 fps camera, with the pipeline still processing 166 of 181
captured frames alongside - and engine CPU DOWN from 90% with no viewer
to 62% with one attached.

Engine version 1.0.0 -> 1.1.0 so a re-run of setup reinstalls it rather
than pip deciding the requirement is already satisfied.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KGcjxF1cNLcuwc3DAPcnfj

behavision/engine.py

@@ -162,7 +162,11 @@ class CameraWorker(threading.Thread):
         # every test using a stubbed worker passed.
         self._stopping = threading.Event()
         self._lock = threading.Lock()
-        self._annotated_jpeg: Optional[bytes] = None
+        # What the live view draws over the freshest frame: the boxes from
+        # the most recent processed frame, and when they were computed. NOT a
+        # pre-rendered JPEG - see latest_jpeg for why.
+        self._overlay: "list[tuple[tuple[int, int, int, int], tuple[int, int, int], str]]" = []
+        self._overlay_ts = 0.0
         self._last_frame_ts = 0.0
         self._was_connected = False
         self.frames_processed = 0
@@ -187,8 +191,46 @@ class CameraWorker(threading.Thread):
         self.source.stop()
 
     def latest_jpeg(self) -> Optional[bytes]:
+        jpeg, _ = self.latest_jpeg_since(0.0)
+        return jpeg
+
+    def latest_jpeg_since(self, known_ts: float) -> "tuple[Optional[bytes], float]":
+        """The freshest captured frame with the latest boxes drawn on it, or
+        (None, known_ts) if the camera has produced nothing newer.
+
+        The live picture is deliberately NOT the frame the pipeline last
+        finished with. That version advanced only when detection, tracking and
+        identification had all completed on a frame - a few times a second on a
+        modest shop PC - and every picture it showed was already as old as that
+        processing. It looked like lag because it was lag. Here the picture runs
+        at the camera's rate off the capture thread's latest frame, and the
+        boxes - which genuinely can only update at pipeline rate - are drawn
+        over it from the last processed frame. Boxes may trail a fast walker by
+        one pipeline period; the picture never does.
+
+        Encoded on demand, per request, so a camera nobody is watching pays for
+        no JPEG at all. The old path encoded every processed frame whether or
+        not a viewer existed - CPU spent on precisely the machine short of it.
+        """
+        frame, ts = self.source.latest_since(known_ts)
+        if frame is None:
+            return None, known_ts
         with self._lock:
-            return self._annotated_jpeg
+            overlay, overlay_ts = list(self._overlay), self._overlay_ts
+        # A stalled pipeline must not leave a box floating over an empty spot.
+        # Older than a second and the person has walked out from under it.
+        draw = overlay if (time.time() - overlay_ts) < 1.0 else []
+        if draw:
+            frame = frame.copy()
+            for (x1, y1, x2, y2), color, text in draw:
+                cv2.rectangle(frame, (x1, y1), (x2, y2), color, 2)
+                if text:
+                    cv2.putText(frame, text, (x1, max(20, y1 - 8)),
+                                cv2.FONT_HERSHEY_SIMPLEX, 0.55, color, 2)
+        ok, buf = cv2.imencode(".jpg", frame, [int(cv2.IMWRITE_JPEG_QUALITY), 80])
+        if not ok:
+            return None, known_ts
+        return buf.tobytes(), ts
 
     def stats(self) -> dict:
         return {
@@ -236,7 +278,7 @@ class CameraWorker(threading.Thread):
                 for track in ended:
                     self._finish_track(track, ts)
 
-                self._publish_annotated(frame, active)
+                self._remember_tracks(active)
                 self.frames_processed += 1
             except Exception:
                 log.exception("[%s] frame processing failed", self.cam_cfg.id)
@@ -426,12 +468,14 @@ class CameraWorker(threading.Thread):
                                            track.quality, rcfg=self.rcfg):
             track.reinforcements += 1
 
-    def _publish_annotated(self, frame: np.ndarray, tracks: "list[Track]") -> None:
-        canvas = frame.copy()
+    def _remember_tracks(self, tracks: "list[Track]") -> None:
+        """Record what to draw. Cheap: a handful of tuples under the lock,
+        no frame copy and no encode. The encode happens in latest_jpeg_since,
+        only when somebody is looking."""
+        overlay = []
         for t in tracks:
             if t.misses > 0:
                 continue  # only draw tracks matched in this frame
-            x1, y1, x2, y2 = t.box
             if t.state == "resolved":
                 color = _COLORS["known"] if t.label and not str(t.label).startswith(
                     "Visitor") else _COLORS["new"]
@@ -440,15 +484,10 @@ class CameraWorker(threading.Thread):
                 color, text = _COLORS["ambiguous"], "?"
             else:
                 color, text = _COLORS["pending"], ""
-            cv2.rectangle(canvas, (x1, y1), (x2, y2), color, 2)
-            if text:
-                cv2.putText(canvas, text, (x1, max(20, y1 - 8)),
-                            cv2.FONT_HERSHEY_SIMPLEX, 0.55, color, 2)
-        ok, buf = cv2.imencode(".jpg", canvas,
-                               [int(cv2.IMWRITE_JPEG_QUALITY), 80])
-        if ok:
-            with self._lock:
-                self._annotated_jpeg = buf.tobytes()
+            overlay.append((tuple(t.box), color, text))
+        with self._lock:
+            self._overlay = overlay
+            self._overlay_ts = time.time()
 
 
 class Engine: