The install finished and then could not download a 230 KB file

With the version ceiling and the widened numpy pin in place, setup succeeded
on the Mac that found them - Python 3.14 chosen and accepted, numpy 2.5.3,
onnxruntime 1.30, faiss 1.15.1, the engine itself - and died on the last step,
fetching the YuNet model:

  ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED]
  certificate verify failed: unable to get local issuer certificate

A python.org macOS build ships its own OpenSSL with NO trust store, and
populates one only when somebody double-clicks Install Certificates.command in
the Python folder. Nobody installing face-recognition software has a reason to
know that exists, and the failure is forty lines of traceback about _ssl.c at
the end of a ten-minute install.

_urlopen tries the default context first and retries with certifi's bundle on
a verification failure. The order is the design:

- Default first, because on Windows and on a system or Homebrew Python the
  default context reads the machine's own certificate store, which is what
  makes a corporate proxy with its own root CA work. Replacing it
  unconditionally would break every site that has one to fix a different
  platform.
- certifi second, because it is already installed: requests is a hard
  dependency and brings it.
- URLError is re-raised untouched. "No route to host" and "no trust store" are
  different problems, and retrying the first with a different CA list only
  delays the real message.

urlretrieve had to go, since it offers no way to pass a context - exactly the
kind of rewrite that silently drops something. The `download: <label> <n>%`
lines are a contract: supervisor.go's progressRe parses them to put first-run
progress in the tray, because the API is not up yet and a shop PC showing a
stopped engine for five minutes looks broken. A test asserts them, and the
rewritten fetch was checked against the real URL: 232,589 bytes, sha256
identical to the model already on disk.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KGcjxF1cNLcuwc3DAPcnfj

CLAUDE.md

@@ -3495,6 +3495,22 @@ an uncapped `>=4.8.1` means every NEW install silently gets a major release
 this project has never run a real camera through while every existing one keeps
 4.11.
 
+### And the fix was defeated by the wreckage of the bug
+
+`makeVenv` reused any environment already on disk, whatever Python built it.
+That machine had a runtime built by **3.14**, left behind by the run that
+failed — so with the ceiling in place setup would choose a good interpreter,
+reach `makeVenv`, find the 3.14 environment, keep it, and die in the same clang
+error as before. A fix a user cannot reach because the bug's own debris is in
+the way is not a fix, and it would have read as the release not working.
+
+It now asks the interpreter inside an existing environment what it is and
+rebuilds when the answer is unsupported, saying so. Rebuilding costs a
+re-download of the libraries and nothing else — the models live in the state
+root, not in there. An environment that cannot be asked counts as unusable
+too: a half-created one answers nothing, and reusing it fails later in pip
+with an error about a package rather than about the environment.
+
 ### One MQTT client id for a whole shop, so two PCs fought over it
 
 `behavision-<client>-<site>` is the same string on every computer claimed to
@@ -3543,3 +3559,43 @@ one, and why installing into it would not survive a restart. Fixed by dragging
 the app to Applications; saying nothing leaves somebody re-running a setup tool
 that cannot win. The product is unsigned, so this is the *normal* first-run
 state on every Mac, not an edge case.
+
+### And then it could not download a 230 KB file
+
+With all of the above fixed the install succeeded on that Mac - Python 3.14
+chosen and accepted, numpy 2.5.3, onnxruntime 1.30, faiss 1.15.1, the engine
+itself - and setup died on the last step, fetching the YuNet model:
+
+```
+ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED]
+certificate verify failed: unable to get local issuer certificate
+```
+
+A python.org macOS build ships its **own OpenSSL with no trust store**, and
+populates one only when somebody double-clicks `Install Certificates.command`
+in the Python folder. Nobody installing face-recognition software has any
+reason to know that exists, and the failure is forty lines of traceback about
+`_ssl.c` at the end of a ten-minute install.
+
+`_urlopen` tries the default context first and retries with **certifi's**
+bundle on a verification failure. The order is the whole design:
+
+- Default first, because on Windows and on a system or Homebrew Python the
+  default context reads the machine's own certificate store - which is what
+  makes a corporate proxy with its own root CA work. Replacing it
+  unconditionally would break every site that has one in order to fix a
+  different platform.
+- certifi second, because it is already installed: `requests` is a hard
+  dependency and brings it.
+- A `URLError` is re-raised untouched. "No route to host" and "no trust store"
+  are different problems, and retrying the first with a different CA list only
+  delays the real message.
+
+`urlretrieve` had to go, since it offers no way to pass a context - and that is
+exactly the kind of rewrite that silently drops something. The
+`download: <label> <n>%` lines are a **contract**: `supervisor.go`'s
+`progressRe` parses them to put first-run progress in the tray, because the API
+is not up yet and a shop PC showing a stopped engine for five minutes after
+install looks broken. `tests/test_model_download.py` asserts them, and the
+rewritten fetch was checked against the real URL: 232,589 bytes, sha256
+identical to the model already on disk.

agent/cmd/behavision-setup/main.go

@@ -473,14 +473,52 @@ func venvPython(venv string) string {
 // engine requires - inside a shared interpreter is how you break the other
 // thing months later, silently.
 func makeVenv(py, venv string) error {
+	// An existing environment is reused - but only if the Python inside it is
+	// one this build supports.
+	//
+	// It used to be reused unconditionally, and that would have made the
+	// version ceiling above look like it did not work. The machine this was
+	// all found on already had a runtime built by Python 3.14, from the run
+	// that failed: with the ceiling in place setup would choose a good
+	// interpreter, reach here, find the 3.14 environment, keep it, and die in
+	// the same clang error as before. A fix that is defeated by the wreckage
+	// of the bug it fixes is not one.
+	//
+	// Rebuilding costs a re-download of the libraries and nothing else. The
+	// models are in the state root, not in here, so they survive.
 	if _, err := os.Stat(venvPython(venv)); err == nil {
-		return nil // already built; pip below brings it up to date
+		ok, ver := venvUsable(venv)
+		if ok {
+			return nil // pip below brings it up to date
+		}
+		fmt.Printf("  [..] %-24s %s\n", "Rebuilding environment",
+			"the existing one uses "+ver+", which is not supported")
+		if err := os.RemoveAll(venv); err != nil {
+			return fmt.Errorf("removing the old environment at %s: %w", venv, err)
+		}
 	}
 	exe, args := splitLauncher(py)
 	args = append(args, "-m", "venv", venv)
 	return stream(exec.Command(exe, args...), "creating the virtual environment")
 }
 
+// venvUsable reports whether the interpreter already inside an environment is
+// one this build supports, and what it is when it is not.
+//
+// An environment that cannot be asked counts as unusable: a half-created or
+// truncated one answers nothing, and reusing it fails later in pip with an
+// error about a package rather than about the environment.
+func venvUsable(venv string) (bool, string) {
+	out, err := exec.Command(venvPython(venv), "-c",
+		"import sys;print('%d.%d'%sys.version_info[:2])").Output()
+	if err != nil {
+		return false, "an interpreter that will not run"
+	}
+	ver := strings.TrimSpace(string(out))
+	major, minor, parsed := parseVer(ver)
+	return pythonVerdict(major, minor, parsed) == verdictOK, "Python " + ver
+}
+
 func pipInstall(vpy, src string) error {
 	fmt.Println("  Installing the engine and its libraries. This downloads a few")
 	fmt.Println("  hundred megabytes and takes a while on a slow connection.")

agent/cmd/behavision-setup/python_test.go

@@ -1,6 +1,10 @@
 package main
 
-import "testing"
+import (
+	"os"
+	"path/filepath"
+	"testing"
+)
 
 // The choice this program makes silently, and got wrong.
 //
@@ -64,3 +68,45 @@ func TestUnknownVersionsAreNotAccepted(t *testing.T) {
 		}
 	}
 }
+
+// An environment already on disk is reused, and that is right until the Python
+// inside it is one this build cannot use.
+//
+// It was reused unconditionally, which would have defeated the ceiling above
+// on the exact machine that found the bug: that Mac already had a runtime
+// built by Python 3.14, left behind by the run that failed. Setup would pick a
+// good interpreter, find the 3.14 environment, keep it, and die in the same
+// clang error as before - a fix defeated by the wreckage of the bug it fixes.
+//
+// Real environments, not a fake: the thing under test is what an interpreter
+// on disk reports about itself.
+func TestAnUnsupportedEnvironmentIsNotReused(t *testing.T) {
+	py, _, err := findPython()
+	if err != nil {
+		t.Skipf("no supported Python on this machine: %v", err)
+	}
+	venv := filepath.Join(t.TempDir(), "runtime")
+	if err := makeVenv(py, venv); err != nil {
+		t.Fatalf("makeVenv: %v", err)
+	}
+	if ok, ver := venvUsable(venv); !ok {
+		t.Fatalf("an environment built from the interpreter setup just chose "+
+			"reported itself unusable (%s)", ver)
+	}
+
+	// The two states that must not be confused with a working one.
+	empty := filepath.Join(t.TempDir(), "gone")
+	if ok, _ := venvUsable(empty); ok {
+		t.Error("a missing environment was reported usable")
+	}
+	broken := filepath.Join(t.TempDir(), "broken")
+	if err := os.MkdirAll(filepath.Dir(venvPython(broken)), 0o755); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.WriteFile(venvPython(broken), []byte("not an interpreter"), 0o755); err != nil {
+		t.Fatal(err)
+	}
+	if ok, ver := venvUsable(broken); ok {
+		t.Errorf("a half-created environment was reported usable (%s)", ver)
+	}
+}

behavision/model_assets.py

@@ -4,6 +4,7 @@ from __future__ import annotations
 
 import logging
 import shutil
+import ssl
 import urllib.request
 from pathlib import Path
 
@@ -35,6 +36,54 @@ _COPY_MAP = {
 }
 
 
+def _https_context() -> "ssl.SSLContext | None":
+    """The CA store to trust, or None to use whatever Python defaults to.
+
+    Returning None first is deliberate. On Windows and on a Homebrew or
+    system Python, the default context reads the machine's own certificate
+    store - which is what makes a corporate proxy with its own root CA work.
+    Replacing that with certifi's bundle unconditionally would break every
+    site that has one, in order to fix a different platform.
+
+    The platform this fixes is a python.org macOS build. It ships its own
+    OpenSSL with NO trust store, and populates one only when somebody
+    double-clicks `Install Certificates.command` in the Python folder -
+    which nobody installing face-recognition software has any reason to know
+    about. Every HTTPS request from that interpreter fails with:
+
+        ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED]
+        certificate verify failed: unable to get local issuer certificate
+
+    Measured on a colleague's Mac: the engine installed perfectly and then
+    could not download a 230 KB model file, ending setup in forty lines of
+    traceback about `_ssl.c`.
+    """
+    try:
+        import certifi
+    except ImportError:  # pragma: no cover - certifi ships with requests
+        return None
+    return ssl.create_default_context(cafile=certifi.where())
+
+
+def _urlopen(url: str, timeout: float = 60.0):
+    """Open a URL, falling back to certifi's CA bundle on a verify failure.
+
+    Default first, certifi second, so the fix is additive: a machine whose
+    own store works keeps using it, and one with no store at all gets a
+    bundle rather than a traceback. certifi is already here - `requests` is a
+    hard dependency and brings it.
+    """
+    try:
+        return urllib.request.urlopen(url, timeout=timeout)
+    except ssl.SSLCertVerificationError:
+        ctx = _https_context()
+        if ctx is None:
+            raise
+        log.info("the system certificate store could not verify %s; "
+                 "using the bundled CA list", url.split("/")[2])
+        return urllib.request.urlopen(url, timeout=timeout, context=ctx)
+
+
 def _fetch(url: str, dest: Path, label: str) -> None:
     """Download with progress on stdout the supervisor can read.
 
@@ -56,7 +105,22 @@ def _fetch(url: str, dest: Path, label: str) -> None:
             last = pct
             log.info("download: %s %d%%", label, pct)
 
-    urllib.request.urlretrieve(url, dest, hook)
+    # Streamed rather than urlretrieve, only because urlretrieve offers no way
+    # to pass an SSL context and the whole point here is choosing one. The
+    # `download: <label> <n>%` lines are a contract: the supervisor parses
+    # them (`progressRe`) to put first-run progress in the tray, and without
+    # them a shop PC shows a stopped engine for five minutes after install.
+    with _urlopen(url) as resp:
+        total = int(resp.headers.get("Content-Length") or 0)
+        blocks, block_size = 0, 64 * 1024
+        with open(dest, "wb") as out:
+            while True:
+                chunk = resp.read(block_size)
+                if not chunk:
+                    break
+                out.write(chunk)
+                blocks += 1
+                hook(blocks, block_size, total)
     log.info("download: %s 100%%", label)
 
 
@@ -91,7 +155,7 @@ def setup_models(models_dir: Path) -> "list[str]":
         import io
         import zipfile
 
-        with urllib.request.urlopen(BUFFALO_SC_URL) as resp:
+        with _urlopen(BUFFALO_SC_URL) as resp:
             payload = io.BytesIO(resp.read())
         with zipfile.ZipFile(payload) as zf, \
                 zf.open("w600k_mbf.onnx") as src, \

tests/test_model_download.py

@@ -0,0 +1,105 @@
+"""Downloading the models is the last step of every fresh install, and it ran
+into the one macOS trap nothing else here does.
+
+A python.org macOS build ships its own OpenSSL with NO trust store, and
+populates one only when somebody double-clicks `Install Certificates.command`
+in the Python folder. Measured on a colleague's Mac: the engine installed
+perfectly - numpy, onnxruntime, faiss, all of it - and then could not fetch a
+230 KB model file, ending setup in forty lines of traceback about `_ssl.c`.
+"""
+import io
+import logging
+import ssl
+import urllib.request
+
+import pytest
+
+from behavision import model_assets
+
+
+class _Resp(io.BytesIO):
+    """Enough of an http response for _fetch: read() and .headers."""
+
+    def __init__(self, payload: bytes):
+        super().__init__(payload)
+        self.headers = {"Content-Length": str(len(payload))}
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, *exc):
+        self.close()
+        return False
+
+
+def _verify_error():
+    return ssl.SSLCertVerificationError(
+        "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: "
+        "unable to get local issuer certificate")
+
+
+def test_a_machine_with_no_trust_store_falls_back_to_the_bundled_one(monkeypatch):
+    calls = []
+
+    def fake(url, timeout=None, context=None):
+        calls.append(context)
+        if context is None:
+            raise _verify_error()
+        return _Resp(b"ok")
+
+    monkeypatch.setattr(urllib.request, "urlopen", fake)
+    with model_assets._urlopen("https://example.invalid/m.onnx") as resp:
+        assert resp.read() == b"ok"
+
+    assert len(calls) == 2, f"expected a retry, got {calls}"
+    # Default FIRST, and that order is the point. On Windows and on a system
+    # Python the default context reads the machine's own certificate store,
+    # which is what makes a corporate proxy with its own root CA work.
+    # Replacing it unconditionally would break every site that has one in
+    # order to fix a different platform.
+    assert calls[0] is None
+    assert isinstance(calls[1], ssl.SSLContext)
+
+
+def test_a_working_trust_store_is_used_as_is(monkeypatch):
+    calls = []
+
+    def fake(url, timeout=None, context=None):
+        calls.append(context)
+        return _Resp(b"ok")
+
+    monkeypatch.setattr(urllib.request, "urlopen", fake)
+    model_assets._urlopen("https://example.invalid/m.onnx").close()
+    assert calls == [None], "the machine's own certificate store was bypassed"
+
+
+def test_a_real_network_failure_is_not_disguised_as_a_certificate_problem(monkeypatch):
+    def fake(url, timeout=None, context=None):
+        raise urllib.error.URLError("no route to host")
+
+    monkeypatch.setattr(urllib.request, "urlopen", fake)
+    with pytest.raises(urllib.error.URLError):
+        model_assets._urlopen("https://example.invalid/m.onnx")
+
+
+def test_progress_lines_survive_the_rewrite(tmp_path, monkeypatch, caplog):
+    """`download: <label> <n>%` is a CONTRACT, not logging.
+
+    The supervisor parses it (progressRe) to put first-run progress in the
+    tray and the window, because the API is not up yet and a shop PC showing
+    a stopped engine for five minutes after install looks broken. Switching
+    off urlretrieve - needed because it offers no way to pass an SSL context -
+    is exactly the kind of change that drops it silently.
+    """
+    payload = b"x" * (64 * 1024 * 8)
+    monkeypatch.setattr(urllib.request, "urlopen",
+                        lambda url, timeout=None, context=None: _Resp(payload))
+    dest = tmp_path / "model.onnx"
+    with caplog.at_level(logging.INFO, logger="behavision.model_assets"):
+        model_assets._fetch("https://example.invalid/m.onnx", dest, "face detector")
+
+    assert dest.read_bytes() == payload
+    lines = [r.getMessage() for r in caplog.records]
+    pct = [ln for ln in lines if ln.startswith("download: face detector ")]
+    assert pct, f"no progress lines at all: {lines}"
+    assert "download: face detector 100%" in pct, pct