4 Commits

Author SHA1 Message Date
0558344dc2 urllib does not let SSLCertVerificationError out, and a fake said it did
The certificate fix shipped and failed on the machine it was written for, with
the exact traceback it was meant to prevent. The retry was written

    except ssl.SSLCertVerificationError:

and urllib never raises that from urlopen. It catches it and re-raises
urllib.error.URLError(err), carrying the original on .reason. So the except
matched nothing, ever, and the fallback could not fire.

The unit test passed throughout, because the stub it used raised the bare SSL
error - a shape real urllib never produces. That is the lesson: a fake that
agrees with the author is worse than no test, because it converts an untested
path into a tested-looking one. This file already says that about
UPDATE ... RETURNING and about the in-memory API fake, and it got written
again anyway.

_is_cert_failure checks the exception and its .reason, and the tests now raise
URLError(SSLCertVerificationError(...)) - what the traceback actually shows. A
plain URLError is re-raised untouched, and a test asserts no second attempt is
made for one.

Beside the stubs there is now a real reproduction, opt-in behind
BEHAVISION_NETWORK_TESTS=1. Python's default context honours SSL_CERT_FILE, so
an empty file gives a context that trusts nobody - the python.org condition
exactly - while certifi is loaded by path and is unaffected. It skips rather
than passes where it cannot reproduce that, and the difference is measured:

    macOS Command Line Tools  LibreSSL 2.8.3   128 CAs with an empty CA file
    python.org / pyenv build  OpenSSL 3.5.8      0 CAs -> reproduces it

Checked for teeth by putting the shipped except back: both the corrected stub
test and the live one fail, and pass again when it is restored.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KGcjxF1cNLcuwc3DAPcnfj
2026-09-30 17:36:53 +05:30
8f07dee048 The engine stopped updating, and said "installed" every time
The SSL fix shipped and did not reach the machine it was written for. That
log said so, one line above the tick:

  behavision is already installed with the same version as the provided
  wheel. Use --force-reinstall to force an installation of the wheel.
   [ok] Engine and dependencies installed

and the traceback below it still pointed at model_assets.py line 59,
urllib.request.urlretrieve - code the fix had deleted.

Two frozen literals caused it: version = "1.1.0" in pyproject.toml and
__version__ = "1.0.0" in behavision/__init__.py. They disagreed with each
other and neither tracked a release, so every release built
behavision-1.1.0-py3-none-any.whl and pip install --upgrade on a machine that
already had 1.1.0 is a no-op. The comment beside that call claimed the
opposite.

The shape of the damage is what makes it bad. The Go binaries - app, agent,
setup tool - are rebuilt every release and updated normally. So a shop PC ran
a current app supervising an engine several releases old, and nothing said
which: /api/health reported the model, the paths, the cameras and the gallery,
and no version at all.

- One version, in the package, read by pyproject through
  [tool.setuptools.dynamic]. In a checkout it reads 0.0.0+dev: a
  plausible-looking number on a developer's /api/health is worse than none.
- pip install --force-reinstall --no-deps <wheel>, after the ordinary
  --upgrade. --upgrade settles the dependencies; the second call guarantees our
  own code is the code in the folder. --no-deps keeps it cheap - forcing the
  dependencies too would re-download ~300 MB every run. A rebuild at an
  unchanged version is the ordinary case while developing, so this must not
  rely on the version moving.
- release.sh stamps the tag: v0.5.6-demo -> 0.5.6+demo, valid PEP 440. Into a
  copy of the line, reverted in a trap, so the tree is never left dirty.

/api/health reports version now. Without it there is no way to tell a shop PC
three releases behind from a current one, which is how this survived several
releases.

Reproduced end to end before fixing, against real wheels on Python 3.12: two
builds of the same version, --upgrade leaves the old code in place and prints
the same sentence the colleague's Mac printed, --force-reinstall --no-deps
replaces it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KGcjxF1cNLcuwc3DAPcnfj
2026-09-30 17:30:24 +05:30
3cddd9c2e1 The install finished and then could not download a 230 KB file
With the version ceiling and the widened numpy pin in place, setup succeeded
on the Mac that found them - Python 3.14 chosen and accepted, numpy 2.5.3,
onnxruntime 1.30, faiss 1.15.1, the engine itself - and died on the last step,
fetching the YuNet model:

  ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED]
  certificate verify failed: unable to get local issuer certificate

A python.org macOS build ships its own OpenSSL with NO trust store, and
populates one only when somebody double-clicks Install Certificates.command in
the Python folder. Nobody installing face-recognition software has a reason to
know that exists, and the failure is forty lines of traceback about _ssl.c at
the end of a ten-minute install.

_urlopen tries the default context first and retries with certifi's bundle on
a verification failure. The order is the design:

- Default first, because on Windows and on a system or Homebrew Python the
  default context reads the machine's own certificate store, which is what
  makes a corporate proxy with its own root CA work. Replacing it
  unconditionally would break every site that has one to fix a different
  platform.
- certifi second, because it is already installed: requests is a hard
  dependency and brings it.
- URLError is re-raised untouched. "No route to host" and "no trust store" are
  different problems, and retrying the first with a different CA list only
  delays the real message.

urlretrieve had to go, since it offers no way to pass a context - exactly the
kind of rewrite that silently drops something. The `download: <label> <n>%`
lines are a contract: supervisor.go's progressRe parses them to put first-run
progress in the tray, because the API is not up yet and a shop PC showing a
stopped engine for five minutes looks broken. A test asserts them, and the
rewritten fetch was checked against the real URL: 232,589 bytes, sha256
identical to the model already on disk.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KGcjxF1cNLcuwc3DAPcnfj
2026-09-30 17:21:41 +05:30
248025cdf9 The Python ceiling was defeated by the venv the failure left behind
makeVenv reused any environment already on disk, whatever Python built it.
The machine that found the version bug already had a runtime built by 3.14,
left there by the run that failed - so with the ceiling in place setup would
choose a good interpreter, reach makeVenv, find the 3.14 environment, keep it,
and die in the same clang error as before.

A fix a user cannot reach because the bug's own debris is in the way is not a
fix, and it would have read as the release not working.

It now asks the interpreter inside an existing environment what it is and
rebuilds when the answer is unsupported, saying so. Rebuilding costs a
re-download of the libraries and nothing else - the models live in the state
root. An environment that cannot be asked counts as unusable too: a
half-created one answers nothing, and reusing it fails later in pip with an
error about a package rather than about the environment.

Tested against real environments rather than a fake, because what is under
test is what an interpreter on disk reports about itself.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KGcjxF1cNLcuwc3DAPcnfj
2026-09-30 17:11:47 +05:30
9 changed files with 570 additions and 8 deletions

156
CLAUDE.md
View File

@@ -3495,6 +3495,22 @@ an uncapped `>=4.8.1` means every NEW install silently gets a major release
this project has never run a real camera through while every existing one keeps
4.11.
### And the fix was defeated by the wreckage of the bug
`makeVenv` reused any environment already on disk, whatever Python built it.
That machine had a runtime built by **3.14**, left behind by the run that
failed — so with the ceiling in place setup would choose a good interpreter,
reach `makeVenv`, find the 3.14 environment, keep it, and die in the same clang
error as before. A fix a user cannot reach because the bug's own debris is in
the way is not a fix, and it would have read as the release not working.
It now asks the interpreter inside an existing environment what it is and
rebuilds when the answer is unsupported, saying so. Rebuilding costs a
re-download of the libraries and nothing else — the models live in the state
root, not in there. An environment that cannot be asked counts as unusable
too: a half-created one answers nothing, and reusing it fails later in pip
with an error about a package rather than about the environment.
### One MQTT client id for a whole shop, so two PCs fought over it
`behavision-<client>-<site>` is the same string on every computer claimed to
@@ -3543,3 +3559,143 @@ one, and why installing into it would not survive a restart. Fixed by dragging
the app to Applications; saying nothing leaves somebody re-running a setup tool
that cannot win. The product is unsigned, so this is the *normal* first-run
state on every Mac, not an edge case.
### And then it could not download a 230 KB file
With all of the above fixed the install succeeded on that Mac - Python 3.14
chosen and accepted, numpy 2.5.3, onnxruntime 1.30, faiss 1.15.1, the engine
itself - and setup died on the last step, fetching the YuNet model:
```
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate
```
A python.org macOS build ships its **own OpenSSL with no trust store**, and
populates one only when somebody double-clicks `Install Certificates.command`
in the Python folder. Nobody installing face-recognition software has any
reason to know that exists, and the failure is forty lines of traceback about
`_ssl.c` at the end of a ten-minute install.
`_urlopen` tries the default context first and retries with **certifi's**
bundle on a verification failure. The order is the whole design:
- Default first, because on Windows and on a system or Homebrew Python the
default context reads the machine's own certificate store - which is what
makes a corporate proxy with its own root CA work. Replacing it
unconditionally would break every site that has one in order to fix a
different platform.
- certifi second, because it is already installed: `requests` is a hard
dependency and brings it.
- A `URLError` is re-raised untouched. "No route to host" and "no trust store"
are different problems, and retrying the first with a different CA list only
delays the real message.
`urlretrieve` had to go, since it offers no way to pass a context - and that is
exactly the kind of rewrite that silently drops something. The
`download: <label> <n>%` lines are a **contract**: `supervisor.go`'s
`progressRe` parses them to put first-run progress in the tray, because the API
is not up yet and a shop PC showing a stopped engine for five minutes after
install looks broken. `tests/test_model_download.py` asserts them, and the
rewritten fetch was checked against the real URL: 232,589 bytes, sha256
identical to the model already on disk.
## The engine stopped updating, and said "installed" every time
The SSL fix above was released and **did not reach the machine it was written
for**. Its log said so, one line above the tick:
```
behavision is already installed with the same version as the provided wheel.
Use --force-reinstall to force an installation of the wheel.
[ok] Engine and dependencies installed
```
and the traceback that followed still pointed at `model_assets.py", line 59,
in _fetch / urllib.request.urlretrieve` — code the fix had deleted.
Two frozen literals caused it: `version = "1.1.0"` in `pyproject.toml` and
`__version__ = "1.0.0"` in `behavision/__init__.py`. They disagreed with each
other and neither tracked a release, so **every release built
`behavision-1.1.0-py3-none-any.whl`** and `pip install --upgrade` on a machine
that already had 1.1.0 is a no-op. The comment beside that call even claimed
the opposite: *"`--upgrade` so re-running after a new release replaces the
engine rather than leaving the old one in place and reporting success."*
The shape of the damage is what makes it bad. The Go binaries — the app, the
agent, the setup tool — are rebuilt every release and updated normally. So a
shop PC ran a current app supervising an engine several releases old, and
nothing anywhere said which: `/api/health` reported the model, the paths, the
cameras and the gallery, and **no version at all**.
Three changes, and the second is the one that does not depend on the first
being remembered:
- **One version, in the package**, read by `pyproject.toml` through
`[tool.setuptools.dynamic]`. Two literals in two files is how they came to
disagree. In a checkout it reads `0.0.0+dev`, because a plausible-looking
number on a developer's `/api/health` would be worse than none.
- **`pip install --force-reinstall --no-deps <wheel>`, after the ordinary
`--upgrade`.** `--upgrade` settles the dependencies; the second call
guarantees our own code is the code in the folder. `--no-deps` is what keeps
it cheap — forcing the dependencies too would re-download ~300 MB of numpy,
OpenCV and onnxruntime on every run. A rebuild at an unchanged version is the
ordinary case while developing, so this must not rely on the version moving.
- **`release.sh` stamps the tag**: `v0.5.6-demo` → `0.5.6+demo`, valid PEP 440
(a local segment takes alphanumerics and dots, never hyphens). Into a copy of
the line, reverted in a `trap`, so the tree is never left dirty.
`/api/health` reports `version` now. Without it there is no way to tell a shop
PC three releases behind from a current one, which is precisely how this
survived several releases.
Reproduced end to end before fixing, against real wheels on Python 3.12: two
builds of the same version, `--upgrade` leaves the old code in place and prints
the same sentence the colleague's Mac printed, `--force-reinstall --no-deps`
replaces it.
### `urllib` does not let `SSLCertVerificationError` out, and a fake said it did
The certificate fix above shipped and **failed on the machine it was written
for, with the exact traceback it was meant to prevent**. The retry was written
```python
except ssl.SSLCertVerificationError:
```
and urllib never raises that from `urlopen`. It catches it and re-raises
`urllib.error.URLError(err)`, carrying the original on `.reason`. So the
`except` matched nothing, ever, and the fallback could not fire.
The unit test passed throughout, because the stub it used raised the bare SSL
error — **a shape real urllib never produces**. That is the whole lesson: a
fake that agrees with the author is worse than no test, because it converts an
untested path into a tested-looking one. The same sentence is already in this
file about `UPDATE ... RETURNING` and about the in-memory API fake, and it was
written again here anyway.
`_is_cert_failure` checks the exception **and** its `.reason`, and the tests
now raise `URLError(SSLCertVerificationError(...))`, which is what his
traceback shows. A plain `URLError` — "no route to host" — is re-raised
untouched: retrying that with a different CA list changes nothing except how
long the operator waits for the real message, and a test asserts the second
attempt never happens.
Beside the stubs there is now a **real** reproduction,
`test_against_a_real_machine_with_no_trust_store`, opt-in behind
`BEHAVISION_NETWORK_TESTS=1` because it reaches github.com. Python's default
context honours `SSL_CERT_FILE`, so pointing it at an empty file gives a
default context that trusts nobody — the python.org condition exactly — while
certifi is loaded by path and is unaffected.
It skips rather than passes where it cannot reproduce that, and the difference
is measured rather than assumed:
```
macOS Command Line Tools LibreSSL 2.8.3 128 CAs with SSL_CERT_FILE empty
(reads the system keychain)
python.org / pyenv build OpenSSL 3.5.8 0 CAs -> reproduces it
```
Checked for teeth by putting the shipped `except` back: both the corrected stub
test and the live one fail, and pass again when it is restored.

View File

@@ -473,14 +473,52 @@ func venvPython(venv string) string {
// engine requires - inside a shared interpreter is how you break the other
// thing months later, silently.
func makeVenv(py, venv string) error {
// An existing environment is reused - but only if the Python inside it is
// one this build supports.
//
// It used to be reused unconditionally, and that would have made the
// version ceiling above look like it did not work. The machine this was
// all found on already had a runtime built by Python 3.14, from the run
// that failed: with the ceiling in place setup would choose a good
// interpreter, reach here, find the 3.14 environment, keep it, and die in
// the same clang error as before. A fix that is defeated by the wreckage
// of the bug it fixes is not one.
//
// Rebuilding costs a re-download of the libraries and nothing else. The
// models are in the state root, not in here, so they survive.
if _, err := os.Stat(venvPython(venv)); err == nil {
return nil // already built; pip below brings it up to date
ok, ver := venvUsable(venv)
if ok {
return nil // pip below brings it up to date
}
fmt.Printf(" [..] %-24s %s\n", "Rebuilding environment",
"the existing one uses "+ver+", which is not supported")
if err := os.RemoveAll(venv); err != nil {
return fmt.Errorf("removing the old environment at %s: %w", venv, err)
}
}
exe, args := splitLauncher(py)
args = append(args, "-m", "venv", venv)
return stream(exec.Command(exe, args...), "creating the virtual environment")
}
// venvUsable reports whether the interpreter already inside an environment is
// one this build supports, and what it is when it is not.
//
// An environment that cannot be asked counts as unusable: a half-created or
// truncated one answers nothing, and reusing it fails later in pip with an
// error about a package rather than about the environment.
func venvUsable(venv string) (bool, string) {
out, err := exec.Command(venvPython(venv), "-c",
"import sys;print('%d.%d'%sys.version_info[:2])").Output()
if err != nil {
return false, "an interpreter that will not run"
}
ver := strings.TrimSpace(string(out))
major, minor, parsed := parseVer(ver)
return pythonVerdict(major, minor, parsed) == verdictOK, "Python " + ver
}
func pipInstall(vpy, src string) error {
fmt.Println(" Installing the engine and its libraries. This downloads a few")
fmt.Println(" hundred megabytes and takes a while on a slow connection.")
@@ -501,7 +539,30 @@ func pipInstall(vpy, src string) error {
// 'behavision.egg-info': Read-only file system". Falling back to source
// copies it somewhere writable first, for the same reason.
if wheels, _ := filepath.Glob(filepath.Join(src, "behavision-*.whl")); len(wheels) > 0 {
return stream(exec.Command(vpy, "-m", "pip", "install", "--upgrade", wheels[0]),
// TWO calls, and the second is the one that matters.
//
// `--upgrade` alone is not an upgrade when the version has not moved:
// pip skips the wheel and says so, one line above this program
// printing "[ok] Engine and dependencies installed". The wheel version
// was a frozen literal for several releases, so every engine fix in
// them silently failed to reach any machine that had run setup once -
// while the Go binaries beside it, rebuilt every release, updated
// normally. Half the product current, half of it months old, and
// nothing saying which.
//
// release.sh stamps the tag into the version now, so the versions do
// differ. This does not rely on that: a rebuild at the same version is
// the ordinary case while developing, and "installed" has to mean the
// code in this folder either way.
if err := stream(exec.Command(vpy, "-m", "pip", "install", "--upgrade", wheels[0]),
"installing the engine"); err != nil {
return err
}
// --no-deps so this is our own package only: the call above has
// already settled the dependencies, and forcing those too would
// re-download ~300 MB of numpy, OpenCV and onnxruntime every run.
return stream(exec.Command(vpy, "-m", "pip", "install",
"--force-reinstall", "--no-deps", wheels[0]),
"installing the engine")
}
tmp, err := os.MkdirTemp("", "behavision-src-")

View File

@@ -1,6 +1,10 @@
package main
import "testing"
import (
"os"
"path/filepath"
"testing"
)
// The choice this program makes silently, and got wrong.
//
@@ -64,3 +68,45 @@ func TestUnknownVersionsAreNotAccepted(t *testing.T) {
}
}
}
// An environment already on disk is reused, and that is right until the Python
// inside it is one this build cannot use.
//
// It was reused unconditionally, which would have defeated the ceiling above
// on the exact machine that found the bug: that Mac already had a runtime
// built by Python 3.14, left behind by the run that failed. Setup would pick a
// good interpreter, find the 3.14 environment, keep it, and die in the same
// clang error as before - a fix defeated by the wreckage of the bug it fixes.
//
// Real environments, not a fake: the thing under test is what an interpreter
// on disk reports about itself.
func TestAnUnsupportedEnvironmentIsNotReused(t *testing.T) {
py, _, err := findPython()
if err != nil {
t.Skipf("no supported Python on this machine: %v", err)
}
venv := filepath.Join(t.TempDir(), "runtime")
if err := makeVenv(py, venv); err != nil {
t.Fatalf("makeVenv: %v", err)
}
if ok, ver := venvUsable(venv); !ok {
t.Fatalf("an environment built from the interpreter setup just chose "+
"reported itself unusable (%s)", ver)
}
// The two states that must not be confused with a working one.
empty := filepath.Join(t.TempDir(), "gone")
if ok, _ := venvUsable(empty); ok {
t.Error("a missing environment was reported usable")
}
broken := filepath.Join(t.TempDir(), "broken")
if err := os.MkdirAll(filepath.Dir(venvPython(broken)), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(venvPython(broken), []byte("not an interpreter"), 0o755); err != nil {
t.Fatal(err)
}
if ok, ver := venvUsable(broken); ok {
t.Errorf("a half-created environment was reported usable (%s)", ver)
}
}

View File

@@ -4,4 +4,15 @@ Pipeline: capture -> detect (YuNet) -> track (IoU) -> align + encode
(ArcFace ONNX) -> match / auto-enroll (FAISS + SQLite) -> events + API.
"""
__version__ = "1.0.0"
# Stamped by release.sh from the git tag, into a copy of this line, so a
# release always produces a wheel nobody has installed before. It was a frozen
# "1.0.0" here and a frozen "1.1.0" in pyproject.toml - two literals that
# disagreed with each other and tracked nothing - which is how every engine fix
# for several releases silently failed to reach a machine that had run setup
# once. pip skips a wheel whose version is already installed and says so, one
# line above setup printing "[ok] Engine and dependencies installed".
#
# In a checkout it stays obviously a checkout: "0.0.0+dev" on /api/health is
# the truth about a developer's machine, and a plausible-looking number there
# would be worse than none.
__version__ = "0.0.0+dev"

View File

@@ -17,6 +17,7 @@ from fastapi.responses import HTMLResponse, Response, StreamingResponse
from fastapi.security import HTTPBasic, HTTPBasicCredentials
from pydantic import BaseModel, ValidationError
from . import __version__
from .config import ApiSection, CameraConfig, CameraTuning
from .commission import CommissionRun
from .events import Event
@@ -142,7 +143,7 @@ def _reencode(jpeg: bytes, width: int, quality: int) -> "bytes | None":
def create_app(engine: Engine) -> FastAPI:
app = FastAPI(title="Behavision", version="1.0.0",
app = FastAPI(title="Behavision", version=__version__,
dependencies=_auth_dependencies(engine.cfg.api))
def worker_or_404(camera_id: str):
@@ -172,6 +173,10 @@ def create_app(engine: Engine) -> FastAPI:
# are up, and the shop recognises nobody it already knows.
stranded = engine.gallery.health["stranded"]
return {"status": "ok" if engine.started_at else "starting",
# Which build is actually running. Without it there was no way
# to tell a shop PC three releases behind from a current one -
# which is precisely how a silent install failure survived.
"version": __version__,
"recognition_model": engine.encoder.model_name,
"gallery_unreadable_embeddings": stranded,
# "where is my database" must be answerable from the API: the

View File

@@ -4,6 +4,8 @@ from __future__ import annotations
import logging
import shutil
import ssl
import urllib.error
import urllib.request
from pathlib import Path
@@ -35,6 +37,81 @@ _COPY_MAP = {
}
def _https_context() -> "ssl.SSLContext | None":
"""The CA store to trust, or None to use whatever Python defaults to.
Returning None first is deliberate. On Windows and on a Homebrew or
system Python, the default context reads the machine's own certificate
store - which is what makes a corporate proxy with its own root CA work.
Replacing that with certifi's bundle unconditionally would break every
site that has one, in order to fix a different platform.
The platform this fixes is a python.org macOS build. It ships its own
OpenSSL with NO trust store, and populates one only when somebody
double-clicks `Install Certificates.command` in the Python folder -
which nobody installing face-recognition software has any reason to know
about. Every HTTPS request from that interpreter fails with:
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED]
certificate verify failed: unable to get local issuer certificate
Measured on a colleague's Mac: the engine installed perfectly and then
could not download a 230 KB model file, ending setup in forty lines of
traceback about `_ssl.c`.
"""
try:
import certifi
except ImportError: # pragma: no cover - certifi ships with requests
return None
return ssl.create_default_context(cafile=certifi.where())
def _is_cert_failure(err: BaseException) -> bool:
"""Is this a certificate-verification failure, however it is wrapped?
`urllib` does NOT let `ssl.SSLCertVerificationError` out. It catches it and
re-raises `urllib.error.URLError(err)`, carrying the original on `.reason`
- so `except ssl.SSLCertVerificationError` around `urlopen` matches
nothing, ever.
That is not a subtlety this file gets to record academically: the first
version of the fallback below was written exactly that way, shipped, and
failed on the machine it was written for with the very traceback it was
meant to prevent. The unit test passed throughout, because the fake it
used raised the bare SSL error - a shape real urllib never produces. A
stub that agrees with the author is worse than no test, and the test now
raises what urllib raises.
"""
reason = getattr(err, "reason", None)
return isinstance(err, ssl.SSLCertVerificationError) or \
isinstance(reason, ssl.SSLCertVerificationError)
def _urlopen(url: str, timeout: float = 60.0):
"""Open a URL, falling back to certifi's CA bundle on a verify failure.
Default first, certifi second, so the fix is additive: a machine whose
own store works keeps using it, and one with no store at all gets a
bundle rather than a traceback. certifi is already here - `requests` is a
hard dependency and brings it.
"""
try:
return urllib.request.urlopen(url, timeout=timeout)
except (urllib.error.URLError, ssl.SSLCertVerificationError) as err:
# Only a certificate problem is worth a second attempt. "No route to
# host" and "connection refused" arrive as URLError too, and retrying
# those with a different CA list changes nothing except how long the
# operator waits for the real message.
if not _is_cert_failure(err):
raise
ctx = _https_context()
if ctx is None:
raise
log.info("the system certificate store could not verify %s; "
"using the bundled CA list", url.split("/")[2])
return urllib.request.urlopen(url, timeout=timeout, context=ctx)
def _fetch(url: str, dest: Path, label: str) -> None:
"""Download with progress on stdout the supervisor can read.
@@ -56,7 +133,22 @@ def _fetch(url: str, dest: Path, label: str) -> None:
last = pct
log.info("download: %s %d%%", label, pct)
urllib.request.urlretrieve(url, dest, hook)
# Streamed rather than urlretrieve, only because urlretrieve offers no way
# to pass an SSL context and the whole point here is choosing one. The
# `download: <label> <n>%` lines are a contract: the supervisor parses
# them (`progressRe`) to put first-run progress in the tray, and without
# them a shop PC shows a stopped engine for five minutes after install.
with _urlopen(url) as resp:
total = int(resp.headers.get("Content-Length") or 0)
blocks, block_size = 0, 64 * 1024
with open(dest, "wb") as out:
while True:
chunk = resp.read(block_size)
if not chunk:
break
out.write(chunk)
blocks += 1
hook(blocks, block_size, total)
log.info("download: %s 100%%", label)
@@ -91,7 +183,7 @@ def setup_models(models_dir: Path) -> "list[str]":
import io
import zipfile
with urllib.request.urlopen(BUFFALO_SC_URL) as resp:
with _urlopen(BUFFALO_SC_URL) as resp:
payload = io.BytesIO(resp.read())
with zipfile.ZipFile(payload) as zf, \
zf.open("w600k_mbf.onnx") as src, \

View File

@@ -1,6 +1,6 @@
[project]
name = "behavision"
version = "1.1.0"
dynamic = ["version"]
description = "Production face recognition over RTSP"
requires-python = ">=3.10"
dependencies = [
@@ -32,3 +32,9 @@ behavision = ["static/*"]
[tool.pytest.ini_options]
testpaths = ["tests"]
# One version for the package and the wheel. Two literals in two files is how
# they came to disagree (1.0.0 here, 1.1.0 there) and how neither tracked a
# release.
[tool.setuptools.dynamic]
version = {attr = "behavision.__version__"}

View File

@@ -50,7 +50,28 @@ step "2. Agent and setup tool"
step "3. Engine source and wheel"
# The wheel is built with the checkout's own interpreter; requires-python is a
# statement about the SHOP PC, which setup enforces when it finds Python there.
# Stamp the tag into the wheel version. Without this every release built
# behavision-1.1.0-py3-none-any.whl, and `pip install --upgrade` on a machine
# that already had 1.1.0 is a no-op - so an engine fix reached nobody who had
# ever run setup, while the Go binaries beside it updated normally. Nothing
# reported a version either, so there was no way to tell a shop PC three
# releases behind from a current one.
#
# v0.5.6-demo -> 0.5.6+demo, which is valid PEP 440: a local segment takes
# alphanumerics and dots, never hyphens.
TMPVER=$(mktemp)
PEP440=$(printf '%s' "${TAG#v}" | sed 's/-/+/; s/[^0-9A-Za-z.+]/./g')
trap 'git checkout -- behavision/__init__.py 2>/dev/null || true; rm -f "$TMPVER"' EXIT
# A temp file rather than `sed -i`, whose argument handling differs between
# BSD and GNU - this script is run from a Mac today and that is not a reason
# to plant a portability trap in a release path.
sed "s/^__version__ = .*/__version__ = \"$PEP440\"/" behavision/__init__.py > "$TMPVER"
cat "$TMPVER" > behavision/__init__.py
.venv/bin/python -m pip wheel --no-deps --ignore-requires-python -q -w "$STAGE/engine-src" . 2>&1 | grep -v "DEPRECATION\|WARNING: Ignoring" || true
git checkout -- behavision/__init__.py
rm -f "$TMPVER"
trap - EXIT
echo " engine wheel version $PEP440"
ls "$STAGE"/engine-src/behavision-*.whl >/dev/null || { echo "wheel was not built" >&2; exit 1; }
cp pyproject.toml requirements.txt "$STAGE/engine-src/"
mkdir -p "$STAGE/engine-src/config" && cp config/default.yaml "$STAGE/engine-src/config/"

View File

@@ -0,0 +1,164 @@
"""Downloading the models is the last step of every fresh install, and it ran
into the one macOS trap nothing else here does.
A python.org macOS build ships its own OpenSSL with NO trust store, and
populates one only when somebody double-clicks `Install Certificates.command`
in the Python folder. Measured on a colleague's Mac: the engine installed
perfectly - numpy, onnxruntime, faiss, all of it - and then could not fetch a
230 KB model file, ending setup in forty lines of traceback about `_ssl.c`.
"""
import io
import logging
import os
import ssl
import urllib.request
import pytest
from behavision import model_assets
class _Resp(io.BytesIO):
"""Enough of an http response for _fetch: read() and .headers."""
def __init__(self, payload: bytes):
super().__init__(payload)
self.headers = {"Content-Length": str(len(payload))}
def __enter__(self):
return self
def __exit__(self, *exc):
self.close()
return False
def _verify_error():
"""What urlopen ACTUALLY raises, which is not what it looks like.
urllib catches ssl.SSLCertVerificationError and re-raises
urllib.error.URLError(err), carrying the original on `.reason`. The first
version of this test raised the bare SSL error - a shape real urllib never
produces - so it passed against a fallback that could never fire, and the
fix shipped and failed on the machine it was written for with the exact
traceback it was meant to prevent.
"""
return urllib.error.URLError(ssl.SSLCertVerificationError(
"[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: "
"unable to get local issuer certificate"))
def test_a_machine_with_no_trust_store_falls_back_to_the_bundled_one(monkeypatch):
calls = []
def fake(url, timeout=None, context=None):
calls.append(context)
if context is None:
raise _verify_error()
return _Resp(b"ok")
monkeypatch.setattr(urllib.request, "urlopen", fake)
with model_assets._urlopen("https://example.invalid/m.onnx") as resp:
assert resp.read() == b"ok"
assert len(calls) == 2, f"expected a retry, got {calls}"
# Default FIRST, and that order is the point. On Windows and on a system
# Python the default context reads the machine's own certificate store,
# which is what makes a corporate proxy with its own root CA work.
# Replacing it unconditionally would break every site that has one in
# order to fix a different platform.
assert calls[0] is None
assert isinstance(calls[1], ssl.SSLContext)
def test_a_working_trust_store_is_used_as_is(monkeypatch):
calls = []
def fake(url, timeout=None, context=None):
calls.append(context)
return _Resp(b"ok")
monkeypatch.setattr(urllib.request, "urlopen", fake)
model_assets._urlopen("https://example.invalid/m.onnx").close()
assert calls == [None], "the machine's own certificate store was bypassed"
def test_a_real_network_failure_is_not_disguised_as_a_certificate_problem(monkeypatch):
"""A URLError is not automatically a certificate problem.
"No route to host" and "connection refused" arrive as URLError too, and
retrying those with a different CA list changes nothing except how long
the operator waits for the real message.
"""
calls = []
def fake(url, timeout=None, context=None):
calls.append(context)
raise urllib.error.URLError("no route to host")
monkeypatch.setattr(urllib.request, "urlopen", fake)
with pytest.raises(urllib.error.URLError):
model_assets._urlopen("https://example.invalid/m.onnx")
assert calls == [None], f"a dead network was retried as a CA problem: {calls}"
def test_progress_lines_survive_the_rewrite(tmp_path, monkeypatch, caplog):
"""`download: <label> <n>%` is a CONTRACT, not logging.
The supervisor parses it (progressRe) to put first-run progress in the
tray and the window, because the API is not up yet and a shop PC showing
a stopped engine for five minutes after install looks broken. Switching
off urlretrieve - needed because it offers no way to pass an SSL context -
is exactly the kind of change that drops it silently.
"""
payload = b"x" * (64 * 1024 * 8)
monkeypatch.setattr(urllib.request, "urlopen",
lambda url, timeout=None, context=None: _Resp(payload))
dest = tmp_path / "model.onnx"
with caplog.at_level(logging.INFO, logger="behavision.model_assets"):
model_assets._fetch("https://example.invalid/m.onnx", dest, "face detector")
assert dest.read_bytes() == payload
lines = [r.getMessage() for r in caplog.records]
pct = [ln for ln in lines if ln.startswith("download: face detector ")]
assert pct, f"no progress lines at all: {lines}"
assert "download: face detector 100%" in pct, pct
@pytest.mark.skipif(not os.environ.get("BEHAVISION_NETWORK_TESTS"),
reason="set BEHAVISION_NETWORK_TESTS=1 to reach github.com")
def test_against_a_real_machine_with_no_trust_store(tmp_path, monkeypatch):
"""The whole thing, over the real network, with the real failure.
Every stub above is a statement about what I believe urllib does, and the
first version of this file proved how much that is worth. Python's default
context honours SSL_CERT_FILE, so pointing it at an EMPTY file reproduces
a python.org macOS build exactly: a default context that trusts nobody.
certifi is loaded by path and is unaffected by the variable, so if the
fallback works here it works there.
"""
empty = tmp_path / "no-cas.pem"
empty.write_text("")
monkeypatch.setenv("SSL_CERT_FILE", str(empty))
monkeypatch.setenv("SSL_CERT_DIR", str(tmp_path / "nothing"))
# Not every interpreter can be put into that state, and pretending
# otherwise would turn this into a test that passes by not running. A
# Python linked against LibreSSL - which is what macOS Command Line Tools
# ships - reads the system keychain and ignores SSL_CERT_FILE entirely:
# measured, 128 CAs loaded with the variable pointing at an empty file. A
# python.org build links OpenSSL and honours it: 0 CAs, which is the
# condition being reproduced.
if ssl.create_default_context().get_ca_certs():
pytest.skip("this interpreter ignores SSL_CERT_FILE (%s), so the "
"no-trust-store condition cannot be reproduced here"
% ssl.OPENSSL_VERSION)
# The premise: the default context really is broken in this process.
with pytest.raises(urllib.error.URLError) as caught:
urllib.request.urlopen(model_assets.YUNET_URL, timeout=30)
assert model_assets._is_cert_failure(caught.value), caught.value
dest = tmp_path / "yunet.onnx"
model_assets._fetch(model_assets.YUNET_URL, dest, "face detector")
assert dest.stat().st_size > 200_000