paho delivers on one goroutine, so bills were committed strictly one
after another. Each is a full Postgres transaction — advisory lock,
dedup, stock row locks, availability check, four inserts, commit — which
is 10-30ms, so the ceiling was roughly 30-100 bills a second and a
shop's backlog draining after an outage took minutes to land.
A fixed pool behind a bounded queue, rather than a goroutine per
message. Unbounded concurrency would open a transaction per message and
exhaust the connection pool under a storm, stalling every one of them at
once — a slow minute turned into a dead one. When the queue fills,
submit blocks: paho stops acknowledging, the broker's in-flight window
fills, it stops sending, and the backpressure reaches the till, which
holds its bills and retries. Slow, but nothing is dropped.
Heartbeats get their own pool. Sharing one would let a backlog of bills
delay presence, so every till would appear to go dark at exactly the
moment the system was busiest — the worst time to be blind to which
counters are alive.
Payloads are copied on the way in. paho reuses its buffer once a handler
returns and the work now happens after that, so a queued bill would
otherwise be read as whatever message arrived next — silently, and as
valid JSON often enough to commit the wrong sale.
One bug found by its own test: submit-after-stop selected between a
done-channel and the job channel, and once both were ready Go picks at
random. Picking the send panics on a closed channel. It would have shown
up in production as an occasional crash during shutdown and nowhere
else. Now guarded by an RWMutex held across the send, so the queue
cannot be closed under one in progress.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Deployed as a StatefulSet with three replicas, and MQTT has no queue
groups — every subscriber receives every message. All three pods would
commit the same bill and publish three acks. Nothing double-counts,
because the ingest deduplicates on the till's UUID and holds an advisory
lock, but it is three times the database work and three times the
traffic for one sale.
Ordinal 0 consumes; the others stay idle. A StatefulSet already
guarantees stable unique ordinals, so this is a deterministic election
with no lock, no lease and no new dependency. If that pod dies the set
recreates it and tills hold their bills meanwhile, which is what they
are built to do.
POS_MQTT_CONSUMER=always/never overrides it for deployments that are not
a StatefulSet. Anything without an ordinal name — a Deployment pod, a
bare container, local development — consumes, because a lone instance
that silently refused to would be a far more confusing failure than one
that did.
The client id now defaults to the pod name rather than a constant. Two
connections sharing an id evict each other in a reconnect loop that
looks exactly like a flapping network, and takes a while to recognise as
anything else.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The ingest only ever wrote. A bill that reached pos_orders was safe and
completely unreachable — no screen in the product could show it, and the
only way to see a day's counter takings was to query the database by
hand.
Three endpoints: a paged bill list, one bill with its lines, and a
summary split the three ways somebody actually asks for — by tender for
reconciling a drawer, by day for a chart, by till for an outlet running
several counters.
locationid is required on all of them and is the authorisation boundary,
so a caller cannot page through another shop's takings by omitting a
parameter. Fetching a bill under the wrong outlet returns 404 even when
the reference is a real one.
Dates match businessdate rather than arrival, because a till that was
offline overnight uploads yesterday's bills this morning and they belong
to yesterday. The list is ordered by billedat for the same reason —
sorting by arrival would interleave a recovered backlog through today.
Unlike the ingest handlers these answer in the usual envelope: they are
read by the web app, not by a terminal, and nothing about them is bound
to the till's contract.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A till holds every bill in its own SQLite database and keeps it for
seven days after we acknowledge it, marking one synced only when its id
comes back in an ack. Everything here follows from that.
Silence is not acceptance, so a failing ingest publishes nothing at all
and the terminal simply sends again. A duplicate is a success, because
at-least-once delivery means a lost ack legitimately re-delivers bills
we already hold, and calling those failures would strand a day of
takings on the till. Deduplication is a unique index on the terminal's
UUID plus an advisory lock held for the transaction.
Bills land in pos_orders / pos_order_items rather than orders: a counter
bill carries a cashier, a terminal, a rounding adjustment, promos,
loyalty movement and a payment split that orders has nowhere to put, and
forcing one into the other loses whatever does not fit. Stock is *not*
split — a counter sale writes the same productstocks rows an app order
does, through helpers extracted from createOrderTx so the rule that
prevents overselling has one implementation rather than two.
GetRevenueSummary and GetSalesSummary were extended to union the new
table in; any new report has to remember the same.
Terminal health goes to Redis under a 90-second TTL, sharing the
instance the express backend uses. A heartbeat is a fact with an expiry
date: a till that loses power stops refreshing and ages off the board by
itself, where a Postgres row would need ~288k writes a day and a reaper.
Proven end to end against the live estate before commit: a bill over
HTTP and one over the real Mosquitto broker, the same bill three times
producing one row and one stock movement, and a heartbeat arriving on
the health endpoint. All probe data was removed afterwards.
Four things that only surfaced against real data. An unset jsonb column
failed the very first bill. Product SKUs are unusable as barcodes — 6,245
products share 93 SKUs and "1" covers 5,794 of them — against the till's
unique index, so barcodes fall back to the product id. A taxpercent of
-1 exists and would have put negative GST in a filed slab. And a product
with id 0 exists, which can never be billed and is now skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
