queue.workolik.com was served by three separate routing definitions that
didn't agree: nginx-queue-proxy.conf and the classic queue-ingress both sent
everything to the deliveries app, but the Gateway API HTTPRoute
(deliveries-route) had a carve-out sending /live/api/v1/mob/orders and
/live/api/v1/web/products to fiesta's raw backend in the nearle namespace
instead (3 fixed replicas, no autoscaling, no resource limits) - a
completely different capacity profile from deliveries (HPA'd, 4-20
replicas). Depending on which router won for a given request, orders could
land on two backends with very different ability to absorb a burst,
plausibly explaining partial order loss / 429s under concurrent load.
Removed the carve-out so all three routing paths agree: everything goes to
deliveries-service.
Also raised deliveries-hpa minReplicas 4->8 and added an explicit
aggressive scaleUp behavior (no stabilization delay, up to 4 pods or 100%
every 15s). Autoscaling reacts to sustained load over roughly 30-60s
(metric polling + pod scheduling + readiness delay), so it does very
little for a burst that's over in seconds - minReplicas is the actual
defense; the behavior block just makes any further scaling land as fast as
possible.
c38a36709b