Fix queue.workolik.com routing conflict and give deliveries HPA burst headroom
queue.workolik.com was served by three separate routing definitions that didn't agree: nginx-queue-proxy.conf and the classic queue-ingress both sent everything to the deliveries app, but the Gateway API HTTPRoute (deliveries-route) had a carve-out sending /live/api/v1/mob/orders and /live/api/v1/web/products to fiesta's raw backend in the nearle namespace instead (3 fixed replicas, no autoscaling, no resource limits) - a completely different capacity profile from deliveries (HPA'd, 4-20 replicas). Depending on which router won for a given request, orders could land on two backends with very different ability to absorb a burst, plausibly explaining partial order loss / 429s under concurrent load. Removed the carve-out so all three routing paths agree: everything goes to deliveries-service. Also raised deliveries-hpa minReplicas 4->8 and added an explicit aggressive scaleUp behavior (no stabilization delay, up to 4 pods or 100% every 15s). Autoscaling reacts to sustained load over roughly 30-60s (metric polling + pod scheduling + readiness delay), so it does very little for a burst that's over in seconds - minReplicas is the actual defense; the behavior block just makes any further scaling land as fast as possible.
This commit is contained in:
@@ -244,8 +244,29 @@ spec:
|
|||||||
apiVersion: apps/v1
|
apiVersion: apps/v1
|
||||||
kind: StatefulSet
|
kind: StatefulSet
|
||||||
name: deliveries
|
name: deliveries
|
||||||
minReplicas: 4
|
minReplicas: 8
|
||||||
maxReplicas: 20
|
maxReplicas: 20
|
||||||
|
# Kubernetes autoscaling reacts to sustained load over ~30-60s (metric
|
||||||
|
# polling + pod scheduling + readiness delay) - it does very little for a
|
||||||
|
# burst that's over in seconds. This behavior block removes the scale-up
|
||||||
|
# stabilization delay and allows adding pods aggressively, so the cluster
|
||||||
|
# reacts as fast as the metrics pipeline allows rather than waiting extra
|
||||||
|
# cycles. minReplicas above is the main defense for bursts (baseline
|
||||||
|
# capacity that's already there before a burst starts); this just makes
|
||||||
|
# whatever additional scaling happens land as quickly as possible.
|
||||||
|
behavior:
|
||||||
|
scaleUp:
|
||||||
|
stabilizationWindowSeconds: 0
|
||||||
|
policies:
|
||||||
|
- type: Percent
|
||||||
|
value: 100
|
||||||
|
periodSeconds: 15
|
||||||
|
- type: Pods
|
||||||
|
value: 4
|
||||||
|
periodSeconds: 15
|
||||||
|
selectPolicy: Max
|
||||||
|
scaleDown:
|
||||||
|
stabilizationWindowSeconds: 300
|
||||||
metrics:
|
metrics:
|
||||||
- type: Resource
|
- type: Resource
|
||||||
resource:
|
resource:
|
||||||
@@ -330,24 +351,6 @@ spec:
|
|||||||
- name: deliveries-service
|
- name: deliveries-service
|
||||||
port: 8000
|
port: 8000
|
||||||
weight: 100
|
weight: 100
|
||||||
- matches:
|
|
||||||
- path:
|
|
||||||
type: PathPrefix
|
|
||||||
value: /live/api/v1/mob/orders
|
|
||||||
backendRefs:
|
|
||||||
- name: fiesta
|
|
||||||
namespace: nearle
|
|
||||||
port: 80
|
|
||||||
weight: 100
|
|
||||||
- matches:
|
|
||||||
- path:
|
|
||||||
type: PathPrefix
|
|
||||||
value: /live/api/v1/web/products
|
|
||||||
backendRefs:
|
|
||||||
- name: fiesta
|
|
||||||
namespace: nearle
|
|
||||||
port: 80
|
|
||||||
weight: 100
|
|
||||||
- matches:
|
- matches:
|
||||||
- path:
|
- path:
|
||||||
type: PathPrefix
|
type: PathPrefix
|
||||||
|
|||||||
Reference in New Issue
Block a user