Skip-login was stripped in an earlier "harden security" pass, which is
why the dashboard started demanding a token. Re-added it; it now runs
as the dashboard's own view-only ServiceAccount (get/list/watch), so
opening it needs no token but write access still requires the
admin-user token as before.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Login stopped working the moment this shipped. Suspect this dashboard
version treats token-ttl=0 as "expire immediately" rather than "never
expire". Reverting to the default (900s) to restore working login;
the idle-timeout annoyance can be revisited with a large finite value
instead of 0, tested before shipping.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Default --token-ttl is 900s (15min idle), which was forcing re-entry of
the token repeatedly. Setting it to 0 makes a logged-in session
persist instead of expiring back to the login screen.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
replicas: 4 was fighting the HPA's minReplicas: 8 on every reconcile -
GitOps tooling (Flux) reapplies the manifest on an interval, which would
keep yanking capacity back down between HPA corrections and quietly
undo the burst-headroom fix from c38a367.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Prepares clusters/production/ for `flux bootstrap git` - Kustomizations
for manifests/alaska and manifests/core, plus a Gitea push Receiver so
new commits reconcile immediately instead of waiting on the poll
interval. Webhook exposed on a dedicated host (flux-webhook.workolik.com)
to avoid the existing queue.workolik.com Gateway/Ingress overlap.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
queue.workolik.com was served by three separate routing definitions that
didn't agree: nginx-queue-proxy.conf and the classic queue-ingress both sent
everything to the deliveries app, but the Gateway API HTTPRoute
(deliveries-route) had a carve-out sending /live/api/v1/mob/orders and
/live/api/v1/web/products to fiesta's raw backend in the nearle namespace
instead (3 fixed replicas, no autoscaling, no resource limits) - a
completely different capacity profile from deliveries (HPA'd, 4-20
replicas). Depending on which router won for a given request, orders could
land on two backends with very different ability to absorb a burst,
plausibly explaining partial order loss / 429s under concurrent load.
Removed the carve-out so all three routing paths agree: everything goes to
deliveries-service.
Also raised deliveries-hpa minReplicas 4->8 and added an explicit
aggressive scaleUp behavior (no stabilization delay, up to 4 pods or 100%
every 15s). Autoscaling reacts to sustained load over roughly 30-60s
(metric polling + pod scheduling + readiness delay), so it does very
little for a burst that's over in seconds - minReplicas is the actual
defense; the behavior block just makes any further scaling land as fast as
possible.
- Rebuild manifests/doormile/miletruth.yaml (was corrupted since the
initial commit - contained pasted AI/terminal output, truncated env
var names/values, duplicate keys). Rebuilt from the confirmed-live
config, secrets sourced via a Secret instead of plaintext values.
- Lock down the Kubernetes Dashboard: remove --enable-skip-login /
--enable-insecure-login / --insecure-port=9090, remove the extra
cluster-admin binding on the dashboard's own ServiceAccount, remove
the now-dead insecure NodePort Service. Token-based login via the
existing admin-user ServiceAccount is unaffected.
- Fix the duplicate `backendRefs` key under the same HTTPRoute rule in
alaska.yaml (invalid/redundant YAML).
- Delete 6 redundant duplicate manifests (fiesta-sts/svc,
atlantis-sts/svc, jupiter-sts/svc) that were partial, stale subsets
of nearle-fiesta/atlantis/jupiter.yaml - one pair disagreed on the
fiesta image tag entirely (v1.3.50 vs v1.3.67, neither of which
matched what's actually live).
- Reconcile nearle-fiesta.yaml and nearle-jupiter.yaml image tags to
the confirmed-live versions (v1.3.78 / v2.7.55).
- Add allowPrivilegeEscalation:false + drop-all-capabilities to
fiesta/atlantis/jupiter/titan/ariane and the 5 specialized core
workers, which previously ran with no securityContext at all.
- Add terminationGracePeriodSeconds:45 to the worker StatefulSets so
Kubernetes gives the new graceful-shutdown drain (see worker.py
changes) enough time before SIGKILL.
