jupiter and Fiesta are separate applications (separate repos, separate
binaries) that happen to share one Postgres instance. The previous fix
(661a08d) repointed queue.workolik.com's createorder from jupiter's v1 to
jupiter's v3 handler to stop items being silently dropped - that worked
because they share a database, but it was never the right target: jupiter
has no item-required guard, no atomic order-number allocation, no
stock-insufficient check, because none of that was ever written for
jupiter. Fiesta's own CreateOrderv3 already has all of it.
Added explicit tenant-based routing in worker.py: FIESTA_TENANT_IDS (seeded
with 1147/R mart and 1135/Suriya Store, the two confirmed so far) forces
createorder for those tenants to FIESTA_BASE_URL instead of jupiter's
mapping. Explicit allowlist rather than a DB heuristic, since jupiter and
Fiesta share one `tenants` table with no single column that cleanly
separates the two populations (checked: tenanttype/moduleid/categoryid/
configid are all inconsistent across the tenants that are known to belong
to each app). Non-Fiesta tenants keep going to jupiter's v3 endpoint
(661a08d), unaffected.
Verified live through the real queue.workolik.com path:
- Zero-item order (tenant 1147): worker log shows "Routing tenant 1147
createorder to Fiesta backend", Fiesta correctly returns 400 "Order must
contain at least one item", no phantom order created.
- Order with items (tenant 1147): itemcount=1, detail_count=1, product 7076
stock ledger moved 25->24, then restored to 25 on cancel.
Expand FIESTA_TENANT_IDS as more Fiesta tenants are identified - there's no
programmatic way to auto-detect them from the shared tenants table.
worker-script's ENDPOINT_MAPPING sent /live/api/v1/mob/orders/createorder to
jupiter's v1 CreateOrder, which only ever writes the order header - it never
loops over "items", so orderdetails/productstocks were silently never
touched for any order placed through queue.workolik.com. itemcount (a plain
scalar on the same Orders struct) persisted, making this easy to miss:
header looked fine, items silently vanished.
Confirmed live before the fix: order 147157/147159 (tenant 1147) had
itemcount=1, detail_count=0, stock unchanged. Root-caused via jupiter's own
log line (orderController.go:393, the log statement inside CreateOrder) at
the exact timestamp of the test request.
Repointed to /live/api/v3/orders/createorder (CreateOrderv3), which does
loop over Items and write orderdetails + productstocks. The loop is a no-op
when Items is empty, so tenants whose entire order history is header-only
(916, 908 - confirmed ~99.5%+ itemless across 100k+ and 6k orders
respectively) are unaffected. Verified live post-fix: order 147160 got
itemcount=1, detail_count=1, and product 7076's stock ledger moved 25->24
on create, restored to 25 on cancel. Tenant 916 traffic unaffected by design
(empty-Items loop is a no-op), not yet re-observed live since the restart
pending their next natural order batch.
Test orders (147157, 147158, 147159, 147160) cancelled, not deleted.
The "gateway" sidecar in the fiesta StatefulSet (workolik360/alaska image,
running fiesta-gateway-script) and the k8s routing objects meant to reach it
(fiesta-route HTTPRoute, the port-8000 rules in nearle-ingress) never
actually served real traffic. fiesta.nearle.app is routed by a standalone
Docker/nginx proxy (conf/nginx-fiesta.conf, unchanged since the initial
commit) straight to NodePort 30823 - the real backend, no path splitting.
The Kubernetes-native routing objects have no working controller on this
cluster (no Traefik/Envoy pod; the shared Gateway resource's own status is
"Waiting for controller"), so they were inert either way.
The sidecar's only real effect, when briefly reachable, was publishing
Fiesta orders onto the same NATS subject worker-orders drains - which
forwards to jupiter, not this backend - producing the header-only phantom
orders fixed in the app on 2026-07-29. It's not a working parallel path,
it was the source of that bug.
Removed: the gateway container + gateway-script volume from the fiesta
StatefulSet, port 8000 from the fiesta Service, the fiesta-route HTTPRoute,
the fiesta-gateway-script ConfigMap (fiesta-gateway.yaml deleted entirely),
and the two dead port-8000 path rules in nearle-ingress's fiesta host block
(kept the correct catch-all). queue.workolik.com and the core-namespace NATS
worker pipeline are untouched - separate system entirely.
Verified live: fiesta pods rolled to single-container, service has only
port 80, and a real request through fiesta.nearle.app still behaves
correctly post-cleanup.
jupiter v2.7.59 fixes an empty-string deliverytime being rejected by
Postgres on every order creation (see backend_jupiter commit 3da5876).
worker-script.yaml ENDPOINT_MAPPING had four entries pointing at
jupiter paths that don't exist (extra /mob/ or /web/ segments):
createorder, customers/login, customers/create, products/create.
createorder was silently dropping real orders with no retry (worker
treats 404 as a client error and calls msg.term()); the other three
had no observed traffic but had the same bug. Corrected all four to
jupiter's actual registered routes. Verified end-to-end: sent a real
order through queue.workolik.com and confirmed it landed in the
orders table.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Workers hardcoded jupiter's ClusterIP (10.43.224.63 / 10.43.229.168),
which had gone stale and pointed at nothing. Every request forwarded
from worker-orders, worker-deliveries, worker-customers,
worker-rider-logs, worker-products, and worker-notifications to
jupiter was timing out silently, breaking order creation, delivery
logs, and rider online status. Repointed at the stable in-cluster DNS
name (jupiter.nearle) instead of a ClusterIP so this can't go stale
again after a future service recreation.
Also bumps jupiter to v2.7.57 (Redis client timeout/pool fix) to
match what's already deployed live.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This StatefulSet was deployed manually outside git at some point before
this repo's GitOps work began, and was destroyed when deleting the core
Kustomization triggered a full namespace recreation (Flux prune-on-delete
cascades regardless of kubectl's --cascade flag - that only affects
Kubernetes' own owner-reference GC, not Flux's finalizer).
Reconstructed from its own logs (NATS_STREAM=NOTIFICATIONS,
NATS_CONSUMER=notifications-worker, FILTER_SUBJECT=api.v1.notifications.push)
plus the same pattern as its sibling workers. Resource limits and
WORKER_CONCURRENCY are a best-guess match to worker-rider-logs, since the
original values were never version-controlled anywhere. Adding it to git
now so this can't happen again.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
kubectl api-resources shows no Middleware kind under any API group, so
this manifest could never apply and was blocking the entire core
Kustomization. The Ingress annotation referencing it has been a
pre-existing no-op; CORS is actually handled by nginx-queue-proxy.conf
and each app's own CORS middleware.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- manifests/core/kustomization.yaml: add ingress-unified.yaml and
traefik-middlewares.yaml, which were never in any kustomization and so
were never actually GitOps-managed - the queue.workolik.com routing fix
from c38a367 turned out to be live already (likely applied manually
before this session), but was completely undetected by Flux until now.
Dropped the redundant top-level `namespace: core` override since every
existing resource already sets its own namespace explicitly, and the
two new files span alaska/nearle.
- manifests/nearle/kustomization.yaml + clusters/production/apps-nearle.yaml:
same GitOps treatment already applied to alaska/core, so a fiesta/jupiter/
atlantis/titan/ariane version bump in git now auto-deploys instead of
requiring manual kubectl apply.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
worker.py (both the ConfigMap copy and conf/worker.py):
- Set an explicit ack_wait=60s on the JetStream pull consumer. It was
previously left at the implicit default (~30s), the same ballpark
as the outbound HTTP timeout - a slow-but-legitimate external call
could cause JetStream to redeliver the message to another worker
while the first was still mid-request, double-processing a
non-idempotent call (e.g. duplicate order creation).
- Track in-flight tasks and drain them (bounded wait) before closing
the NATS/HTTP connections on shutdown, instead of cutting them off
immediately - avoids dropped/duplicated messages on pod restarts.
- Generic exception handler now does nak(delay=5) instead of an
undelayed nak(), avoiding a tight redelivery loop on a persistent
bug.
- Missing 'data' field in a message now explicitly drops with a log
line instead of silently forwarding the entire internal envelope.
- Removed the hardcoded NATS password fallback baked into the source
(every deployment already supplies it via a Secret at runtime, so
this was a redundant plaintext copy sitting in a ConfigMap).
app.py (both the ConfigMap copy and conf/app.py):
- Fixed "NATS by connected" typo -> "NATS not connected".
- Same hardcoded-password-fallback removal as worker.py.
CORS:
- conf/nginx-jupiter.conf and the in-cluster jupiter-cors-proxy nginx
config both add their own CORS headers without stripping any the
upstream might set, unlike nginx-queue-proxy.conf which does this
correctly. Added proxy_hide_header for the ACA-* headers in both -
browsers reject a response with duplicate Access-Control-* values.
docker-compose.yml:
- Added the missing doormile-proxy service (doormile.com -> :8206 ->
NodePort 30830). nginx-doormile.conf existed but had no service
wiring it into Traefik, unlike every other app.
- Rebuild manifests/doormile/miletruth.yaml (was corrupted since the
initial commit - contained pasted AI/terminal output, truncated env
var names/values, duplicate keys). Rebuilt from the confirmed-live
config, secrets sourced via a Secret instead of plaintext values.
- Lock down the Kubernetes Dashboard: remove --enable-skip-login /
--enable-insecure-login / --insecure-port=9090, remove the extra
cluster-admin binding on the dashboard's own ServiceAccount, remove
the now-dead insecure NodePort Service. Token-based login via the
existing admin-user ServiceAccount is unaffected.
- Fix the duplicate `backendRefs` key under the same HTTPRoute rule in
alaska.yaml (invalid/redundant YAML).
- Delete 6 redundant duplicate manifests (fiesta-sts/svc,
atlantis-sts/svc, jupiter-sts/svc) that were partial, stale subsets
of nearle-fiesta/atlantis/jupiter.yaml - one pair disagreed on the
fiesta image tag entirely (v1.3.50 vs v1.3.67, neither of which
matched what's actually live).
- Reconcile nearle-fiesta.yaml and nearle-jupiter.yaml image tags to
the confirmed-live versions (v1.3.78 / v2.7.55).
- Add allowPrivilegeEscalation:false + drop-all-capabilities to
fiesta/atlantis/jupiter/titan/ariane and the 5 specialized core
workers, which previously ran with no securityContext at all.
- Add terminationGracePeriodSeconds:45 to the worker StatefulSets so
Kubernetes gives the new graceful-shutdown drain (see worker.py
changes) enough time before SIGKILL.
