doormile_backend/CLAUDE.md

# Doormile — Full Project Memory

Portable context for a fresh Claude session working on this project on any
machine or account. Covers the **whole** Doormile system, not just one slice
of it. Read fully before touching the codebase.

A note on provenance: sections marked **[verified this session]** were
confirmed by directly reading this repo's code. Sections marked **[carried
forward]** come from Suriya's own prior-session notes and have NOT been
independently re-verified against source — treat them as reported state, not
confirmed state, until checked.

---

## 1. What Doormile is

Suriya is building **Doormile**, a real commercial logistics platform for the
Indian market (Coimbatore, Hyderabad, Bengaluru, Chennai) — hub-based parcel
courier/delivery, not a prototype. Treat everything as production-grade from
the start. Stakeholders: Doormile's own ops team, partner tenants (client
logistics/travel companies), hub staff, milers (delivery riders), and end
customers.

**Suriya's standing preferences** (apply without being asked):
- Direct, honest technical assessments over diplomatic framing. If something
  isn't done, say so plainly. Don't declare victory early.
- Production-grade code from the start, not "refactor later."
- **Warn before any consequential server/schema change** before making it.
- Concise answers — don't over-explain.
- Once a decision is made, don't keep re-asking for confirmation on the same
  scope — proceed and report back. But *do* ask when a decision is genuinely
  ambiguous or touches money/data-correctness in a way that can't be safely
  guessed.
- Minimal-effort, highest-leverage fixes over big rewrites, until he
  explicitly asks for the big rewrite.
- Works across multiple Claude sessions/machines simultaneously, treating
  Claude as a co-architect across the full stack — this file exists so any
  of those sessions can pick up full context.

---

## 2. System architecture — the whole stack

**[carried forward, substantial infra reported as built and verified in
prior sessions]**

- **Backend**: Go + Fiber, deployed on **Kubernetes**, at `api.doormile.com`.
  200 registered routes **[verified this session, exact count]** — see §7.
  This is the primary booking/assignment API and the primary trigger for
  miler assignment, calling the AI decision layer with a 5-second timeout
  fallback so a slow AI response never blocks a booking.
- **AI dispatch layer**: a Python autonomous agent swarm (8 agents, **JARVIS**
  as master orchestrator) on a dedicated server, connected to NATS
  JetStream. The `DispatchAgent` operates as a NATS *watcher*, not the
  assignment trigger — the Go backend triggers assignment directly (see §4);
  the agent swarm reasons about it, doesn't gate it.
- **AI decision engine**: `routemate.workolik.com` — runs Claude Sonnet for
  miler-assignment reasoning, scoring candidates using hub load, on-time
  rate, and hub capacity. Uses pgvector-backed RAG memory with local
  embeddings (`all-MiniLM-L6-v2`, 384-dim) specifically to avoid ongoing
  per-call API cost. **[verified this session]**: the Go side of this is
  `models.AgentDecision` (`agentdecisions` table) with a
  `context_embedding vector(1536)` column and an ivfflat index (raw SQL in
  `migrations/migrate.go`, not GORM AutoMigrate) — note the dimension
  mismatch worth flagging (1536 in the Go model/migration vs 384 for
  MiniLM); worth confirming which embedding model is actually populating
  that column before trusting vector search quality.
- **Event bus**: NATS JetStream, 6 persistent streams, dedicated server.
  **[verified this session]**: `db.Js` is the package-level JetStream handle;
  convention across the codebase is `if db.Js != nil { ... }` and warn-log
  on publish failure rather than failing the request — publishing is
  best-effort, never blocking. Confirmed publish call sites include
  `booking.assigned` (from `internal/assignment`) and NATS publishes inside
  `AssignMilerToBooking` (`controllers/booking_assignment_service.go`).
- **Data stores**: PostgreSQL + pgvector 0.7.4, dedicated server, 37 tables
  under GORM AutoMigrate **[verified this session, exact count from
  `migrations/migrate.go`]**. Redis (`go-redis/v9`) for high-frequency
  ephemeral state — GPS pings, live miler status/location (GEO-indexed at
  `milers:locations`, used by `queryNearbyMilers` for assignment radius
  search), booking cache. Durable business state always lands in Postgres;
  Redis is never the system of record for anything that needs to survive a
  flush.
- **Ingress/infra**: Kubernetes, Traefik, nginx.

---

## 3. The client-facing surfaces

Five distinct front doors into this system. Only the backend API surface for
each has been directly inspected this session (via `routes.go`); the actual
client codebases (Flutter, React) have not been opened in this session.

1. **Customer app (B2C)** — Flutter. Auth via Firebase OTP (phone). Backend
   surface: 19 customer routes **[verified this session]**
   (`customer`/`customerAuth` groups) — booking creation, tracking,
   `AppCustomer`/`AppCustomerLocation`. **[carried forward]**: reported built
   and verified in prior sessions; the last live end-to-end test was
   blocked here — see §9.
2. **Miler app** — Flutter, for delivery riders. Backend surface: 38 routes
   **[verified this session]** (`miler`/`milerAuth` groups) — duty
   start/stop, GPS pings, assignment accept/reject/cancel, delivery
   complete/skip, break logs, support tickets. This session added
   `ResetMilerPin`, `MilerCancelAssignment`, `MilerSkipDelivery` to close
   gaps found against the old Nearle rider app (§8).
3. **Admin console** — React, converted from the old `NearlExpress`
   /`doormile_express_console` codebase. Backend surface: 89 routes
   **[verified this session]** (`admin`/`adminAuth` groups) — bookings,
   partner management, reports, pricing, express (formerly "CRM") bookings.
   This session added partner CRUD, bulk booking create/cancel, reports,
   password change, miler notify (§8).
4. **Hub console** — React, separate from the admin console. Backend
   surface: 31 routes **[verified this session]** (`hub`/`hubAuth` groups)
   — per-hub booking queues, tripsheet building, manual/batch assignment,
   hub messaging. Auth is separate from admin (`middlewares.HubStaffAuth`,
   sets `c.Locals("hubid")`). This session added tenant-scoping (closed a
   real cross-tenant leak) and `HubBatchAssign` (§8).
5. **CRM (field sales)** — a genuinely separate feature from "CRM bookings"
   (see §8 naming note). `POST/GET /crm/clients`, `GET/PUT/DELETE
   /crm/clients/:id` — 5 routes, all open/no-auth by design: field reps
   register leads from the Flutter side, the web console reads without a
   separate CRM login. Models: `DoormileClient`, `DoormileAuth`. **Not
   deeply audited** — only the route registration was read this session.

Total: 200 routes = miler 38 + admin 89 + hub 31 + customer 19 + crm 5 +
internal 5 + redisUsers 4 + bookingCache 3 + 2 top-level pricing routes +
websocket routes. **[verified this session]**

---

## 4. AI dispatch & assignment — how a booking actually gets a miler

**[verified this session]** — read directly from
`internal/assignment/crm_assignment.go` and `booking_assignment_service.go`.

- Two entry points, same core: `AssignCustomerMiler` (B2C bookings) and
  `AssignCRMMiler` (express/console-originated bookings — name predates this
  session's "CRM"→"express" rename, deliberately left unrenamed since it's a
  working assignment engine, not just a label). Both are fire-and-forget
  goroutines called after transaction commit, retrying up to 5 times, 2
  minutes apart (~10 minutes total), before giving up and publishing a
  failure event (`publishAssignmentFailed`, `reasonNoMilerAvailable`) to NATS
  so the failure isn't silently invisible.
- `TryAssignOnce` is the synchronous single-attempt variant — used when a
  hub staff member manually triggers "auto-assign" from the console and
  needs an immediate answer rather than the ~10-minute retry loop. Returns a
  rich `AutoAssignResult` (assigned/escalated, miler name, distance, AI
  reasoning text, candidates found) for the UI to show.
- Core flow (`tryAssign`): Redis `GEOSEARCH` on `milers:locations` (10km
  radius, top 10, sorted nearest-first) → `selectMilerWithAI` (the actual
  Claude Sonnet call to `routemate.workolik.com`, logs an `AgentDecision`
  row with reasoning + optional embedding) → `commitAssignment` (single DB
  transaction: create `BookingAssignment`, update `PickupBooking.status` to
  `BookingMilerAssigned`, flip `MilerProfile.availabilitystatus`) → publish
  `booking.assigned` to NATS (best-effort) → push notifications to miler and
  customer.
- `HubBatchAssign` (**built this session**, `controllers/hubController.go`)
  is a separate, simpler path: a greedy nearest-available-rider heuristic
  (haversine distance, no AI call, no retry loop) for clearing a hub's
  pending-pickup queue in one batch call, capped per rider (default 5),
  reusing `AssignMilerToBooking` for the actual transactional assignment.
  **Explicitly not** a replacement for `selectMilerWithAI`'s reasoning, and
  **not** a multi-stop route/VRP solver — see the routing gap below.

**Logistics/routing — what exists vs what doesn't:**
- What exists: hub network (`Hub` model, `originhubid`/`currenthubid`/
  `destinationhubid` on bookings/consignments for tracking a parcel's hub
  path), `Tripsheet`/`TripsheetItem` for hub-to-hub batch transport, the
  nearest-rider assignment logic above, `HubBatchAssign`'s greedy queue
  clearer.
- What does **not** exist in Doormile yet: any multi-stop route sequencing /
  vehicle-routing-problem solver. Nearle used external paid services
  (`routes.workolik.com`, and `routemate.workolik.com` was itself originally
  a Nearle-side concept) for that; nothing in Doormile replaces true
  stop-sequencing optimization today. `HubBatchAssign` only decides *who*
  gets which single booking, not *what order* a rider should run multiple
  stops in. Flag this clearly if asked whether routing is "done" — it isn't,
  by design, pending a real conversation about whether it's needed yet.

---

## 5. Data model reference (`models/*.go`)

**[verified this session]**

- `PickupBooking` (`pickupbookings`) — customer's pickup request, pre-hub.
  Has `Tenantid *int` (**added this session**, see §8). `Bookingsource`:
  `"Customer_App"` (B2C) or `"CRM_Console"` (console-created — deliberately
  left as `"CRM_Console"` even though the outward API/route naming was
  changed to "express", see §8).
- `BookingParcel`, `BookingServiceOption`, `BookingPayment`,
  `BookingAssignment` (status: Assigned/Accepted/Rejected/Reassigned/
  Completed/Cancelled, carries `AgentDecisionID *uint64` linking back to the
  AI reasoning that produced it), `BookingVehicleRequirement`.
- `Consignment` (`consignments`) — the shipment once picked up. Has its own
  `Tenantid int` (not nullable, pre-existing). `Attemptcount int` (used by
  `MilerSkipDelivery`, added this session). `ConsignmentHistory` (event
  log), `ConsignmentException` (Lost/Damaged/Misrouted/Receiver_Refused/
  Missing_Contents/Undeliverable).
- `Hub`, `Vehicle`, `Tripsheet`, `TripsheetItem`, `DeliveryProof`.
- `AppUser` (`appusers`) — shared login table for staff/miler/admin roles
  (`Roleid`: 1 admin, 3 manager, 4 rep/exec, 5 miler, 6 hub staff via a
  separate `HubStaffAccount` table). `MilerProfile` — actual rider profile
  (availability status, current lat/lng, rating). `MilerDutyLog`,
  `MilerBreakLog`, `MilerSupportTicket`.
- `AppCustomer`/`AppCustomerLocation` — B2C app customers (separate from
  legacy `Customer`/`CustomerLocation` kept for backward compat — flagged as
  a future duplicate-data risk, not resolved).
- `HubStaffAccount` — `Tenantid *int`: **nil = Doormile staff (sees
  everything)**, **set = partner-tenant staff (own tenant's data only)**.
  Load-bearing distinction — see the tenant-leak fix in §8.
- `PartnerInfo` (`partnerinfo`) — fleet/vehicle-supplying partner companies.
  **Distinct concept from `Tenant`** (`tenants` — client companies Doormile
  delivers *for*). Confusingly close names, flagged, not acted on.
- `DoormileClient`/`DoormileAuth` — the real, separate CRM feature (§3.5),
  not to be confused with "CRM bookings" (renamed to "express bookings" this
  session specifically to end that naming collision).
- `Pricing`, `DoormilePricing`, `CompetitorBranch`, `CarrierPricing` —
  pricing engine + competitive intel.
- `AgentDecision` — AI dispatch-reasoning log (§4), pgvector
  `context_embedding` column.
- Redis-only ephemeral structs (`models/redis.go`): `MilerLog`,
  `MilerStatus`, `ConsignmentLog`, `CachedUser` — never durable state.

---

## 6. Current environment / seeded data state

**[carried forward, not re-verified this session]**

- Database seeded with 16 hubs (4 per city across the 4 target cities), 10+
  milers, 10 tenants (4 Doormile-ops-owned + 6 named partner tenants), 5 hub
  staff accounts, 55+ pricing rules.
- Credentials on file from prior sessions (values not repeated here —
  confirm current values rather than assuming): admin login at
  `suriya@doormile.com`; hub staff accounts pattern `hub.[city]@doormile.in`
  plus partner variants; miler test phone numbers + PINs.
- Auth: Firebase OTP for customer login — this cannot be scripted/bypassed
  from the command line, which is why the last live E2E test stalled (§9).

---

## 7. Codebase conventions (read before writing any Go here)

**[verified this session]**

- **Response helpers** (`utils` package): `utils.OK(c, data)`,
  `utils.Created(c, data)`, `utils.Message(c, "text")`,
  `utils.List(c, slice, total)`, `utils.Paginated(c, slice, total, page)`,
  `utils.BadRequest/NotFound/Internal/Unauthorized/Forbidden/Conflict(c,
  msg)`. Always use these, never hand-roll `c.JSON`.
- **DB**: `db.DB` is the package-level `*gorm.DB`. `db.Rdb` is Redis
  (`*redis.Client`, `go-redis/v9`). `db.Js` is NATS JetStream (nil-check
  before publishing, warn-log on failure, never fail the request over it).
  `db.Ctx` for Redis calls needing a context.
- **Auth**: `c.Locals("userid")` (int), `c.Locals("tenantid")` (int, only
  set via `middlewares.AuthMiddleware` — the *requesting user's own*
  tenant, not necessarily the resource's tenant; conflating these two was
  exactly the bug fixed this session, §8). Hub staff auth is separate
  (`middlewares.HubStaffAuth`), sets `c.Locals("hubid")` and
  `c.Locals("userid")` = `hubstaffaccountid`.
- **DTOs**: `dto/*.go` (`admin.go`, `auth.go`, `booking.go`, `client.go`),
  one struct per request shape.
- **Constants** (`constants/constants.go`): every status enum lives here as
  typed string constants. Always use these, never inline string literals.
- **Soft delete**: only some models have `Deletedat *time.Time` (`Hub`,
  `Vehicle`, `Consignment`, `Tripsheet`, `TripsheetItem`,
  `ConsignmentException`, `DoormilePricing`, `CarrierPricing`). Others
  (`PartnerInfo`, `Tenant`) have no soft-delete column — hard `Delete`.
  Check the struct before assuming either way.
- **Assignment logic reuse**: `AssignMilerToBooking(bookingID,
  milerUserID int, assignedByUserID *int)` in
  `controllers/booking_assignment_service.go` is the one transactional path
  for "assign this miler to this booking." Reuse it, don't reimplement —
  both `HubAssignMiler`/`AdminAssignMiler` and `HubBatchAssign` call it. The
  AI-driven `commitAssignment` in `internal/assignment` is a separate
  transactional writer used by the auto-assignment retry path — don't
  conflate the two; know which one a given call site needs.
- **Distance calc**: `haversineKM(lat1, lon1, lat2, lon2)` defined once in
  `hubController.go`, used package-wide. Don't redefine it.
- **Hub tenant scoping**: `scopeBookingsToOwnTenant(c, query)` in
  `hubController.go` (**added this session**) — restricts a bookings query
  to the requesting hub staff's own tenant if partner-scoped, no-ops for
  Doormile staff. Use on any new hub-console bookings query.
- **No Go toolchain was available in the prior sandbox session.** Every
  change was verified by hand (field/column names cross-checked against
  real structs, brace-balance via `grep -o "{" | wc -l` vs `}`) but **never
  actually compiled**. Run `go build ./...` before trusting any of it — the
  single most important unfinished step from that session.

---

## 8. The Jupiter → Doormile migration (this session's scoped work)

This section is what one specific session did: closing API/schema gaps
between the legacy Nearle ("jupiter") system and the new Doormile backend,
for the **rider app and console specifically** (hub console gap-closure was
also done incidentally while fixing the tenant leak, but wasn't the primary
target).

### 8.1 Why jupiter/Nearle was being replaced
Legacy Go+Fiber backend (`backend_jupiter`, module `nearle`), Postgres,
Redis, base URL `jupiter.nearle.app` (+ a separate write path
`queue.workolik.com` with TLS verification disabled and a hardcoded IP
pin). Consumed by a Flutter rider app and a React admin console
(`doormile_express_console` — literal ancestor of the current Doormile admin
console). Found to be structurally broken on live-DB analysis:
- `riderlogs`: 1.17M rows, 643MB, **zero indexes**, 68M lifetime UPDATEs vs
  762K inserts, caused by an unindexed `UPDATE ... WHERE userid=?` rewriting
  ~97K rows per call — the real root cause of read timeouts previously
  blamed on Redis.
- Every table had only its primary key indexed; `orders`/`deliveries` never
  autovacuumed.
- `getdeliveries` returned every row **21×** (unconstrained `LEFT JOIN
  tenantpricing`, `DISTINCT` over 87 columns that didn't dedupe anything).
- `createdeliveries` had a quadratic insert bug (slice declared outside a
  loop kept accumulating) — confirmed live: 66,446 deliveries → 132,826
  `deliveryqueues` rows (~2×) with duplicate `deliveryid`s.
- v2 endpoints wrote **only to Redis**, invisible to v1/v3 Postgres reads —
  genuine split-brain, with a Redis `INCR` ID space independent of the
  Postgres sequence (collision risk).
- `orders` had 75 columns (~20 never populated once across 137K rows).
  `deliveries` had 92 columns, six lat/lng pairs for 3 real points, status
  spread across 6 separate text+timestamp columns instead of an events
  table.
- `PUT /deliveries/updatedelivery` was overloaded for **11 different real
  actions** (8 rider status transitions + 3 unrelated console actions),
  distinguished only by which JSON fields happened to be non-empty.
- The "exhaustive" API docs undercounted real usage by ~8 endpoints
  (including the login endpoint itself and a whole `/v1/substitutions` CRUD
  feature), found only by cross-checking against actual console source.

Doormile's new schema was already solving most of this structurally before
this session (Redis-only telemetry, real event-log tables, 37 normalized
tables with real FKs and typed columns, parcel/courier domain model instead
of retail/food-delivery shaped).

### 8.2 What was built this session (14 new endpoints)

| Method | Path | Handler |
|---|---|---|
| POST | `/miler/reset-pin` | `ResetMilerPin` |
| POST | `/miler/bookings/:bookingid/cancel` | `MilerCancelAssignment` |
| POST | `/miler/consignments/:id/skip` | `MilerSkipDelivery` |
| GET | `/admin/reports` | `GetAdminReports` |
| PUT | `/admin/profile/password` | `AdminChangePassword` |
| GET | `/admin/partners` | `GetPartners` |
| POST | `/admin/partners` | `CreatePartner` |
| GET | `/admin/partners/:id` | `GetPartnerDetails` |
| PUT | `/admin/partners/:id` | `UpdatePartner` |
| DELETE | `/admin/partners/:id` | `DeletePartner` |
| POST | `/admin/milers/:id/notify` | `AdminNotifyMiler` |
| POST | `/admin/expressbooking/bulk` | `AdminBulkCreateBookings` |
| POST | `/admin/bookings/bulk-cancel` | `AdminBulkCancelBookings` |
| POST | `/hub/bookings/batch-assign` | `HubBatchAssign` |

Plus two real bugs found and fixed incidentally while doing tenant-
separation work (not requested, found along the way):
1. **Data mis-attribution**: `BookingPickupComplete` set the resulting
   `Consignment.Tenantid` from the *completing miler's own* tenant rather
   than the booking's actual tenant — silently mis-attributed shipments for
   any miler carrying parcels across tenants. Fixed to use
   `booking.Tenantid` when set.
2. **Cross-tenant data leak**: `GetHubUnassignedBookings`/
   `GetHubBookingsRange` had no tenant scoping — a partner tenant's hub
   staff could see every other tenant's bookings at the same hub. Fixed via
   `scopeBookingsToOwnTenant`.

Naming cleanup: `CreateCRMBooking`→`CreateExpressBooking`,
`/admin/crmbooking`→`/admin/expressbooking` (+ `/bulk`), because "CRM
bookings" collided with the genuinely separate CRM feature (§3.5).
Deliberately **not** renamed: the stored `Bookingsource: "CRM_Console"`
value and `internal/assignment/crm_assignment.go`'s `AssignCRMMiler` — a
working assignment engine and an already-written DB value, not just a
label; renaming those is a deeper change than was asked for.

Deliberately skipped: rider substitutions (`/v1/substitutions` in Nearle) —
Suriya's own call, looked low-traffic in the old system.

### 8.3 Verified vs not, for this migration slice
**Verified**: every new handler's field/column names checked by hand
against real structs; brace-balance confirmed after every edit; no
duplicate symbol definitions.
**Not verified**:
1. `go build ./...` — never run, no Go toolchain in that sandbox.
2. No integration test has hit any of the 14 new endpoints.
3. The `Tenantid` migration hasn't executed against a real DB yet — will
   run automatically via `AutoMigrate` next deploy (additive, nullable,
   safe).
4. **Nothing on the client side has changed.** The rider Flutter app and
   `doormile_express_console` still call `jupiter.nearle.app`. API coverage
   existing on Doormile does not mean traffic is using it. Not a
   flip-a-URL cutover either — response shapes are completely different
   (flat 87-column Nearle rows vs nested Doormile JSON) — every screen that
   parses a response needs rewriting, not just repointing.

---

## 9. Current blockers & open work (whole-project level)

**[carried forward]**
- Last live end-to-end system test stalled on **customer JWT acquisition**
  — customer login requires Firebase OTP on a real phone, can't be scripted
  from the command line. Needs either a real-phone run or a load-test
  workaround.
- A **load test** targeting high concurrent bookings (capacity check before
  go-live) was in progress at the end of the last relevant session, not
  finished.
- Go-live preparation across the 4 target cities is still ahead.

**Migration-slice-specific (§8.3)**: `go build` verification, integration
testing of the 14 new endpoints, running the `Tenantid` migration, and the
client-app rewrites are all still pending.

---

## 10. Deliberately skipped / open decisions

- **Rider substitutions** — skipped, low old-system traffic. Revisit if it
  turns out to matter.
- **B2C tenant attribution** — `PickupBooking.Tenantid` stays nil for B2C
  bookings; whether direct-to-consumer traffic should be attributed to one
  of the 4 Doormile-ops tenants (per city) is a business decision, not
  something to guess at.
- **Batch route optimization** — `HubBatchAssign` is a single-booking
  nearest-rider heuristic, not a multi-stop VRP solver (§4). Untested
  against real volume vs. whatever the old paid external services provided.
- **`PartnerInfo` vs `Tenant` naming confusion** — flagged, not acted on.
- **`Customer`/`CustomerLocation` (legacy) vs `AppCustomer` (new B2C)** —
  two customer-shaped tables coexisting, flagged as a future duplicate-data
  risk, not resolved.
- **`AgentDecision.context_embedding` dimension (1536) vs the reported
  MiniLM embedding size (384)** — flagged this session (§2), not
  investigated further; worth resolving before relying on vector search
  quality from that column.

---

## 11. Suggested next steps, in order

1. Resolve the customer-JWT E2E test blocker (real phone or load-test
   workaround) and finish the load test.
2. `go build ./...` in `DoormileBackend`, fix anything that doesn't
   compile — nothing from the migration session has ever been compiled.
3. Stand up a test/staging DB, let `AutoMigrate` run, smoke-test the 14 new
   migration-session endpoints with real requests.
4. Pick one low-risk console slice (e.g. Reports or Partner management —
   net-new UI, not replacing something live) and wire it to Doormile
   instead of jupiter — first real proof the cutover works end to end.
5. Only after that: rewrite the higher-traffic screens (orders/deliveries
   list, rider status updates) and the rider app's request/response
   handling.
6. Resolve the B2C tenant-attribution question before it's load-bearing for
   real revenue reporting.
7. Confirm the `AgentDecision` embedding-dimension question.
8. Decide on substitutions and batch-optimization sophistication once real
   usage data says whether they're actually needed.
9. Go-live preparation across the 4 target cities.