Files
doormile_backend/CLAUDE.md

465 lines
25 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Doormile — Full Project Memory
Portable context for a fresh Claude session working on this project on any
machine or account. Covers the **whole** Doormile system, not just one slice
of it. Read fully before touching the codebase.
A note on provenance: sections marked **[verified this session]** were
confirmed by directly reading this repo's code. Sections marked **[carried
forward]** come from Suriya's own prior-session notes and have NOT been
independently re-verified against source — treat them as reported state, not
confirmed state, until checked.
---
## 1. What Doormile is
Suriya is building **Doormile**, a real commercial logistics platform for the
Indian market (Coimbatore, Hyderabad, Bengaluru, Chennai) — hub-based parcel
courier/delivery, not a prototype. Treat everything as production-grade from
the start. Stakeholders: Doormile's own ops team, partner tenants (client
logistics/travel companies), hub staff, milers (delivery riders), and end
customers.
**Suriya's standing preferences** (apply without being asked):
- Direct, honest technical assessments over diplomatic framing. If something
isn't done, say so plainly. Don't declare victory early.
- Production-grade code from the start, not "refactor later."
- **Warn before any consequential server/schema change** before making it.
- Concise answers — don't over-explain.
- Once a decision is made, don't keep re-asking for confirmation on the same
scope — proceed and report back. But *do* ask when a decision is genuinely
ambiguous or touches money/data-correctness in a way that can't be safely
guessed.
- Minimal-effort, highest-leverage fixes over big rewrites, until he
explicitly asks for the big rewrite.
- Works across multiple Claude sessions/machines simultaneously, treating
Claude as a co-architect across the full stack — this file exists so any
of those sessions can pick up full context.
---
## 2. System architecture — the whole stack
**[carried forward, substantial infra reported as built and verified in
prior sessions]**
- **Backend**: Go + Fiber, deployed on **Kubernetes**, at `api.doormile.com`.
200 registered routes **[verified this session, exact count]** — see §7.
This is the primary booking/assignment API and the primary trigger for
miler assignment, calling the AI decision layer with a 5-second timeout
fallback so a slow AI response never blocks a booking.
- **AI dispatch layer**: a Python autonomous agent swarm (8 agents, **JARVIS**
as master orchestrator) on a dedicated server, connected to NATS
JetStream. The `DispatchAgent` operates as a NATS *watcher*, not the
assignment trigger — the Go backend triggers assignment directly (see §4);
the agent swarm reasons about it, doesn't gate it.
- **AI decision engine**: `routemate.workolik.com` — runs Claude Sonnet for
miler-assignment reasoning, scoring candidates using hub load, on-time
rate, and hub capacity. Uses pgvector-backed RAG memory with local
embeddings (`all-MiniLM-L6-v2`, 384-dim) specifically to avoid ongoing
per-call API cost. **[verified this session]**: the Go side of this is
`models.AgentDecision` (`agentdecisions` table) with a
`context_embedding vector(1536)` column and an ivfflat index (raw SQL in
`migrations/migrate.go`, not GORM AutoMigrate) — note the dimension
mismatch worth flagging (1536 in the Go model/migration vs 384 for
MiniLM); worth confirming which embedding model is actually populating
that column before trusting vector search quality.
- **Event bus**: NATS JetStream, 6 persistent streams, dedicated server.
**[verified this session]**: `db.Js` is the package-level JetStream handle;
convention across the codebase is `if db.Js != nil { ... }` and warn-log
on publish failure rather than failing the request — publishing is
best-effort, never blocking. Confirmed publish call sites include
`booking.assigned` (from `internal/assignment`) and NATS publishes inside
`AssignMilerToBooking` (`controllers/booking_assignment_service.go`).
- **Data stores**: PostgreSQL + pgvector 0.7.4, dedicated server, 37 tables
under GORM AutoMigrate **[verified this session, exact count from
`migrations/migrate.go`]**. Redis (`go-redis/v9`) for high-frequency
ephemeral state — GPS pings, live miler status/location (GEO-indexed at
`milers:locations`, used by `queryNearbyMilers` for assignment radius
search), booking cache. Durable business state always lands in Postgres;
Redis is never the system of record for anything that needs to survive a
flush.
- **Ingress/infra**: Kubernetes, Traefik, nginx.
---
## 3. The client-facing surfaces
Five distinct front doors into this system. Only the backend API surface for
each has been directly inspected this session (via `routes.go`); the actual
client codebases (Flutter, React) have not been opened in this session.
1. **Customer app (B2C)** — Flutter. Auth via Firebase OTP (phone). Backend
surface: 19 customer routes **[verified this session]**
(`customer`/`customerAuth` groups) — booking creation, tracking,
`AppCustomer`/`AppCustomerLocation`. **[carried forward]**: reported built
and verified in prior sessions; the last live end-to-end test was
blocked here — see §9.
2. **Miler app** — Flutter, for delivery riders. Backend surface: 38 routes
**[verified this session]** (`miler`/`milerAuth` groups) — duty
start/stop, GPS pings, assignment accept/reject/cancel, delivery
complete/skip, break logs, support tickets. This session added
`ResetMilerPin`, `MilerCancelAssignment`, `MilerSkipDelivery` to close
gaps found against the old Nearle rider app (§8).
3. **Admin console** — React, converted from the old `NearlExpress`
/`doormile_express_console` codebase. Backend surface: 89 routes
**[verified this session]** (`admin`/`adminAuth` groups) — bookings,
partner management, reports, pricing, express (formerly "CRM") bookings.
This session added partner CRUD, bulk booking create/cancel, reports,
password change, miler notify (§8).
4. **Hub console** — React, separate from the admin console. Backend
surface: 31 routes **[verified this session]** (`hub`/`hubAuth` groups)
— per-hub booking queues, tripsheet building, manual/batch assignment,
hub messaging. Auth is separate from admin (`middlewares.HubStaffAuth`,
sets `c.Locals("hubid")`). This session added tenant-scoping (closed a
real cross-tenant leak) and `HubBatchAssign` (§8).
5. **CRM (field sales)** — a genuinely separate feature from "CRM bookings"
(see §8 naming note). `POST/GET /crm/clients`, `GET/PUT/DELETE
/crm/clients/:id` — 5 routes, all open/no-auth by design: field reps
register leads from the Flutter side, the web console reads without a
separate CRM login. Models: `DoormileClient`, `DoormileAuth`. **Not
deeply audited** — only the route registration was read this session.
Total: 200 routes = miler 38 + admin 89 + hub 31 + customer 19 + crm 5 +
internal 5 + redisUsers 4 + bookingCache 3 + 2 top-level pricing routes +
websocket routes. **[verified this session]**
---
## 4. AI dispatch & assignment — how a booking actually gets a miler
**[verified this session]** — read directly from
`internal/assignment/crm_assignment.go` and `booking_assignment_service.go`.
- Two entry points, same core: `AssignCustomerMiler` (B2C bookings) and
`AssignCRMMiler` (express/console-originated bookings — name predates this
session's "CRM"→"express" rename, deliberately left unrenamed since it's a
working assignment engine, not just a label). Both are fire-and-forget
goroutines called after transaction commit, retrying up to 5 times, 2
minutes apart (~10 minutes total), before giving up and publishing a
failure event (`publishAssignmentFailed`, `reasonNoMilerAvailable`) to NATS
so the failure isn't silently invisible.
- `TryAssignOnce` is the synchronous single-attempt variant — used when a
hub staff member manually triggers "auto-assign" from the console and
needs an immediate answer rather than the ~10-minute retry loop. Returns a
rich `AutoAssignResult` (assigned/escalated, miler name, distance, AI
reasoning text, candidates found) for the UI to show.
- Core flow (`tryAssign`): Redis `GEOSEARCH` on `milers:locations` (10km
radius, top 10, sorted nearest-first) → `selectMilerWithAI` (the actual
Claude Sonnet call to `routemate.workolik.com`, logs an `AgentDecision`
row with reasoning + optional embedding) → `commitAssignment` (single DB
transaction: create `BookingAssignment`, update `PickupBooking.status` to
`BookingMilerAssigned`, flip `MilerProfile.availabilitystatus`) → publish
`booking.assigned` to NATS (best-effort) → push notifications to miler and
customer.
- `HubBatchAssign` (**built this session**, `controllers/hubController.go`)
is a separate, simpler path: a greedy nearest-available-rider heuristic
(haversine distance, no AI call, no retry loop) for clearing a hub's
pending-pickup queue in one batch call, capped per rider (default 5),
reusing `AssignMilerToBooking` for the actual transactional assignment.
**Explicitly not** a replacement for `selectMilerWithAI`'s reasoning, and
**not** a multi-stop route/VRP solver — see the routing gap below.
**Logistics/routing — what exists vs what doesn't:**
- What exists: hub network (`Hub` model, `originhubid`/`currenthubid`/
`destinationhubid` on bookings/consignments for tracking a parcel's hub
path), `Tripsheet`/`TripsheetItem` for hub-to-hub batch transport, the
nearest-rider assignment logic above, `HubBatchAssign`'s greedy queue
clearer.
- What does **not** exist in Doormile yet: any multi-stop route sequencing /
vehicle-routing-problem solver. Nearle used external paid services
(`routes.workolik.com`, and `routemate.workolik.com` was itself originally
a Nearle-side concept) for that; nothing in Doormile replaces true
stop-sequencing optimization today. `HubBatchAssign` only decides *who*
gets which single booking, not *what order* a rider should run multiple
stops in. Flag this clearly if asked whether routing is "done" — it isn't,
by design, pending a real conversation about whether it's needed yet.
---
## 5. Data model reference (`models/*.go`)
**[verified this session]**
- `PickupBooking` (`pickupbookings`) — customer's pickup request, pre-hub.
Has `Tenantid *int` (**added this session**, see §8). `Bookingsource`:
`"Customer_App"` (B2C) or `"CRM_Console"` (console-created — deliberately
left as `"CRM_Console"` even though the outward API/route naming was
changed to "express", see §8).
- `BookingParcel`, `BookingServiceOption`, `BookingPayment`,
`BookingAssignment` (status: Assigned/Accepted/Rejected/Reassigned/
Completed/Cancelled, carries `AgentDecisionID *uint64` linking back to the
AI reasoning that produced it), `BookingVehicleRequirement`.
- `Consignment` (`consignments`) — the shipment once picked up. Has its own
`Tenantid int` (not nullable, pre-existing). `Attemptcount int` (used by
`MilerSkipDelivery`, added this session). `ConsignmentHistory` (event
log), `ConsignmentException` (Lost/Damaged/Misrouted/Receiver_Refused/
Missing_Contents/Undeliverable).
- `Hub`, `Vehicle`, `Tripsheet`, `TripsheetItem`, `DeliveryProof`.
- `AppUser` (`appusers`) — shared login table for staff/miler/admin roles
(`Roleid`: 1 admin, 3 manager, 4 rep/exec, 5 miler, 6 hub staff via a
separate `HubStaffAccount` table). `MilerProfile` — actual rider profile
(availability status, current lat/lng, rating). `MilerDutyLog`,
`MilerBreakLog`, `MilerSupportTicket`.
- `AppCustomer`/`AppCustomerLocation` — B2C app customers (separate from
legacy `Customer`/`CustomerLocation` kept for backward compat — flagged as
a future duplicate-data risk, not resolved).
- `HubStaffAccount``Tenantid *int`: **nil = Doormile staff (sees
everything)**, **set = partner-tenant staff (own tenant's data only)**.
Load-bearing distinction — see the tenant-leak fix in §8.
- `PartnerInfo` (`partnerinfo`) — fleet/vehicle-supplying partner companies.
**Distinct concept from `Tenant`** (`tenants` — client companies Doormile
delivers *for*). Confusingly close names, flagged, not acted on.
- `DoormileClient`/`DoormileAuth` — the real, separate CRM feature (§3.5),
not to be confused with "CRM bookings" (renamed to "express bookings" this
session specifically to end that naming collision).
- `Pricing`, `DoormilePricing`, `CompetitorBranch`, `CarrierPricing`
pricing engine + competitive intel.
- `AgentDecision` — AI dispatch-reasoning log (§4), pgvector
`context_embedding` column.
- Redis-only ephemeral structs (`models/redis.go`): `MilerLog`,
`MilerStatus`, `ConsignmentLog`, `CachedUser` — never durable state.
---
## 6. Current environment / seeded data state
**[carried forward, not re-verified this session]**
- Database seeded with 16 hubs (4 per city across the 4 target cities), 10+
milers, 10 tenants (4 Doormile-ops-owned + 6 named partner tenants), 5 hub
staff accounts, 55+ pricing rules.
- Credentials on file from prior sessions (values not repeated here —
confirm current values rather than assuming): admin login at
`suriya@doormile.com`; hub staff accounts pattern `hub.[city]@doormile.in`
plus partner variants; miler test phone numbers + PINs.
- Auth: Firebase OTP for customer login — this cannot be scripted/bypassed
from the command line, which is why the last live E2E test stalled (§9).
---
## 7. Codebase conventions (read before writing any Go here)
**[verified this session]**
- **Response helpers** (`utils` package): `utils.OK(c, data)`,
`utils.Created(c, data)`, `utils.Message(c, "text")`,
`utils.List(c, slice, total)`, `utils.Paginated(c, slice, total, page)`,
`utils.BadRequest/NotFound/Internal/Unauthorized/Forbidden/Conflict(c,
msg)`. Always use these, never hand-roll `c.JSON`.
- **DB**: `db.DB` is the package-level `*gorm.DB`. `db.Rdb` is Redis
(`*redis.Client`, `go-redis/v9`). `db.Js` is NATS JetStream (nil-check
before publishing, warn-log on failure, never fail the request over it).
`db.Ctx` for Redis calls needing a context.
- **Auth**: `c.Locals("userid")` (int), `c.Locals("tenantid")` (int, only
set via `middlewares.AuthMiddleware` — the *requesting user's own*
tenant, not necessarily the resource's tenant; conflating these two was
exactly the bug fixed this session, §8). Hub staff auth is separate
(`middlewares.HubStaffAuth`), sets `c.Locals("hubid")` and
`c.Locals("userid")` = `hubstaffaccountid`.
- **DTOs**: `dto/*.go` (`admin.go`, `auth.go`, `booking.go`, `client.go`),
one struct per request shape.
- **Constants** (`constants/constants.go`): every status enum lives here as
typed string constants. Always use these, never inline string literals.
- **Soft delete**: only some models have `Deletedat *time.Time` (`Hub`,
`Vehicle`, `Consignment`, `Tripsheet`, `TripsheetItem`,
`ConsignmentException`, `DoormilePricing`, `CarrierPricing`). Others
(`PartnerInfo`, `Tenant`) have no soft-delete column — hard `Delete`.
Check the struct before assuming either way.
- **Assignment logic reuse**: `AssignMilerToBooking(bookingID,
milerUserID int, assignedByUserID *int)` in
`controllers/booking_assignment_service.go` is the one transactional path
for "assign this miler to this booking." Reuse it, don't reimplement —
both `HubAssignMiler`/`AdminAssignMiler` and `HubBatchAssign` call it. The
AI-driven `commitAssignment` in `internal/assignment` is a separate
transactional writer used by the auto-assignment retry path — don't
conflate the two; know which one a given call site needs.
- **Distance calc**: `haversineKM(lat1, lon1, lat2, lon2)` defined once in
`hubController.go`, used package-wide. Don't redefine it.
- **Hub tenant scoping**: `scopeBookingsToOwnTenant(c, query)` in
`hubController.go` (**added this session**) — restricts a bookings query
to the requesting hub staff's own tenant if partner-scoped, no-ops for
Doormile staff. Use on any new hub-console bookings query.
- **Go toolchain was unavailable in the sandbox that wrote this session's
changes.** Every change was verified by hand (field/column names
cross-checked against real structs, brace-balance via `grep -o "{" |
wc -l` vs `}`) at write time. **[verified separately, on Suriya's own
machine]**: `go build ./...` and `go vet ./...` were both run afterward
and passed clean (only pulled two missing indirect modules,
`tinylib/msgp` and `philhofer/fwd`). This was the first actual compiler
verification of this code — commit `c272a33`, pushed to `origin/main`.
---
## 8. The Jupiter → Doormile migration (this session's scoped work)
This section is what one specific session did: closing API/schema gaps
between the legacy Nearle ("jupiter") system and the new Doormile backend,
for the **rider app and console specifically** (hub console gap-closure was
also done incidentally while fixing the tenant leak, but wasn't the primary
target).
### 8.1 Why jupiter/Nearle was being replaced
Legacy Go+Fiber backend (`backend_jupiter`, module `nearle`), Postgres,
Redis, base URL `jupiter.nearle.app` (+ a separate write path
`queue.workolik.com` with TLS verification disabled and a hardcoded IP
pin). Consumed by a Flutter rider app and a React admin console
(`doormile_express_console` — literal ancestor of the current Doormile admin
console). Found to be structurally broken on live-DB analysis:
- `riderlogs`: 1.17M rows, 643MB, **zero indexes**, 68M lifetime UPDATEs vs
762K inserts, caused by an unindexed `UPDATE ... WHERE userid=?` rewriting
~97K rows per call — the real root cause of read timeouts previously
blamed on Redis.
- Every table had only its primary key indexed; `orders`/`deliveries` never
autovacuumed.
- `getdeliveries` returned every row **21×** (unconstrained `LEFT JOIN
tenantpricing`, `DISTINCT` over 87 columns that didn't dedupe anything).
- `createdeliveries` had a quadratic insert bug (slice declared outside a
loop kept accumulating) — confirmed live: 66,446 deliveries → 132,826
`deliveryqueues` rows (~2×) with duplicate `deliveryid`s.
- v2 endpoints wrote **only to Redis**, invisible to v1/v3 Postgres reads —
genuine split-brain, with a Redis `INCR` ID space independent of the
Postgres sequence (collision risk).
- `orders` had 75 columns (~20 never populated once across 137K rows).
`deliveries` had 92 columns, six lat/lng pairs for 3 real points, status
spread across 6 separate text+timestamp columns instead of an events
table.
- `PUT /deliveries/updatedelivery` was overloaded for **11 different real
actions** (8 rider status transitions + 3 unrelated console actions),
distinguished only by which JSON fields happened to be non-empty.
- The "exhaustive" API docs undercounted real usage by ~8 endpoints
(including the login endpoint itself and a whole `/v1/substitutions` CRUD
feature), found only by cross-checking against actual console source.
Doormile's new schema was already solving most of this structurally before
this session (Redis-only telemetry, real event-log tables, 37 normalized
tables with real FKs and typed columns, parcel/courier domain model instead
of retail/food-delivery shaped).
### 8.2 What was built this session (14 new endpoints)
| Method | Path | Handler |
|---|---|---|
| POST | `/miler/reset-pin` | `ResetMilerPin` |
| POST | `/miler/bookings/:bookingid/cancel` | `MilerCancelAssignment` |
| POST | `/miler/consignments/:id/skip` | `MilerSkipDelivery` |
| GET | `/admin/reports` | `GetAdminReports` |
| PUT | `/admin/profile/password` | `AdminChangePassword` |
| GET | `/admin/partners` | `GetPartners` |
| POST | `/admin/partners` | `CreatePartner` |
| GET | `/admin/partners/:id` | `GetPartnerDetails` |
| PUT | `/admin/partners/:id` | `UpdatePartner` |
| DELETE | `/admin/partners/:id` | `DeletePartner` |
| POST | `/admin/milers/:id/notify` | `AdminNotifyMiler` |
| POST | `/admin/expressbooking/bulk` | `AdminBulkCreateBookings` |
| POST | `/admin/bookings/bulk-cancel` | `AdminBulkCancelBookings` |
| POST | `/hub/bookings/batch-assign` | `HubBatchAssign` |
Plus two real bugs found and fixed incidentally while doing tenant-
separation work (not requested, found along the way):
1. **Data mis-attribution**: `BookingPickupComplete` set the resulting
`Consignment.Tenantid` from the *completing miler's own* tenant rather
than the booking's actual tenant — silently mis-attributed shipments for
any miler carrying parcels across tenants. Fixed to use
`booking.Tenantid` when set.
2. **Cross-tenant data leak**: `GetHubUnassignedBookings`/
`GetHubBookingsRange` had no tenant scoping — a partner tenant's hub
staff could see every other tenant's bookings at the same hub. Fixed via
`scopeBookingsToOwnTenant`.
Naming cleanup: `CreateCRMBooking`→`CreateExpressBooking`,
`/admin/crmbooking`→`/admin/expressbooking` (+ `/bulk`), because "CRM
bookings" collided with the genuinely separate CRM feature (§3.5).
Deliberately **not** renamed: the stored `Bookingsource: "CRM_Console"`
value and `internal/assignment/crm_assignment.go`'s `AssignCRMMiler` — a
working assignment engine and an already-written DB value, not just a
label; renaming those is a deeper change than was asked for.
Deliberately skipped: rider substitutions (`/v1/substitutions` in Nearle) —
Suriya's own call, looked low-traffic in the old system.
### 8.3 Verified vs not, for this migration slice
**Verified**: every new handler's field/column names checked by hand
against real structs; brace-balance confirmed after every edit; no
duplicate symbol definitions. **`go build ./...` and `go vet ./...` both
pass clean** (verified on Suriya's machine, not the sandbox that wrote the
code) — committed as `c272a33`, pushed to `origin/main`. The hand-verified
code compiled correctly the first time it hit a real toolchain.
**Not verified**:
1. No integration test has hit any of the 14 new endpoints — a clean
compile says the code is well-formed, not that it behaves correctly
against a real DB/Redis/NATS.
2. The `Tenantid` migration hasn't executed against a real DB yet — will
run automatically via `AutoMigrate` next deploy (additive, nullable,
safe).
3. **Nothing on the client side has changed.** The rider Flutter app and
`doormile_express_console` still call `jupiter.nearle.app`. API coverage
existing on Doormile does not mean traffic is using it. Not a
flip-a-URL cutover either — response shapes are completely different
(flat 87-column Nearle rows vs nested Doormile JSON) — every screen that
parses a response needs rewriting, not just repointing.
---
## 9. Current blockers & open work (whole-project level)
**[carried forward]**
- Last live end-to-end system test stalled on **customer JWT acquisition**
— customer login requires Firebase OTP on a real phone, can't be scripted
from the command line. Needs either a real-phone run or a load-test
workaround.
- A **load test** targeting high concurrent bookings (capacity check before
go-live) was in progress at the end of the last relevant session, not
finished.
- Go-live preparation across the 4 target cities is still ahead.
**Migration-slice-specific (§8.3)**: `go build` verification is now done
(commit `c272a33`). Still pending: integration testing of the 14 new
endpoints, running the `Tenantid` migration, and the client-app rewrites.
---
## 10. Deliberately skipped / open decisions
- **Rider substitutions** — skipped, low old-system traffic. Revisit if it
turns out to matter.
- **B2C tenant attribution** — `PickupBooking.Tenantid` stays nil for B2C
bookings; whether direct-to-consumer traffic should be attributed to one
of the 4 Doormile-ops tenants (per city) is a business decision, not
something to guess at.
- **Batch route optimization** — `HubBatchAssign` is a single-booking
nearest-rider heuristic, not a multi-stop VRP solver (§4). Untested
against real volume vs. whatever the old paid external services provided.
- **`PartnerInfo` vs `Tenant` naming confusion** — flagged, not acted on.
- **`Customer`/`CustomerLocation` (legacy) vs `AppCustomer` (new B2C)** —
two customer-shaped tables coexisting, flagged as a future duplicate-data
risk, not resolved.
- **`AgentDecision.context_embedding` dimension (1536) vs the reported
MiniLM embedding size (384)** — flagged this session (§2), not
investigated further; worth resolving before relying on vector search
quality from that column.
---
## 11. Suggested next steps, in order
1. Resolve the customer-JWT E2E test blocker (real phone or load-test
workaround) and finish the load test.
2. ~~`go build ./...` in `DoormileBackend`~~ — **done**, clean pass,
commit `c272a33` pushed to `origin/main`.
3. Stand up a test/staging DB, let `AutoMigrate` run, smoke-test the 14 new
migration-session endpoints with real requests.
4. Pick one low-risk console slice (e.g. Reports or Partner management —
net-new UI, not replacing something live) and wire it to Doormile
instead of jupiter — first real proof the cutover works end to end.
5. Only after that: rewrite the higher-traffic screens (orders/deliveries
list, rider status updates) and the rider app's request/response
handling.
6. Resolve the B2C tenant-attribution question before it's load-bearing for
real revenue reporting.
7. Confirm the `AgentDecision` embedding-dimension question.
8. Decide on substitutions and batch-optimization sophistication once real
usage data says whether they're actually needed.
9. Go-live preparation across the 4 target cities.