Doormile/doormile_backend
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
doormile_backend/CLAUDE.md

25 KiB
Raw Permalink Blame History

Doormile — Full Project Memory

Portable context for a fresh Claude session working on this project on any machine or account. Covers the whole Doormile system, not just one slice of it. Read fully before touching the codebase.

A note on provenance: sections marked [verified this session] were confirmed by directly reading this repo's code. Sections marked [carried forward] come from Suriya's own prior-session notes and have NOT been independently re-verified against source — treat them as reported state, not confirmed state, until checked.

1. What Doormile is

Suriya is building Doormile, a real commercial logistics platform for the Indian market (Coimbatore, Hyderabad, Bengaluru, Chennai) — hub-based parcel courier/delivery, not a prototype. Treat everything as production-grade from the start. Stakeholders: Doormile's own ops team, partner tenants (client logistics/travel companies), hub staff, milers (delivery riders), and end customers.

Suriya's standing preferences (apply without being asked):

  • Direct, honest technical assessments over diplomatic framing. If something isn't done, say so plainly. Don't declare victory early.
  • Production-grade code from the start, not "refactor later."
  • Warn before any consequential server/schema change before making it.
  • Concise answers — don't over-explain.
  • Once a decision is made, don't keep re-asking for confirmation on the same scope — proceed and report back. But do ask when a decision is genuinely ambiguous or touches money/data-correctness in a way that can't be safely guessed.
  • Minimal-effort, highest-leverage fixes over big rewrites, until he explicitly asks for the big rewrite.
  • Works across multiple Claude sessions/machines simultaneously, treating Claude as a co-architect across the full stack — this file exists so any of those sessions can pick up full context.

2. System architecture — the whole stack

[carried forward, substantial infra reported as built and verified in prior sessions]

  • Backend: Go + Fiber, deployed on Kubernetes, at api.doormile.com. 200 registered routes [verified this session, exact count] — see §7. This is the primary booking/assignment API and the primary trigger for miler assignment, calling the AI decision layer with a 5-second timeout fallback so a slow AI response never blocks a booking.
  • AI dispatch layer: a Python autonomous agent swarm (8 agents, JARVIS as master orchestrator) on a dedicated server, connected to NATS JetStream. The DispatchAgent operates as a NATS watcher, not the assignment trigger — the Go backend triggers assignment directly (see §4); the agent swarm reasons about it, doesn't gate it.
  • AI decision engine: routemate.workolik.com — runs Claude Sonnet for miler-assignment reasoning, scoring candidates using hub load, on-time rate, and hub capacity. Uses pgvector-backed RAG memory with local embeddings (all-MiniLM-L6-v2, 384-dim) specifically to avoid ongoing per-call API cost. [verified this session]: the Go side of this is models.AgentDecision (agentdecisions table) with a context_embedding vector(1536) column and an ivfflat index (raw SQL in migrations/migrate.go, not GORM AutoMigrate) — note the dimension mismatch worth flagging (1536 in the Go model/migration vs 384 for MiniLM); worth confirming which embedding model is actually populating that column before trusting vector search quality.
  • Event bus: NATS JetStream, 6 persistent streams, dedicated server. [verified this session]: db.Js is the package-level JetStream handle; convention across the codebase is if db.Js != nil { ... } and warn-log on publish failure rather than failing the request — publishing is best-effort, never blocking. Confirmed publish call sites include booking.assigned (from internal/assignment) and NATS publishes inside AssignMilerToBooking (controllers/booking_assignment_service.go).
  • Data stores: PostgreSQL + pgvector 0.7.4, dedicated server, 37 tables under GORM AutoMigrate [verified this session, exact count from migrations/migrate.go]. Redis (go-redis/v9) for high-frequency ephemeral state — GPS pings, live miler status/location (GEO-indexed at milers:locations, used by queryNearbyMilers for assignment radius search), booking cache. Durable business state always lands in Postgres; Redis is never the system of record for anything that needs to survive a flush.
  • Ingress/infra: Kubernetes, Traefik, nginx.

3. The client-facing surfaces

Five distinct front doors into this system. Only the backend API surface for each has been directly inspected this session (via routes.go); the actual client codebases (Flutter, React) have not been opened in this session.

  1. Customer app (B2C) — Flutter. Auth via Firebase OTP (phone). Backend surface: 19 customer routes [verified this session] (customer/customerAuth groups) — booking creation, tracking, AppCustomer/AppCustomerLocation. [carried forward]: reported built and verified in prior sessions; the last live end-to-end test was blocked here — see §9.
  2. Miler app — Flutter, for delivery riders. Backend surface: 38 routes [verified this session] (miler/milerAuth groups) — duty start/stop, GPS pings, assignment accept/reject/cancel, delivery complete/skip, break logs, support tickets. This session added ResetMilerPin, MilerCancelAssignment, MilerSkipDelivery to close gaps found against the old Nearle rider app (§8).
  3. Admin console — React, converted from the old NearlExpress /doormile_express_console codebase. Backend surface: 89 routes [verified this session] (admin/adminAuth groups) — bookings, partner management, reports, pricing, express (formerly "CRM") bookings. This session added partner CRUD, bulk booking create/cancel, reports, password change, miler notify (§8).
  4. Hub console — React, separate from the admin console. Backend surface: 31 routes [verified this session] (hub/hubAuth groups) — per-hub booking queues, tripsheet building, manual/batch assignment, hub messaging. Auth is separate from admin (middlewares.HubStaffAuth, sets c.Locals("hubid")). This session added tenant-scoping (closed a real cross-tenant leak) and HubBatchAssign (§8).
  5. CRM (field sales) — a genuinely separate feature from "CRM bookings" (see §8 naming note). POST/GET /crm/clients, GET/PUT/DELETE /crm/clients/:id — 5 routes, all open/no-auth by design: field reps register leads from the Flutter side, the web console reads without a separate CRM login. Models: DoormileClient, DoormileAuth. Not deeply audited — only the route registration was read this session.

Total: 200 routes = miler 38 + admin 89 + hub 31 + customer 19 + crm 5 + internal 5 + redisUsers 4 + bookingCache 3 + 2 top-level pricing routes + websocket routes. [verified this session]

4. AI dispatch & assignment — how a booking actually gets a miler

[verified this session] — read directly from internal/assignment/crm_assignment.go and booking_assignment_service.go.

  • Two entry points, same core: AssignCustomerMiler (B2C bookings) and AssignCRMMiler (express/console-originated bookings — name predates this session's "CRM"→"express" rename, deliberately left unrenamed since it's a working assignment engine, not just a label). Both are fire-and-forget goroutines called after transaction commit, retrying up to 5 times, 2 minutes apart (~10 minutes total), before giving up and publishing a failure event (publishAssignmentFailed, reasonNoMilerAvailable) to NATS so the failure isn't silently invisible.
  • TryAssignOnce is the synchronous single-attempt variant — used when a hub staff member manually triggers "auto-assign" from the console and needs an immediate answer rather than the ~10-minute retry loop. Returns a rich AutoAssignResult (assigned/escalated, miler name, distance, AI reasoning text, candidates found) for the UI to show.
  • Core flow (tryAssign): Redis GEOSEARCH on milers:locations (10km radius, top 10, sorted nearest-first) → selectMilerWithAI (the actual Claude Sonnet call to routemate.workolik.com, logs an AgentDecision row with reasoning + optional embedding) → commitAssignment (single DB transaction: create BookingAssignment, update PickupBooking.status to BookingMilerAssigned, flip MilerProfile.availabilitystatus) → publish booking.assigned to NATS (best-effort) → push notifications to miler and customer.
  • HubBatchAssign (built this session, controllers/hubController.go) is a separate, simpler path: a greedy nearest-available-rider heuristic (haversine distance, no AI call, no retry loop) for clearing a hub's pending-pickup queue in one batch call, capped per rider (default 5), reusing AssignMilerToBooking for the actual transactional assignment. Explicitly not a replacement for selectMilerWithAI's reasoning, and not a multi-stop route/VRP solver — see the routing gap below.

Logistics/routing — what exists vs what doesn't:

  • What exists: hub network (Hub model, originhubid/currenthubid/ destinationhubid on bookings/consignments for tracking a parcel's hub path), Tripsheet/TripsheetItem for hub-to-hub batch transport, the nearest-rider assignment logic above, HubBatchAssign's greedy queue clearer.
  • What does not exist in Doormile yet: any multi-stop route sequencing / vehicle-routing-problem solver. Nearle used external paid services (routes.workolik.com, and routemate.workolik.com was itself originally a Nearle-side concept) for that; nothing in Doormile replaces true stop-sequencing optimization today. HubBatchAssign only decides who gets which single booking, not what order a rider should run multiple stops in. Flag this clearly if asked whether routing is "done" — it isn't, by design, pending a real conversation about whether it's needed yet.

5. Data model reference (models/*.go)

[verified this session]

  • PickupBooking (pickupbookings) — customer's pickup request, pre-hub. Has Tenantid *int (added this session, see §8). Bookingsource: "Customer_App" (B2C) or "CRM_Console" (console-created — deliberately left as "CRM_Console" even though the outward API/route naming was changed to "express", see §8).
  • BookingParcel, BookingServiceOption, BookingPayment, BookingAssignment (status: Assigned/Accepted/Rejected/Reassigned/ Completed/Cancelled, carries AgentDecisionID *uint64 linking back to the AI reasoning that produced it), BookingVehicleRequirement.
  • Consignment (consignments) — the shipment once picked up. Has its own Tenantid int (not nullable, pre-existing). Attemptcount int (used by MilerSkipDelivery, added this session). ConsignmentHistory (event log), ConsignmentException (Lost/Damaged/Misrouted/Receiver_Refused/ Missing_Contents/Undeliverable).
  • Hub, Vehicle, Tripsheet, TripsheetItem, DeliveryProof.
  • AppUser (appusers) — shared login table for staff/miler/admin roles (Roleid: 1 admin, 3 manager, 4 rep/exec, 5 miler, 6 hub staff via a separate HubStaffAccount table). MilerProfile — actual rider profile (availability status, current lat/lng, rating). MilerDutyLog, MilerBreakLog, MilerSupportTicket.
  • AppCustomer/AppCustomerLocation — B2C app customers (separate from legacy Customer/CustomerLocation kept for backward compat — flagged as a future duplicate-data risk, not resolved).
  • HubStaffAccount — Tenantid *int: nil = Doormile staff (sees everything), set = partner-tenant staff (own tenant's data only). Load-bearing distinction — see the tenant-leak fix in §8.
  • PartnerInfo (partnerinfo) — fleet/vehicle-supplying partner companies. Distinct concept from Tenant (tenants — client companies Doormile delivers for). Confusingly close names, flagged, not acted on.
  • DoormileClient/DoormileAuth — the real, separate CRM feature (§3.5), not to be confused with "CRM bookings" (renamed to "express bookings" this session specifically to end that naming collision).
  • Pricing, DoormilePricing, CompetitorBranch, CarrierPricing — pricing engine + competitive intel.
  • AgentDecision — AI dispatch-reasoning log (§4), pgvector context_embedding column.
  • Redis-only ephemeral structs (models/redis.go): MilerLog, MilerStatus, ConsignmentLog, CachedUser — never durable state.

6. Current environment / seeded data state

[carried forward, not re-verified this session]

  • Database seeded with 16 hubs (4 per city across the 4 target cities), 10+ milers, 10 tenants (4 Doormile-ops-owned + 6 named partner tenants), 5 hub staff accounts, 55+ pricing rules.
  • Credentials on file from prior sessions (values not repeated here — confirm current values rather than assuming): admin login at suriya@doormile.com; hub staff accounts pattern hub.[city]@doormile.in plus partner variants; miler test phone numbers + PINs.
  • Auth: Firebase OTP for customer login — this cannot be scripted/bypassed from the command line, which is why the last live E2E test stalled (§9).

7. Codebase conventions (read before writing any Go here)

[verified this session]

  • Response helpers (utils package): utils.OK(c, data), utils.Created(c, data), utils.Message(c, "text"), utils.List(c, slice, total), utils.Paginated(c, slice, total, page), utils.BadRequest/NotFound/Internal/Unauthorized/Forbidden/Conflict(c, msg). Always use these, never hand-roll c.JSON.
  • DB: db.DB is the package-level *gorm.DB. db.Rdb is Redis (*redis.Client, go-redis/v9). db.Js is NATS JetStream (nil-check before publishing, warn-log on failure, never fail the request over it). db.Ctx for Redis calls needing a context.
  • Auth: c.Locals("userid") (int), c.Locals("tenantid") (int, only set via middlewares.AuthMiddleware — the requesting user's own tenant, not necessarily the resource's tenant; conflating these two was exactly the bug fixed this session, §8). Hub staff auth is separate (middlewares.HubStaffAuth), sets c.Locals("hubid") and c.Locals("userid") = hubstaffaccountid.
  • DTOs: dto/*.go (admin.go, auth.go, booking.go, client.go), one struct per request shape.
  • Constants (constants/constants.go): every status enum lives here as typed string constants. Always use these, never inline string literals.
  • Soft delete: only some models have Deletedat *time.Time (Hub, Vehicle, Consignment, Tripsheet, TripsheetItem, ConsignmentException, DoormilePricing, CarrierPricing). Others (PartnerInfo, Tenant) have no soft-delete column — hard Delete. Check the struct before assuming either way.
  • Assignment logic reuse: AssignMilerToBooking(bookingID, milerUserID int, assignedByUserID *int) in controllers/booking_assignment_service.go is the one transactional path for "assign this miler to this booking." Reuse it, don't reimplement — both HubAssignMiler/AdminAssignMiler and HubBatchAssign call it. The AI-driven commitAssignment in internal/assignment is a separate transactional writer used by the auto-assignment retry path — don't conflate the two; know which one a given call site needs.
  • Distance calc: haversineKM(lat1, lon1, lat2, lon2) defined once in hubController.go, used package-wide. Don't redefine it.
  • Hub tenant scoping: scopeBookingsToOwnTenant(c, query) in hubController.go (added this session) — restricts a bookings query to the requesting hub staff's own tenant if partner-scoped, no-ops for Doormile staff. Use on any new hub-console bookings query.
  • Go toolchain was unavailable in the sandbox that wrote this session's changes. Every change was verified by hand (field/column names cross-checked against real structs, brace-balance via grep -o "{" | wc -l vs }) at write time. [verified separately, on Suriya's own machine]: go build ./... and go vet ./... were both run afterward and passed clean (only pulled two missing indirect modules, tinylib/msgp and philhofer/fwd). This was the first actual compiler verification of this code — commit c272a33, pushed to origin/main.

8. The Jupiter → Doormile migration (this session's scoped work)

This section is what one specific session did: closing API/schema gaps between the legacy Nearle ("jupiter") system and the new Doormile backend, for the rider app and console specifically (hub console gap-closure was also done incidentally while fixing the tenant leak, but wasn't the primary target).

8.1 Why jupiter/Nearle was being replaced

Legacy Go+Fiber backend (backend_jupiter, module nearle), Postgres, Redis, base URL jupiter.nearle.app (+ a separate write path queue.workolik.com with TLS verification disabled and a hardcoded IP pin). Consumed by a Flutter rider app and a React admin console (doormile_express_console — literal ancestor of the current Doormile admin console). Found to be structurally broken on live-DB analysis:

  • riderlogs: 1.17M rows, 643MB, zero indexes, 68M lifetime UPDATEs vs 762K inserts, caused by an unindexed UPDATE ... WHERE userid=? rewriting ~97K rows per call — the real root cause of read timeouts previously blamed on Redis.
  • Every table had only its primary key indexed; orders/deliveries never autovacuumed.
  • getdeliveries returned every row 21× (unconstrained LEFT JOIN tenantpricing, DISTINCT over 87 columns that didn't dedupe anything).
  • createdeliveries had a quadratic insert bug (slice declared outside a loop kept accumulating) — confirmed live: 66,446 deliveries → 132,826 deliveryqueues rows (~2×) with duplicate deliveryids.
  • v2 endpoints wrote only to Redis, invisible to v1/v3 Postgres reads — genuine split-brain, with a Redis INCR ID space independent of the Postgres sequence (collision risk).
  • orders had 75 columns (~20 never populated once across 137K rows). deliveries had 92 columns, six lat/lng pairs for 3 real points, status spread across 6 separate text+timestamp columns instead of an events table.
  • PUT /deliveries/updatedelivery was overloaded for 11 different real actions (8 rider status transitions + 3 unrelated console actions), distinguished only by which JSON fields happened to be non-empty.
  • The "exhaustive" API docs undercounted real usage by ~8 endpoints (including the login endpoint itself and a whole /v1/substitutions CRUD feature), found only by cross-checking against actual console source.

Doormile's new schema was already solving most of this structurally before this session (Redis-only telemetry, real event-log tables, 37 normalized tables with real FKs and typed columns, parcel/courier domain model instead of retail/food-delivery shaped).

8.2 What was built this session (14 new endpoints)

Method Path Handler
POST /miler/reset-pin ResetMilerPin
POST /miler/bookings/:bookingid/cancel MilerCancelAssignment
POST /miler/consignments/:id/skip MilerSkipDelivery
GET /admin/reports GetAdminReports
PUT /admin/profile/password AdminChangePassword
GET /admin/partners GetPartners
POST /admin/partners CreatePartner
GET /admin/partners/:id GetPartnerDetails
PUT /admin/partners/:id UpdatePartner
DELETE /admin/partners/:id DeletePartner
POST /admin/milers/:id/notify AdminNotifyMiler
POST /admin/expressbooking/bulk AdminBulkCreateBookings
POST /admin/bookings/bulk-cancel AdminBulkCancelBookings
POST /hub/bookings/batch-assign HubBatchAssign

Plus two real bugs found and fixed incidentally while doing tenant- separation work (not requested, found along the way):

  1. Data mis-attribution: BookingPickupComplete set the resulting Consignment.Tenantid from the completing miler's own tenant rather than the booking's actual tenant — silently mis-attributed shipments for any miler carrying parcels across tenants. Fixed to use booking.Tenantid when set.
  2. Cross-tenant data leak: GetHubUnassignedBookings/ GetHubBookingsRange had no tenant scoping — a partner tenant's hub staff could see every other tenant's bookings at the same hub. Fixed via scopeBookingsToOwnTenant.

Naming cleanup: CreateCRMBooking→CreateExpressBooking, /admin/crmbooking→/admin/expressbooking (+ /bulk), because "CRM bookings" collided with the genuinely separate CRM feature (§3.5). Deliberately not renamed: the stored Bookingsource: "CRM_Console" value and internal/assignment/crm_assignment.go's AssignCRMMiler — a working assignment engine and an already-written DB value, not just a label; renaming those is a deeper change than was asked for.

Deliberately skipped: rider substitutions (/v1/substitutions in Nearle) — Suriya's own call, looked low-traffic in the old system.

8.3 Verified vs not, for this migration slice

Verified: every new handler's field/column names checked by hand against real structs; brace-balance confirmed after every edit; no duplicate symbol definitions. go build ./... and go vet ./... both pass clean (verified on Suriya's machine, not the sandbox that wrote the code) — committed as c272a33, pushed to origin/main. The hand-verified code compiled correctly the first time it hit a real toolchain. Not verified:

  1. No integration test has hit any of the 14 new endpoints — a clean compile says the code is well-formed, not that it behaves correctly against a real DB/Redis/NATS.
  2. The Tenantid migration hasn't executed against a real DB yet — will run automatically via AutoMigrate next deploy (additive, nullable, safe).
  3. Nothing on the client side has changed. The rider Flutter app and doormile_express_console still call jupiter.nearle.app. API coverage existing on Doormile does not mean traffic is using it. Not a flip-a-URL cutover either — response shapes are completely different (flat 87-column Nearle rows vs nested Doormile JSON) — every screen that parses a response needs rewriting, not just repointing.

9. Current blockers & open work (whole-project level)

[carried forward]

  • Last live end-to-end system test stalled on customer JWT acquisition — customer login requires Firebase OTP on a real phone, can't be scripted from the command line. Needs either a real-phone run or a load-test workaround.
  • A load test targeting high concurrent bookings (capacity check before go-live) was in progress at the end of the last relevant session, not finished.
  • Go-live preparation across the 4 target cities is still ahead.

Migration-slice-specific (§8.3): go build verification is now done (commit c272a33). Still pending: integration testing of the 14 new endpoints, running the Tenantid migration, and the client-app rewrites.

10. Deliberately skipped / open decisions

  • Rider substitutions — skipped, low old-system traffic. Revisit if it turns out to matter.
  • B2C tenant attribution — PickupBooking.Tenantid stays nil for B2C bookings; whether direct-to-consumer traffic should be attributed to one of the 4 Doormile-ops tenants (per city) is a business decision, not something to guess at.
  • Batch route optimization — HubBatchAssign is a single-booking nearest-rider heuristic, not a multi-stop VRP solver (§4). Untested against real volume vs. whatever the old paid external services provided.
  • PartnerInfo vs Tenant naming confusion — flagged, not acted on.
  • Customer/CustomerLocation (legacy) vs AppCustomer (new B2C) — two customer-shaped tables coexisting, flagged as a future duplicate-data risk, not resolved.
  • AgentDecision.context_embedding dimension (1536) vs the reported MiniLM embedding size (384) — flagged this session (§2), not investigated further; worth resolving before relying on vector search quality from that column.

11. Suggested next steps, in order

  1. Resolve the customer-JWT E2E test blocker (real phone or load-test workaround) and finish the load test.
  2. go build ./... in DoormileBackend — done, clean pass, commit c272a33 pushed to origin/main.
  3. Stand up a test/staging DB, let AutoMigrate run, smoke-test the 14 new migration-session endpoints with real requests.
  4. Pick one low-risk console slice (e.g. Reports or Partner management — net-new UI, not replacing something live) and wire it to Doormile instead of jupiter — first real proof the cutover works end to end.
  5. Only after that: rewrite the higher-traffic screens (orders/deliveries list, rider status updates) and the rider app's request/response handling.
  6. Resolve the B2C tenant-attribution question before it's load-bearing for real revenue reporting.
  7. Confirm the AgentDecision embedding-dimension question.
  8. Decide on substitutions and batch-optimization sophistication once real usage data says whether they're actually needed.
  9. Go-live preparation across the 4 target cities.
Reference in New Issue View Git Blame Copy Permalink