986 lines
31 KiB
Plaintext
986 lines
31 KiB
Plaintext
================================================================================
|
|
NATS MESSAGE QUEUE SYSTEM
|
|
Intermediate-Level Guide
|
|
================================================================================
|
|
|
|
TABLE OF CONTENTS
|
|
-----------------
|
|
1. What This System Does (Simple Explanation)
|
|
2. The Big Picture - How Everything Works Together
|
|
3. Each Component Explained Simply
|
|
4. How Messages Flow Through the System
|
|
5. Why We Use Each Technology
|
|
6. How to Deploy and Use
|
|
7. Common Questions Answered
|
|
8. Troubleshooting Made Easy
|
|
|
|
================================================================================
|
|
1. WHAT THIS SYSTEM DOES (SIMPLE EXPLANATION)
|
|
================================================================================
|
|
|
|
Imagine you have a restaurant:
|
|
- Customers place orders (HTTP requests)
|
|
- Orders go to the kitchen queue (NATS message queue)
|
|
- Chefs process orders (Workers)
|
|
- Food gets delivered (Forwarded to external API)
|
|
|
|
OUR SYSTEM:
|
|
-----------
|
|
1. Your mobile app or website sends a request (like "update delivery status")
|
|
2. FastAPI receives it and puts it in a message queue (NATS)
|
|
3. Workers pick up messages from the queue
|
|
4. Workers forward the request to the actual external API
|
|
5. Everything is monitored and can scale automatically
|
|
|
|
WHY USE A QUEUE?
|
|
---------------
|
|
✓ If the external API is slow, your app doesn't wait
|
|
✓ If the external API is down, messages wait in queue (won't be lost)
|
|
✓ You can process many requests without overloading anything
|
|
✓ Easy to add more workers if you have lots of messages
|
|
|
|
================================================================================
|
|
2. THE BIG PICTURE - HOW EVERYTHING WORKS TOGETHER
|
|
================================================================================
|
|
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ SIMPLE FLOW DIAGRAM │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
|
|
[Your App]
|
|
│
|
|
│ Sends HTTP request: "Update delivery #123"
|
|
▼
|
|
[FastAPI Server]
|
|
│
|
|
│ Puts message in queue
|
|
▼
|
|
[NATS Message Queue]
|
|
│
|
|
│ Stores message safely
|
|
│ (like a mailbox)
|
|
▼
|
|
[Worker Process]
|
|
│
|
|
│ Picks up message
|
|
│ Forwards to external API
|
|
▼
|
|
[External API]
|
|
│
|
|
│ Processes the request
|
|
▼
|
|
[Done!]
|
|
|
|
COMPONENTS BREAKDOWN:
|
|
---------------------
|
|
|
|
1. NATS = The Message Queue (like a post office)
|
|
- Receives messages from FastAPI
|
|
- Stores them safely
|
|
- Gives them to workers when ready
|
|
|
|
2. FastAPI = The API Server (like a receptionist)
|
|
- Receives requests from your app
|
|
- Quickly puts them in the queue
|
|
- Returns "OK, got it!" immediately
|
|
|
|
3. Workers = The Processors (like workers in a factory)
|
|
- Take messages from queue
|
|
- Forward to external API
|
|
- Handle retries if something fails
|
|
|
|
4. Nginx = The Helper (like a translator)
|
|
- Helps the dashboard talk to NATS
|
|
- Adds security headers (CORS)
|
|
|
|
5. Dashboard = The Monitor (like a control panel)
|
|
- Shows you what's happening
|
|
- See how many messages are waiting
|
|
- Monitor if everything is working
|
|
|
|
================================================================================
|
|
3. EACH COMPONENT EXPLAINED SIMPLY
|
|
================================================================================
|
|
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ 3.1 NATS - THE MESSAGE QUEUE │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
|
|
WHAT IT IS:
|
|
-----------
|
|
Think of NATS like a smart mailbox system:
|
|
- FastAPI puts messages in
|
|
- Workers take messages out
|
|
- Messages are stored safely (won't be lost)
|
|
- Can handle millions of messages
|
|
|
|
WHY WE USE IT:
|
|
-------------
|
|
✓ Super fast (messages delivered in less than 1 millisecond)
|
|
✓ Reliable (messages won't disappear)
|
|
✓ Simple to use (no complicated setup)
|
|
✓ Lightweight (doesn't need much resources)
|
|
|
|
HOW IT WORKS:
|
|
-------------
|
|
1. FastAPI publishes a message: "Here's a delivery update"
|
|
2. NATS stores it in a "stream" (like a folder)
|
|
3. Worker asks: "Any messages for me?"
|
|
4. NATS gives worker the message
|
|
5. Worker processes it and says "Done!" (ACK)
|
|
6. NATS removes the message from queue
|
|
|
|
REAL EXAMPLE:
|
|
-------------
|
|
Message looks like this:
|
|
{
|
|
"endpoint": "/live/api/v1/deliveries/updatedelivery",
|
|
"method": "PUT",
|
|
"data": {"delivery_id": 123, "status": "delivered"},
|
|
"received_at": 1234567890
|
|
}
|
|
|
|
This gets stored in NATS and workers pick it up.
|
|
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ 3.2 FASTAPI - THE API SERVER │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
|
|
WHAT IT IS:
|
|
-----------
|
|
FastAPI is a Python web server that:
|
|
- Listens for HTTP requests
|
|
- Validates the data
|
|
- Puts messages in NATS queue
|
|
- Returns response immediately
|
|
|
|
WHY WE USE IT:
|
|
-------------
|
|
✓ Fast (handles many requests per second)
|
|
✓ Easy to write (Python is simple)
|
|
✓ Automatic documentation (shows all endpoints)
|
|
✓ Built-in validation (catches bad data)
|
|
|
|
HOW IT WORKS:
|
|
-------------
|
|
1. Your app sends: PUT /live/api/v1/deliveries/updatedelivery
|
|
2. FastAPI receives it
|
|
3. Checks if data is valid
|
|
4. Creates a message with all the info
|
|
5. Publishes to NATS (super fast, < 1ms)
|
|
6. Returns: {"status": "accepted", "message_id": 12345}
|
|
|
|
ENDPOINTS WE HAVE:
|
|
------------------
|
|
- PUT /live/api/v1/deliveries/updatedelivery
|
|
- POST /live/api/v1/deliveries/createdeliveries
|
|
- POST /live/api/v2/partners/createriderlog
|
|
- POST /live/api/v2/deliveries/createdeliverylog
|
|
- POST /live/api/v2/partners/createbreaklog
|
|
- POST /live/api/v2/partners/updatebreaklog
|
|
|
|
Each endpoint does the same thing: receives request → puts in queue → returns OK
|
|
|
|
HEALTH CHECKS:
|
|
--------------
|
|
- GET /health → Is the server running? (Yes/No)
|
|
- GET /ready → Is NATS connected? (Yes/No)
|
|
- GET /metrics → Statistics (for monitoring)
|
|
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ 3.3 WORKERS - THE MESSAGE PROCESSORS │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
|
|
WHAT IT IS:
|
|
-----------
|
|
Workers are background processes that:
|
|
- Watch the NATS queue for new messages
|
|
- Take messages out
|
|
- Forward them to the external API
|
|
- Handle errors and retries
|
|
|
|
WHY WE USE THEM:
|
|
---------------
|
|
✓ Keeps your API fast (doesn't wait for external API)
|
|
✓ Handles failures automatically (retries if API is down)
|
|
✓ Can scale independently (add more workers if needed)
|
|
✓ Separates concerns (API vs processing)
|
|
|
|
HOW IT WORKS:
|
|
-------------
|
|
1. Worker asks NATS: "Any messages?"
|
|
2. NATS gives worker a batch (up to 10 messages)
|
|
3. For each message:
|
|
a. Reads the endpoint and data
|
|
b. Maps to external API URL
|
|
c. Sends HTTP request (PUT or POST)
|
|
d. Waits for response
|
|
4. If success: Tells NATS "Done!" (ACK) → message removed
|
|
5. If failure: Tells NATS "Try again" (NAK) → message stays in queue
|
|
|
|
RETRY LOGIC:
|
|
------------
|
|
If external API fails:
|
|
- Wait 5 seconds, try again
|
|
- If fails, wait 10 seconds, try again
|
|
- If fails, wait 15 seconds, try again
|
|
- Up to 5 attempts total
|
|
- After 5 failures, message stays in queue for manual review
|
|
|
|
EXAMPLE:
|
|
--------
|
|
Message says: "Update delivery #123"
|
|
Worker:
|
|
1. Reads message
|
|
2. Maps to: https://jupiter.nearle.app/live/api/v1/deliveries/updatedelivery
|
|
3. Sends PUT request with the data
|
|
4. External API responds: "OK"
|
|
5. Worker tells NATS: "Done!" → message removed
|
|
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ 3.4 NGINX - THE HELPER PROXY │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
|
|
WHAT IT IS:
|
|
-----------
|
|
Nginx is a web server that acts as a middleman:
|
|
- Sits between Dashboard and NATS monitoring
|
|
- Adds security headers (CORS)
|
|
- Handles browser security requirements
|
|
|
|
WHY WE NEED IT:
|
|
---------------
|
|
The NATS Dashboard runs in your browser (JavaScript).
|
|
Browsers have security rules (CORS) that prevent websites from
|
|
talking to other websites unless they have permission.
|
|
|
|
Nginx adds the permission headers so the dashboard can work.
|
|
|
|
HOW IT WORKS:
|
|
-------------
|
|
1. Dashboard (in browser) wants to check NATS status
|
|
2. Browser blocks it (security rule)
|
|
3. Nginx intercepts the request
|
|
4. Adds headers: "Yes, dashboard is allowed to talk to NATS"
|
|
5. Browser says "OK" and allows the request
|
|
6. Dashboard gets the data it needs
|
|
|
|
SIMPLE ANALOGY:
|
|
---------------
|
|
Like a bouncer at a club:
|
|
- Dashboard wants to enter (access NATS)
|
|
- Bouncer (Nginx) checks the list
|
|
- Adds your name to the VIP list (CORS headers)
|
|
- You get in!
|
|
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ 3.5 NATS DASHBOARD - THE MONITORING SCREEN │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
|
|
WHAT IT IS:
|
|
-----------
|
|
A web page that shows you:
|
|
- How many messages are in the queue
|
|
- How fast messages are being processed
|
|
- If workers are running
|
|
- Statistics and graphs
|
|
|
|
WHY WE USE IT:
|
|
-------------
|
|
✓ See what's happening in real-time
|
|
✓ Know if something is wrong
|
|
✓ Monitor performance
|
|
✓ Debug issues
|
|
|
|
WHAT YOU CAN SEE:
|
|
-----------------
|
|
- Stream Statistics: How many messages total
|
|
- Consumer Status: Are workers processing?
|
|
- Message Rate: Messages per second
|
|
- Pending Messages: How many waiting to be processed
|
|
- Consumer Lag: How far behind are workers?
|
|
|
|
HOW TO ACCESS:
|
|
--------------
|
|
URL: https://natsadmin.workolik.com
|
|
Username: admin
|
|
Password: package@321#
|
|
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ 3.6 DOCKER - CONTAINERIZATION │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
|
|
WHAT IT IS:
|
|
-----------
|
|
Docker packages everything into containers:
|
|
- Like shipping containers for software
|
|
- Each service runs in its own container
|
|
- All containers work together
|
|
|
|
WHY WE USE IT:
|
|
-------------
|
|
✓ Same environment everywhere (dev, staging, production)
|
|
✓ Easy to deploy (just run docker-compose up)
|
|
✓ Isolated services (if one crashes, others keep running)
|
|
✓ Easy to update (just rebuild container)
|
|
|
|
HOW IT WORKS:
|
|
-------------
|
|
docker-compose.yml defines:
|
|
- NATS container (the message queue)
|
|
- Nginx container (the proxy)
|
|
- Dashboard container (the monitoring)
|
|
|
|
All containers talk to each other via a network.
|
|
|
|
SIMPLE ANALOGY:
|
|
---------------
|
|
Like apartments in a building:
|
|
- Each apartment (container) is separate
|
|
- But they share utilities (network)
|
|
- You can move apartments (containers) easily
|
|
- If one apartment has issues, others are fine
|
|
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ 3.7 KUBERNETES - CONTAINER ORCHESTRATION │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
|
|
WHAT IT IS:
|
|
-----------
|
|
Kubernetes manages containers across multiple servers:
|
|
- Like a manager for a team of workers
|
|
- Automatically starts/stops containers
|
|
- Spreads work across multiple servers
|
|
- Auto-scales when busy
|
|
|
|
WHY WE USE IT:
|
|
-------------
|
|
✓ High availability (if one server dies, others take over)
|
|
✓ Auto-scaling (adds more workers when busy)
|
|
✓ Self-healing (restarts crashed containers)
|
|
✓ Load balancing (spreads requests evenly)
|
|
|
|
HOW IT WORKS:
|
|
-------------
|
|
1. You define: "I want 4 FastAPI servers and 2 workers"
|
|
2. Kubernetes creates them across different servers
|
|
3. If one crashes, Kubernetes restarts it
|
|
4. If load increases, Kubernetes adds more
|
|
5. If load decreases, Kubernetes removes some
|
|
|
|
AUTO-SCALING EXAMPLE:
|
|
---------------------
|
|
Normal load: 4 FastAPI pods, 2 worker pods
|
|
High load: Kubernetes sees CPU at 80%
|
|
→ Adds more pods automatically
|
|
→ Now: 10 FastAPI pods, 5 worker pods
|
|
Low load: Kubernetes sees CPU at 20%
|
|
→ Removes some pods
|
|
→ Back to: 4 FastAPI pods, 2 worker pods
|
|
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ 3.8 TRAEFIK - THE ROUTER │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
|
|
WHAT IT IS:
|
|
-----------
|
|
Traefik is like a smart receptionist:
|
|
- Receives all incoming requests
|
|
- Routes them to the right service
|
|
- Handles HTTPS certificates automatically
|
|
- Adds security (like password protection)
|
|
|
|
WHY WE USE IT:
|
|
-------------
|
|
✓ Automatic HTTPS (free SSL certificates)
|
|
✓ Easy routing (just add labels to containers)
|
|
✓ Handles authentication
|
|
✓ One entry point for everything
|
|
|
|
HOW IT WORKS:
|
|
-------------
|
|
1. Request comes in: https://queue.workolik.com
|
|
2. Traefik receives it
|
|
3. Checks: "This goes to FastAPI"
|
|
4. Routes to FastAPI service
|
|
5. Returns response
|
|
|
|
HTTPS CERTIFICATES:
|
|
-------------------
|
|
Traefik automatically gets free SSL certificates from Let's Encrypt.
|
|
You don't need to do anything - it just works!
|
|
|
|
AUTHENTICATION:
|
|
---------------
|
|
For the dashboard (natsadmin.workolik.com):
|
|
- Traefik adds password protection
|
|
- Username: admin
|
|
- Password: package@321#
|
|
|
|
================================================================================
|
|
4. HOW MESSAGES FLOW THROUGH THE SYSTEM
|
|
================================================================================
|
|
|
|
STEP-BY-STEP EXAMPLE:
|
|
---------------------
|
|
|
|
Let's say your app wants to update a delivery status.
|
|
|
|
STEP 1: YOUR APP SENDS REQUEST
|
|
--------------------------------
|
|
Your mobile app sends:
|
|
PUT https://queue.workolik.com/live/api/v1/deliveries/updatedelivery
|
|
Body: {"delivery_id": 123, "status": "delivered"}
|
|
|
|
STEP 2: TRAEFIK RECEIVES IT
|
|
----------------------------
|
|
Traefik sees: "This is for queue.workolik.com"
|
|
Routes to: FastAPI service
|
|
|
|
STEP 3: FASTAPI PROCESSES IT
|
|
-----------------------------
|
|
FastAPI:
|
|
- Receives the request
|
|
- Validates the data (checks it's valid JSON)
|
|
- Creates a message:
|
|
{
|
|
"endpoint": "/live/api/v1/deliveries/updatedelivery",
|
|
"method": "PUT",
|
|
"data": {"delivery_id": 123, "status": "delivered"},
|
|
"received_at": 1234567890
|
|
}
|
|
- Publishes to NATS (super fast!)
|
|
- Returns: {"status": "accepted", "message_id": 12345}
|
|
|
|
STEP 4: YOUR APP GETS RESPONSE
|
|
-------------------------------
|
|
Your app receives: "OK, got it!" (takes < 1 second)
|
|
Your app doesn't wait for the external API - it's done!
|
|
|
|
STEP 5: NATS STORES MESSAGE
|
|
---------------------------
|
|
NATS:
|
|
- Receives message
|
|
- Stores in stream "EVENTS"
|
|
- Subject: "api.v1.deliveries.updatedelivery"
|
|
- Message is safe and won't be lost
|
|
|
|
STEP 6: WORKER PICKS IT UP
|
|
---------------------------
|
|
Worker:
|
|
- Asks NATS: "Any messages?"
|
|
- NATS gives worker the message
|
|
- Worker reads: "Update delivery #123"
|
|
|
|
STEP 7: WORKER FORWARDS TO EXTERNAL API
|
|
----------------------------------------
|
|
Worker:
|
|
- Maps endpoint to: https://jupiter.nearle.app/live/api/v1/deliveries/updatedelivery
|
|
- Sends PUT request with the data
|
|
- Waits for response
|
|
|
|
STEP 8: EXTERNAL API RESPONDS
|
|
------------------------------
|
|
External API:
|
|
- Processes the request
|
|
- Updates delivery status
|
|
- Returns: 200 OK
|
|
|
|
STEP 9: WORKER ACKNOWLEDGES
|
|
----------------------------
|
|
Worker:
|
|
- Sees success (200 OK)
|
|
- Tells NATS: "Done!" (ACK)
|
|
- NATS removes message from queue
|
|
- Done!
|
|
|
|
IF SOMETHING GOES WRONG:
|
|
------------------------
|
|
If external API is down:
|
|
- Worker gets error
|
|
- Tells NATS: "Try again later" (NAK)
|
|
- NATS keeps message in queue
|
|
- Worker waits 5 seconds
|
|
- Tries again
|
|
- Repeats up to 5 times
|
|
- If still fails, message stays for manual review
|
|
|
|
================================================================================
|
|
5. WHY WE USE EACH TECHNOLOGY
|
|
================================================================================
|
|
|
|
NATS (Message Queue):
|
|
---------------------
|
|
✓ Super fast (messages in < 1ms)
|
|
✓ Reliable (messages won't disappear)
|
|
✓ Simple (easy to set up and use)
|
|
✓ Lightweight (doesn't need much resources)
|
|
✓ Perfect for this use case
|
|
|
|
Why not RabbitMQ? Too heavy, more complex
|
|
Why not Kafka? Overkill, too complex for this
|
|
Why not Redis? Less mature, fewer features
|
|
|
|
FASTAPI (API Server):
|
|
---------------------
|
|
✓ Fast (handles many requests)
|
|
✓ Easy (Python is simple to write)
|
|
✓ Modern (async/await support)
|
|
✓ Automatic docs (shows all endpoints)
|
|
✓ Type validation (catches errors early)
|
|
|
|
Why not Flask? Slower, no async support
|
|
Why not Django? Too heavy, overkill
|
|
Why not Node.js? Team knows Python better
|
|
|
|
WORKERS (Message Processors):
|
|
------------------------------
|
|
✓ Decouples API from external service
|
|
✓ Handles retries automatically
|
|
✓ Can scale independently
|
|
✓ Easy to monitor separately
|
|
|
|
Why separate workers? Keeps API fast, handles failures better
|
|
|
|
NGINX (Proxy):
|
|
--------------
|
|
✓ Needed for CORS (browser security)
|
|
✓ Lightweight (small container)
|
|
✓ Fast (high performance)
|
|
✓ Simple config
|
|
|
|
Why needed? Browsers block cross-origin requests without CORS headers
|
|
|
|
DOCKER (Containers):
|
|
--------------------
|
|
✓ Same environment everywhere
|
|
✓ Easy deployment
|
|
✓ Isolated services
|
|
✓ Easy updates
|
|
|
|
Why containers? Consistency, easy deployment, isolation
|
|
|
|
KUBERNETES (Orchestration):
|
|
----------------------------
|
|
✓ High availability (multiple servers)
|
|
✓ Auto-scaling (adds workers when busy)
|
|
✓ Self-healing (restarts crashed services)
|
|
✓ Load balancing
|
|
|
|
Why Kubernetes? Production needs reliability and scaling
|
|
|
|
TRAEFIK (Router):
|
|
-----------------
|
|
✓ Automatic HTTPS (free certificates)
|
|
✓ Easy routing (just labels)
|
|
✓ Handles authentication
|
|
✓ One entry point
|
|
|
|
Why Traefik? Simplifies HTTPS and routing
|
|
|
|
================================================================================
|
|
6. HOW TO DEPLOY AND USE
|
|
================================================================================
|
|
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ 6.1 DOCKER COMPOSE (SIMPLE DEPLOYMENT) │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
|
|
FOR: Local development, small deployments, single server
|
|
|
|
STEPS:
|
|
------
|
|
1. Make sure Docker is installed
|
|
2. Go to the kubernetes folder
|
|
3. Run: docker-compose up -d
|
|
4. That's it! Everything starts automatically
|
|
|
|
WHAT GETS STARTED:
|
|
-----------------
|
|
- NATS server (message queue)
|
|
- Nginx proxy (CORS helper)
|
|
- NATS Dashboard (monitoring)
|
|
|
|
ACCESS:
|
|
-------
|
|
- Dashboard: https://natsadmin.workolik.com
|
|
- NATS Monitoring: https://nats.workolik.com
|
|
|
|
CHECK STATUS:
|
|
-------------
|
|
docker-compose ps # See what's running
|
|
docker-compose logs -f # See logs
|
|
docker-compose restart # Restart everything
|
|
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ 6.2 KUBERNETES (PRODUCTION DEPLOYMENT) │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
|
|
FOR: Production, high availability, auto-scaling
|
|
|
|
STEPS:
|
|
------
|
|
1. Build Docker images
|
|
2. Push to registry
|
|
3. Update secrets.yaml with your credentials
|
|
4. Run: ./deploy.sh
|
|
5. Wait for pods to start
|
|
|
|
WHAT GETS DEPLOYED:
|
|
-------------------
|
|
- FastAPI pods (4-20, auto-scales)
|
|
- Worker pods (2-10, auto-scales)
|
|
- Services (load balancers)
|
|
- Auto-scaling rules
|
|
|
|
CHECK STATUS:
|
|
-------------
|
|
kubectl get pods -n nats-backend # See pods
|
|
kubectl logs -f deployment/fastapi-backend # See logs
|
|
kubectl get hpa -n nats-backend # See auto-scaling
|
|
|
|
TEST ENDPOINTS:
|
|
---------------
|
|
curl -X PUT https://queue.workolik.com/live/api/v1/deliveries/updatedelivery \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"delivery_id": 123, "status": "delivered"}'
|
|
|
|
Expected response:
|
|
{"status": "accepted", "message_id": 12345, "endpoint": "..."}
|
|
|
|
================================================================================
|
|
7. COMMON QUESTIONS ANSWERED
|
|
================================================================================
|
|
|
|
Q: Why use a message queue instead of calling the API directly?
|
|
A:
|
|
- Your app doesn't have to wait for slow external API
|
|
- If external API is down, messages wait (won't be lost)
|
|
- Can handle traffic spikes better
|
|
- Easy to add more workers if needed
|
|
|
|
Q: What happens if NATS goes down?
|
|
A:
|
|
- Messages are stored on disk (persistent)
|
|
- When NATS restarts, messages are still there
|
|
- Workers reconnect automatically
|
|
- No data loss
|
|
|
|
Q: What happens if a worker crashes?
|
|
A:
|
|
- Kubernetes restarts it automatically
|
|
- Messages stay in queue
|
|
- Other workers keep processing
|
|
- No data loss
|
|
|
|
Q: How fast is it?
|
|
A:
|
|
- FastAPI responds in < 1 second (usually < 100ms)
|
|
- NATS stores message in < 1 millisecond
|
|
- Worker processes in seconds (depends on external API)
|
|
- Total: Usually under 2 seconds end-to-end
|
|
|
|
Q: Can I add more endpoints?
|
|
A:
|
|
- Yes! Add endpoint in FastAPI (app.py)
|
|
- Add mapping in worker (worker.py)
|
|
- Deploy and it works
|
|
|
|
Q: How do I monitor if everything is working?
|
|
A:
|
|
- Check NATS Dashboard: https://natsadmin.workolik.com
|
|
- Check FastAPI metrics: GET /metrics
|
|
- Check worker metrics: Port 9090
|
|
- Check Kubernetes: kubectl get pods
|
|
|
|
Q: What if external API is slow?
|
|
A:
|
|
- Messages wait in queue
|
|
- Workers retry automatically
|
|
- Your API still responds fast (doesn't wait)
|
|
- No impact on your users
|
|
|
|
Q: How many messages can it handle?
|
|
A:
|
|
- NATS: Millions per second
|
|
- FastAPI: Thousands per second (depends on hardware)
|
|
- Workers: Depends on external API speed
|
|
- Can scale workers automatically if needed
|
|
|
|
Q: Is it secure?
|
|
A:
|
|
- HTTPS for all external access (Traefik)
|
|
- Password protection for dashboard
|
|
- NATS authentication required
|
|
- CORS properly configured
|
|
|
|
Q: Can I test locally?
|
|
A:
|
|
- Yes! Use Docker Compose
|
|
- Everything runs on your machine
|
|
- Same as production (just smaller scale)
|
|
|
|
================================================================================
|
|
8. TROUBLESHOOTING MADE EASY
|
|
================================================================================
|
|
|
|
PROBLEM: FastAPI returns 503 "NATS not connected"
|
|
-----------------------------------------------
|
|
WHAT IT MEANS: FastAPI can't talk to NATS
|
|
|
|
HOW TO FIX:
|
|
1. Check if NATS is running: docker ps | grep nats
|
|
2. Check NATS logs: docker logs nats
|
|
3. Check network: Can FastAPI reach NATS?
|
|
4. Check credentials: Are username/password correct?
|
|
|
|
PROBLEM: Messages not being processed
|
|
-------------------------------------
|
|
WHAT IT MEANS: Workers aren't picking up messages
|
|
|
|
HOW TO FIX:
|
|
1. Check if workers are running: kubectl get pods | grep worker
|
|
2. Check worker logs: kubectl logs -f deployment/nats-worker
|
|
3. Check NATS Dashboard: Are messages in queue?
|
|
4. Check consumer: Is worker_consumer active?
|
|
|
|
PROBLEM: CORS errors in browser
|
|
--------------------------------
|
|
WHAT IT MEANS: Dashboard can't access NATS monitoring
|
|
|
|
HOW TO FIX:
|
|
1. Check Nginx proxy is running: docker ps | grep nats-proxy
|
|
2. Check nginx-nats.conf: Are CORS headers correct?
|
|
3. Check NATS_MONITORING_URL in dashboard config
|
|
4. Restart nginx-proxy: docker-compose restart nats-proxy
|
|
|
|
PROBLEM: External API not receiving requests
|
|
--------------------------------------------
|
|
WHAT IT MEANS: Workers aren't forwarding messages
|
|
|
|
HOW TO FIX:
|
|
1. Check worker logs: Look for forwarding errors
|
|
2. Check EXTERNAL_BASE_URL: Is it correct?
|
|
3. Test connectivity: Can workers reach external API?
|
|
4. Check endpoint mapping: Is endpoint in worker.py?
|
|
|
|
PROBLEM: Auto-scaling not working
|
|
---------------------------------
|
|
WHAT IT MEANS: Pods aren't scaling up/down
|
|
|
|
HOW TO FIX:
|
|
1. Check HPA: kubectl get hpa -n nats-backend
|
|
2. Check metrics-server: kubectl top pods
|
|
3. Check resource limits: Are they set correctly?
|
|
4. Check if max replicas reached
|
|
|
|
QUICK HEALTH CHECK:
|
|
-------------------
|
|
1. Are all pods running? → kubectl get pods
|
|
2. Are messages processing? → Check dashboard
|
|
3. Are there errors? → Check logs
|
|
4. Is external API reachable? → Test from worker pod
|
|
|
|
COMMON COMMANDS:
|
|
----------------
|
|
# Docker Compose
|
|
docker-compose ps # Status
|
|
docker-compose logs -f # Logs
|
|
docker-compose restart # Restart
|
|
|
|
# Kubernetes
|
|
kubectl get pods # Pod status
|
|
kubectl logs -f [pod-name] # Pod logs
|
|
kubectl describe pod [pod] # Pod details
|
|
kubectl get hpa # Auto-scaling status
|
|
|
|
# NATS
|
|
# Check dashboard: https://natsadmin.workolik.com
|
|
|
|
================================================================================
|
|
9. KEY CONCEPTS EXPLAINED SIMPLY
|
|
================================================================================
|
|
|
|
MESSAGE QUEUE:
|
|
--------------
|
|
Like a post office:
|
|
- You drop off a letter (message)
|
|
- Post office stores it safely
|
|
- Mail carrier picks it up
|
|
- Delivers to destination
|
|
|
|
In our system:
|
|
- FastAPI drops off message
|
|
- NATS stores it
|
|
- Worker picks it up
|
|
- Delivers to external API
|
|
|
|
PUBLISH/SUBSCRIBE:
|
|
------------------
|
|
Like a radio station:
|
|
- Radio station broadcasts (publishes)
|
|
- Radios listen (subscribe)
|
|
- Many radios can listen to same station
|
|
|
|
In our system:
|
|
- FastAPI publishes messages
|
|
- Workers subscribe to messages
|
|
- Many workers can process same queue
|
|
|
|
ACKNOWLEDGMENT (ACK/NAK):
|
|
-------------------------
|
|
Like a receipt:
|
|
- You send a package
|
|
- Recipient signs for it (ACK)
|
|
- If rejected, you get it back (NAK)
|
|
|
|
In our system:
|
|
- Worker processes message
|
|
- If success: ACK → message removed
|
|
- If failure: NAK → message stays for retry
|
|
|
|
STREAM:
|
|
-------
|
|
Like a folder:
|
|
- All related messages go in one stream
|
|
- Easy to organize
|
|
- Can have multiple consumers
|
|
|
|
In our system:
|
|
- Stream: "EVENTS"
|
|
- All API messages go here
|
|
- Workers read from this stream
|
|
|
|
CONSUMER:
|
|
---------
|
|
Like a reader:
|
|
- Reads messages from stream
|
|
- Processes them
|
|
- Acknowledges when done
|
|
|
|
In our system:
|
|
- Consumer: "worker_consumer"
|
|
- Workers use this to get messages
|
|
- Tracks which messages are processed
|
|
|
|
AUTO-SCALING:
|
|
-------------
|
|
Like a restaurant:
|
|
- Few customers → Few waiters
|
|
- Many customers → More waiters
|
|
- Automatically adjusts
|
|
|
|
In our system:
|
|
- Low load → Few pods
|
|
- High load → More pods
|
|
- Kubernetes does it automatically
|
|
|
|
LOAD BALANCING:
|
|
---------------
|
|
Like distributing work:
|
|
- Manager gives tasks to available workers
|
|
- Spreads work evenly
|
|
- No one gets overloaded
|
|
|
|
In our system:
|
|
- Requests go to available FastAPI pod
|
|
- Spreads load evenly
|
|
- Kubernetes handles it
|
|
|
|
================================================================================
|
|
10. REAL-WORLD EXAMPLE
|
|
================================================================================
|
|
|
|
SCENARIO: Your delivery app needs to update delivery status
|
|
|
|
WITHOUT MESSAGE QUEUE:
|
|
----------------------
|
|
1. App sends request to your API
|
|
2. Your API calls external API directly
|
|
3. External API is slow (takes 5 seconds)
|
|
4. Your app waits 5 seconds
|
|
5. User sees loading spinner
|
|
6. If external API is down, your app fails
|
|
|
|
WITH MESSAGE QUEUE (OUR SYSTEM):
|
|
---------------------------------
|
|
1. App sends request to FastAPI
|
|
2. FastAPI puts message in queue (< 100ms)
|
|
3. FastAPI returns "OK" immediately
|
|
4. User sees success right away
|
|
5. Worker processes message in background
|
|
6. If external API is slow/down, message waits
|
|
7. Worker retries automatically
|
|
8. No impact on your users
|
|
|
|
BENEFITS:
|
|
---------
|
|
✓ Users get instant response
|
|
✓ System handles failures gracefully
|
|
✓ Can handle traffic spikes
|
|
✓ Easy to scale
|
|
✓ Messages never lost
|
|
|
|
================================================================================
|
|
11. QUICK REFERENCE
|
|
================================================================================
|
|
|
|
ENDPOINTS:
|
|
----------
|
|
PUT /live/api/v1/deliveries/updatedelivery
|
|
POST /live/api/v1/deliveries/createdeliveries
|
|
POST /live/api/v2/partners/createriderlog
|
|
POST /live/api/v2/deliveries/createdeliverylog
|
|
POST /live/api/v2/partners/createbreaklog
|
|
POST /live/api/v2/partners/updatebreaklog
|
|
|
|
URLS:
|
|
-----
|
|
FastAPI: https://queue.workolik.com
|
|
Dashboard: https://natsadmin.workolik.com
|
|
NATS Monitoring: https://nats.workolik.com
|
|
|
|
CREDENTIALS:
|
|
------------
|
|
NATS:
|
|
- Username: admin
|
|
- Password: package@321#
|
|
|
|
Dashboard:
|
|
- Username: admin
|
|
- Password: package@321#
|
|
|
|
PORTS:
|
|
------
|
|
NATS: 4222 (client), 8222 (monitoring)
|
|
FastAPI: 8000 (internal), 8201 (external)
|
|
Worker: 9090 (metrics)
|
|
Nginx: 80 (internal), 8082 (external)
|
|
|
|
FILES:
|
|
------
|
|
docker-compose.yml # Docker deployment
|
|
scripts/app.py # FastAPI code
|
|
scripts/worker.py # Worker code
|
|
nginx-nats.conf # Nginx config
|
|
manifests/*.yaml # Kubernetes configs
|
|
|
|
================================================================================
|
|
END OF GUIDE
|
|
================================================================================
|
|
|
|
This guide explains the system in simple terms. For more technical details,
|
|
see TECH_STACK_ARCHITECTURE.txt.
|
|
|
|
Remember:
|
|
- FastAPI receives requests and puts them in queue
|
|
- NATS stores messages safely
|
|
- Workers process messages and forward to external API
|
|
- Everything is monitored and can scale automatically
|
|
|
|
Questions? Check Section 7 (Common Questions) or Section 8 (Troubleshooting).
|
|
|
|
Last Updated: 2025-01-XX
|
|
Version: Intermediate Level
|
|
|