Initial commit
This commit is contained in:
985
docs/TECH_STACK_ARCHITECTURE_INTERMEDIATE.txt
Normal file
985
docs/TECH_STACK_ARCHITECTURE_INTERMEDIATE.txt
Normal file
@@ -0,0 +1,985 @@
|
||||
================================================================================
|
||||
NATS MESSAGE QUEUE SYSTEM
|
||||
Intermediate-Level Guide
|
||||
================================================================================
|
||||
|
||||
TABLE OF CONTENTS
|
||||
-----------------
|
||||
1. What This System Does (Simple Explanation)
|
||||
2. The Big Picture - How Everything Works Together
|
||||
3. Each Component Explained Simply
|
||||
4. How Messages Flow Through the System
|
||||
5. Why We Use Each Technology
|
||||
6. How to Deploy and Use
|
||||
7. Common Questions Answered
|
||||
8. Troubleshooting Made Easy
|
||||
|
||||
================================================================================
|
||||
1. WHAT THIS SYSTEM DOES (SIMPLE EXPLANATION)
|
||||
================================================================================
|
||||
|
||||
Imagine you have a restaurant:
|
||||
- Customers place orders (HTTP requests)
|
||||
- Orders go to the kitchen queue (NATS message queue)
|
||||
- Chefs process orders (Workers)
|
||||
- Food gets delivered (Forwarded to external API)
|
||||
|
||||
OUR SYSTEM:
|
||||
-----------
|
||||
1. Your mobile app or website sends a request (like "update delivery status")
|
||||
2. FastAPI receives it and puts it in a message queue (NATS)
|
||||
3. Workers pick up messages from the queue
|
||||
4. Workers forward the request to the actual external API
|
||||
5. Everything is monitored and can scale automatically
|
||||
|
||||
WHY USE A QUEUE?
|
||||
---------------
|
||||
✓ If the external API is slow, your app doesn't wait
|
||||
✓ If the external API is down, messages wait in queue (won't be lost)
|
||||
✓ You can process many requests without overloading anything
|
||||
✓ Easy to add more workers if you have lots of messages
|
||||
|
||||
================================================================================
|
||||
2. THE BIG PICTURE - HOW EVERYTHING WORKS TOGETHER
|
||||
================================================================================
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ SIMPLE FLOW DIAGRAM │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
[Your App]
|
||||
│
|
||||
│ Sends HTTP request: "Update delivery #123"
|
||||
▼
|
||||
[FastAPI Server]
|
||||
│
|
||||
│ Puts message in queue
|
||||
▼
|
||||
[NATS Message Queue]
|
||||
│
|
||||
│ Stores message safely
|
||||
│ (like a mailbox)
|
||||
▼
|
||||
[Worker Process]
|
||||
│
|
||||
│ Picks up message
|
||||
│ Forwards to external API
|
||||
▼
|
||||
[External API]
|
||||
│
|
||||
│ Processes the request
|
||||
▼
|
||||
[Done!]
|
||||
|
||||
COMPONENTS BREAKDOWN:
|
||||
---------------------
|
||||
|
||||
1. NATS = The Message Queue (like a post office)
|
||||
- Receives messages from FastAPI
|
||||
- Stores them safely
|
||||
- Gives them to workers when ready
|
||||
|
||||
2. FastAPI = The API Server (like a receptionist)
|
||||
- Receives requests from your app
|
||||
- Quickly puts them in the queue
|
||||
- Returns "OK, got it!" immediately
|
||||
|
||||
3. Workers = The Processors (like workers in a factory)
|
||||
- Take messages from queue
|
||||
- Forward to external API
|
||||
- Handle retries if something fails
|
||||
|
||||
4. Nginx = The Helper (like a translator)
|
||||
- Helps the dashboard talk to NATS
|
||||
- Adds security headers (CORS)
|
||||
|
||||
5. Dashboard = The Monitor (like a control panel)
|
||||
- Shows you what's happening
|
||||
- See how many messages are waiting
|
||||
- Monitor if everything is working
|
||||
|
||||
================================================================================
|
||||
3. EACH COMPONENT EXPLAINED SIMPLY
|
||||
================================================================================
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ 3.1 NATS - THE MESSAGE QUEUE │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
WHAT IT IS:
|
||||
-----------
|
||||
Think of NATS like a smart mailbox system:
|
||||
- FastAPI puts messages in
|
||||
- Workers take messages out
|
||||
- Messages are stored safely (won't be lost)
|
||||
- Can handle millions of messages
|
||||
|
||||
WHY WE USE IT:
|
||||
-------------
|
||||
✓ Super fast (messages delivered in less than 1 millisecond)
|
||||
✓ Reliable (messages won't disappear)
|
||||
✓ Simple to use (no complicated setup)
|
||||
✓ Lightweight (doesn't need much resources)
|
||||
|
||||
HOW IT WORKS:
|
||||
-------------
|
||||
1. FastAPI publishes a message: "Here's a delivery update"
|
||||
2. NATS stores it in a "stream" (like a folder)
|
||||
3. Worker asks: "Any messages for me?"
|
||||
4. NATS gives worker the message
|
||||
5. Worker processes it and says "Done!" (ACK)
|
||||
6. NATS removes the message from queue
|
||||
|
||||
REAL EXAMPLE:
|
||||
-------------
|
||||
Message looks like this:
|
||||
{
|
||||
"endpoint": "/live/api/v1/deliveries/updatedelivery",
|
||||
"method": "PUT",
|
||||
"data": {"delivery_id": 123, "status": "delivered"},
|
||||
"received_at": 1234567890
|
||||
}
|
||||
|
||||
This gets stored in NATS and workers pick it up.
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ 3.2 FASTAPI - THE API SERVER │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
WHAT IT IS:
|
||||
-----------
|
||||
FastAPI is a Python web server that:
|
||||
- Listens for HTTP requests
|
||||
- Validates the data
|
||||
- Puts messages in NATS queue
|
||||
- Returns response immediately
|
||||
|
||||
WHY WE USE IT:
|
||||
-------------
|
||||
✓ Fast (handles many requests per second)
|
||||
✓ Easy to write (Python is simple)
|
||||
✓ Automatic documentation (shows all endpoints)
|
||||
✓ Built-in validation (catches bad data)
|
||||
|
||||
HOW IT WORKS:
|
||||
-------------
|
||||
1. Your app sends: PUT /live/api/v1/deliveries/updatedelivery
|
||||
2. FastAPI receives it
|
||||
3. Checks if data is valid
|
||||
4. Creates a message with all the info
|
||||
5. Publishes to NATS (super fast, < 1ms)
|
||||
6. Returns: {"status": "accepted", "message_id": 12345}
|
||||
|
||||
ENDPOINTS WE HAVE:
|
||||
------------------
|
||||
- PUT /live/api/v1/deliveries/updatedelivery
|
||||
- POST /live/api/v1/deliveries/createdeliveries
|
||||
- POST /live/api/v2/partners/createriderlog
|
||||
- POST /live/api/v2/deliveries/createdeliverylog
|
||||
- POST /live/api/v2/partners/createbreaklog
|
||||
- POST /live/api/v2/partners/updatebreaklog
|
||||
|
||||
Each endpoint does the same thing: receives request → puts in queue → returns OK
|
||||
|
||||
HEALTH CHECKS:
|
||||
--------------
|
||||
- GET /health → Is the server running? (Yes/No)
|
||||
- GET /ready → Is NATS connected? (Yes/No)
|
||||
- GET /metrics → Statistics (for monitoring)
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ 3.3 WORKERS - THE MESSAGE PROCESSORS │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
WHAT IT IS:
|
||||
-----------
|
||||
Workers are background processes that:
|
||||
- Watch the NATS queue for new messages
|
||||
- Take messages out
|
||||
- Forward them to the external API
|
||||
- Handle errors and retries
|
||||
|
||||
WHY WE USE THEM:
|
||||
---------------
|
||||
✓ Keeps your API fast (doesn't wait for external API)
|
||||
✓ Handles failures automatically (retries if API is down)
|
||||
✓ Can scale independently (add more workers if needed)
|
||||
✓ Separates concerns (API vs processing)
|
||||
|
||||
HOW IT WORKS:
|
||||
-------------
|
||||
1. Worker asks NATS: "Any messages?"
|
||||
2. NATS gives worker a batch (up to 10 messages)
|
||||
3. For each message:
|
||||
a. Reads the endpoint and data
|
||||
b. Maps to external API URL
|
||||
c. Sends HTTP request (PUT or POST)
|
||||
d. Waits for response
|
||||
4. If success: Tells NATS "Done!" (ACK) → message removed
|
||||
5. If failure: Tells NATS "Try again" (NAK) → message stays in queue
|
||||
|
||||
RETRY LOGIC:
|
||||
------------
|
||||
If external API fails:
|
||||
- Wait 5 seconds, try again
|
||||
- If fails, wait 10 seconds, try again
|
||||
- If fails, wait 15 seconds, try again
|
||||
- Up to 5 attempts total
|
||||
- After 5 failures, message stays in queue for manual review
|
||||
|
||||
EXAMPLE:
|
||||
--------
|
||||
Message says: "Update delivery #123"
|
||||
Worker:
|
||||
1. Reads message
|
||||
2. Maps to: https://jupiter.nearle.app/live/api/v1/deliveries/updatedelivery
|
||||
3. Sends PUT request with the data
|
||||
4. External API responds: "OK"
|
||||
5. Worker tells NATS: "Done!" → message removed
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ 3.4 NGINX - THE HELPER PROXY │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
WHAT IT IS:
|
||||
-----------
|
||||
Nginx is a web server that acts as a middleman:
|
||||
- Sits between Dashboard and NATS monitoring
|
||||
- Adds security headers (CORS)
|
||||
- Handles browser security requirements
|
||||
|
||||
WHY WE NEED IT:
|
||||
---------------
|
||||
The NATS Dashboard runs in your browser (JavaScript).
|
||||
Browsers have security rules (CORS) that prevent websites from
|
||||
talking to other websites unless they have permission.
|
||||
|
||||
Nginx adds the permission headers so the dashboard can work.
|
||||
|
||||
HOW IT WORKS:
|
||||
-------------
|
||||
1. Dashboard (in browser) wants to check NATS status
|
||||
2. Browser blocks it (security rule)
|
||||
3. Nginx intercepts the request
|
||||
4. Adds headers: "Yes, dashboard is allowed to talk to NATS"
|
||||
5. Browser says "OK" and allows the request
|
||||
6. Dashboard gets the data it needs
|
||||
|
||||
SIMPLE ANALOGY:
|
||||
---------------
|
||||
Like a bouncer at a club:
|
||||
- Dashboard wants to enter (access NATS)
|
||||
- Bouncer (Nginx) checks the list
|
||||
- Adds your name to the VIP list (CORS headers)
|
||||
- You get in!
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ 3.5 NATS DASHBOARD - THE MONITORING SCREEN │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
WHAT IT IS:
|
||||
-----------
|
||||
A web page that shows you:
|
||||
- How many messages are in the queue
|
||||
- How fast messages are being processed
|
||||
- If workers are running
|
||||
- Statistics and graphs
|
||||
|
||||
WHY WE USE IT:
|
||||
-------------
|
||||
✓ See what's happening in real-time
|
||||
✓ Know if something is wrong
|
||||
✓ Monitor performance
|
||||
✓ Debug issues
|
||||
|
||||
WHAT YOU CAN SEE:
|
||||
-----------------
|
||||
- Stream Statistics: How many messages total
|
||||
- Consumer Status: Are workers processing?
|
||||
- Message Rate: Messages per second
|
||||
- Pending Messages: How many waiting to be processed
|
||||
- Consumer Lag: How far behind are workers?
|
||||
|
||||
HOW TO ACCESS:
|
||||
--------------
|
||||
URL: https://natsadmin.workolik.com
|
||||
Username: admin
|
||||
Password: package@321#
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ 3.6 DOCKER - CONTAINERIZATION │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
WHAT IT IS:
|
||||
-----------
|
||||
Docker packages everything into containers:
|
||||
- Like shipping containers for software
|
||||
- Each service runs in its own container
|
||||
- All containers work together
|
||||
|
||||
WHY WE USE IT:
|
||||
-------------
|
||||
✓ Same environment everywhere (dev, staging, production)
|
||||
✓ Easy to deploy (just run docker-compose up)
|
||||
✓ Isolated services (if one crashes, others keep running)
|
||||
✓ Easy to update (just rebuild container)
|
||||
|
||||
HOW IT WORKS:
|
||||
-------------
|
||||
docker-compose.yml defines:
|
||||
- NATS container (the message queue)
|
||||
- Nginx container (the proxy)
|
||||
- Dashboard container (the monitoring)
|
||||
|
||||
All containers talk to each other via a network.
|
||||
|
||||
SIMPLE ANALOGY:
|
||||
---------------
|
||||
Like apartments in a building:
|
||||
- Each apartment (container) is separate
|
||||
- But they share utilities (network)
|
||||
- You can move apartments (containers) easily
|
||||
- If one apartment has issues, others are fine
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ 3.7 KUBERNETES - CONTAINER ORCHESTRATION │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
WHAT IT IS:
|
||||
-----------
|
||||
Kubernetes manages containers across multiple servers:
|
||||
- Like a manager for a team of workers
|
||||
- Automatically starts/stops containers
|
||||
- Spreads work across multiple servers
|
||||
- Auto-scales when busy
|
||||
|
||||
WHY WE USE IT:
|
||||
-------------
|
||||
✓ High availability (if one server dies, others take over)
|
||||
✓ Auto-scaling (adds more workers when busy)
|
||||
✓ Self-healing (restarts crashed containers)
|
||||
✓ Load balancing (spreads requests evenly)
|
||||
|
||||
HOW IT WORKS:
|
||||
-------------
|
||||
1. You define: "I want 4 FastAPI servers and 2 workers"
|
||||
2. Kubernetes creates them across different servers
|
||||
3. If one crashes, Kubernetes restarts it
|
||||
4. If load increases, Kubernetes adds more
|
||||
5. If load decreases, Kubernetes removes some
|
||||
|
||||
AUTO-SCALING EXAMPLE:
|
||||
---------------------
|
||||
Normal load: 4 FastAPI pods, 2 worker pods
|
||||
High load: Kubernetes sees CPU at 80%
|
||||
→ Adds more pods automatically
|
||||
→ Now: 10 FastAPI pods, 5 worker pods
|
||||
Low load: Kubernetes sees CPU at 20%
|
||||
→ Removes some pods
|
||||
→ Back to: 4 FastAPI pods, 2 worker pods
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ 3.8 TRAEFIK - THE ROUTER │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
WHAT IT IS:
|
||||
-----------
|
||||
Traefik is like a smart receptionist:
|
||||
- Receives all incoming requests
|
||||
- Routes them to the right service
|
||||
- Handles HTTPS certificates automatically
|
||||
- Adds security (like password protection)
|
||||
|
||||
WHY WE USE IT:
|
||||
-------------
|
||||
✓ Automatic HTTPS (free SSL certificates)
|
||||
✓ Easy routing (just add labels to containers)
|
||||
✓ Handles authentication
|
||||
✓ One entry point for everything
|
||||
|
||||
HOW IT WORKS:
|
||||
-------------
|
||||
1. Request comes in: https://queue.workolik.com
|
||||
2. Traefik receives it
|
||||
3. Checks: "This goes to FastAPI"
|
||||
4. Routes to FastAPI service
|
||||
5. Returns response
|
||||
|
||||
HTTPS CERTIFICATES:
|
||||
-------------------
|
||||
Traefik automatically gets free SSL certificates from Let's Encrypt.
|
||||
You don't need to do anything - it just works!
|
||||
|
||||
AUTHENTICATION:
|
||||
---------------
|
||||
For the dashboard (natsadmin.workolik.com):
|
||||
- Traefik adds password protection
|
||||
- Username: admin
|
||||
- Password: package@321#
|
||||
|
||||
================================================================================
|
||||
4. HOW MESSAGES FLOW THROUGH THE SYSTEM
|
||||
================================================================================
|
||||
|
||||
STEP-BY-STEP EXAMPLE:
|
||||
---------------------
|
||||
|
||||
Let's say your app wants to update a delivery status.
|
||||
|
||||
STEP 1: YOUR APP SENDS REQUEST
|
||||
--------------------------------
|
||||
Your mobile app sends:
|
||||
PUT https://queue.workolik.com/live/api/v1/deliveries/updatedelivery
|
||||
Body: {"delivery_id": 123, "status": "delivered"}
|
||||
|
||||
STEP 2: TRAEFIK RECEIVES IT
|
||||
----------------------------
|
||||
Traefik sees: "This is for queue.workolik.com"
|
||||
Routes to: FastAPI service
|
||||
|
||||
STEP 3: FASTAPI PROCESSES IT
|
||||
-----------------------------
|
||||
FastAPI:
|
||||
- Receives the request
|
||||
- Validates the data (checks it's valid JSON)
|
||||
- Creates a message:
|
||||
{
|
||||
"endpoint": "/live/api/v1/deliveries/updatedelivery",
|
||||
"method": "PUT",
|
||||
"data": {"delivery_id": 123, "status": "delivered"},
|
||||
"received_at": 1234567890
|
||||
}
|
||||
- Publishes to NATS (super fast!)
|
||||
- Returns: {"status": "accepted", "message_id": 12345}
|
||||
|
||||
STEP 4: YOUR APP GETS RESPONSE
|
||||
-------------------------------
|
||||
Your app receives: "OK, got it!" (takes < 1 second)
|
||||
Your app doesn't wait for the external API - it's done!
|
||||
|
||||
STEP 5: NATS STORES MESSAGE
|
||||
---------------------------
|
||||
NATS:
|
||||
- Receives message
|
||||
- Stores in stream "EVENTS"
|
||||
- Subject: "api.v1.deliveries.updatedelivery"
|
||||
- Message is safe and won't be lost
|
||||
|
||||
STEP 6: WORKER PICKS IT UP
|
||||
---------------------------
|
||||
Worker:
|
||||
- Asks NATS: "Any messages?"
|
||||
- NATS gives worker the message
|
||||
- Worker reads: "Update delivery #123"
|
||||
|
||||
STEP 7: WORKER FORWARDS TO EXTERNAL API
|
||||
----------------------------------------
|
||||
Worker:
|
||||
- Maps endpoint to: https://jupiter.nearle.app/live/api/v1/deliveries/updatedelivery
|
||||
- Sends PUT request with the data
|
||||
- Waits for response
|
||||
|
||||
STEP 8: EXTERNAL API RESPONDS
|
||||
------------------------------
|
||||
External API:
|
||||
- Processes the request
|
||||
- Updates delivery status
|
||||
- Returns: 200 OK
|
||||
|
||||
STEP 9: WORKER ACKNOWLEDGES
|
||||
----------------------------
|
||||
Worker:
|
||||
- Sees success (200 OK)
|
||||
- Tells NATS: "Done!" (ACK)
|
||||
- NATS removes message from queue
|
||||
- Done!
|
||||
|
||||
IF SOMETHING GOES WRONG:
|
||||
------------------------
|
||||
If external API is down:
|
||||
- Worker gets error
|
||||
- Tells NATS: "Try again later" (NAK)
|
||||
- NATS keeps message in queue
|
||||
- Worker waits 5 seconds
|
||||
- Tries again
|
||||
- Repeats up to 5 times
|
||||
- If still fails, message stays for manual review
|
||||
|
||||
================================================================================
|
||||
5. WHY WE USE EACH TECHNOLOGY
|
||||
================================================================================
|
||||
|
||||
NATS (Message Queue):
|
||||
---------------------
|
||||
✓ Super fast (messages in < 1ms)
|
||||
✓ Reliable (messages won't disappear)
|
||||
✓ Simple (easy to set up and use)
|
||||
✓ Lightweight (doesn't need much resources)
|
||||
✓ Perfect for this use case
|
||||
|
||||
Why not RabbitMQ? Too heavy, more complex
|
||||
Why not Kafka? Overkill, too complex for this
|
||||
Why not Redis? Less mature, fewer features
|
||||
|
||||
FASTAPI (API Server):
|
||||
---------------------
|
||||
✓ Fast (handles many requests)
|
||||
✓ Easy (Python is simple to write)
|
||||
✓ Modern (async/await support)
|
||||
✓ Automatic docs (shows all endpoints)
|
||||
✓ Type validation (catches errors early)
|
||||
|
||||
Why not Flask? Slower, no async support
|
||||
Why not Django? Too heavy, overkill
|
||||
Why not Node.js? Team knows Python better
|
||||
|
||||
WORKERS (Message Processors):
|
||||
------------------------------
|
||||
✓ Decouples API from external service
|
||||
✓ Handles retries automatically
|
||||
✓ Can scale independently
|
||||
✓ Easy to monitor separately
|
||||
|
||||
Why separate workers? Keeps API fast, handles failures better
|
||||
|
||||
NGINX (Proxy):
|
||||
--------------
|
||||
✓ Needed for CORS (browser security)
|
||||
✓ Lightweight (small container)
|
||||
✓ Fast (high performance)
|
||||
✓ Simple config
|
||||
|
||||
Why needed? Browsers block cross-origin requests without CORS headers
|
||||
|
||||
DOCKER (Containers):
|
||||
--------------------
|
||||
✓ Same environment everywhere
|
||||
✓ Easy deployment
|
||||
✓ Isolated services
|
||||
✓ Easy updates
|
||||
|
||||
Why containers? Consistency, easy deployment, isolation
|
||||
|
||||
KUBERNETES (Orchestration):
|
||||
----------------------------
|
||||
✓ High availability (multiple servers)
|
||||
✓ Auto-scaling (adds workers when busy)
|
||||
✓ Self-healing (restarts crashed services)
|
||||
✓ Load balancing
|
||||
|
||||
Why Kubernetes? Production needs reliability and scaling
|
||||
|
||||
TRAEFIK (Router):
|
||||
-----------------
|
||||
✓ Automatic HTTPS (free certificates)
|
||||
✓ Easy routing (just labels)
|
||||
✓ Handles authentication
|
||||
✓ One entry point
|
||||
|
||||
Why Traefik? Simplifies HTTPS and routing
|
||||
|
||||
================================================================================
|
||||
6. HOW TO DEPLOY AND USE
|
||||
================================================================================
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ 6.1 DOCKER COMPOSE (SIMPLE DEPLOYMENT) │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
FOR: Local development, small deployments, single server
|
||||
|
||||
STEPS:
|
||||
------
|
||||
1. Make sure Docker is installed
|
||||
2. Go to the kubernetes folder
|
||||
3. Run: docker-compose up -d
|
||||
4. That's it! Everything starts automatically
|
||||
|
||||
WHAT GETS STARTED:
|
||||
-----------------
|
||||
- NATS server (message queue)
|
||||
- Nginx proxy (CORS helper)
|
||||
- NATS Dashboard (monitoring)
|
||||
|
||||
ACCESS:
|
||||
-------
|
||||
- Dashboard: https://natsadmin.workolik.com
|
||||
- NATS Monitoring: https://nats.workolik.com
|
||||
|
||||
CHECK STATUS:
|
||||
-------------
|
||||
docker-compose ps # See what's running
|
||||
docker-compose logs -f # See logs
|
||||
docker-compose restart # Restart everything
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ 6.2 KUBERNETES (PRODUCTION DEPLOYMENT) │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
FOR: Production, high availability, auto-scaling
|
||||
|
||||
STEPS:
|
||||
------
|
||||
1. Build Docker images
|
||||
2. Push to registry
|
||||
3. Update secrets.yaml with your credentials
|
||||
4. Run: ./deploy.sh
|
||||
5. Wait for pods to start
|
||||
|
||||
WHAT GETS DEPLOYED:
|
||||
-------------------
|
||||
- FastAPI pods (4-20, auto-scales)
|
||||
- Worker pods (2-10, auto-scales)
|
||||
- Services (load balancers)
|
||||
- Auto-scaling rules
|
||||
|
||||
CHECK STATUS:
|
||||
-------------
|
||||
kubectl get pods -n nats-backend # See pods
|
||||
kubectl logs -f deployment/fastapi-backend # See logs
|
||||
kubectl get hpa -n nats-backend # See auto-scaling
|
||||
|
||||
TEST ENDPOINTS:
|
||||
---------------
|
||||
curl -X PUT https://queue.workolik.com/live/api/v1/deliveries/updatedelivery \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"delivery_id": 123, "status": "delivered"}'
|
||||
|
||||
Expected response:
|
||||
{"status": "accepted", "message_id": 12345, "endpoint": "..."}
|
||||
|
||||
================================================================================
|
||||
7. COMMON QUESTIONS ANSWERED
|
||||
================================================================================
|
||||
|
||||
Q: Why use a message queue instead of calling the API directly?
|
||||
A:
|
||||
- Your app doesn't have to wait for slow external API
|
||||
- If external API is down, messages wait (won't be lost)
|
||||
- Can handle traffic spikes better
|
||||
- Easy to add more workers if needed
|
||||
|
||||
Q: What happens if NATS goes down?
|
||||
A:
|
||||
- Messages are stored on disk (persistent)
|
||||
- When NATS restarts, messages are still there
|
||||
- Workers reconnect automatically
|
||||
- No data loss
|
||||
|
||||
Q: What happens if a worker crashes?
|
||||
A:
|
||||
- Kubernetes restarts it automatically
|
||||
- Messages stay in queue
|
||||
- Other workers keep processing
|
||||
- No data loss
|
||||
|
||||
Q: How fast is it?
|
||||
A:
|
||||
- FastAPI responds in < 1 second (usually < 100ms)
|
||||
- NATS stores message in < 1 millisecond
|
||||
- Worker processes in seconds (depends on external API)
|
||||
- Total: Usually under 2 seconds end-to-end
|
||||
|
||||
Q: Can I add more endpoints?
|
||||
A:
|
||||
- Yes! Add endpoint in FastAPI (app.py)
|
||||
- Add mapping in worker (worker.py)
|
||||
- Deploy and it works
|
||||
|
||||
Q: How do I monitor if everything is working?
|
||||
A:
|
||||
- Check NATS Dashboard: https://natsadmin.workolik.com
|
||||
- Check FastAPI metrics: GET /metrics
|
||||
- Check worker metrics: Port 9090
|
||||
- Check Kubernetes: kubectl get pods
|
||||
|
||||
Q: What if external API is slow?
|
||||
A:
|
||||
- Messages wait in queue
|
||||
- Workers retry automatically
|
||||
- Your API still responds fast (doesn't wait)
|
||||
- No impact on your users
|
||||
|
||||
Q: How many messages can it handle?
|
||||
A:
|
||||
- NATS: Millions per second
|
||||
- FastAPI: Thousands per second (depends on hardware)
|
||||
- Workers: Depends on external API speed
|
||||
- Can scale workers automatically if needed
|
||||
|
||||
Q: Is it secure?
|
||||
A:
|
||||
- HTTPS for all external access (Traefik)
|
||||
- Password protection for dashboard
|
||||
- NATS authentication required
|
||||
- CORS properly configured
|
||||
|
||||
Q: Can I test locally?
|
||||
A:
|
||||
- Yes! Use Docker Compose
|
||||
- Everything runs on your machine
|
||||
- Same as production (just smaller scale)
|
||||
|
||||
================================================================================
|
||||
8. TROUBLESHOOTING MADE EASY
|
||||
================================================================================
|
||||
|
||||
PROBLEM: FastAPI returns 503 "NATS not connected"
|
||||
-----------------------------------------------
|
||||
WHAT IT MEANS: FastAPI can't talk to NATS
|
||||
|
||||
HOW TO FIX:
|
||||
1. Check if NATS is running: docker ps | grep nats
|
||||
2. Check NATS logs: docker logs nats
|
||||
3. Check network: Can FastAPI reach NATS?
|
||||
4. Check credentials: Are username/password correct?
|
||||
|
||||
PROBLEM: Messages not being processed
|
||||
-------------------------------------
|
||||
WHAT IT MEANS: Workers aren't picking up messages
|
||||
|
||||
HOW TO FIX:
|
||||
1. Check if workers are running: kubectl get pods | grep worker
|
||||
2. Check worker logs: kubectl logs -f deployment/nats-worker
|
||||
3. Check NATS Dashboard: Are messages in queue?
|
||||
4. Check consumer: Is worker_consumer active?
|
||||
|
||||
PROBLEM: CORS errors in browser
|
||||
--------------------------------
|
||||
WHAT IT MEANS: Dashboard can't access NATS monitoring
|
||||
|
||||
HOW TO FIX:
|
||||
1. Check Nginx proxy is running: docker ps | grep nats-proxy
|
||||
2. Check nginx-nats.conf: Are CORS headers correct?
|
||||
3. Check NATS_MONITORING_URL in dashboard config
|
||||
4. Restart nginx-proxy: docker-compose restart nats-proxy
|
||||
|
||||
PROBLEM: External API not receiving requests
|
||||
--------------------------------------------
|
||||
WHAT IT MEANS: Workers aren't forwarding messages
|
||||
|
||||
HOW TO FIX:
|
||||
1. Check worker logs: Look for forwarding errors
|
||||
2. Check EXTERNAL_BASE_URL: Is it correct?
|
||||
3. Test connectivity: Can workers reach external API?
|
||||
4. Check endpoint mapping: Is endpoint in worker.py?
|
||||
|
||||
PROBLEM: Auto-scaling not working
|
||||
---------------------------------
|
||||
WHAT IT MEANS: Pods aren't scaling up/down
|
||||
|
||||
HOW TO FIX:
|
||||
1. Check HPA: kubectl get hpa -n nats-backend
|
||||
2. Check metrics-server: kubectl top pods
|
||||
3. Check resource limits: Are they set correctly?
|
||||
4. Check if max replicas reached
|
||||
|
||||
QUICK HEALTH CHECK:
|
||||
-------------------
|
||||
1. Are all pods running? → kubectl get pods
|
||||
2. Are messages processing? → Check dashboard
|
||||
3. Are there errors? → Check logs
|
||||
4. Is external API reachable? → Test from worker pod
|
||||
|
||||
COMMON COMMANDS:
|
||||
----------------
|
||||
# Docker Compose
|
||||
docker-compose ps # Status
|
||||
docker-compose logs -f # Logs
|
||||
docker-compose restart # Restart
|
||||
|
||||
# Kubernetes
|
||||
kubectl get pods # Pod status
|
||||
kubectl logs -f [pod-name] # Pod logs
|
||||
kubectl describe pod [pod] # Pod details
|
||||
kubectl get hpa # Auto-scaling status
|
||||
|
||||
# NATS
|
||||
# Check dashboard: https://natsadmin.workolik.com
|
||||
|
||||
================================================================================
|
||||
9. KEY CONCEPTS EXPLAINED SIMPLY
|
||||
================================================================================
|
||||
|
||||
MESSAGE QUEUE:
|
||||
--------------
|
||||
Like a post office:
|
||||
- You drop off a letter (message)
|
||||
- Post office stores it safely
|
||||
- Mail carrier picks it up
|
||||
- Delivers to destination
|
||||
|
||||
In our system:
|
||||
- FastAPI drops off message
|
||||
- NATS stores it
|
||||
- Worker picks it up
|
||||
- Delivers to external API
|
||||
|
||||
PUBLISH/SUBSCRIBE:
|
||||
------------------
|
||||
Like a radio station:
|
||||
- Radio station broadcasts (publishes)
|
||||
- Radios listen (subscribe)
|
||||
- Many radios can listen to same station
|
||||
|
||||
In our system:
|
||||
- FastAPI publishes messages
|
||||
- Workers subscribe to messages
|
||||
- Many workers can process same queue
|
||||
|
||||
ACKNOWLEDGMENT (ACK/NAK):
|
||||
-------------------------
|
||||
Like a receipt:
|
||||
- You send a package
|
||||
- Recipient signs for it (ACK)
|
||||
- If rejected, you get it back (NAK)
|
||||
|
||||
In our system:
|
||||
- Worker processes message
|
||||
- If success: ACK → message removed
|
||||
- If failure: NAK → message stays for retry
|
||||
|
||||
STREAM:
|
||||
-------
|
||||
Like a folder:
|
||||
- All related messages go in one stream
|
||||
- Easy to organize
|
||||
- Can have multiple consumers
|
||||
|
||||
In our system:
|
||||
- Stream: "EVENTS"
|
||||
- All API messages go here
|
||||
- Workers read from this stream
|
||||
|
||||
CONSUMER:
|
||||
---------
|
||||
Like a reader:
|
||||
- Reads messages from stream
|
||||
- Processes them
|
||||
- Acknowledges when done
|
||||
|
||||
In our system:
|
||||
- Consumer: "worker_consumer"
|
||||
- Workers use this to get messages
|
||||
- Tracks which messages are processed
|
||||
|
||||
AUTO-SCALING:
|
||||
-------------
|
||||
Like a restaurant:
|
||||
- Few customers → Few waiters
|
||||
- Many customers → More waiters
|
||||
- Automatically adjusts
|
||||
|
||||
In our system:
|
||||
- Low load → Few pods
|
||||
- High load → More pods
|
||||
- Kubernetes does it automatically
|
||||
|
||||
LOAD BALANCING:
|
||||
---------------
|
||||
Like distributing work:
|
||||
- Manager gives tasks to available workers
|
||||
- Spreads work evenly
|
||||
- No one gets overloaded
|
||||
|
||||
In our system:
|
||||
- Requests go to available FastAPI pod
|
||||
- Spreads load evenly
|
||||
- Kubernetes handles it
|
||||
|
||||
================================================================================
|
||||
10. REAL-WORLD EXAMPLE
|
||||
================================================================================
|
||||
|
||||
SCENARIO: Your delivery app needs to update delivery status
|
||||
|
||||
WITHOUT MESSAGE QUEUE:
|
||||
----------------------
|
||||
1. App sends request to your API
|
||||
2. Your API calls external API directly
|
||||
3. External API is slow (takes 5 seconds)
|
||||
4. Your app waits 5 seconds
|
||||
5. User sees loading spinner
|
||||
6. If external API is down, your app fails
|
||||
|
||||
WITH MESSAGE QUEUE (OUR SYSTEM):
|
||||
---------------------------------
|
||||
1. App sends request to FastAPI
|
||||
2. FastAPI puts message in queue (< 100ms)
|
||||
3. FastAPI returns "OK" immediately
|
||||
4. User sees success right away
|
||||
5. Worker processes message in background
|
||||
6. If external API is slow/down, message waits
|
||||
7. Worker retries automatically
|
||||
8. No impact on your users
|
||||
|
||||
BENEFITS:
|
||||
---------
|
||||
✓ Users get instant response
|
||||
✓ System handles failures gracefully
|
||||
✓ Can handle traffic spikes
|
||||
✓ Easy to scale
|
||||
✓ Messages never lost
|
||||
|
||||
================================================================================
|
||||
11. QUICK REFERENCE
|
||||
================================================================================
|
||||
|
||||
ENDPOINTS:
|
||||
----------
|
||||
PUT /live/api/v1/deliveries/updatedelivery
|
||||
POST /live/api/v1/deliveries/createdeliveries
|
||||
POST /live/api/v2/partners/createriderlog
|
||||
POST /live/api/v2/deliveries/createdeliverylog
|
||||
POST /live/api/v2/partners/createbreaklog
|
||||
POST /live/api/v2/partners/updatebreaklog
|
||||
|
||||
URLS:
|
||||
-----
|
||||
FastAPI: https://queue.workolik.com
|
||||
Dashboard: https://natsadmin.workolik.com
|
||||
NATS Monitoring: https://nats.workolik.com
|
||||
|
||||
CREDENTIALS:
|
||||
------------
|
||||
NATS:
|
||||
- Username: admin
|
||||
- Password: package@321#
|
||||
|
||||
Dashboard:
|
||||
- Username: admin
|
||||
- Password: package@321#
|
||||
|
||||
PORTS:
|
||||
------
|
||||
NATS: 4222 (client), 8222 (monitoring)
|
||||
FastAPI: 8000 (internal), 8201 (external)
|
||||
Worker: 9090 (metrics)
|
||||
Nginx: 80 (internal), 8082 (external)
|
||||
|
||||
FILES:
|
||||
------
|
||||
docker-compose.yml # Docker deployment
|
||||
scripts/app.py # FastAPI code
|
||||
scripts/worker.py # Worker code
|
||||
nginx-nats.conf # Nginx config
|
||||
manifests/*.yaml # Kubernetes configs
|
||||
|
||||
================================================================================
|
||||
END OF GUIDE
|
||||
================================================================================
|
||||
|
||||
This guide explains the system in simple terms. For more technical details,
|
||||
see TECH_STACK_ARCHITECTURE.txt.
|
||||
|
||||
Remember:
|
||||
- FastAPI receives requests and puts them in queue
|
||||
- NATS stores messages safely
|
||||
- Workers process messages and forward to external API
|
||||
- Everything is monitored and can scale automatically
|
||||
|
||||
Questions? Check Section 7 (Common Questions) or Section 8 (Troubleshooting).
|
||||
|
||||
Last Updated: 2025-01-XX
|
||||
Version: Intermediate Level
|
||||
|
||||
Reference in New Issue
Block a user