409 lines
12 KiB
Markdown
409 lines
12 KiB
Markdown
# Professional k3s Target Architecture
|
|
|
|
This document is a practical upgrade path from the current repo layout to a more professional, resilient platform.
|
|
|
|
It is written for the current setup in this repository:
|
|
|
|
- k3s is being used as the Kubernetes distribution.
|
|
- ingress/proxy traffic is still partly handled outside the cluster with Docker Compose and nginx.
|
|
- app workloads are running in Kubernetes.
|
|
- workers depend on NATS/JetStream.
|
|
- some services still use `NodePort`.
|
|
- the core workloads point to a single external NATS IP.
|
|
|
|
## 1. What the current repo already does well
|
|
|
|
There is already a good foundation here:
|
|
|
|
- API and worker responsibilities are separated.
|
|
- asynchronous processing with NATS/JetStream is the right pattern.
|
|
- workers are split by business area in [`manifests/core/workers.yaml`](E:/Birock/kubernetes/manifests/core/workers.yaml).
|
|
- you already have some resource requests and limits.
|
|
- you already use PodDisruptionBudget for one worker set.
|
|
- you already think in namespaces such as `core` and `nearle`.
|
|
|
|
That means you are not starting from zero. You are mostly at the stage of cleaning up platform boundaries and removing single points of failure.
|
|
|
|
## 2. Main problems in the current architecture
|
|
|
|
### Problem A: single point of failure in messaging
|
|
|
|
Your core workloads use one external NATS endpoint:
|
|
|
|
- [`manifests/core/core-config.yaml#L10`](E:/Birock/kubernetes/manifests/core/core-config.yaml#L10)
|
|
|
|
If that node or disk fails, the Kubernetes pods can still be healthy but the platform is still down.
|
|
|
|
### Problem B: node failover is not immediate today
|
|
|
|
You want "if current node is not working, switch immediately to backup node". Right now that is not really true because:
|
|
|
|
- some traffic still depends on host-level Docker Compose proxies
|
|
- some services are exposed with `NodePort`
|
|
- some workloads use `hostPath`
|
|
- there is no clear control-plane and worker-plane separation
|
|
- there is no shared HA datastore for k3s control-plane state
|
|
|
|
For example:
|
|
|
|
- [`manifests/nearle/nearle-jupiter.yaml#L58`](E:/Birock/kubernetes/manifests/nearle/nearle-jupiter.yaml#L58) exposes `jupiter` with `NodePort`
|
|
- [`manifests/nearle/nearle-jupiter.yaml#L43`](E:/Birock/kubernetes/manifests/nearle/nearle-jupiter.yaml#L43) uses `hostPath`
|
|
|
|
`NodePort` is not wrong, but it is usually not the best long-term edge pattern for a professional HA setup.
|
|
|
|
### Problem C: CPU spikes can still hurt the whole node
|
|
|
|
Your workers are separated logically, but they still share the same physical node resources unless you explicitly isolate them with:
|
|
|
|
- dedicated worker nodes
|
|
- taints and tolerations
|
|
- node labels and node affinity
|
|
- tighter requests/limits
|
|
- autoscaling based on the right metrics
|
|
|
|
Some workers also have relatively high concurrency values:
|
|
|
|
- [`manifests/core/workers.yaml#L42`](E:/Birock/kubernetes/manifests/core/workers.yaml#L42)
|
|
- [`manifests/core/workers.yaml#L192`](E:/Birock/kubernetes/manifests/core/workers.yaml#L192)
|
|
|
|
If the external target slows down or retries increase, these workers can create CPU and network pressure.
|
|
|
|
### Problem D: secrets are stored in repo manifests
|
|
|
|
There are inline credentials here:
|
|
|
|
- [`manifests/core/core-secrets.yaml`](E:/Birock/kubernetes/manifests/core/core-secrets.yaml)
|
|
|
|
For a professional setup, secrets should move to a proper secret manager or at least be injected outside git.
|
|
|
|
## 3. What "professional architecture" should look like here
|
|
|
|
For your use case, the clean target is not "one main node and one cold backup node".
|
|
|
|
The cleaner target is:
|
|
|
|
1. Highly available k3s control plane
|
|
2. Separate worker nodes for workloads
|
|
3. NATS deployed as an HA cluster
|
|
4. Ingress handled inside Kubernetes
|
|
5. Backups for both cluster state and NATS data
|
|
6. Workload placement rules so noisy workers do not affect all services
|
|
|
|
## 4. Recommended target layout
|
|
|
|
### Minimum professional layout
|
|
|
|
- `3` k3s server nodes
|
|
- `2` workload worker nodes
|
|
- `3` NATS pods with JetStream replication
|
|
- `1` in-cluster ingress controller
|
|
- `1` backup system for cluster resources and persistent volumes
|
|
|
|
### Example node roles
|
|
|
|
- `cp-1`: k3s server
|
|
- `cp-2`: k3s server
|
|
- `cp-3`: k3s server
|
|
- `app-1`: app services and ingress
|
|
- `wrk-1`: NATS workers and heavy async jobs
|
|
|
|
Better:
|
|
|
|
- `app-1`, `app-2`: app/service nodes
|
|
- `wrk-1`, `wrk-2`: dedicated worker nodes
|
|
|
|
### Planes
|
|
|
|
Control plane:
|
|
|
|
- k3s server nodes only
|
|
- no application workloads if possible
|
|
|
|
Service plane:
|
|
|
|
- FastAPI / business services
|
|
- ingress controller
|
|
- gateway
|
|
- observability stack
|
|
|
|
Worker plane:
|
|
|
|
- NATS worker consumers
|
|
- batch jobs
|
|
- CPU-heavy or retry-heavy services
|
|
|
|
Messaging plane:
|
|
|
|
- NATS cluster with JetStream
|
|
- dedicated storage
|
|
|
|
## 5. Best failover model for your requirement
|
|
|
|
You asked for immediate switch to a backup node if one node fails.
|
|
|
|
There are two ways to think about that:
|
|
|
|
### Option 1: active-passive node failover
|
|
|
|
This means one main node and one backup node waiting.
|
|
|
|
This is simpler to understand, but it is still not the best Kubernetes design because:
|
|
|
|
- one active node is still a bottleneck
|
|
- failover is not truly instant
|
|
- stateful components are harder to fail over cleanly
|
|
- you will still need shared storage or replicated state
|
|
|
|
### Option 2: active-active cluster with multiple nodes
|
|
|
|
This is the better professional design.
|
|
|
|
Instead of "switching to backup", Kubernetes simply reschedules workloads onto healthy nodes because:
|
|
|
|
- services already run across more than one node
|
|
- ingress already points at the cluster, not one host
|
|
- NATS data is replicated
|
|
- pods have anti-affinity and replicas across nodes
|
|
|
|
This is the model I recommend for you.
|
|
|
|
## 6. Recommended production architecture for this repo
|
|
|
|
### Edge and ingress
|
|
|
|
Move ingress fully into Kubernetes:
|
|
|
|
- install Traefik or nginx ingress inside k3s
|
|
- stop depending on host-level Docker Compose nginx proxies for production routing
|
|
- expose ingress through a proper load balancer or a floating IP/VIP
|
|
|
|
Good choices:
|
|
|
|
- `kube-vip` for virtual IP failover on bare metal
|
|
- `MetalLB` for service load balancers on bare metal
|
|
|
|
This is the clean replacement for current host-side proxying in [`docker-compose.yml`](E:/Birock/kubernetes/docker-compose.yml).
|
|
|
|
### Kubernetes control plane
|
|
|
|
Use HA k3s server nodes:
|
|
|
|
- `3` k3s server nodes
|
|
- external datastore or embedded etcd in HA mode
|
|
|
|
For small-to-medium production, HA k3s with embedded etcd is often enough and simpler than trying to maintain a single-node server plus a backup.
|
|
|
|
### App services
|
|
|
|
Run FastAPI and business services on app nodes:
|
|
|
|
- Deployment instead of StatefulSet unless stable pod identity is required
|
|
- `replicas >= 2`
|
|
- topology spread constraints
|
|
- anti-affinity across nodes
|
|
- proper readiness and liveness probes
|
|
|
|
### Worker services
|
|
|
|
Run workers only on worker nodes:
|
|
|
|
- label worker nodes, for example `workload-type=async`
|
|
- taint worker nodes
|
|
- add tolerations to worker pods
|
|
- add node affinity so worker pods land only there
|
|
|
|
This gives you the "separate worker plane" you asked for.
|
|
|
|
### NATS
|
|
|
|
Deploy NATS inside Kubernetes as an HA cluster, not as a single external endpoint.
|
|
|
|
Use:
|
|
|
|
- `3` NATS pods
|
|
- JetStream replication factor `3`
|
|
- persistent volumes
|
|
- pod anti-affinity
|
|
- dedicated node pool if possible
|
|
|
|
This is the biggest architecture improvement you can make for service survival.
|
|
|
|
### Backups
|
|
|
|
Use two backup layers:
|
|
|
|
1. Kubernetes resource/state backup
|
|
2. JetStream or persistent volume backup
|
|
|
|
Recommended tools:
|
|
|
|
- Velero for cluster resource backup and restore
|
|
- CSI snapshots or storage-level snapshots for persistent volumes
|
|
- scheduled export/backup for critical NATS data if needed
|
|
|
|
## 7. How to separate service plane and worker plane
|
|
|
|
This is a very good idea for your stack.
|
|
|
|
### Service plane should host
|
|
|
|
- ingress controller
|
|
- FastAPI/API gateway
|
|
- frontend-facing services
|
|
- dashboard/observability tools
|
|
|
|
### Worker plane should host
|
|
|
|
- NATS consumers
|
|
- retry-heavy jobs
|
|
- long-running async processors
|
|
- any CPU-heavy integration jobs
|
|
|
|
### Basic implementation pattern
|
|
|
|
On nodes:
|
|
|
|
- label app nodes: `node-role.workolik/app=true`
|
|
- label worker nodes: `node-role.workolik/worker=true`
|
|
|
|
Optionally taint worker nodes:
|
|
|
|
- `dedicated=workers:NoSchedule`
|
|
|
|
Then:
|
|
|
|
- app manifests use node affinity for `app=true`
|
|
- worker manifests use node affinity and toleration for worker nodes
|
|
|
|
## 8. CPU spike reduction strategy
|
|
|
|
The CPU spike problem is usually not solved by "adding one backup node". It is solved by isolation, limits, and scaling.
|
|
|
|
### Do these first
|
|
|
|
1. Put workers on separate nodes.
|
|
2. Tighten worker CPU limits and requests based on real usage.
|
|
3. Reduce high default concurrency for the noisiest workers.
|
|
4. Add HPA or KEDA scaling from queue depth, not only CPU.
|
|
5. Ensure retries do not cause synchronized storms.
|
|
|
|
### Very likely spike sources in this repo
|
|
|
|
- high worker concurrency
|
|
- many worker StatefulSets sharing the same node
|
|
- retries against slow external APIs
|
|
- host-level proxying plus cluster-level routing mix
|
|
- single-node k3s carrying ingress, apps, workers, and maybe NATS responsibilities together
|
|
|
|
### Better autoscaling choice
|
|
|
|
For queue workers, KEDA is often better than plain HPA because it can scale on:
|
|
|
|
- NATS lag
|
|
- queue depth
|
|
- custom Prometheus metrics
|
|
|
|
That is usually more useful than only scaling from CPU percentage.
|
|
|
|
## 9. Concrete migration path
|
|
|
|
### Phase 1: stabilize current cluster
|
|
|
|
Do this before any big redesign:
|
|
|
|
- move secrets out of git
|
|
- standardize on ingress instead of many host nginx proxies
|
|
- remove unnecessary `NodePort` exposure where possible
|
|
- add resource dashboards and alerting
|
|
- capture actual CPU and memory usage for each worker
|
|
|
|
### Phase 2: separate worker nodes
|
|
|
|
- add a second or third node
|
|
- label and taint worker nodes
|
|
- move worker workloads there
|
|
- keep app services on separate nodes
|
|
|
|
This alone will already reduce blast radius from worker CPU spikes.
|
|
|
|
### Phase 3: make k3s highly available
|
|
|
|
- create `3` k3s server nodes
|
|
- use HA embedded etcd
|
|
- put ingress behind `kube-vip` or `MetalLB`
|
|
|
|
Now loss of one server does not mean cluster loss.
|
|
|
|
### Phase 4: make NATS highly available
|
|
|
|
- deploy NATS cluster inside Kubernetes
|
|
- enable JetStream replication
|
|
- use persistent volumes
|
|
- update workloads to connect to in-cluster NATS service
|
|
|
|
This replaces the current single external NATS dependency from [`manifests/core/core-config.yaml#L10`](E:/Birock/kubernetes/manifests/core/core-config.yaml#L10).
|
|
|
|
### Phase 5: add backup and restore
|
|
|
|
- install Velero
|
|
- schedule cluster backups
|
|
- schedule volume snapshots
|
|
- test restore into a fresh environment
|
|
|
|
If restore has never been tested, backup is not yet reliable.
|
|
|
|
## 10. What I would choose for you
|
|
|
|
Because you said you are a beginner, I would not jump straight into a very large platform.
|
|
|
|
I would choose this as the practical target:
|
|
|
|
- `3` k3s server nodes
|
|
- `2` worker nodes
|
|
- in-cluster Traefik
|
|
- `MetalLB` or `kube-vip`
|
|
- NATS HA cluster with JetStream replication
|
|
- Velero backups
|
|
- node separation for app plane and worker plane
|
|
|
|
This is modern, realistic, and still manageable.
|
|
|
|
## 11. What I would not recommend
|
|
|
|
I would avoid these patterns for your next version:
|
|
|
|
- one main Kubernetes node plus one cold backup node as the final design
|
|
- storing secrets directly in yaml in git
|
|
- relying on `NodePort` as the main production exposure pattern
|
|
- mixing host Docker Compose production routing with cluster routing long-term
|
|
- keeping NATS as a single external IP with no clear HA story
|
|
|
|
## 12. Immediate next actions for this repo
|
|
|
|
If we continue from this document, the best implementation order is:
|
|
|
|
1. Add node placement rules for workers and apps.
|
|
2. Convert external exposure to a single in-cluster ingress pattern.
|
|
3. Remove hard dependence on the single external NATS IP.
|
|
4. Add observability for CPU, memory, restart count, and queue lag.
|
|
5. Introduce backup tooling and test restore.
|
|
|
|
## 13. Summary in simple words
|
|
|
|
The clean professional version of your stack is:
|
|
|
|
- Kubernetes control plane on multiple HA server nodes
|
|
- app services on one node group
|
|
- NATS workers on another node group
|
|
- NATS itself running as a replicated cluster
|
|
- ingress and failover handled at cluster level, not manually by switching servers
|
|
- backups for both Kubernetes state and message data
|
|
|
|
That gives you what you want:
|
|
|
|
- less CPU blast radius
|
|
- better reliability
|
|
- faster failover
|
|
- cleaner separation of responsibilities
|
|
- a more modern production setup
|