Nearle/kubernetes
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit

Browse Source
This commit is contained in:
Suriya
2026-07-18 12:00:33 +05:30
commit caac8413e9
83 changed files with 10262 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

408
docs/PROFESSIONAL_K3S_TARGET_ARCHITECTURE.md Normal file
View File

@@ -0,0 +1,408 @@
# Professional k3s Target Architecture
This document is a practical upgrade path from the current repo layout to a more professional, resilient platform.
It is written for the current setup in this repository:
- k3s is being used as the Kubernetes distribution.
- ingress/proxy traffic is still partly handled outside the cluster with Docker Compose and nginx.
- app workloads are running in Kubernetes.
- workers depend on NATS/JetStream.
- some services still use `NodePort`.
- the core workloads point to a single external NATS IP.
## 1. What the current repo already does well
There is already a good foundation here:
- API and worker responsibilities are separated.
- asynchronous processing with NATS/JetStream is the right pattern.
- workers are split by business area in [`manifests/core/workers.yaml`](E:/Birock/kubernetes/manifests/core/workers.yaml).
- you already have some resource requests and limits.
- you already use PodDisruptionBudget for one worker set.
- you already think in namespaces such as `core` and `nearle`.
That means you are not starting from zero. You are mostly at the stage of cleaning up platform boundaries and removing single points of failure.
## 2. Main problems in the current architecture
### Problem A: single point of failure in messaging
Your core workloads use one external NATS endpoint:
- [`manifests/core/core-config.yaml#L10`](E:/Birock/kubernetes/manifests/core/core-config.yaml#L10)
If that node or disk fails, the Kubernetes pods can still be healthy but the platform is still down.
### Problem B: node failover is not immediate today
You want "if current node is not working, switch immediately to backup node". Right now that is not really true because:
- some traffic still depends on host-level Docker Compose proxies
- some services are exposed with `NodePort`
- some workloads use `hostPath`
- there is no clear control-plane and worker-plane separation
- there is no shared HA datastore for k3s control-plane state
For example:
- [`manifests/nearle/nearle-jupiter.yaml#L58`](E:/Birock/kubernetes/manifests/nearle/nearle-jupiter.yaml#L58) exposes `jupiter` with `NodePort`
- [`manifests/nearle/nearle-jupiter.yaml#L43`](E:/Birock/kubernetes/manifests/nearle/nearle-jupiter.yaml#L43) uses `hostPath`
`NodePort` is not wrong, but it is usually not the best long-term edge pattern for a professional HA setup.
### Problem C: CPU spikes can still hurt the whole node
Your workers are separated logically, but they still share the same physical node resources unless you explicitly isolate them with:
- dedicated worker nodes
- taints and tolerations
- node labels and node affinity
- tighter requests/limits
- autoscaling based on the right metrics
Some workers also have relatively high concurrency values:
- [`manifests/core/workers.yaml#L42`](E:/Birock/kubernetes/manifests/core/workers.yaml#L42)
- [`manifests/core/workers.yaml#L192`](E:/Birock/kubernetes/manifests/core/workers.yaml#L192)
If the external target slows down or retries increase, these workers can create CPU and network pressure.
### Problem D: secrets are stored in repo manifests
There are inline credentials here:
- [`manifests/core/core-secrets.yaml`](E:/Birock/kubernetes/manifests/core/core-secrets.yaml)
For a professional setup, secrets should move to a proper secret manager or at least be injected outside git.
## 3. What "professional architecture" should look like here
For your use case, the clean target is not "one main node and one cold backup node".
The cleaner target is:
1. Highly available k3s control plane
2. Separate worker nodes for workloads
3. NATS deployed as an HA cluster
4. Ingress handled inside Kubernetes
5. Backups for both cluster state and NATS data
6. Workload placement rules so noisy workers do not affect all services
## 4. Recommended target layout
### Minimum professional layout
- `3` k3s server nodes
- `2` workload worker nodes
- `3` NATS pods with JetStream replication
- `1` in-cluster ingress controller
- `1` backup system for cluster resources and persistent volumes
### Example node roles
- `cp-1`: k3s server
- `cp-2`: k3s server
- `cp-3`: k3s server
- `app-1`: app services and ingress
- `wrk-1`: NATS workers and heavy async jobs
Better:
- `app-1`, `app-2`: app/service nodes
- `wrk-1`, `wrk-2`: dedicated worker nodes
### Planes
Control plane:
- k3s server nodes only
- no application workloads if possible
Service plane:
- FastAPI / business services
- ingress controller
- gateway
- observability stack
Worker plane:
- NATS worker consumers
- batch jobs
- CPU-heavy or retry-heavy services
Messaging plane:
- NATS cluster with JetStream
- dedicated storage
## 5. Best failover model for your requirement
You asked for immediate switch to a backup node if one node fails.
There are two ways to think about that:
### Option 1: active-passive node failover
This means one main node and one backup node waiting.
This is simpler to understand, but it is still not the best Kubernetes design because:
- one active node is still a bottleneck
- failover is not truly instant
- stateful components are harder to fail over cleanly
- you will still need shared storage or replicated state
### Option 2: active-active cluster with multiple nodes
This is the better professional design.
Instead of "switching to backup", Kubernetes simply reschedules workloads onto healthy nodes because:
- services already run across more than one node
- ingress already points at the cluster, not one host
- NATS data is replicated
- pods have anti-affinity and replicas across nodes
This is the model I recommend for you.
## 6. Recommended production architecture for this repo
### Edge and ingress
Move ingress fully into Kubernetes:
- install Traefik or nginx ingress inside k3s
- stop depending on host-level Docker Compose nginx proxies for production routing
- expose ingress through a proper load balancer or a floating IP/VIP
Good choices:
- `kube-vip` for virtual IP failover on bare metal
- `MetalLB` for service load balancers on bare metal
This is the clean replacement for current host-side proxying in [`docker-compose.yml`](E:/Birock/kubernetes/docker-compose.yml).
### Kubernetes control plane
Use HA k3s server nodes:
- `3` k3s server nodes
- external datastore or embedded etcd in HA mode
For small-to-medium production, HA k3s with embedded etcd is often enough and simpler than trying to maintain a single-node server plus a backup.
### App services
Run FastAPI and business services on app nodes:
- Deployment instead of StatefulSet unless stable pod identity is required
- `replicas >= 2`
- topology spread constraints
- anti-affinity across nodes
- proper readiness and liveness probes
### Worker services
Run workers only on worker nodes:
- label worker nodes, for example `workload-type=async`
- taint worker nodes
- add tolerations to worker pods
- add node affinity so worker pods land only there
This gives you the "separate worker plane" you asked for.
### NATS
Deploy NATS inside Kubernetes as an HA cluster, not as a single external endpoint.
Use:
- `3` NATS pods
- JetStream replication factor `3`
- persistent volumes
- pod anti-affinity
- dedicated node pool if possible
This is the biggest architecture improvement you can make for service survival.
### Backups
Use two backup layers:
1. Kubernetes resource/state backup
2. JetStream or persistent volume backup
Recommended tools:
- Velero for cluster resource backup and restore
- CSI snapshots or storage-level snapshots for persistent volumes
- scheduled export/backup for critical NATS data if needed
## 7. How to separate service plane and worker plane
This is a very good idea for your stack.
### Service plane should host
- ingress controller
- FastAPI/API gateway
- frontend-facing services
- dashboard/observability tools
### Worker plane should host
- NATS consumers
- retry-heavy jobs
- long-running async processors
- any CPU-heavy integration jobs
### Basic implementation pattern
On nodes:
- label app nodes: `node-role.workolik/app=true`
- label worker nodes: `node-role.workolik/worker=true`
Optionally taint worker nodes:
- `dedicated=workers:NoSchedule`
Then:
- app manifests use node affinity for `app=true`
- worker manifests use node affinity and toleration for worker nodes
## 8. CPU spike reduction strategy
The CPU spike problem is usually not solved by "adding one backup node". It is solved by isolation, limits, and scaling.
### Do these first
1. Put workers on separate nodes.
2. Tighten worker CPU limits and requests based on real usage.
3. Reduce high default concurrency for the noisiest workers.
4. Add HPA or KEDA scaling from queue depth, not only CPU.
5. Ensure retries do not cause synchronized storms.
### Very likely spike sources in this repo
- high worker concurrency
- many worker StatefulSets sharing the same node
- retries against slow external APIs
- host-level proxying plus cluster-level routing mix
- single-node k3s carrying ingress, apps, workers, and maybe NATS responsibilities together
### Better autoscaling choice
For queue workers, KEDA is often better than plain HPA because it can scale on:
- NATS lag
- queue depth
- custom Prometheus metrics
That is usually more useful than only scaling from CPU percentage.
## 9. Concrete migration path
### Phase 1: stabilize current cluster
Do this before any big redesign:
- move secrets out of git
- standardize on ingress instead of many host nginx proxies
- remove unnecessary `NodePort` exposure where possible
- add resource dashboards and alerting
- capture actual CPU and memory usage for each worker
### Phase 2: separate worker nodes
- add a second or third node
- label and taint worker nodes
- move worker workloads there
- keep app services on separate nodes
This alone will already reduce blast radius from worker CPU spikes.
### Phase 3: make k3s highly available
- create `3` k3s server nodes
- use HA embedded etcd
- put ingress behind `kube-vip` or `MetalLB`
Now loss of one server does not mean cluster loss.
### Phase 4: make NATS highly available
- deploy NATS cluster inside Kubernetes
- enable JetStream replication
- use persistent volumes
- update workloads to connect to in-cluster NATS service
This replaces the current single external NATS dependency from [`manifests/core/core-config.yaml#L10`](E:/Birock/kubernetes/manifests/core/core-config.yaml#L10).
### Phase 5: add backup and restore
- install Velero
- schedule cluster backups
- schedule volume snapshots
- test restore into a fresh environment
If restore has never been tested, backup is not yet reliable.
## 10. What I would choose for you
Because you said you are a beginner, I would not jump straight into a very large platform.
I would choose this as the practical target:
- `3` k3s server nodes
- `2` worker nodes
- in-cluster Traefik
- `MetalLB` or `kube-vip`
- NATS HA cluster with JetStream replication
- Velero backups
- node separation for app plane and worker plane
This is modern, realistic, and still manageable.
## 11. What I would not recommend
I would avoid these patterns for your next version:
- one main Kubernetes node plus one cold backup node as the final design
- storing secrets directly in yaml in git
- relying on `NodePort` as the main production exposure pattern
- mixing host Docker Compose production routing with cluster routing long-term
- keeping NATS as a single external IP with no clear HA story
## 12. Immediate next actions for this repo
If we continue from this document, the best implementation order is:
1. Add node placement rules for workers and apps.
2. Convert external exposure to a single in-cluster ingress pattern.
3. Remove hard dependence on the single external NATS IP.
4. Add observability for CPU, memory, restart count, and queue lag.
5. Introduce backup tooling and test restore.
## 13. Summary in simple words
The clean professional version of your stack is:
- Kubernetes control plane on multiple HA server nodes
- app services on one node group
- NATS workers on another node group
- NATS itself running as a replicated cluster
- ingress and failover handled at cluster level, not manually by switching servers
- backups for both Kubernetes state and message data
That gives you what you want:
- less CPU blast radius
- better reliability
- faster failover
- cleaner separation of responsibilities
- a more modern production setup