What actually happens when you run a container?

Namespaces — "What You Can See"

A namespace wraps a global resource and presents processes inside it with an illusion they have their own isolated instance. The key syscall is clone(CLONE_NEWPID | CLONE_NEWNET | ...).

When PID namespace is created: the first process inside it gets PID 1 — even though the host kernel sees it as PID 7823. Two different realities, same machine. That's the trick.

cgroups v2 — "What You Can Use"

Control groups enforce resource accounting and limits. The cgroup hierarchy is exposed via a virtual filesystem at /sys/fs/cgroup/. Every container gets its own subtree.

# Find container's cgroup path
cat /proc/$(docker inspect \
  --format='{{.State.Pid}}' nginx)/cgroup

# Read memory limit directly
cat /sys/fs/cgroup/system.slice/\
  docker-<id>.scope/memory.max
# → 268435456 (256MB)

① kubectl sends HTTP PATCH to kube-apiserver

kubectl serializes your YAML to JSON, sends a REST request. The API server runs: Authentication → Authorization (RBAC) → Admission controllers (webhooks can mutate/validate) → Validation against OpenAPI schema → Persist to etcd.

② Deployment Controller wakes up

The Deployment controller in controller-manager has a watch on Deployment resources. etcd sends a watch event. Controller reconciles: desired replicas = 3, current ReplicaSets = 0. Creates a new ReplicaSet with pod template + podTemplateHash label.

③ ReplicaSet Controller creates Pod objects

ReplicaSet controller sees it owns 0 pods but wants 3. Creates 3 Pod objects in etcd with status.phase: Pending and no spec.nodeName. These pods don't exist anywhere yet — they're just records in a database.

④ Scheduler binds pods to nodes

Scheduler watches for unscheduled pods. Runs filter plugins (nodeSelector, taints, resource availability), score plugins (spreading, affinity), picks winner, writes spec.nodeName: worker-2 back to etcd.

⑤ kubelet on worker-2 notices its pod

kubelet watches pods assigned to its node. Calls containerd CRI gRPC API: RunPodSandbox (create pause container + network namespace), PullImage, CreateContainer, StartContainer. Updates pod status in etcd.

⑥ CNI plugin sets up networking

kubelet calls the CNI plugin (Calico/Cilium/Flannel) to assign an IP from the pod CIDR, set up veth pairs, configure routing. Now the pod has an IP and can communicate with the cluster network.

⑦ Pod is Running — Endpoints updated

Endpoint controller detects the pod is ready (readinessProbe passes), adds its IP:port to the Service's Endpoints object. kube-proxy programs iptables rules. Traffic can now reach the pod.

The Pause Container — Why does every pod have a mystery container?

etcd Watch API — How controllers get notified without polling

Services, ClusterIP and iptables — The magic of virtual IPs

Rolling Updates — Zero-downtime deployment mechanics

GitOps — Git as Source of Truth

In traditional CD, the pipeline runs kubectl apply directly. If someone does kubectl edit manually, drift is invisible.

GitOps flips this: ArgoCD runs inside the cluster and continuously watches a Git repo. If the cluster drifts from Git, the controller reconciles back. Git is the only way to change production.

Observability: The Three Pillars

# API error rate > 1% over 5 minutes → PagerDuty
alert: HighErrorRate
expr: |
  sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
  /
  sum(rate(http_requests_total[5m])) by (service)
  > 0.01
for: 2m  # must be true for 2 min (avoid flaps)
labels:
  severity: critical

# p95 latency degradation
alert: SlowRequests
expr: |
  histogram_quantile(0.95,
    sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
  ) > 0.5  # 500ms SLO