$ cat "Case Study: Monolith Migration to Kubernetes with Zero-Downtime CI/CD.md"
DevOps Case Studies

Monolith to Kubernetes Migration: A Noxet Engineering Case Study

In this case study, Noxet infrastructure engineers share the technical methodology behind transitioning a high-traffic monolithic backend from bare-metal servers into Kubernetes orchestration. The project addressed classic enterprise challenges: lengthy manual deployment cycles, performance bottlenecks under traffic surges, and severe outage risks during service updates.

Baseline State and Architectural Bottlenecks

Prior to our modernization effort, the application ran as a monolithic service across two dedicated bare-metal servers managed via systemd and Nginx:

  • In-Memory Stateful Dependencies: User sessions were stored in process memory, forcing load balancers to rely on sticky sessions (IP/cookie hashing), resulting in uneven traffic distribution.
  • Local Disk File I/O: File uploads were saved directly to host storage, requiring fragile network file shares (NFS) to keep the nodes synchronized.
  • Manual Deployments: Releases took upwards of 3 hours, required scheduled late-night maintenance windows, and frequently incurred downtime due to service restart lag.
  • Absence of Elasticity: Traffic surges during marketing campaigns pushed CPU and memory usage to 100%, causing cascading 502/504 Bad Gateway errors.

Phase 1. 12-Factor Application Refactoring

Containerizing a monolith “as is” without adapting its internal architecture merely relocates technical debt to a new abstraction layer. We began with targeted architectural refactoring:

  1. Externalizing Session State (Stateless): Session data was migrated into an in-memory Redis Sentinel cluster. This allowed horizontally scaling application instances without interrupting active user authentication.
  2. Object Storage Adoption: Local disk file operations were replaced with an S3 API interface connected to an on-premise MinIO cluster.
  3. Graceful Shutdown: The service was refactored to intercept SIGTERM signals. Upon receipt, the pod ceases accepting new ingress connections, completes active transactions within a 15-second grace period, and exits cleanly.
  4. Structured JSON Telemetry: Log output was redirected to stdout/stderr in structured JSON format, enabling unified log streaming via Vector into VictoriaLogs / OpenSearch.

Phase 2. Container Build Optimization

Security compliance and container image size directly govern pod scheduling latency during scale-out events. We designed an optimized multi-stage Dockerfile:

# --- Build Stage (Builder) ---
FROM golang:1.22-alpine AS builder
WORKDIR /app
RUN apk add --no-cache git ca-certificates
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-w -s" -o /bin/server ./cmd/server

# --- Minimal Runtime Image ---
FROM gcr.io/distroless/static-debian12:nonroot
WORKDIR /app
COPY --from=builder /bin/server /app/server
USER nonroot:nonroot
EXPOSE 8080
ENTRYPOINT ["/app/server"]

Build Optimization Outcomes:

  • Final image size plummeted from 1.4 GB to 28 MB.
  • Complete removal of package managers, debug tools, and shell interpreters (sh, bash, curl) eliminated common lateral movement attack vectors.
  • Process runs under an unprivileged user context (nonroot:nonroot).

Phase 3. Kubernetes Deployment Architecture

To guarantee zero-downtime rolling updates and automated scaling, deployment manifests were standardized via Helm:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: core-api
  labels:
    app: core-api
spec:
  replicas: 4
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
  selector:
    matchLabels:
      app: core-api
  template:
    metadata:
      labels:
        app: core-api
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - name: app
          image: registry.noxet.ru/project/core-api:v2.4.1
          resources:
            requests:
              cpu: "300m"
              memory: "512Mi"
            limits:
              cpu: "1500m"
              memory: "1536Mi"
          readinessProbe:
            httpGet:
              path: /healthz/ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
            failureThreshold: 2
          livenessProbe:
            httpGet:
              path: /healthz/live
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 10

High-Availability Safeguards:

  • maxUnavailable: 0: Pre-existing pods remain operational until newly created pods pass their readinessProbe and successfully join the endpoint slice.
  • Horizontal Pod Autoscaling (HPA): Automatically scales the deployment between 4 and 24 replicas based on 70% CPU target utilization or custom RPS metrics.
  • PodDisruptionBudget (PDB): Enforces a minimum availability threshold of 75% operational pods during routine node maintenance and cluster upgrades.

Phase 4. Automated GitLab CI/CD Pipeline

Build, testing, and deployment workflows are automated via GitLab CI leveraging Kaniko (privileged socket-free container compilation):

stages:
  - test
  - build
  - deploy

run_tests:
  stage: test
  image: golang:1.22-alpine
  script:
    - go test -v -race ./...

build_image:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:v1.20.0-debug
    entrypoint: [""]
  script:
    - /kaniko/executor
        --context "${CI_PROJECT_DIR}"
        --dockerfile "${CI_PROJECT_DIR}/Dockerfile"
        --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHORT_SHA}"
        --destination "${CI_REGISTRY_IMAGE}:latest"
        --cache=true
  only:
    - main

deploy_production:
  stage: deploy
  image: alpine/helm:3.14.0
  script:
    - helm upgrade --install core-api ./helm \
        --namespace production \
        --set image.tag=${CI_COMMIT_SHORT_SHA} \
        --wait --timeout 5m
  environment:
    name: production
  only:
    - main

Measurable Infrastructure Outcomes

  1. Cycle Time Acceleration: Release pipeline duration dropped from 2–3 hours of manual intervention down to 6 minutes for the complete automated workflow.
  2. Zero Downtime Deployments: Leveraging RollingUpdate, readinessProbe, and preStop hooks eliminated dropped connections during application releases.
  3. Automated Elasticity: During promotional traffic surges, the cluster automatically expands replica pools within 90 seconds, maintaining p99 API latency under 45 ms.
  4. Enhanced Security Posture: Enforced network boundaries via Kubernetes NetworkPolicies and read-only container root filesystems.

Planning a migration to Kubernetes or seeking zero-downtime CI/CD architecture? Consult with Noxet infrastructure engineers.