Cloud Infrastructure

Kubernetes in Production: Security, Monitoring, and Cost Optimization

20 March 202516 min readBy Bayseian Engineering

Production-hardened Kubernetes deployment strategies covering service mesh, observability, auto-scaling, and infrastructure-as-code best practices.

Production Kubernetes: Beyond the Tutorial

Running Kubernetes in production is vastly different from local development. This guide covers the security, reliability, cost optimization, and operational practices that separate toy clusters from enterprise-grade infrastructure.

•Security hardening (RBAC, Pod Security Standards, network policies, secrets)
•Reliability (probes, disruption budgets, autoscaling)
•Monitoring and observability (metrics, logs, traces)
•Cost optimization (right-sizing, autoscaling, spot instances)

Security Hardening

A note on Pod Security: PodSecurityPolicy was removed from Kubernetes in v1.25. Its replacement is Pod Security Standards, enforced by the built-in Pod Security Admission controller. You label namespaces with the level you want (privileged, baseline, restricted) and the API server enforces it at admission time. For policies beyond what PSS covers (image registries, required labels, custom rules), add a policy engine such as Kyverno or OPA Gatekeeper.

On the networking side, eBPF-based CNIs, Cilium in particular, have become the production default on new clusters: network policy enforcement, service mesh capabilities and deep observability (Hubble) without sidecar overhead.

YAML

# Production Security Configuration

# 1. Pod Security Standards: enforced per namespace (PSP's replacement)
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

# 2. Pod-level securityContext: what 'restricted' expects
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: production
spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: api
          image: registry.internal/api:1.42.0
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]

# 3. Network Policies: zero-trust networking
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-network-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: production
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: database
      ports:
        - protocol: TCP
          port: 5432

# 4. RBAC: least-privilege access
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: production
  name: developer-role
rules:
  - apiGroups: ["", "apps", "batch"]
    resources: ["pods", "deployments", "jobs"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get"]
# Note: no delete, no secrets access

# 5. Secrets Management with External Secrets Operator
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: api-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: api-secrets
    creationPolicy: Owner
  data:
    - secretKey: database-url
      remoteRef:
        key: prod/api/database-url

# 6. Resource Limits: prevent noisy neighbors
apiVersion: v1
kind: LimitRange
metadata:
  name: resource-limits
  namespace: production
spec:
  limits:
    - max:
        cpu: "2"
        memory: 2Gi
      min:
        cpu: 100m
        memory: 128Mi
      default:
        cpu: 500m
        memory: 512Mi
      defaultRequest:
        cpu: 200m
        memory: 256Mi
      type: Container

Reliability: Probes, Budgets and Autoscaling

The outages that hurt aren't exotic. They're rollouts that route traffic to pods that aren't ready, node drains that take down every replica at once, and workloads that were never sized.

•Readiness gates traffic. It should check the things the pod needs to serve (DB connection, cache warm), and fail fast when they're gone.
•Liveness restarts stuck processes. Keep it dumb (process responsive?), and never point it at downstream dependencies or you convert a database blip into a cluster-wide restart storm.
•Startup probes protect slow-booting apps from liveness kills during initialisation.

PodDisruptionBudgets keep voluntary disruptions (node upgrades, cluster autoscaler scale-downs) from evicting too many replicas at once. Without one, a routine node rotation can take your service to zero.

Autoscaling layers: HPA for pods (CPU/memory or custom metrics), VPA in recommendation mode to inform right-sizing, and a cluster autoscaler (or Karpenter on EKS, which provisions right-sized nodes directly) for capacity.

YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: api
          resources:
            requests: { cpu: 250m, memory: 512Mi }
            limits: { memory: 512Mi }   # memory limit = request; no CPU limit (throttling hurts p99)
          readinessProbe:
            httpGet: { path: /ready, port: 8080 }
            periodSeconds: 5
            failureThreshold: 2
          livenessProbe:
            httpGet: { path: /healthz, port: 8080 }
            periodSeconds: 10
            failureThreshold: 3
          startupProbe:
            httpGet: { path: /healthz, port: 8080 }
            failureThreshold: 30      # up to 5 min to boot before liveness applies
            periodSeconds: 10
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: api
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: { type: Utilization, averageUtilization: 70 }

Monitoring and Observability

The Three Pillars of Observability:

1.Metrics (Prometheus + Grafana): the RED method (rate, errors, duration) for request-driven services and the USE method (utilization, saturation, errors) for resources catch different failure classes, so run both, not one.

2.Logs (Loki + Promtail): centralized, structured JSON logs across every pod, searchable by label instead of by SSH-ing into a node. Alert on error-pattern spikes, not just individual error lines. One bad deploy produces a pattern, not a blip.

3.Traces (Tempo + OpenTelemetry): distributed tracing turns "the API is slow" into "this specific downstream call is slow," mapping service dependencies and pinpointing the actual bottleneck instead of guessing from aggregate latency.

On Cilium clusters, Hubble adds a fourth lens: flow-level network observability (who talks to whom, dropped packets, policy denials) with no app changes.

•Cluster level: node CPU/memory and PVC usage catch capacity problems before they become scheduling failures.
•Application level: request rate, error rate, and latency at p50/p95/p99. p50 tells you the typical experience, p99 tells you who's actually suffering, and averages hide both.
•Business level: sign-ups, transactions, revenue tracked alongside infra metrics, so an incident review can answer "did this cost us anything" without a separate meeting.
•Cost: resource utilization, waste, and spot instance savings. Without this, cost optimization stays a quarterly fire drill instead of a standing signal.

•Only alert on actionable issues: every alert that fires and gets ignored trains the on-call engineer to ignore the next one too.
•Use runbooks for all alerts: the person paged at 3am shouldn't have to reconstruct the fix from memory.
•Escalation policies (PagerDuty, Opsgenie): a page that nobody acknowledges in N minutes should automatically reach someone who can.
•Alert grouping and deduplication: one root cause should page once, not fan out into fifty correlated alerts that bury the signal.

Cost Optimization

Kubernetes makes it easy to overspend invisibly, and requested-but-unused CPU is the classic leak. The discipline:

1. Right-size from data, not guesses. Deploy VPA in recommendation mode (or a tool like Goldilocks on top of it) and reconcile requests against actual usage monthly. Most clusters we audit run at 20-35% real utilisation before this exercise.

2. Spot/preemptible nodes for interruptible work. Batch jobs, workers and stateless replicas tolerate interruption; run them on spot node groups at a fraction of on-demand price, keep a small on-demand baseline for the critical path. Karpenter automates the mixing.

3. Namespace-level visibility. Deploy OpenCost/Kubecost so every team sees its own spend. Cost falls fastest when it's attributed.

4. Kill idle capacity. Dev/staging clusters that run nights and weekends, orphaned PVCs, and load balancers for deleted services are pure waste. Automate their cleanup.

5. Set LimitRanges and ResourceQuotas per namespace so one team's experiment can't silently double the node count.

## Conclusion Running Kubernetes in production requires careful attention to reliability, security, and operational best practices. While Kubernetes provides powerful primitives for container orchestration, production success depends on properly configuring health checks, resource management, security policies, and observability from day one. The essential practices for production Kubernetes: - Health checks: correctly separated liveness, readiness, and startup probes stop a routine rollout or node drain from turning into a cascading outage. - Resource management: requests, limits, and HPA sized from real usage data (not guesses) is what keeps utilization efficient and cost predictable. - Security hardening: RBAC, Pod Security Standards, network policies, and externalized secrets close the gaps that turn a compromised pod into a compromised cluster. - Comprehensive observability: metrics, logs, and traces together cut incident response from hours to minutes. Any one alone leaves blind spots. The difference between a Kubernetes cluster that "works" and one that's truly production-ready is often these operational details. Proper health checks prevent outages, resource limits prevent noisy neighbor problems, and good monitoring reduces mean time to resolution from hours to minutes. At Bayseian, we've operated production Kubernetes clusters for clients handling millions of requests per day, maintaining 99.9%+ uptime through careful implementation of these best practices. Our approach emphasizes infrastructure as code (using Helm and Terraform), automated testing, and comprehensive monitoring with Prometheus and Grafana. Whether you're migrating to Kubernetes or optimizing an existing cluster, these best practices provide a solid foundation for reliable, scalable production deployments. Start with the basics (health checks, resource limits, basic monitoring) and gradually layer in more sophisticated patterns as your needs grow. Ready to build production-grade Kubernetes infrastructure? Contact us at contact@bayseian.com to discuss your deployment strategy.

KubernetesDevOpsSecurityProductionInfrastructure

Cloud Infrastructure

Multi-Cloud Infrastructure as Code with Terraform: Lessons Learned

Best practices for managing infrastructure across AWS, GCP, and Azure using Terraform, including state management, modules, and CI/CD integration.

Working on something like this?

No pitch, just a practical conversation with the team that builds and operates these systems in production.

Start a conversation