Troubleshooting Kubernetes Pod Eviction in High-Traffic Clusters

Kubernetes cluster managing pod resources and eviction pressure

Kubernetes pod eviction in a high-traffic cluster is usually a protective action rather than a random application crash. When a node runs dangerously low on memory, local storage, filesystem inodes, or available process IDs, the kubelet may terminate selected pods to recover enough resources for the node to remain operational.

The word “eviction” is also used for several different Kubernetes behaviors. A pod may be removed because of node pressure, a planned node drain, a NoExecute taint, scheduler preemption, a vertical autoscaling operation, or another controller using the Eviction API.

These situations do not have the same cause or solution. Increasing memory limits will not solve an inode shortage. Adding a PodDisruptionBudget will not prevent node-pressure eviction. Scaling the Deployment may make the incident worse when the cluster has no node capacity for the additional pods.

The safest troubleshooting process begins by identifying who terminated the pod, which resource or policy triggered the action, which node was involved, and whether the replacement pod could be scheduled successfully.

This guide explains how to distinguish eviction from container restarts, investigate memory and disk pressure, understand Kubernetes eviction ordering, review autoscaling and scheduling, protect critical workloads, and build a repeatable incident-response process.

Operational scope The commands and configurations in this guide are illustrative. Kubernetes behavior, kubelet thresholds, filesystem layouts, autoscaling features, metrics, and administrative permissions vary by cluster version and managed provider. Review production changes in a safe environment and confirm current behavior in the official documentation for your platform.

Last reviewed: July 16, 2026. Kubernetes features, managed-cluster defaults, metrics, and autoscaling behavior may change. Confirm the final configuration against the current documentation for your cluster version and provider.

What Kubernetes Pod Eviction Means

Eviction is the termination of a pod because a Kubernetes component or authorized actor has decided that the pod should no longer remain on its current node.

During node-pressure eviction, the kubelet marks the selected pod as failed and terminates it. When the pod belongs to a Deployment, StatefulSet, or another workload controller, that controller normally creates a replacement pod.

The replacement may start on another node, return to the same node after pressure clears, or remain Pending when the cluster lacks suitable capacity.

Eviction Is Not the Same as a Container Restart

Symptom What Happened First Investigation
Pod reason is Evicted The entire pod was terminated because of a policy or resource condition Inspect the pod message, node conditions, events, and resource pressure
Container reason is OOMKilled A container was killed after exceeding its memory limit or during an operating-system OOM event Check memory limits, working-set history, heap behavior, and node-level OOM evidence
Pod remains Running but restarts increase One or more containers are restarting inside the same pod Inspect the previous container logs, exit code, probes, and restart policy
Pod is Pending The scheduler cannot currently place the pod Review scheduling events, requests, taints, affinity, quotas, and node capacity
Pod disappears during drain An administrator or automation requested a voluntary eviction Review drain activity, PodDisruptionBudgets, audit logs, and maintenance events

Identify Which Kind of Eviction Occurred

Before changing resource values, determine which Kubernetes mechanism initiated the disruption.

Eviction or Disruption Type Typical Cause Important Detail
Node-pressure eviction Low available memory, disk space, inodes, or process IDs Initiated by the kubelet and not prevented by a PodDisruptionBudget
API-initiated eviction Node drain, cluster maintenance, autoscaling controller, or administrator action Normally uses the Eviction API and can respect a PodDisruptionBudget
Scheduler preemption A higher-priority pending pod cannot schedule without removing lower-priority pods Related to Pod Priority rather than current node-pressure thresholds
Taint-based eviction A node receives a NoExecute taint that the pod does not tolerate A toleration may delay or prevent the eviction, depending on policy
Autoscaler-initiated replacement A vertical autoscaler or node-consolidation process needs to recreate the pod May be expected behavior rather than a capacity incident
Infrastructure interruption Node shutdown, spot-instance termination, hardware failure, or network loss May appear as a node failure or pod disappearance rather than an Evicted reason

Node-Pressure Eviction Signals

The kubelet monitors specific node resources and compares their available quantity with configured soft or hard eviction thresholds.

Signal Group Examples Possible Cause
Available memory memory.available Traffic growth, memory leaks, caches, sidecars, large workloads, or insufficient system reservation
Node filesystem space nodefs.available Container logs, local volumes, temporary files, dead containers, or host-level data
Image filesystem space imagefs.available Unused images, large images, rapid deployments, or failed image garbage collection
Container filesystem space containerfs.available where supported Writable layers, logs, and local container data
Filesystem inodes nodefs.inodesFree, imagefs.inodesFree Large numbers of small log, cache, session, or temporary files
Available process IDs pid.available on supported Linux nodes Process leaks, uncontrolled worker creation, or fork-heavy applications
CPU usage is not a direct node-pressure eviction signal High CPU usage can cause throttling, latency, failed probes, additional HPA replicas, and more node demand. It may contribute indirectly to instability, but the kubelet does not normally evict pods simply because node CPU utilization is high.

Soft and Hard Eviction Thresholds

A soft threshold must remain breached for a configured grace period before the kubelet begins eviction. A hard threshold has no grace period and may cause immediate pod termination.

An illustrative threshold expression is:

memory.available<1Gi
nodefs.available<10%
nodefs.inodesFree<5%

These are examples of syntax, not recommended universal values. Suitable thresholds depend on node size, workload density, traffic bursts, image size, recovery speed, and system-daemon requirements.

Before evicting application pods, the kubelet may attempt node-level reclamation such as removing dead containers and unused images. If this does not reduce pressure enough, it begins selecting pods for eviction.

How Kubernetes Selects Pods for Node-Pressure Eviction

The kubelet does not simply remove all BestEffort pods first and stop there. Its selection considers:

  1. Whether the pod’s usage of the constrained resource exceeds its request.
  2. The pod’s Priority.
  3. The pod’s usage relative to its request.

For resources without pod requests, such as process IDs and filesystem inodes, Priority has a larger role because Kubernetes cannot compare usage against a declared request.

QoS Classes Are an Indicator, Not the Complete Algorithm

QoS Class Common Configuration Eviction Interpretation
BestEffort No CPU or memory requests or limits Often highly exposed during memory pressure because any positive usage exceeds a zero request
Burstable Some requests or limits are set, but the pod does not meet Guaranteed requirements Risk depends on Priority and how far usage exceeds requests
Guaranteed CPU and memory requests equal limits for the relevant containers Normally protected while lower-priority or over-request candidates remain, but not immune to extreme node pressure

QoS is most useful as an approximate indicator under memory pressure. It does not directly determine eviction ordering for disk, inode, or PID pressure.

Step 1: Capture Evidence Before It Disappears

Evicted pods and Kubernetes events may be removed later. Collect evidence before deleting failed pods or restarting nodes.

List failed pods and their recorded reasons:

kubectl get pods -A \
  --field-selector=status.phase=Failed \
  -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,NODE:.spec.nodeName,REASON:.status.reason,MESSAGE:.status.message'

Inspect the affected pod:

kubectl describe pod POD_NAME -n NAMESPACE

kubectl get pod POD_NAME -n NAMESPACE -o yaml

Review recent events:

kubectl get events -A \
  --sort-by=.metadata.creationTimestamp

Record:

  • Pod name, namespace, and owning workload.
  • Node name.
  • Eviction reason and message.
  • Start and termination times.
  • Resource requests and limits for every container.
  • PriorityClass and QoS class.
  • Volumes and temporary-storage paths.
  • Recent Deployment, configuration, or traffic changes.

Step 2: Inspect the Affected Node

Describe the node that hosted the evicted pod:

kubectl describe node NODE_NAME

Focus on:

  • MemoryPressure
  • DiskPressure
  • PIDPressure
  • Ready
  • Node taints.
  • Capacity and Allocatable resources.
  • Allocated requests and limits.
  • Recent node events.

A concise view of node conditions can be retrieved with:

kubectl get node NODE_NAME \
  -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" reason="}{.reason}{"\n"}{end}'

Compare the node’s total Capacity with Allocatable. Allocatable represents the resources available for pods after Kubernetes and system reservations are considered.

Step 3: Review Historical Resource Usage

Current usage alone may not explain an eviction that occurred during an earlier traffic spike.

When the resource metrics API is available, use:

kubectl top node

kubectl top pod -A --containers

These commands provide recent CPU and memory usage but are not a replacement for historical monitoring. An evicted pod may no longer have current metrics.

Review historical charts for:

  • Node available memory and working set.
  • Pod and container memory.
  • CPU usage and throttling.
  • Filesystem capacity and available bytes.
  • Filesystem inode usage.
  • Container writable-layer usage.
  • Log volume.
  • Process count.
  • Pending pods.
  • Replica count and autoscaler activity.
  • Traffic volume and request concurrency.

Diagnosing Memory-Pressure Eviction

Memory pressure occurs when the node’s available memory crosses a kubelet eviction threshold.

Common causes include:

  • Requests that are much lower than normal peak usage.
  • Application memory leaks.
  • Large in-memory caches.
  • Traffic bursts that increase concurrent request state.
  • Sidecars or service-mesh proxies with underestimated memory use.
  • Batch jobs sharing nodes with interactive services.
  • System daemons consuming more memory than reserved.
  • Too many pods scheduled on the node.

Requests and Limits Serve Different Purposes

A memory request influences scheduling and establishes the workload’s expected reserved memory. A memory limit restricts how much memory the container may use.

If the request is unrealistically low, the scheduler may place too many pods on the node. If the limit is too low, the container may be OOMKilled during normal peaks.

Do not raise a memory limit without checking whether the node has enough Allocatable memory to support the new total usage.

Compare Every Container in the Pod

The pod’s resource profile includes:

  • Main application containers.
  • Sidecar containers.
  • Native sidecars where used.
  • Init-container scheduling requirements.
  • Runtime overhead.

A logging or service-mesh sidecar can become the largest memory consumer during high traffic even when the application itself remains stable.

Diagnosing Disk and Ephemeral-Storage Eviction

Local ephemeral storage can include:

  • Container writable layers.
  • Node-level container logs.
  • Non-memory-backed emptyDir volumes.
  • Temporary files.
  • Application caches.
  • Downloaded or generated files.

A memory-backed emptyDir using tmpfs is normally counted as memory usage rather than local ephemeral storage.

Common Disk-Pressure Causes

  • Access logs growing rapidly during traffic spikes.
  • Applications writing unbounded files to /tmp.
  • Large request or response caches.
  • Crash dumps.
  • Dead containers and unused images.
  • High deployment frequency with large container images.
  • File leaks that create many small files and exhaust inodes.
  • Sidecars buffering data locally during an external outage.

Configure Ephemeral-Storage Requests and Limits

An illustrative container resource definition is:

resources:
  requests:
    cpu: "500m"
    memory: "768Mi"
    ephemeral-storage: "1Gi"
  limits:
    cpu: "2"
    memory: "1Gi"
    ephemeral-storage: "2Gi"

The scheduler can use ephemeral-storage requests when selecting a node. If Kubernetes is correctly measuring local storage, a pod that exceeds its configured storage limit may be marked for eviction.

Storage quantities are case-sensitive. For example, 400m represents a fractional byte rather than 400 megabytes. Use values such as 400Mi or 400M intentionally.

Limit Temporary Volumes

A controlled temporary volume may use:

volumes:
  - name: temporary-data
    emptyDir:
      sizeLimit: 1Gi

An emptyDir limit is not a replacement for log rotation, cache eviction, or application cleanup. It only creates a boundary for the volume.

Diagnosing PID Pressure

Every process and thread consumes a process identifier. A node may enter PID pressure when workloads or system services create too many processes.

Possible causes include:

  • A process leak.
  • Unbounded worker creation.
  • A fork loop.
  • Each request launching a new operating-system process.
  • Large numbers of shell scripts or helper processes.
  • Sidecars creating unmanaged subprocesses.

PID pressure cannot be solved by increasing memory limits. Inspect process creation behavior, per-container process counts, node process limits, and application concurrency controls.

Investigating OOMKilled Containers

An OOMKilled container is not automatically an evicted pod.

Check the previous container state:

kubectl get pod POD_NAME -n NAMESPACE \
  -o jsonpath='{range .status.containerStatuses[*]}{.name}{" reason="}{.lastState.terminated.reason}{" exit="}{.lastState.terminated.exitCode}{"\n"}{end}'

Retrieve logs from the previous container instance:

kubectl logs POD_NAME -n NAMESPACE \
  -c CONTAINER_NAME --previous

Determine whether:

  • The container exceeded its memory limit.
  • The node reached an operating-system OOM condition before kubelet eviction.
  • A memory leak increased usage across time.
  • The application legitimately requires a higher peak limit.
  • The runtime heap is configured independently from the container limit.

Requests, Limits, and High-Traffic Behavior

Resource settings should be based on observed workload behavior across normal and peak periods.

Configuration Problem Possible Effect Safer Direction
Missing requests Scheduler has no realistic reservation signal and pods may be packed densely Add measured CPU, memory, and storage requests where relevant
Requests far below normal use Node appears schedulable while real consumption is already high Base requests on representative percentiles and peak patterns
Requests far above real use Pods remain Pending or node autoscaling becomes unnecessarily expensive Review historical utilization and workload guarantees
Memory limit too low Normal traffic causes repeated OOM kills Profile the application and increase only after confirming node capacity
CPU limit too low Throttling increases response time and may cause failed probes Measure throttling and burst requirements before changing limits
No storage boundaries Logs, caches, or temporary files consume the node filesystem Set storage requests and limits and control file lifecycle

Autoscaling Can Help or Worsen Eviction

Horizontal Pod Autoscaler

The Horizontal Pod Autoscaler increases or decreases replicas based on configured metrics.

When CPU or memory utilization is used, the calculation depends on resource requests. Missing or inaccurate requests can produce unreliable scaling behavior.

Scaling from five pods to twenty pods reduces per-pod traffic only when:

  • The load balancer distributes traffic appropriately.
  • The additional pods can be scheduled.
  • Dependencies such as databases can support the extra concurrency.
  • The workload is not constrained by one shared bottleneck.

If nodes are already full, additional replicas may remain Pending and provide no relief.

Node Autoscaling

A node autoscaler can provision additional nodes for unschedulable workloads. It usually reasons from pod requests and scheduling constraints rather than future real usage.

See also  Step-by-Step Guide to Migrating Legacy Monoliths to AWS Serverless

Review:

  • Node-provisioning delay.
  • Maximum node-group size.
  • Cloud quota and regional capacity.
  • Available instance types.
  • Taints and node selectors.
  • Zone and volume constraints.
  • Whether requests accurately represent demand.

Avoid an Autoscaling Race

During a sudden traffic increase:

  1. Existing pods use more memory.
  2. The HPA requests additional replicas.
  3. The scheduler attempts to place them on existing nodes.
  4. Nodes cross memory or disk thresholds.
  5. Existing pods are evicted before new nodes become ready.

Possible mitigations include adequate headroom, faster scaling signals, realistic requests, minimum replica capacity, scheduled scaling for predictable peaks, and node provisioning that begins before the cluster reaches pressure.

Scheduling and Workload Placement

Resource tuning alone may not help when important replicas are concentrated on one node or one zone.

Review:

  • Topology spread constraints.
  • Pod anti-affinity.
  • Node affinity and selectors.
  • Taints and tolerations.
  • Persistent-volume topology.
  • Priority classes.
  • DaemonSet overhead.
  • Maximum pods per node.

Illustrative Topology Spread Constraint

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: checkout-api

  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: checkout-api

This example must be adapted to the available zones, node labels, replica count, and desired scheduling behavior. Strict constraints can leave pods Pending when the cluster cannot satisfy them.

Pod Priority and Preemption

Priority helps Kubernetes decide which workloads are more important.

A PriorityClass may be appropriate for essential customer-facing services, infrastructure components, and less critical batch workloads.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: customer-api
value: 100000
globalDefault: false
description: Priority for critical customer-facing API workloads

The workload can reference it:

spec:
  template:
    spec:
      priorityClassName: customer-api
Do not give every workload high priority Priority is meaningful only when workloads are differentiated. Excessive use can cause important services to compete equally, allow one team to displace others, or leave low-priority workloads permanently unschedulable.

PodDisruptionBudgets: What They Protect

A PodDisruptionBudget controls how many selected replicas may be unavailable during voluntary evictions that use the Eviction API.

An illustrative PDB is:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-api
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: checkout-api

A PDB can help during:

  • Planned node drains.
  • Some node-autoscaler consolidations.
  • Administrative maintenance using the Eviction API.
  • Controllers that voluntarily evict pods.

A PDB does not prevent:

  • Node-pressure eviction.
  • Node hardware failure.
  • Operating-system crashes.
  • Network partition.
  • Container OOM kills.
  • Application Deployment rolling-update behavior.

A PDB requiring zero unavailable pods can block a node drain indefinitely. It should reflect the application’s real replica count, readiness behavior, and ability to tolerate planned disruption.

Taints, Tolerations, and Node Conditions

Kubernetes can map node conditions to taints, preventing new pods from scheduling onto unhealthy nodes.

Examples include:

  • node.kubernetes.io/memory-pressure
  • node.kubernetes.io/disk-pressure
  • node.kubernetes.io/pid-pressure
  • node.kubernetes.io/not-ready
  • node.kubernetes.io/unreachable

A NoSchedule taint prevents new pods without a matching toleration. A NoExecute taint may also evict already-running pods that do not tolerate it.

A toleration with tolerationSeconds can allow a pod to remain bound temporarily while a transient node problem is evaluated.

Do not broadly tolerate pressure taints simply to keep pods running. Doing so may place critical workloads on a node that is already unable to support them.

Reserve Resources for Kubernetes and the Operating System

The scheduler uses node Allocatable rather than assuming every byte of machine capacity is available to pods.

Cluster administrators can reserve resources for:

  • Kubernetes node components.
  • The operating system.
  • Logging and monitoring agents.
  • Container runtime processes.
  • Storage and networking daemons.

If system reservations are too small, system services may consume resources that the scheduler believed were available for workloads. This can create node pressure even when pods are not greatly exceeding their requests.

Managed Kubernetes services may restrict or abstract these kubelet settings. Use the provider’s documented node configuration rather than changing unmanaged flags blindly.

Worked Example: DiskPressure During a Traffic Spike

Consider an illustrative checkout API with eight replicas. Each pod contains:

  • The checkout application.
  • A service-mesh proxy.
  • A log-forwarding sidecar.
  • An emptyDir cache.

During a promotion campaign:

  1. Traffic increases rapidly.
  2. The application writes more access and diagnostic logs.
  3. The log destination becomes slow.
  4. The sidecar buffers logs locally.
  5. The application cache also grows.
  6. Several pods are concentrated on the same node.
  7. The node reaches a disk-space threshold.
  8. The kubelet garbage-collects unused images but cannot reclaim enough storage.
  9. Pods using large amounts of local storage are evicted.

Evidence Collected

Observation Meaning Action
Pod message reports ephemeral-storage pressure The issue is not primarily CPU or application memory Investigate logs, writable layers, and temporary volumes
Node reports DiskPressure The node crossed a filesystem threshold Remove the node from new scheduling and inspect storage
Log sidecar has no storage limit Local buffering can grow without a pod-level boundary Configure buffering, storage boundaries, and alerting
Several replicas occupy one node The incident removes a large portion of application capacity Add topology spreading and verify node headroom

Controlled Remediation

The team could:

  • Reduce or sample unnecessary logs.
  • Apply log rotation and bounded sidecar buffering.
  • Add measured ephemeral-storage requests and limits to every container.
  • Set an appropriate emptyDir size limit.
  • Spread replicas across nodes and zones.
  • Increase node storage only after controlling unbounded growth.
  • Alert on disk and inode trends before eviction thresholds are reached.
  • Test behavior when the remote logging service becomes slow.

Illustrative Deployment Configuration

The following fragment demonstrates several protective settings. The values are examples and must be based on actual measurements.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
spec:
  replicas: 8

  selector:
    matchLabels:
      app: checkout-api

  template:
    metadata:
      labels:
        app: checkout-api

    spec:
      priorityClassName: customer-api
      terminationGracePeriodSeconds: 30

      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app: checkout-api

      containers:
        - name: application
          image: registry.example.com/checkout@sha256:IMMUTABLE_DIGEST

          resources:
            requests:
              cpu: "500m"
              memory: "768Mi"
              ephemeral-storage: "1Gi"
            limits:
              cpu: "2"
              memory: "1Gi"
              ephemeral-storage: "2Gi"

          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            periodSeconds: 5
            timeoutSeconds: 2

          volumeMounts:
            - name: temporary-data
              mountPath: /tmp/application

        - name: log-forwarder
          image: registry.example.com/log-forwarder@sha256:IMMUTABLE_DIGEST

          resources:
            requests:
              cpu: "100m"
              memory: "128Mi"
              ephemeral-storage: "512Mi"
            limits:
              cpu: "500m"
              memory: "256Mi"
              ephemeral-storage: "1Gi"

      volumes:
        - name: temporary-data
          emptyDir:
            sizeLimit: 1Gi

Monitoring Evictions Before Users Notice

Do not alert only after pods have already been evicted. Monitor leading indicators.

Signal What It Reveals Possible Response
Available node memory Whether the node is approaching memory thresholds Investigate pod growth, system use, and scaling headroom
Filesystem available bytes Whether nodefs, imagefs, or containerfs is filling Inspect logs, images, writable layers, and local volumes
Free inodes Whether many small files are exhausting filesystem metadata Find file-producing workloads and clean bounded data
Pod usage versus request Which pods are likely to become eviction candidates Adjust requests or fix abnormal resource growth
Pending pods Whether replacements or HPA replicas cannot schedule Review scheduler events and node-autoscaling capacity
Eviction events by reason Whether incidents are concentrated by node, resource, workload, or region Apply a cause-specific runbook
User-facing availability Whether pod replacement maintains real service health Escalate when disruption exceeds the service objective

Incident-Response Sequence

Pod Eviction Investigation Flow
Confirm Eviction Type
Capture Pod Evidence
Inspect Node Pressure
Review Capacity and Placement
Apply One Controlled Fix
  1. Confirm the disruption type.

    Separate node pressure, OOMKilled containers, scheduler preemption, planned eviction, taint-based eviction, and infrastructure failure.

  2. Capture pod and node evidence.

    Save manifests, events, conditions, resource definitions, timestamps, and ownership information.

  3. Identify the constrained resource.

    Begin with the recorded eviction message rather than assuming the problem is memory.

  4. Review historical behavior.

    Compare the incident with traffic, deployments, autoscaling, logging, batch jobs, and node capacity.

  5. Stabilize the cluster.

    Reduce nonessential load, pause heavy jobs, add safe capacity, stop unbounded file growth, or remove an unhealthy node from service.

  6. Apply a cause-specific correction.

    Change requests, limits, scheduling, application behavior, or node configuration based on evidence.

  7. Verify under representative traffic.

    Confirm that node pressure, replacement scheduling, application latency, and user availability remain stable.

Common Mistakes That Make Evictions Worse

Mistake Possible Result Better Direction
Treating Evicted and OOMKilled as identical The team changes the wrong resource setting Inspect pod reason, container state, and node events separately
Assuming CPU caused node-pressure eviction Memory, storage, inode, or PID pressure remains untreated Begin with the recorded kubelet eviction signal
Increasing every memory limit Pods consume more memory and node pressure becomes worse Profile memory and check total node Allocatable capacity
Scaling replicas without node headroom New pods remain Pending or increase pressure Coordinate HPA, requests, and node autoscaling
Ignoring sidecars The largest resource consumer remains unconfigured Measure and configure every container in the pod
Ignoring inode usage Disk appears to have free bytes while the filesystem cannot create files Monitor both bytes and free inodes
Using a PDB as node-pressure protection Critical pods are still evicted under resource starvation Use capacity, requests, Priority, spreading, and monitoring
Adding broad tolerations Pods remain on nodes that cannot support them safely Tolerate only reviewed taints for a clear operational reason
Deleting failed pods before collecting evidence Eviction reason and node association are lost Export the pod status, events, and node data first

Production Readiness Checklist

Before the next traffic peak, confirm:
✓ Every container has reviewed resource requests
✓ Memory limits match measured peak behavior
✓ CPU throttling is monitored separately from eviction
✓ Ephemeral-storage usage and limits are reviewed
✓ Log and temporary-file growth is bounded
✓ Disk bytes and inode availability are monitored
✓ PID usage is observable where relevant
✓ Sidecars are included in capacity calculations
✓ Critical replicas are spread across nodes and zones
✓ Priority classes reflect real business importance
✓ PDBs are configured for planned disruption only
✓ HPA resource metrics have valid requests
✓ Node autoscaling can provision required capacity in time
✓ Kubernetes and system resources are reserved appropriately
✓ Eviction events are retained in centralized monitoring
✓ The incident runbook has been tested

When to Request Specialist Support

Involve an experienced Kubernetes, platform, cloud, storage, or application-performance professional when:

  • Evictions affect payment, identity, healthcare, or other critical services.
  • Nodes repeatedly enter pressure after resource tuning.
  • The cluster has enough apparent capacity but replacements remain Pending.
  • The kubelet reports pressure without an obvious workload source.
  • Managed-cluster eviction thresholds or node settings need modification.
  • Service-mesh, logging, security, or storage sidecars consume significant resources.
  • Autoscaling creates repeated oscillation between Pending pods and node pressure.
  • The workload uses local persistent state or specialized hardware.
  • Several teams share nodes with different priority and availability requirements.
  • The team cannot determine whether the incident was eviction, preemption, or infrastructure failure.

Conclusion

Troubleshooting Kubernetes pod eviction begins by identifying the exact disruption mechanism. Node-pressure eviction, OOMKilled containers, voluntary eviction, scheduler preemption, taint-based eviction, and infrastructure loss require different responses.

For node pressure, begin with the resource recorded by the kubelet. Inspect memory availability, local storage, filesystem inodes, and process IDs before changing workload limits.

Configure realistic requests, but do not assume requests and limits alone guarantee stability. Include sidecars, system services, temporary storage, traffic bursts, scheduling concentration, and autoscaling delays in the capacity model.

Use Priority and topology spreading to protect critical workloads, and use PodDisruptionBudgets for planned voluntary disruption. A PDB does not prevent resource-pressure eviction.

Finally, retain historical metrics and events. The most useful evidence often exists only during the incident. A tested runbook and early warning signals are more effective than increasing every resource value after pods have already been removed.

Frequently Asked Questions

What does Evicted mean in Kubernetes?
It means that Kubernetes terminated the entire pod because of a resource condition or eviction policy. Inspect the pod status message, events, node conditions, and audit or maintenance activity to determine which mechanism initiated it.
Is Evicted the same as OOMKilled?
No. Evicted normally describes termination of the complete pod. OOMKilled normally describes a container being killed because of memory pressure or a memory limit. The kubelet may restart an OOMKilled container inside the same pod.
Does high CPU usage cause pod eviction?
CPU saturation is not normally a direct node-pressure eviction signal. It can cause throttling, latency, failed probes, and more autoscaling demand. Node-pressure eviction focuses on resources such as available memory, filesystem space, inodes, and process IDs.
Does a PodDisruptionBudget prevent node-pressure eviction?
No. A PodDisruptionBudget protects selected workloads during voluntary evictions that use the Eviction API. The kubelet does not respect the PDB when it must reclaim resources during node-pressure eviction.
Are Guaranteed pods impossible to evict?
No. They are generally protected while lower-priority or over-request candidates are available, but severe resource pressure or excessive system-daemon usage can still require their eviction. Pod Priority also affects selection.
Why do replacement pods remain Pending after an eviction?
The cluster may lack sufficient requested resources or a suitable node. Taints, affinity, topology constraints, volume zones, quotas, unavailable instance types, and slow node autoscaling can also prevent scheduling.

Official References

Editorial note: The workloads, resource quantities, PriorityClass values, topology rules, commands, and incident scenarios in this guide are illustrative. Production configuration must be adapted to the application’s measured resource usage, cluster version, node type, managed provider, availability requirements, security controls, and operational procedures.