Troubleshooting Docker Container Memory Leaks in Production

Troubleshooting Docker container memory leaks in production begins by determining whether the container is actually leaking memory. A rising graph can also represent a growing workload, filesystem cache, native allocations, buffered requests, allocator fragmentation, additional threads, mapped files, or memory that the runtime has retained for future reuse.

Docker does not create a separate memory-management system for the application. On Linux, the container’s processes are accounted for and constrained through control groups, while the application runtime continues managing its own heap, native libraries, threads, sockets, and memory-mapped regions.

This creates several layers of evidence. An application heap can remain stable while container memory grows because of native allocations. A process RSS value can remain high after objects are freed because the allocator has not returned pages to the operating system. A container metric can include file-backed or kernel memory that is not part of the managed-language heap.

The correct investigation therefore compares container-level metrics, cgroup accounting, process memory, application-runtime telemetry, workload volume, garbage-collection behavior, and deployment history before changing resource limits or restarting the service.

This guide explains how to distinguish a real leak from normal memory behavior, preserve evidence before an OOM event, inspect Docker and cgroup metrics, profile common application runtimes, configure protective limits, deploy a correction safely, and prevent repeated production incidents.

Last reviewed: July 16, 2026. Docker Engine, Linux cgroups, application runtimes, container metrics, and profiler behavior may change. Confirm the final procedure against the current documentation for your operating system and runtime.

What a Memory Leak Actually Is

A memory leak occurs when an application continues retaining memory that is no longer required for useful work.

Examples include:

• Objects that remain reachable because they were added to a global collection and never removed.
• Caches without a size limit or expiration policy.
• Event listeners, callbacks, subscriptions, or timers that are never released.
• Requests or messages retained after processing has finished.
• Threads, goroutines, subprocesses, or tasks that never terminate.
• Native-library allocations that are not freed.
• Database result sets, file buffers, or network buffers held indefinitely.
• Memory-mapped files that remain referenced.

The important characteristic is not simply that memory increased. It is that the application’s retained memory continues increasing relative to comparable workload cycles and does not return to a stable operating range.

Memory Growth That Is Not Necessarily a Leak

Pattern	Possible Explanation	How to Investigate
Memory rises with traffic and later stabilizes	Expected working-set growth, connection pools, caches, or additional concurrency	Compare equal traffic periods and confirm that memory reaches a repeatable plateau
Managed heap falls but RSS stays high	Allocator retention, fragmentation, native allocations, stacks, or mapped memory	Compare heap telemetry with process RSS, cgroup memory, native profiling, and allocation patterns
Container memory grows during heavy file access	Filesystem cache or memory-mapped files	Review the file-related fields in cgroup memory statistics
Memory grows while a dependency is slow	Requests, messages, retries, or response bodies are accumulating in memory	Measure active requests, queue depth, buffer sizes, deadlines, and dependency latency
Memory rises during startup and remains stable	Runtime initialization, class loading, compiled code, connection pools, or warmed caches	Establish a normal post-start baseline before evaluating later growth
Memory increases after every deployment	New code, dependency changes, changed runtime flags, higher traffic, or larger caches	Compare versions under equivalent traffic and configuration

Understand the Different Memory Measurements

There is no single metric that completely explains a container’s memory use.

Measurement	What It Represents	Important Limitation
Managed heap	Objects managed by a runtime such as the JVM, V8, Python, or Go	Does not necessarily include native libraries, stacks, sockets, mapped files, or all runtime overhead
RSS	Physical pages currently resident for a process	Shared pages and allocator behavior can make totals difficult to interpret across processes
Virtual memory size	Address space reserved or mapped by the process	A large virtual size does not mean all of that memory is physically resident
Docker CLI memory usage	A container-oriented view based on cgroup accounting	On Linux, the CLI subtracts inactive file cache from the total, so it may differ from the Engine API
cgroup memory.current	Total memory currently charged to a cgroup v2 hierarchy	It combines several memory categories and must be interpreted with memory.stat
Host available memory	Memory available across the complete machine	Host pressure may involve several containers and system services rather than one application

Step 1: Stabilize the Service Without Destroying Evidence

When production memory is close to a limit, reliability comes first. However, immediately restarting every container can erase the evidence needed to identify the cause.

A controlled response may include:

• Remove one affected replica from the load balancer.
• Keep healthy replicas serving traffic.
• Pause noncritical batch work.
• Reduce incoming concurrency if requests are accumulating.
• Capture runtime and cgroup evidence from the isolated replica.
• Restart or replace the replica after evidence collection.
• Scale additional clean replicas only when the host or cluster has safe capacity.

Do not attach a heavy profiler to the only healthy production instance unless the risk and recovery path are understood.

Record the Incident Context

Before changing the system, record:

• Container name and ID.
• Image digest and application version.
• Host name and environment.
• Container start time.
• Current and peak memory.
• Configured memory and swap limits.
• Recent restart count.
• Traffic and concurrency.
• Queue depth and dependency latency.
• Recent configuration or dependency changes.
• Whether the container was OOM-killed.

Step 2: Establish a Time-Based Memory Pattern

A memory leak is normally diagnosed through behavior over time rather than one measurement.

Compare memory against:

• Requests per second.
• Active requests.
• Batch sizes.
• Connected clients.
• Queue depth.
• Cache entries.
• Loaded tenants or datasets.
• Garbage-collection cycles.
• Application uptime.

Look for a Rising Post-Garbage-Collection Baseline

For garbage-collected applications, one useful signal is the amount of live memory remaining after comparable full or major collection cycles.

If the post-collection baseline rises continuously under equivalent workload, the process may be retaining additional live objects.

However, a stable managed heap does not rule out:

• Native-memory leaks.
• Thread-stack growth.
• Direct or off-heap buffers.
• Memory-mapped files.
• Allocator fragmentation.
• Kernel or socket memory.

Step 3: Inspect Docker Runtime Metrics

Start with a live overview:

docker stats

Inspect selected containers only:

docker stats api-worker payment-worker

Capture one non-streaming sample:

docker stats --no-stream \
  --format 'table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}\t{{.PIDs}}\t{{.BlockIO}}'

Review:

• Memory usage and configured limit.
• Memory percentage.
• Process and thread count shown in the PIDS column.
• CPU use and throttling indicators from other monitoring.
• Block I/O and network activity.

A rising PIDS value can indicate increasing thread or process creation. It is not proof of a memory leak, but it may explain growing stack and kernel memory.

Understand the Docker CLI Cache Adjustment

On Linux, the Docker CLI subtracts inactive file cache from the memory total it displays. The Docker Engine statistics API exposes the total and cache-related values separately.

Consequently:

• A monitoring platform using the Engine API may display a different number from the CLI.
• Two dashboards may disagree while both are reading valid but differently adjusted metrics.
• A memory alert must document which definition it uses.

Step 4: Inspect Container State and OOM Evidence

Inspect the recorded container state:

docker inspect \
  --format 'pid={{.State.Pid}} oom={{.State.OOMKilled}} exit={{.State.ExitCode}} started={{.State.StartedAt}} finished={{.State.FinishedAt}} error={{json .State.Error}}' \
  CONTAINER_NAME

Also inspect the resource configuration:

docker inspect \
  --format 'memory={{.HostConfig.Memory}} reservation={{.HostConfig.MemoryReservation}} swap={{.HostConfig.MemorySwap}} restart={{.HostConfig.RestartPolicy.Name}}' \
  CONTAINER_NAME

Exit Code 137 Is Not Proof of an OOM Kill

An exit code of 137 commonly means that the process ended after receiving SIGKILL. An OOM kill can produce this result, but so can:

• A manual docker kill.
• An orchestrator forcibly terminating the process.
• A shutdown that exceeded its grace period.
• Another privileged process sending SIGKILL.

Stronger evidence includes:

• The Docker OOMKilled state.
• cgroup memory.events counters.
• Kernel OOM messages.
• Host or orchestrator events.
• A memory graph reaching the configured hard limit.

Step 5: Determine the cgroup Version

Check whether the host uses cgroup v2:

if [ -f /sys/fs/cgroup/cgroup.controllers ]; then
  echo "cgroup v2"
else
  echo "cgroup v1 or hybrid layout"
fi

Do not assume that every Docker host uses this hard-coded path:

/sys/fs/cgroup/memory/docker/CONTAINER_ID/memory.usage_in_bytes

The real layout depends on:

• cgroup version.
• systemd integration.
• Docker configuration.
• Rootless Docker.
• Parent cgroups.
• The operating system.

Step 6: Locate and Inspect the cgroup

Get the container’s host process ID:

PID=$(docker inspect --format '{{.State.Pid}}' CONTAINER_NAME)

echo "$PID"

cat "/proc/$PID/cgroup"

On a typical cgroup v2 host, derive the relative path:

CGROUP_PATH=$(awk -F: '$1 == "0" { print $3 }' "/proc/$PID/cgroup")
CGROUP_DIR="/sys/fs/cgroup${CGROUP_PATH}"

echo "$CGROUP_DIR"

test -d "$CGROUP_DIR" && echo "cgroup directory found"

Administrative permissions may be required to read all files.

Read the Main cgroup v2 Memory Files

cat "$CGROUP_DIR/memory.current"
cat "$CGROUP_DIR/memory.peak"
cat "$CGROUP_DIR/memory.max"
cat "$CGROUP_DIR/memory.high"
cat "$CGROUP_DIR/memory.swap.current"
cat "$CGROUP_DIR/memory.swap.max"

The values are normally expressed in bytes. A limit may contain the value max when no finite boundary is configured.

Inspect Memory Categories

grep -E '^(anon|file|kernel|kernel_stack|pagetables|sock|shmem|slab) ' \
  "$CGROUP_DIR/memory.stat"

Field	General Meaning	Possible Investigation
anon	Anonymous mappings commonly associated with application heaps and native allocations	Compare with runtime heap, native profilers, queues, and allocator behavior
file	Cached filesystem data, including some shared-memory use	Review file access, mmap use, temporary files, and cache reclaim behavior
kernel	Kernel memory charged to the cgroup	Inspect threads, sockets, page tables, slab use, and process count
sock	Network transmission buffers	Review active connections, slow clients, buffering, and backpressure
kernel_stack	Memory used by kernel stacks	Check whether thread or process counts are increasing
pagetables	Memory used to map virtual addresses	Review very large address spaces, mappings, process count, and allocation patterns

Inspect Pressure and OOM Events

cat "$CGROUP_DIR/memory.events"
cat "$CGROUP_DIR/memory.events.local"

Relevant counters may include:

• high: allocations encountered the high boundary and reclaim pressure.
• max: usage approached or exceeded the hard maximum.
• oom: allocation reached an OOM condition.
• oom_kill: a process in the cgroup was killed by an OOM killer.

Record these counters over time rather than checking only after the container has already been replaced.

Step 7: Inspect Process Memory

Use the host PID obtained from Docker:

cat "/proc/$PID/status"

Useful values can include:

• VmRSS
• VmSize
• VmSwap
• Threads
• RssAnon
• RssFile
• RssShmem

When available, view an aggregated mapping breakdown:

cat "/proc/$PID/smaps_rollup"

For a mapping-by-mapping investigation:

pmap -x "$PID"

Tool availability and permissions vary by image and host. Prefer host-side diagnostics or a controlled debugging environment rather than installing arbitrary tools into a running production container.

Step 8: Identify the Responsible Process

A container can run more than one process. Confirm which process is consuming memory.

docker top CONTAINER_NAME -eo pid,ppid,comm,rss,vsz,args

From the host, inspect processes inside the container’s PID namespace or cgroup using the approved administration tools.

Look for:

• An unexpected child process.
• Repeatedly created workers.
• Shell processes that never exit.
• Growing thread count.
• A sidecar or helper process consuming more memory than the primary application.
• Defunct processes indicating inadequate process reaping.

Use an Init Process Where Appropriate

When the main application does not correctly reap child processes, Docker can run a minimal init process with:

docker run --init IMAGE_NAME

This can help manage child-process lifecycle, but it does not fix memory retained inside the application.

Step 9: Compare Managed Heap with Container Memory

Observed Pattern	Likely Direction	Next Evidence
Heap and cgroup memory rise together	Managed objects, workload growth, or managed cache	Heap profiles, object-retention paths, cache size, and queue depth
Heap is stable but anonymous memory rises	Native allocations, direct buffers, runtime overhead, or allocator fragmentation	Native-memory tools, runtime-specific native metrics, and process mappings
File memory rises	Page cache, mapped files, shared memory, or tmpfs	File and mmap activity, temporary volumes, and reclaim behavior
Kernel or socket memory rises	Connections, network buffers, threads, or kernel structures	Connection count, slow clients, backpressure, thread count, and socket statistics
Heap falls after GC but RSS falls slowly or not at all	Runtime or allocator retains committed pages for reuse	Repeated workload cycles, allocator metrics, fragmentation, and long-term RSS plateau

Step 10: Use the Correct Application Profiler

Container metrics identify where memory is charged. Application profilers help identify which allocations or objects are responsible.

Runtime	Useful Official Capability	Important Limitation
Node.js	V8 heap snapshots, allocation profiling, GC traces, and runtime memory statistics	A heap snapshot can pause the event loop and require substantial additional memory
Java	Java Flight Recorder, JDK Mission Control, GC logs, class histograms, and heap dumps	Heap analysis does not automatically explain every native, direct-buffer, thread, or mapped-memory allocation
Python	tracemalloc snapshots and allocation differences	Python-level tracing may not account for all memory allocated inside native extensions
Go	pprof heap, allocation, goroutine, and thread-creation profiles	Go heap profiles do not automatically explain all cgo or external native allocations

Node.js Memory Investigation

Compare:

• heapUsed
• heapTotal
• external
• arrayBuffers
• rss

An illustrative diagnostic endpoint or periodic metric can use:

const usage = process.memoryUsage();

console.log({
  rss: usage.rss,
  heapTotal: usage.heapTotal,
  heapUsed: usage.heapUsed,
  external: usage.external,
  arrayBuffers: usage.arrayBuffers
});

Interpret the pattern:

• Growing heapUsed after comparable garbage collections can indicate retained JavaScript objects.
• Stable heap with growing external or arrayBuffers may indicate buffers or native-backed allocations.
• Stable managed values with growing RSS may require native-memory and allocator investigation.

Capture Heap Snapshots Carefully

A heap snapshot can be compared at two points in time to identify growing object types and retaining paths.

Do not capture it casually on a heavily loaded instance. Snapshot creation can:

• Pause JavaScript execution.
• Require memory in addition to the existing heap.
• Cause the process to exceed its container limit.
• Create a large file containing sensitive application data.

Prefer:

• An isolated production replica.
• A staging environment reproducing the same workload.
• A preconfigured secure diagnostic method.
• Encrypted and access-controlled storage for the snapshot.

Java Memory Investigation

Compare:

• Heap used before and after major collections.
• Old-generation or long-lived object growth.
• Garbage-collection frequency and pause duration.
• Metaspace.
• Direct buffer pools.
• Thread count and stack memory.
• Native-memory tracking where enabled.
• Container memory compared with JVM memory categories.

Java Flight Recorder and JDK Mission Control can help identify object types whose live population grows over time.

A heap dump may contain:

• Credentials.
• Customer information.
• Request bodies.
• Session data.
• Application secrets.

Handle heap dumps as sensitive production data.

Leave Room Outside the Java Heap

Do not configure the maximum Java heap equal to the complete container memory limit.

The container also needs memory for:

• Metaspace.
• Code cache.
• Thread stacks.
• Direct buffers.
• JNI and native libraries.
• Garbage-collector structures.
• Shared libraries and process overhead.
• Kernel and socket memory charged to the cgroup.

Python Memory Investigation

Python’s tracemalloc module can compare allocation snapshots:

import tracemalloc

tracemalloc.start()

before = tracemalloc.take_snapshot()

# Run a representative workload.

after = tracemalloc.take_snapshot()

for statistic in after.compare_to(before, "lineno")[:20]:
    print(statistic)

This can identify Python allocation sites that grew between snapshots.

If container memory rises while traced Python allocations remain stable, investigate:

• Native extensions.
• Image or data-processing libraries.
• Database drivers.
• Subprocesses.
• Memory-mapped files.
• Allocator fragmentation.
• Large temporary buffers.

Go Memory Investigation

Go applications can expose controlled profiling data through pprof or write profiles to files.

Useful profiles include:

• Heap.
• Allocations.
• Goroutines.
• Thread creation.
• Blocking and mutex contention where relevant.

A growing goroutine count can retain stacks, request state, channels, timers, and referenced objects even when individual goroutines appear small.

Review:

• Live heap after garbage collection.
• Total allocated bytes.
• Goroutine count.
• Objects retained by queues and channels.
• cgo allocations.
• Buffers retained in pools.

Do not expose profiling endpoints publicly. Restrict them to an approved administrative path.

Common Application-Level Causes

Cause	Typical Signal	Correction Direction
Unbounded cache	Entry count and heap rise with distinct keys or tenants	Define maximum size, expiration, eviction, ownership, and observability
Listener or subscription leak	Listeners increase after reconnects, requests, or deployments	Unsubscribe during lifecycle cleanup and test repeated creation and destruction
Queue without backpressure	Memory increases when a dependency or consumer becomes slow	Bound the queue, reject or persist excess work, and apply deadlines
Thread or goroutine leak	Thread, task, or goroutine count rises with uptime	Add cancellation, completion, timeouts, and lifecycle ownership
Large request buffering	Memory spikes with uploads, exports, or large API responses	Stream data, bound request size, and limit concurrent large operations
Native allocation leak	Container and RSS grow while managed heap remains stable	Use native profiling and review external libraries and buffers
Connection leak	Open connections, sockets, buffers, and pending operations increase	Close resources deterministically and configure pool limits and deadlines

Backpressure and Dependency Failures

Some apparent leaks are caused by work entering the process faster than it can leave.

For example:

1. An external API becomes slow.
2. Requests remain active longer.
3. The application stores response buffers and request context in memory.
4. Retries create additional requests.
5. Memory grows until the dependency recovers or the container is killed.

The correct fix may involve:

• Per-request deadlines.
• Bounded concurrency.
• Bounded queues.
• Retry limits.
• Backoff with jitter.
• Streaming instead of full buffering.
• Load shedding.
• Durable external queues for background work.

Increasing the container limit without controlling admission may only allow a larger backlog to accumulate.

Allocator Fragmentation and Memory Retention

An application can free objects without immediately reducing RSS.

Possible reasons include:

• The runtime keeps memory committed for later reuse.
• Free blocks are distributed in a way that prevents large pages from being returned.
• Different allocation sizes create fragmentation.
• Several threads maintain separate allocator arenas or caches.
• A process touched pages that remain resident until the kernel reclaims them.

This pattern differs from a classic object-retention leak.

Evidence may show:

• The managed live heap returns to a stable baseline.
• Application-level object counts remain stable.
• RSS reaches a higher plateau but eventually stops growing.
• Memory is reused during a later workload cycle without equivalent new RSS growth.

Before replacing allocators or changing low-level runtime settings, reproduce the behavior under representative load and confirm that the change improves the complete service rather than only one metric.

Step 11: Configure Memory Limits as Guardrails

By default, a Docker container may use memory allowed by the host unless a constraint is configured.

A hard limit protects the host and neighboring workloads:

docker run \
  --name example-api \
  --memory=768m \
  --memory-reservation=512m \
  --memory-swap=768m \
  IMAGE_NAME

In this example:

• --memory=768m defines the hard container memory limit.
• --memory-reservation=512m defines a softer reservation boundary used during contention.
• Setting --memory-swap to the same value as --memory prevents additional swap use.

The values are illustrative. Measure the application’s normal and peak working set before selecting a limit.

Do Not Set the Limit Equal to the Normal Peak

Include headroom for:

• Runtime overhead.
• Garbage-collection behavior.
• Temporary allocations.
• Traffic bursts.
• Thread stacks.
• Direct and native buffers.
• Diagnostic collection.
• Kernel and socket memory charged to the cgroup.

Excessively loose limits reduce protection. Excessively tight limits create avoidable OOM kills and can prevent useful profiling.

Docker Compose Memory Example

services:
  api:
    image: registry.example.com/api:APPROVED_VERSION

    mem_limit: 768m
    mem_reservation: 512m
    memswap_limit: 768m

    init: true
    restart: "on-failure:3"
    stop_grace_period: 30s

    logging:
      driver: local
      options:
        max-size: "10m"
        max-file: "3"

This example provides a resource boundary and limited restart behavior. It does not repair retained memory.

Recreate the Container After Configuration Changes

Running only:

docker compose restart

does not necessarily apply modified Compose configuration. Recreate the service through the approved deployment process so the new resource settings are used.

Swap Is Not a Leak Fix

Swap can provide additional protection against immediate memory exhaustion, but frequent swapping can substantially increase latency.

Consider:

• Whether the host has swap configured.
• The application’s latency requirements.
• Whether memory contains sensitive information.
• Host-level contention.
• Whether swap hides an undersized host or leaking workload.

Do not assume that disabling swap is always correct or that enabling unlimited swap is safe. Test the behavior for the actual workload and operating environment.

Restart Policies Are Recovery Controls

A restart policy can restore a failed replica, but it must not become the permanent memory-management strategy.

Docker supports policies such as:

• no
• on-failure[:max-retries]
• always
• unless-stopped

A limited policy may look like:

docker run \
  --restart=on-failure:3 \
  IMAGE_NAME

Why Endless Restarts Are Dangerous

• They erase evidence before it is collected.
• They can create a rapid crash loop.
• They may repeatedly overload a dependency during startup.
• They hide customer impact behind a changing container ID.
• They do not prevent the leak from returning.
• They can make monitoring look like many short incidents instead of one persistent defect.

Alert on restart frequency and preserve the reason for every termination.

Log Growth Is a Separate Resource Problem

Container logs primarily consume disk, not the application’s managed heap. However, logging can also contribute to memory pressure when:

• The application buffers logs before writing them.
• A logging library has an unbounded queue.
• A remote log destination becomes slow.
• A sidecar buffers data in memory.
• Very large log records create temporary allocations.

Docker’s default json-file logging driver does not perform log rotation unless configured. For suitable environments, Docker recommends the local driver, which performs rotation by default.

Log rotation protects disk space but does not fix an in-process logging queue or buffer leak.

Step 12: Reproduce the Pattern Safely

A useful reproduction should reflect the production condition that causes memory growth.

Include:

• Representative request payloads.
• Similar concurrency.
• Comparable dependency latency.
• The same runtime and application flags.
• The same container memory limit.
• The same cache configuration.
• The same image architecture.
• A sufficiently long test duration.

Use Repeated Workload Cycles

A practical test may follow:

1. Start a clean container.
2. Wait for normal initialization.
3. Record baseline heap, RSS, cgroup memory, threads, and cache entries.
4. Run a representative workload.
5. Allow queues to drain and garbage collection to occur naturally.
6. Record the new baseline.
7. Repeat several cycles.
8. Compare retained memory and object profiles.

A real leak often becomes clearer when each equivalent cycle leaves behind additional retained state.

Step 13: Correct the Root Cause

The correction should match the evidence.

Finding	Weak Response	Better Correction
Unbounded cache	Increase the memory limit	Add maximum size, expiration, eviction, metrics, and ownership
Requests accumulate during dependency failure	Add more application replicas without limits	Add deadlines, bounded concurrency, backpressure, and controlled retries
Thread leak	Restart every few hours	Correct cancellation, lifecycle cleanup, and worker ownership
Large upload buffering	Raise the heap size	Stream the body and limit request size and concurrent uploads
Native-library leak	Tune managed garbage collection	Upgrade, replace, configure, or correct the native component
Allocator reaches a stable high plateau	Classify it immediately as a leak	Verify reuse, fragmentation, latency, and long-term stability before low-level tuning

Step 14: Deploy the Correction Gradually

Memory corrections can introduce new reliability problems. For example, a smaller cache may increase database load, a lower concurrency limit may reduce throughput, and a new allocator setting may affect latency.

A controlled deployment should:

1. Build one reviewed image.
2. Deploy it to a small number of replicas.
3. Keep the previous version available for rollback.
4. Compare memory under equivalent traffic.
5. Monitor latency, errors, throughput, CPU, garbage collection, and dependency load.
6. Expand only after the new memory baseline is stable.

Define Rollback Conditions

Rollback may be required when the correction causes:

• Higher error rates.
• Unacceptable latency.
• Reduced throughput.
• Excessive garbage collection.
• Database or dependency overload.
• A different memory-growth pattern.
• Repeated container restarts.

Worked Example: A Node.js Session Cache Leak

Consider an illustrative API running with a 768 MiB container limit.

The service shows this pattern:

• Memory starts near 250 MiB.
• Traffic remains within its normal range.
• Container memory increases by approximately 40 MiB each hour.
• Garbage collection reduces temporary allocations but the post-GC heap baseline continues rising.
• The container reaches its limit and is OOM-killed after several hours.

Evidence Collected

Evidence	Interpretation	Next Step
Heap and anonymous cgroup memory rise together	The growth is likely in managed JavaScript objects	Compare heap snapshots from an isolated replica
Session-cache entry count rises continuously	Expired sessions remain stored	Inspect cache lifecycle and deletion logic
Heap snapshot shows session objects retained by one global map	The map is the retaining path	Add expiration and a maximum entry count
OOM event counter increases at the hard limit	The limit is containing the process but availability is affected	Keep the limit as a guardrail while deploying the application fix

Correction

The team:

• Adds a bounded cache with expiration.
• Records cache entry count and estimated size.
• Removes sessions when logout or expiration events occur.
• Runs repeated load-and-idle cycles.
• Deploys the change to one replica.
• Confirms that post-GC heap returns to a stable range.
• Expands the deployment gradually.

The memory limit remains in place. It continues protecting the host if another defect appears.

Monitoring That Should Remain After the Incident

Metric	What It Reveals	Alert Direction
Container memory versus limit	Proximity to the hard boundary	Alert before the application has insufficient diagnostic and recovery headroom
Managed live heap	Retained runtime-managed objects	Detect a rising post-collection baseline
Native or external memory	Memory outside the principal managed heap	Compare against RSS and cgroup anonymous memory
Threads, processes, or goroutines	Leaking execution units and stack growth	Alert on unexpected growth relative to workload
Queue depth and active requests	Work accumulating inside the process	Alert before the backlog creates memory exhaustion
OOM and memory.events counters	Pressure and cgroup-level allocation failures	Alert on any unexpected OOM kill and repeated high-boundary pressure
Restart count	Whether automated recovery is hiding instability	Escalate repeated restarts rather than treating them as normal

Common Docker Memory Troubleshooting Mistakes

Mistake	Possible Result	Better Direction
Calling every increase a leak	Normal cache, workload, or allocator behavior is misdiagnosed	Compare repeated workload cycles and retained baselines
Watching only docker stats	The responsible heap, native allocation, cache, or kernel category remains unknown	Combine Docker, cgroup, process, runtime, and workload metrics
Using a fixed cgroup path	Commands fail or inspect the wrong hierarchy	Determine cgroup version and derive the process path
Assuming exit code 137 proves OOM	A forced termination is incorrectly classified	Confirm Docker state, cgroup events, and kernel evidence
Restarting before collecting evidence	Heap, mappings, counters, and retaining paths are lost	Isolate one replica and collect evidence safely first
Increasing the limit as the final fix	The leak takes longer to fail and affects more host memory	Use the limit as containment while correcting retention
Capturing heap snapshots at the limit	The profiler itself triggers an OOM or long pause	Capture earlier on an isolated replica with adequate headroom
Looking only at managed heap	Native, mapped, socket, thread, and kernel memory is ignored	Compare heap with RSS and cgroup categories
Running unlimited retries	Pending requests and buffers create rapid memory growth	Use deadlines, bounded queues, backoff, and admission control

Production Readiness Checklist

When to Request Specialist Support

Involve an experienced application-performance, Linux, runtime, platform, or reliability engineer when:

• The container is repeatedly OOM-killed despite application-level corrections.
• Managed heap remains stable while container memory continues growing.
• Native libraries, image-processing tools, database drivers, or cgo are involved.
• Allocator fragmentation appears to be significant.
• The cgroup layout or accounting differs from the expected environment.
• Several containers create host-wide memory pressure.
• Profiling may expose regulated or highly sensitive information.
• The application cannot be profiled without affecting critical traffic.
• Memory growth depends on rare production-only workload conditions.
• The team cannot distinguish an application leak from kernel, runtime, or allocator behavior.

Conclusion

Troubleshooting Docker container memory leaks requires separating application retention from normal runtime, operating-system, and workload behavior.

Begin with a time-based pattern. Compare container memory with traffic, active work, queue depth, garbage collection, process count, deployment version, and application uptime.

Use Docker metrics for an initial view, then inspect the actual cgroup. On cgroup v2 systems, memory.current, memory.stat, and memory.events help separate memory categories and confirm pressure or OOM activity.

Compare the managed heap with process RSS and cgroup memory. If the heap grows, use the runtime’s object and allocation profiler. If the heap remains stable, investigate native allocations, direct buffers, stacks, mapped files, page cache, sockets, and allocator behavior.

Memory limits and restart policies remain important production guardrails. They contain a defective process and protect neighboring workloads, but they do not remove retained objects or correct unbounded queues.

Finally, reproduce the issue through repeated workload cycles, deploy the correction gradually, retain rollback capability, and keep memory, OOM, restart, queue, and runtime metrics after the incident has ended.

Frequently Asked Questions

Does rising Docker memory always mean the application has a leak?

No. The increase may come from normal working-set growth, filesystem cache, native allocations, buffered work, mapped files, additional threads, socket buffers, or allocator retention. A leak requires evidence that unnecessary memory remains retained and continues growing.

Why does docker stats differ from my monitoring dashboard?

On Linux, the Docker CLI subtracts inactive file cache from the total it displays. A monitoring platform using the Engine API or direct cgroup data may use a different calculation. Document the metric definition before comparing values.

Does exit code 137 prove the container was OOM-killed?

No. It commonly indicates termination by SIGKILL, which may result from an OOM event, a manual kill, an orchestrator action, or an expired shutdown grace period. Confirm the Docker OOMKilled state, cgroup events, and host evidence.

Why is RSS high after garbage collection?

The runtime or native allocator may retain committed pages for future reuse, memory may be fragmented, or native buffers, stacks, mapped files, and shared libraries may still be resident. Compare live heap, RSS, cgroup categories, and repeated workload cycles.

Should I increase the Docker memory limit?

Increase it only when measurements show that the legitimate working set needs more headroom and the host has sufficient capacity. A higher limit does not fix retained memory and can allow a leak to affect more of the host.

Are automatic restarts a valid solution?

They are a recovery mechanism, not the root correction. Restarts can temporarily restore service, but the leak will return. Repeated restarts should trigger investigation and preserve termination evidence.

Can a heap snapshot crash the container?

Yes. Some snapshot operations pause application work and require significant additional memory. For Node.js, snapshot generation can require approximately twice the current heap size. Capture it earlier on an isolated replica with adequate memory headroom.

Official References

• Docker: Container resource constraints
• Docker: docker container stats
• Docker: Runtime metrics and cgroups
• Docker: docker container inspect
• Docker: Restart policies
• Docker Compose: Service memory and restart configuration
• Docker: Configure logging drivers and rotation
• Linux kernel: Control Group v2
• Node.js: Memory diagnostics
• Node.js: Using heap snapshots
• Node.js: Using the heap profiler
• Oracle Java: Troubleshooting memory leaks
• Python: tracemalloc
• Go: Diagnostics and profiling
• Go: runtime/pprof