Back to blog

When a Security Alert Sent Me Down the Docker Rabbit Hole

·5 min read
DockerDevOpsCybersecurityCI/CD

Listen to this article· 3 min

Photo by Sébastien Lavalaye on Unsplash

It started with a message from the SOC team.

"We found a file detected by SentinelOne on several devices, including Kubernetes pods."

One of the machines was a CI/CD runner.

The file was called:

pam_exec.so

And the reported path looked like this:

/var/lib/docker/overlay2/.../diff/usr/lib/security/pam_exec.so

At first glance, that sounds bad.

A security tool found a suspicious file on a machine used for builds and deployments.

My first thought was basically:

"Okay... where did this come from?"

That question turned into a surprisingly useful Docker lesson.

The First Clue Was the Path

The first thing I noticed was that the file wasn't reported under a normal Linux system path like:

/usr/lib/security/pam_exec.so

It was under:

/var/lib/docker/overlay2/

That changed the investigation completely.

Docker uses overlay2 to store filesystem data for images and containers.

(That's the classic overlay2 storage driver, which long-lived hosts like our runner still use. Fresh installs of Docker Engine 29 and later default to the containerd image store instead, which keeps layers under /var/lib/containerd.)

So instead of assuming:

Security alertMalware on the server

I started thinking:

A better question
  1. 01Security alert
  2. 02File inside Docker storage
  3. 03Which image or container does it belong to?

We checked the same path on the host itself, and the file wasn't present there.

So the investigation had already moved from "something was installed on the server" to "something exists somewhere inside Docker's filesystem."

Then I Had to Understand What a Docker Layer Actually Is

This was probably the most useful Docker concept I took away from the investigation.

A Docker image isn't one giant filesystem.

It is built from layers.

For example:

FROM debian
RUN apt-get update
RUN apt-get install -y curl
COPY app /app

The base image brings its own layers, and each filesystem-changing instruction (RUN, COPY, ADD) adds one more. Conceptually, you can think of it as:

Layers from that Dockerfile
  1. 01Layer 1base OS
  2. 02Layer 2package updates
  3. 03Layer 3curl
  4. 04Layer 4application files

The layers are stacked together so that the container sees what looks like one normal filesystem.

When a container starts, Docker adds a writable layer on top of those image layers for changes made by the running container.

So the mental model becomes:

Container
  1. Writable container layer
  2. Image layer
  3. Image layer
  4. Image layer
  5. Base image layer

Docker stores this underlying filesystem data under:

/var/lib/docker/overlay2/

and the diff/ directory contains the actual filesystem content for a layer. That explains why the SOC alert contained a path such as:

overlay2/<layer-id>/diff/usr/lib/security/pam_exec.so

In other words:

The path itself was already telling us that the file belonged to Docker storage and potentially to an image layer.

This sounds obvious after you understand it.

It wasn't quite so obvious before.

Image, Container, Layer, Cache

I also had to stop mixing these concepts together.

A simple mental model is:

Host → Docker
  1. 01Host
  2. 02Docker
  3. 03
    Image
    LayerLayerLayer
  4. 04
    Container
    Image layersWritable layer

The image is the packaged blueprint.

The container is the running instance.

The layers are the filesystem pieces that make up the image.

And Docker can keep images and layers around as cached artifacts even after containers have stopped.

That last part is especially important on CI/CD runners.

A runner can accumulate:

  • Images
  • Containers
  • Layers
  • Build cache
  • Temporary artifacts

So a security scan can find a file somewhere in Docker's stored data even when that file isn't part of the host operating system and isn't being executed by a running application.

But What Exactly Is pam_exec.so?

The filename certainly didn't help.

pam_exec.so is a legitimate Linux PAM module. PAM, or Pluggable Authentication Modules, is part of Linux authentication, and pam_exec.so can be used to execute commands or scripts during authentication.

So the presence of the file alone wasn't enough to conclude that something malicious had happened.

That gave me another important security lesson:

A detection is the beginning of an investigation, not the conclusion.

The real questions became:

  • Where is the file?
  • Where did it come from?
  • Which image contains it?
  • Was that image expected?
  • Is the file actually being used?
  • Is the host itself affected?

Then I Started Following the Docker Layers

The SOC team had provided the Docker layer ID associated with the alert.

That was the key.

Instead of searching every file manually, I could trace the layer back toward its source:

Tracing it back
  1. 01pam_exec.so
  2. 02overlay2
  3. 03Layer ID
  4. 04Docker image

This was probably the most useful thing I learned from the whole incident.

When something suspicious appears inside Docker, don't stop at the file.

Trace its provenance.

In the development environment, we found the file inside one of the standard CI images used by the runner.

In production, we went a step further and mapped the exact layer reported by SOC to a security-agent image already expected on the environment.

And suddenly the story looked very different.

The Difference Between "Found" and "Compromised"

This was probably the most important lesson.

A scanner can tell you:

"I found this file."

That doesn't automatically mean:

"An attacker installed this file."

And it definitely doesn't automatically mean:

"The server is compromised."

We still needed to establish the file's origin and context.

That is why tracing:

filelayerimagecontainer

is so useful.

You want to move from:

  • Unknown file

to:

Known, end to end
  1. 01Known file
  2. 02Known layer
  3. 03Known image
  4. 04Expected component

The more you can establish that chain, the stronger your conclusion becomes.

Then Docker Gave Me Another Lesson

During the investigation, I also ran into something much less glamorous:

permission denied while trying to connect to the Docker daemon socket

The famous:

/var/run/docker.sock

appeared.

Using elevated Docker access allowed the inspection commands to run.

That was another reminder that access to the Docker daemon is powerful.

If you can control the Docker daemon, you are dealing with something much more privileged than simply being able to start a container.

So even while investigating a security alert, Docker was quietly teaching me another security lesson.

Very efficient teacher.

CI/CD Runners Make Investigations Messier

The machine involved wasn't a normal application server.

It was a CI/CD runner.

That means Docker tends to get messy:

  • Build images
  • Test images
  • Helper images
  • Cached layers
  • Temporary containers
  • Docker-in-Docker
  • Security tooling
  • Dangling images
  • Old artifacts

All of that can accumulate over time.

So a security scanner looking through Docker storage may find files that are:

  • inside an image
  • inside an old layer
  • cached locally
  • part of a build environment
  • not currently running at all

That doesn't mean alerts should be ignored.

It means the environment needs context.

A CI runner is naturally much noisier than a clean application VM.

The Mental Model I Took Away

Before this incident, my Docker mental model was mostly:

ImageContainerApplication

After it:

After
  1. 01Host
  2. 02Docker
  3. 03Image
  4. 04Layers
  5. 05Container
  6. 06Application

And when Kubernetes enters the picture:

With Kubernetes
  1. 01Cloud
  2. 02VM / Node
  3. 03Container Runtime
  4. 04Image Layers
  5. 05Container
  6. 06Pod
  7. 07Application

That extra depth matters when something goes wrong.

Because troubleshooting often means moving down the stack until you find the source of the problem.

What Started as a Malware Alert

The original message was basically:

Security toolSuspicious file detected

What I eventually had to understand was:

What it actually took
  1. 01Security alert
  2. 02Linux filesystem
  3. 03Docker storage
  4. 04overlay2
  5. 05Layer
  6. 06Image
  7. 07Container / CI environment
  8. 08Provenance

That was much more valuable than simply finding the file.

I learned how Docker actually stores things.

I learned how images and layers relate to each other.

I learned that CI/CD runners can contain a lot of Docker artifacts.

And I learned that security investigations often depend less on the filename and more on the context surrounding the filename.

Final Thought

The most useful question in this whole investigation wasn't:

"Is this file malicious?"

It was:

"Where did this file come from?"

Once you start answering that question systematically, the investigation becomes much clearer.

The questions, in order
  1. 01Where is it?
  2. 02Host or container?
  3. 03Which layer?
  4. 04Which image?
  5. 05Where did the image come from?
  6. 06Is it expected?
  7. 07Is it actually being used?

That is a mindset I want to keep as I move deeper into DevOps and platform engineering.

Because infrastructure isn't just about deploying things.

Sometimes it's about figuring out why a file exists three layers underneath a container on a machine you didn't even realize was storing it.

And yes...

I now look at /var/lib/docker/overlay2 a little differently.

…

Up to 50 claps per reader

All posts
Share