Caching is hard: docker edition

Kamil Podlesak

So, what happens when you run docker build twice with different sources, when the Dockerfile files happen to have the same size and timestamp?
Well, the second run will of course use the Dockerfile cached from the previous run! And the other files too, for a good measure (as long as their sizes and timestamps match).

And if you ask "what is the chance that the timestamp matches" then the answer is "quite high, actually" - it's quite common pattern that the CI/release build server builds several variants with just a slightly different Dockerfile (usually different version of some crucial library, or some switch changed from true to false). And the timestamp is, of course, taken from the commit time of their last change (which is something like "changed maintainer email address").

Wrong dockerfile used when building two images in the same context folder · Issue #1368 · moby/buildkit

When building different images with different dockerfiles in the same context, the wrong dockerfile will be used (probably due to caching problems). I created a minimal reproduction here. Please se...

Bonus points: The github issue marks this marked almost one year ago, but it's not part of any official docker release so far.

Extra bonus points if the difference between variants is actually important for security and you run production with a build that spews debug info to everyone. Luckily, I have dodged that bullet (the build actually failed in my case).

konnichimade

@Kamil-Podlesak :|

Steve_The_Cynic

@Kamil-Podlesak Could be worse.

Àt $JOB we use cmake to manage the main build process. There are many things wrong with cmake, or at least with the way our build system uses it, and among these wrongnesses is this:

Tell cmake (via make) that you want to build the firmware.
There is a momentary network glitch that means that the server from which the build downloads ==> this archive full of stuff is unavailable.
The build fails as a result.
In a bid to help us, cmake caches this failure.
Fix the network.
Relaunch the build.
The server is definitely available.
Nothing changed in the local build environment, so cmake uses the cached result and doesn't even try to download the archive.
The build fails.

TheCPUWizard

The 1 second resolution means that time really can not be used in a who slew of cases, not just this one.

sh_code

@Kamil-Podlesak what utterly fascinates me is that i don't have even the slightest most surface-level idea how docker actually does its magic, but even i am aware that a better identifiers, such as checksums, or even filenames, exist.

Gribnit

@sh_code somebody managed to convince themselves that size and time were dimensions of uniqueness. I wonder what they think the third one was.

TheCPUWizard

@Gribnit said in Caching is hard: docker edition:

@sh_code somebody managed to convince themselves that size and time were dimensions of uniqueness. I wonder what they think the third one was.

IT is a dimension, not only of sight and of sound, but of Mind!

HardwareGeek

@TheCPUWizard And it lies between the pit of man's fears and the summit of his knowledge.

Bulb

@sh_code said in Caching is hard: docker edition:

checksums

The problem with checksums is that you have to read the content to calculate them and that defeats most of the purpose of caching. So it's pretty rare that someone actually uses them.

@Kamil-Podlesak said in Caching is hard: docker edition:

And the timestamp is, of course, taken from the commit time of their last change

Don't do that. It breaks most build systems I've ever seen. Because, you see, the build systems check whether the build product is newer then the sources. But when the timestamp is set to the time of commit, it can change while staying behind the build product timestamp (the next commit was made before the previous build completed) or even go backward (e.g. switching branches). And then the build system won't rebuild it.

Kamil Podlesak

@Kamil-Podlesak said in Caching is hard: docker edition:

And the timestamp is, of course, taken from the commit time of their last change

Don't do that. It breaks most build systems I've ever seen. Because, you see, the build systems check whether the build product is newer then the sources. But when the timestamp is set to the time of commit, it can change while staying behind the build product timestamp (the next commit was made before the previous build completed) or even go backward (e.g. switching branches). And then the build system won't rebuild it.

Yes, timestamp-based incremental build systems are... ehm... tricky, everyone knows that. Also, everyone knows that CI/CD/release builds should not be incremental, ever

The issue here is that the docker build suddenly, unexpectedly and pretty much without warning became a timestamp-based incremental build in a random minor release. Also, it's an "incremental build" in the scope of whole docker host, regardless of source directory (so cleaning up workspace, creating new jobs, etc... is completely irrelevant; but getting different job runner host is).

Bulb

@Kamil-Podlesak said in Caching is hard: docker edition:

Yes, timestamp-based incremental build systems are... ehm... tricky, everyone knows that.

You have to treat them like the shitted stick, but if you avoid the idea that setting file timestamps to the value in the originating system was ever sane and sensible, it can be done.

@Kamil-Podlesak said in Caching is hard: docker edition:

Also, everyone knows that CI/CD/release builds should not be incremental, ever

Not everyone can afford it though. It looks like you haven't seen enough builds that take 4+ hours from clean state (which is, like, the norm for me; I always did release builds clean, but for CI definitely not an option).

@Kamil-Podlesak said in Caching is hard: docker edition:

The issue here is that the docker build suddenly, unexpectedly and pretty much without warning became a timestamp-based incremental build in a random minor release.

The cache's been there literally for years.

Also, you can tell it not to use the cache (docker build --no-cache).

@Kamil-Podlesak said in Caching is hard: docker edition:

Also, it's an "incremental build" in the scope of whole docker host, regardless of source directory

Yeah, that aspect is ugly.

Steve_The_Cynic

@Bulb said in Caching is hard: docker edition:

Don't do that. It breaks most build systems I've ever seen. Because, you see, the build systems check whether the build product is newer then the sources. But when the timestamp is set to the time of commit, it can change while staying behind the build product timestamp (the next commit was made before the previous build completed) or even go backward (e.g. switching branches). And then the build system won't rebuild it.

The only build system that I've ever encountered that would handle that was the internal one in Turbo C for MSDOS. It wrote timestamp information about the sources into the compiled .OBJ files, and recompiled if the timestamps were different.

Kamil Podlesak

The cache's been there literally for years.

The new "builder" was in "incubation" phase for a few years, so I suppose , but...

Also, you can tell it not to use the cache (docker build --no-cache).

No, this is a completely different topic and completely different cache!

docker builder prune -f is the correct way to avoid the cache. There used to be an environment variable to disable the "docker builder", but it does not work anymore.

Bulb

@Kamil-Podlesak said in Caching is hard: docker edition:

The cache's been there literally for years.

The new "builder" was in "incubation" phase for a few years, so I suppose , but...

No, I mean the build was always caching. So if they broke it (more than it was) with the “builder”, it's a bug in there.

Also, you can tell it not to use the cache (docker build --no-cache).

No, this is a completely different topic and completely different cache!

Ok, but that means they introduced a flat out bug in the builder. Because the previous whatever implementation of the build command was caching and the logic was (supposed to be) the same.

docker builder prune -f is the correct way to avoid the cache. There used to be an environment variable to disable the "docker builder", but it does not work anymore.

I guess it's about time to stop using the code from the Docker company, or the Moby project, to actually implement docker.

Kubernetes isn't using docker any more, only the lower level containerd component, and there is a docker-build-compatible buildkit-for-kubectl.

In Linux, I've replaced docker with podman (and optionally buildah). They work rootless by default, and support building inside a container (I'd have to look up and check some instructions, but I am sure it's still easier than docker-in-docker).

And on Windows, well, I am still using rancher-desktop with docker (moby) backend enabled (rather than just the default containerd + nerdctl), because colleague told me they had some incompatibility with docker-compose, but then I am not actually using docker-compose myself, so maybe I should give it a try.

Yours sincerely,
Docker Must Die!

Kamil Podlesak

@Bulb said in Caching is hard: docker edition:

@Kamil-Podlesak said in Caching is hard: docker edition:

The cache's been there literally for years.

The new "builder" was in "incubation" phase for a few years, so I suppose , but...

No, I mean the build was always caching. So if they broke it (more than it was) with the “builder”, it's a bug in there.

Nope, I am pretty sure that the Dockerfile has never been cached.

The layers produced by the build have been always cached.
This new cache is a completely different thing that caches the Dockerfile and other "source" files, before the build even starts.

Also, you can tell it not to use the cache (docker build --no-cache).

No, this is a completely different topic and completely different cache!

Ok, but that means they introduced a flat out bug in the builder. Because the previous whatever implementation of the build command was caching and the logic was (supposed to be) the same.

Yes, they did indeed introduce a bug.

Actually this new build is rather nice, especially on Docker Desktop where this cache actually saves a lot of time. If it only used checksum (yes, it's slower, but still faster than copying the file into VM).

docker builder prune -f is the correct way to avoid the cache. There used to be an environment variable to disable the "docker builder", but it does not work anymore.

I guess it's about time to stop using the code from the Docker company, or the Moby project, to actually implement docker.

And on Windows, well, I am still using rancher-desktop with docker (moby) backend enabled (rather than just the default containerd + nerdctl), because colleague told me they had some incompatibility with docker-compose, but then I am not actually using docker-compose myself, so maybe I should give it a try.

Yes, docker-compose is one of the main things that keeps me at docker.
The other one is that I am often working on a product sold to customers (ie customer run it on their own infrastructure) and we cannot really afford to support all possible variations. So if we want to switch to, say, podman, we need to clearly declare it and handle the pushback.

This is made complicated by the fact that we don't have official docker repository available and providing image for download is obviously not legally viable. So the distribution is actually a .tgz with all files and instructions "unpack it and run docker build -t .

Bulb

@Kamil-Podlesak said in Caching is hard: docker edition:

The other one is that I am often working on a product sold to customers (ie customer run it on their own infrastructure) and we cannot really afford to support all possible variations. So if we want to switch to, say, podman, we need to clearly declare it and handle the pushback.

… Kubernetes is used by many customers, and while not sold per-se, is maintained for many customers, and it basically silently dropped the dockerd layer and switched to containerd alone.

As far as I can tell, podman can run any image docker can and vice versa, and with containerd there is no difference at all, because dockerd calls containerd and you can simply bypass the dockerd. So it's just about how you build the images, and unless you are building them at the customer, it shouldn't affect the customer at all.

Kamil Podlesak

@Bulb said in Caching is hard: docker edition:

So it's just about how you build the images, and unless you are building them at the customer, it shouldn't affect the customer at all.

Unfortunately we do... so far it's the only way to avoid GNU viral license in a way that is approved by Legal. Theoretically the docker repository API should be OK, but I am not a lawyer (much less a EU+UK+CH IP-lawyer).

Gribnit

@HardwareGeek said in Caching is hard: docker edition:

@TheCPUWizard And it lies between the pit of man's fears and the summit of his knowledge.

Thanks for the scope restriction. I'll just check all the code between there, shall I.