Docker network timeouts

boomzilla

17.05 is really Ancient, but my impression is that that was the first version that worked reliably (another project around here is still using 1.13 under its OpenShift 3.11, which was pretty broken, but RedHat somehow managed to kick it into a, working, square ball and make the whole house of cards stand). I have no known issues with that.

The server is still on Ubuntu 18.04LTS (as is my VM, in order to match).

Did you build the container on the server or on your test VM and copied it over?

The containers were all built where they're running. No copying.

Bulb

@boomzilla said in Docker network timeouts:

@Bulb said in Docker network timeouts:

17.05 is really Ancient, but my impression is that that was the first version that worked reliably (another project around here is still using 1.13 under its OpenShift 3.11, which was pretty broken, but RedHat somehow managed to kick it into a, working, square ball and make the whole house of cards stand). I have no known issues with that.

The server is still on Ubuntu 18.04LTS (as is my VM, in order to match).

Bionic Beaver has docker.io 19.03.6 in its updates. And even the less supported platforms have 17.12. See

Ubuntu – Error

If kernel was/is/will be updated, Docker should too.

Did you build the container on the server or on your test VM and copied it over?

The containers were all built where they're running. No copying.

Ok, that means it can't use any features the daemon does not support yet.

boomzilla

@Bulb yeah, I saw docker.io while poking around in apt.

boomzilla@what:~$ apt search docker | grep ^docker

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

docker/bionic 1.5-1build1 amd64
docker-compose/bionic,bionic 1.17.1-2 all
docker-containerd/bionic 0.2.3+git+docker1.13.1~ds1-1 amd64
docker-doc/bionic-updates,bionic-updates 20.10.2-0ubuntu1~18.04.2 all
docker-engine/now 17.05.0~ce-0~ubuntu-xenial amd64 [installed,local]
docker-registry/bionic 2.6.2~ds1-1 amd64
docker-runc/bionic 1.0.0~rc2+git+docker1.13.1~ds1-3 amd64
docker.io/bionic-updates 20.10.2-0ubuntu

Hmm...now I notice that docker-engine was from the xenial repository.

Bulb

@boomzilla said in Docker network timeouts:

Hmm...now I notice that docker-engine was from the xenial repository.

No, it must have been from upstream repository. The package is called docker.io even in Xenial. The version there, 1.10.3, was thoroughly botched though, so I understand pulling the upstream one instead.

boomzilla

@Bulb you mean xenial was pulling from upstream? My point is that the server is on bionic and that's coming from the xenial repo.

Hmm...so now I need a plan about how to update.

If I uninstall docker-engine will that stop the currently running containers?

If it doesn't, would the updated docker cleanly restart the containers running on the older version?

One annoyance is that Ben's script assumes the old container is running. If it isn't, I'll probably have to do some finagling.

Bulb

@boomzilla said in Docker network timeouts:

@Bulb you mean xenial was pulling from upstream? My point is that the server is on bionic and that's coming from the xenial repo.

No, I mean it was coming from xenial release in docker.com (or .org or whatever; the history is convoluted) repo.

@boomzilla said in Docker network timeouts:

If I uninstall docker-engine will that stop the currently running containers?
If it doesn't, would the updated docker cleanly restart the containers running on the older version?

It will stop them when the old daemon is stopped, which must happen for upgrade.

It Should™ be able to start them back after the upgrade, but I wouldn't trust it too much.

@boomzilla said in Docker network timeouts:

One annoyance is that Ben's script assumes the old container is running. If it isn't, I'll probably have to do some finagling.

Unless the script runs under -e (abort on failure) shell flag the rename, stop and rm -v at the end failing because the old container isn't there shouldn't matter too much. Or if it does have set -e, you can just temporarily comment out the three commands or add || true to them.

What I don't like in the script are those cps. I'd probably do it with a second Dockerfile to minimize the amount of things done with a wrapper script, because that would be more obvious for someone else, or me three years later.

boomzilla

@Bulb said in Docker network timeouts:

@boomzilla said in Docker network timeouts:

@Bulb you mean xenial was pulling from upstream? My point is that the server is on bionic and that's coming from the xenial repo.

No, I mean it was coming from xenial release in docker.com (or .org or whatever; the history is convoluted) repo.

Hmm...ah, yes...ok...

$ grep -rn docker /etc/apt/
Binary file /etc/apt/trusted.gpg~ matches
/etc/apt/sources.list.d/docker.list:1:# deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable # disabled on upgrade to bionic
/etc/apt/sources.list.d/docker.list.distUpgrade:1:deb [arch=amd64] https://download.docker.com/linux/ubuntu xenial stable

@boomzilla said in Docker network timeouts:

If I uninstall docker-engine will that stop the currently running containers?
If it doesn't, would the updated docker cleanly restart the containers running on the older version?

It will stop them when the old daemon is stopped, which must happen for upgrade.

It Should™ be able to start them back after the upgrade, but I wouldn't trust it too much.

@boomzilla said in Docker network timeouts:

One annoyance is that Ben's script assumes the old container is running. If it isn't, I'll probably have to do some finagling.

Unless the script runs under -e (abort on failure) shell flag the rename, stop and rm -v at the end failing because the old container isn't there shouldn't matter too much. Or if it does have set -e, you can just temporarily comment out the three commands or add || true to them.

Yeah, it does.

What I don't like in the script are those cps. I'd probably do it with a second Dockerfile to minimize the amount of things done with a wrapper script, because that would be more obvious for someone else, or me three years later.

Definitely don't look at the Docker file we use for nodebb.

Bulb

@boomzilla said in Docker network timeouts:

/etc/apt/sources.list.d/docker.list:1:# deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable # disabled on upgrade to bionic

Yeah, Ubuntu . On my work notebook the upgrade script never ran correctly to completion anyway (at home I have Debian everywhere).

Either way it should be updated every now and then. Docker runs with root permissions, so it is somewhat sensitive itself.

@boomzilla said in Docker network timeouts:

Definitely don't look at the Docker file we use for nodebb.

The Dockerfile is OK. It's long and messy, but it's the standard way of doing things, so it's fairly easy to sort out when you need to. The *.bash scripts is what I don't like, because they can do absolutely anything and follow no sensible convention.

Here I most dislike the volumes. Volumes mean you have directories with precious data stored at some unknown place managed by docker. This makes some sense in docker-for-desktop (Windows) where the volumes live inside the Linux VM and so have all the expected Linux filesystem features, but on Linux just mounting specific directories is easier to work with.

There is one gotcha with ancient Docker that you might need to do sooner or later: old docker used aufs filesystem, but that is broken and not supported on newer kernels any more and replaced by overlay2.

You can check what you are using with docker system info | grep Storage. Unfortunately the procedure for the switch calls for nuking the docker data directory. And that means you have to backup the volumes and the containers and import them back later.

boomzilla

@Bulb said in Docker network timeouts:

@boomzilla said in Docker network timeouts:
/etc/apt/sources.list.d/docker.list:1:# deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable # disabled on upgrade to bionic
Yeah, Ubuntu . On my work notebook the upgrade script never ran correctly to completion anyway (at home I have Debian everywhere).

This looks like something Ben did in order to get a reasonable version of docker.

@boomzilla said in Docker network timeouts:

Definitely don't look at the Docker file we use for nodebb.

The Dockerfile is OK. It's long and messy, but it's the standard way of doing things, so it's fairly easy to sort out when you need to. The *.bash scripts is what I don't like, because they can do absolutely anything and follow no sensible convention.

Here I most dislike the volumes. Volumes mean you have directories with precious data stored at some unknown place managed by docker. This makes some sense in docker-for-desktop (Windows) where the volumes live inside the Linux VM and so have all the expected Linux filesystem features, but on Linux just mounting specific directories is easier to work with.

There is one gotcha with ancient Docker that you might need to do sooner or later: old docker used aufs filesystem, but that is broken and not supported on newer kernels any more and replaced by overlay2.

You can check what you are using with docker system info | grep Storage. Unfortunately the procedure for the switch calls for nuking the docker data directory. And that means you have to backup the volumes and the containers and import them back later.

boomzilla@what:~$ docker system info | grep Storage
Storage Driver: overlay2
WARNING: No swap limit support

Bulb

@boomzilla said in Docker network timeouts:

This looks like something Ben did in order to get a reasonable version of docker.

I don't mean that. I mean the fact the upgrade script replaced the release name there, but commented the line out at the same time and left it commented out.

@boomzilla said in Docker network timeouts:

Storage Driver: overlay2

Good. That means an upgrade should be able to pick up the images, containers and volumes fine.

Gribnit

But all this time it was actually a left Reebok Pump sneaker.