1. Home
  2. Categories
  3. Coding Help
  4. Linux server occasionally losing its network stack

Linux server occasionally losing its network stack

  • ben_lubar

    Here's what appears to be the relevant part of the system log.

    Oct 20 02:56:01 urist.lubar.me systemd-timesyncd[976]: Synchronized to time server [2001:67c:1560:8003::c7]:123 (ntp.ubuntu.com).
Oct 20 02:56:01 urist.lubar.me dockerd[1374]: time="2018-10-20T02:56:01-05:00" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/7929dbfda12b4851637ef1343d5ea2c8bf30c82d21e8f6111113f202b075bbad/shim.sock" debug=false pid=31657
Oct 20 02:56:01 urist.lubar.me kernel: br-f2cfca096cb9: port 1(vethc06806b) entered disabled state
Oct 20 02:56:01 urist.lubar.me kernel: br-f1a30e78d2c0: port 5(veth803694f) entered disabled state
Oct 20 02:56:02 urist.lubar.me systemd-timesyncd[976]: Network configuration changed, trying to establish connection.
Oct 20 02:56:02 urist.lubar.me kernel: eth0: renamed from veth97a4109
Oct 20 02:56:02 urist.lubar.me systemd-networkd[902]: vethc06806b: Gained carrier
Oct 20 02:56:02 urist.lubar.me kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethc06806b: link becomes ready
Oct 20 02:56:02 urist.lubar.me kernel: br-f2cfca096cb9: port 1(vethc06806b) entered blocking state
Oct 20 02:56:02 urist.lubar.me kernel: br-f2cfca096cb9: port 1(vethc06806b) entered forwarding state
Oct 20 02:56:02 urist.lubar.me kernel: eth1: renamed from veth2490df6
Oct 20 02:56:02 urist.lubar.me kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth803694f: link becomes ready
Oct 20 02:56:02 urist.lubar.me kernel: br-f1a30e78d2c0: port 5(veth803694f) entered blocking state
Oct 20 02:56:02 urist.lubar.me kernel: br-f1a30e78d2c0: port 5(veth803694f) entered forwarding state
Oct 20 02:56:02 urist.lubar.me systemd-networkd[902]: veth803694f: Gained carrier
Oct 20 02:56:02 urist.lubar.me systemd-timesyncd[976]: Synchronized to time server [2001:67c:1560:8003::c7]:123 (ntp.ubuntu.com).
Oct 20 02:56:03 urist.lubar.me systemd-networkd[902]: veth803694f: Gained IPv6LL
Oct 20 02:56:03 urist.lubar.me systemd-timesyncd[976]: Network configuration changed, trying to establish connection.
Oct 20 02:56:03 urist.lubar.me systemd-timesyncd[976]: Synchronized to time server [2001:67c:1560:8003::c7]:123 (ntp.ubuntu.com).
Oct 20 02:56:04 urist.lubar.me systemd-networkd[902]: vethc06806b: Gained IPv6LL
Oct 20 02:56:04 urist.lubar.me systemd-timesyncd[976]: Network configuration changed, trying to establish connection.
Oct 20 02:56:04 urist.lubar.me systemd-timesyncd[976]: Synchronized to time server [2001:67c:1560:8003::c7]:123 (ntp.ubuntu.com).
Oct 20 03:00:01 urist.lubar.me systemd-resolved[1011]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP+EDNS0.
Oct 20 03:00:01 urist.lubar.me systemd-resolved[1011]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP+EDNS0.
Oct 20 03:00:01 urist.lubar.me systemd-resolved[1011]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Oct 20 03:00:01 urist.lubar.me systemd-resolved[1011]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Oct 20 03:00:02 urist.lubar.me systemd-resolved[1011]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP+EDNS0.
Oct 20 03:00:02 urist.lubar.me systemd-resolved[1011]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP+EDNS0.
Oct 20 03:00:02 urist.lubar.me systemd-resolved[1011]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Oct 20 03:00:02 urist.lubar.me systemd-resolved[1011]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Oct 20 03:00:02 urist.lubar.me systemd-resolved[1011]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP+EDNS0.
Oct 20 03:00:02 urist.lubar.me systemd-resolved[1011]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP+EDNS0.
Oct 20 03:00:02 urist.lubar.me systemd-resolved[1011]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Oct 20 03:00:02 urist.lubar.me systemd-resolved[1011]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.
Oct 20 03:00:45 urist.lubar.me dockerd[1374]: time="2018-10-20T03:00:45-05:00" level=info msg="shim reaped" id=8447cff066c1f016f1543059a788938d7f9890b4a6ea195e1f80d20158dc736e
Oct 20 03:00:45 urist.lubar.me dockerd[1374]: time="2018-10-20T03:00:45.353305987-05:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Oct 20 03:00:45 urist.lubar.me systemd-networkd[902]: veth2baa768: Lost carrier
Oct 20 03:00:45 urist.lubar.me systemd-timesyncd[976]: Network configuration changed, trying to establish connection.
Oct 20 03:00:45 urist.lubar.me kernel: br-f2cfca096cb9: port 7(veth2baa768) entered disabled state
Oct 20 03:00:45 urist.lubar.me kernel: vethdadfdef: renamed from eth0
Oct 20 03:00:45 urist.lubar.me systemd-udevd[4436]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Oct 20 03:00:45 urist.lubar.me networkd-dispatcher[1119]: WARNING:Unknown index 2539 seen, reloading interface list
Oct 20 03:00:45 urist.lubar.me systemd-timesyncd[976]: Synchronized to time server [2001:67c:1560:8003::c7]:123 (ntp.ubuntu.com).
Oct 20 03:00:45 urist.lubar.me systemd-timesyncd[976]: Network configuration changed, trying to establish connection.
Oct 20 03:00:45 urist.lubar.me kernel: br-f2cfca096cb9: port 7(veth2baa768) entered disabled state
Oct 20 03:00:45 urist.lubar.me kernel: device veth2baa768 left promiscuous mode
Oct 20 03:00:45 urist.lubar.me kernel: br-f2cfca096cb9: port 7(veth2baa768) entered disabled state
Oct 20 03:00:45 urist.lubar.me systemd-timesyncd[976]: Synchronized to time server [2001:67c:1560:8003::c7]:123 (ntp.ubuntu.com).
Oct 20 03:00:45 urist.lubar.me systemd-networkd[902]: vethb97449a: Lost carrier
Oct 20 03:00:45 urist.lubar.me systemd-timesyncd[976]: Network configuration changed, trying to establish connection.
Oct 20 03:00:45 urist.lubar.me kernel: br-f1a30e78d2c0: port 9(vethb97449a) entered disabled state
Oct 20 03:00:45 urist.lubar.me kernel: veth239f107: renamed from eth1
Oct 20 03:00:45 urist.lubar.me systemd-udevd[4533]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Oct 20 03:00:45 urist.lubar.me networkd-dispatcher[1119]: WARNING:Unknown index 2541 seen, reloading interface list
Oct 20 03:00:45 urist.lubar.me kernel: br-f1a30e78d2c0: port 9(vethb97449a) entered disabled state
Oct 20 03:00:45 urist.lubar.me kernel: device vethb97449a left promiscuous mode
Oct 20 03:00:45 urist.lubar.me kernel: br-f1a30e78d2c0: port 9(vethb97449a) entered disabled state
Oct 20 03:00:45 urist.lubar.me systemd-timesyncd[976]: Synchronized to time server [2001:67c:1560:8003::c7]:123 (ntp.ubuntu.com).
Oct 20 03:00:45 urist.lubar.me systemd-timesyncd[976]: Network configuration changed, trying to establish connection.
Oct 20 03:00:45 urist.lubar.me systemd-udevd[4543]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Oct 20 03:00:45 urist.lubar.me networkd-dispatcher[1119]: WARNING:Unknown index 2548 seen, reloading interface list
Oct 20 03:00:45 urist.lubar.me systemd-udevd[4543]: Could not generate persistent MAC address for veth8b0e1b6: No such file or directory
Oct 20 03:00:45 urist.lubar.me systemd-udevd[4544]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Oct 20 03:00:45 urist.lubar.me systemd-udevd[4544]: Could not generate persistent MAC address for veth46d2d4d: No such file or directory
Oct 20 03:00:45 urist.lubar.me kernel: br-f1a30e78d2c0: port 9(veth46d2d4d) entered blocking state
Oct 20 03:00:45 urist.lubar.me kernel: br-f1a30e78d2c0: port 9(veth46d2d4d) entered disabled state
Oct 20 03:00:45 urist.lubar.me kernel: device veth46d2d4d entered promiscuous mode
Oct 20 03:00:45 urist.lubar.me kernel: IPv6: ADDRCONF(NETDEV_UP): veth46d2d4d: link is not ready
Oct 20 03:00:45 urist.lubar.me kernel: br-f1a30e78d2c0: port 9(veth46d2d4d) entered blocking state
Oct 20 03:00:45 urist.lubar.me kernel: br-f1a30e78d2c0: port 9(veth46d2d4d) entered forwarding state
Oct 20 03:00:46 urist.lubar.me dockerd[1374]: time="2018-10-20T03:00:46.051004641-05:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Oct 20 03:00:46 urist.lubar.me dockerd[1374]: time="2018-10-20T03:00:46.051040038-05:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Oct 20 03:00:46 urist.lubar.me systemd-udevd[4552]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Oct 20 03:00:46 urist.lubar.me kernel: br-f2cfca096cb9: port 7(vetha4cf22c) entered blocking state
Oct 20 03:00:46 urist.lubar.me kernel: br-f2cfca096cb9: port 7(vetha4cf22c) entered disabled state
Oct 20 03:00:46 urist.lubar.me kernel: device vetha4cf22c entered promiscuous mode
Oct 20 03:00:46 urist.lubar.me kernel: IPv6: ADDRCONF(NETDEV_UP): vetha4cf22c: link is not ready
Oct 20 03:00:46 urist.lubar.me kernel: br-f2cfca096cb9: port 7(vetha4cf22c) entered blocking state
Oct 20 03:00:46 urist.lubar.me kernel: br-f2cfca096cb9: port 7(vetha4cf22c) entered forwarding state
Oct 20 03:00:46 urist.lubar.me networkd-dispatcher[1119]: WARNING:Unknown index 2550 seen, reloading interface list
Oct 20 03:00:46 urist.lubar.me systemd-udevd[4552]: Could not generate persistent MAC address for veth6deba9b: No such file or directory
Oct 20 03:00:46 urist.lubar.me systemd-udevd[4553]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Oct 20 03:00:46 urist.lubar.me systemd-udevd[4553]: Could not generate persistent MAC address for vetha4cf22c: No such file or directory
Oct 20 03:00:46 urist.lubar.me systemd-timesyncd[976]: Synchronized to time server [2001:67c:1560:8003::c7]:123 (ntp.ubuntu.com).
Oct 20 03:00:46 urist.lubar.me dockerd[1374]: time="2018-10-20T03:00:46-05:00" level=info msg="shim docker-containerd-shim started" address="/containerd-shim/moby/8447cff066c1f016f1543059a788938d7f9890b4a6ea195e1f80d20158dc736e/shim.sock" debug=false pid=4561
Oct 20 03:00:46 urist.lubar.me kernel: br-f1a30e78d2c0: port 9(veth46d2d4d) entered disabled state
Oct 20 03:00:46 urist.lubar.me kernel: br-f2cfca096cb9: port 7(vetha4cf22c) entered disabled state
Oct 20 03:00:46 urist.lubar.me systemd-timesyncd[976]: Network configuration changed, trying to establish connection.
Oct 20 03:00:46 urist.lubar.me kernel: eth0: renamed from veth6deba9b
Oct 20 03:00:46 urist.lubar.me systemd-networkd[902]: vetha4cf22c: Gained carrier
Oct 20 03:00:46 urist.lubar.me kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vetha4cf22c: link becomes ready
Oct 20 03:00:46 urist.lubar.me kernel: br-f2cfca096cb9: port 7(vetha4cf22c) entered blocking state
Oct 20 03:00:46 urist.lubar.me kernel: br-f2cfca096cb9: port 7(vetha4cf22c) entered forwarding state
Oct 20 03:00:46 urist.lubar.me kernel: eth1: renamed from veth8b0e1b6
Oct 20 03:00:46 urist.lubar.me systemd-networkd[902]: veth46d2d4d: Gained carrier
Oct 20 03:00:46 urist.lubar.me kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth46d2d4d: link becomes ready
Oct 20 03:00:46 urist.lubar.me kernel: br-f1a30e78d2c0: port 9(veth46d2d4d) entered blocking state
Oct 20 03:00:46 urist.lubar.me kernel: br-f1a30e78d2c0: port 9(veth46d2d4d) entered forwarding state
Oct 20 03:00:46 urist.lubar.me systemd-timesyncd[976]: Synchronized to time server [2001:67c:1560:8003::c7]:123 (ntp.ubuntu.com).
Oct 20 03:00:48 urist.lubar.me systemd-networkd[902]: vetha4cf22c: Gained IPv6LL
Oct 20 03:00:48 urist.lubar.me systemd-timesyncd[976]: Network configuration changed, trying to establish connection.
Oct 20 03:00:48 urist.lubar.me systemd-timesyncd[976]: Synchronized to time server [2001:67c:1560:8003::c7]:123 (ntp.ubuntu.com).
Oct 20 03:00:48 urist.lubar.me systemd-networkd[902]: veth46d2d4d: Gained IPv6LL
Oct 20 03:00:48 urist.lubar.me systemd-timesyncd[976]: Network configuration changed, trying to establish connection.
Oct 20 03:00:48 urist.lubar.me systemd-timesyncd[976]: Synchronized to time server [2001:67c:1560:8003::c7]:123 (ntp.ubuntu.com).
Oct 20 03:01:03 urist.lubar.me dockerd[1374]: time="2018-10-20T03:01:03-05:00" level=info msg="shim reaped" id=7929dbfda12b4851637ef1343d5ea2c8bf30c82d21e8f6111113f202b075bbad
Oct 20 03:01:03 urist.lubar.me dockerd[1374]: time="2018-10-20T03:01:03.113509746-05:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Oct 20 03:01:03 urist.lubar.me systemd-networkd[902]: vethc06806b: Lost carrier
Oct 20 03:01:03 urist.lubar.me systemd-timesyncd[976]: Network configuration changed, trying to establish connection.
Oct 20 03:01:03 urist.lubar.me kernel: br-f2cfca096cb9: port 1(vethc06806b) entered disabled state
Oct 20 03:01:03 urist.lubar.me kernel: veth97a4109: renamed from eth0

    Any idea what's happening here?

    The server seems to have still been running for the last 6 hours but not receiving or sending anything on the network.

    2 Luhmann 1 Reply
  • Luhmann
    BINNED

    @ben_lubar
    Linux hardware?

    5 loopback0 Polygeekery 2 Replies
  • loopback0
    Discourse touched me in a no-no place

    @Luhmann said in Linux server occasionally losing its network stack:

    Linux hardware?

    </thread>

    https://media.giphy.com/media/xT9DPhONuz1SpCONiM/giphy.gif

    5
  • Polygeekery
    Grade A Premium Asshole

    @Luhmann said in Linux server occasionally losing its network stack:

    @ben_lubar
    Linux hardware?

    My bet is on it lost its connection to the secret servers in China.

    4 pie_flavor 1 Reply
  • wft

    For the love of whatever deity, why do your real hardware interfaces get renamed to „veth*”? Does docker then fuck them up?

    Ahem, could it also be that you have something real fucked up, like MAC address of one of your schmocker containers being the same as that of your physical interface?

    4 ben_lubar 1 Reply
  • pie_flavor
    Considered Harmful

    @Polygeekery said in Linux server occasionally losing its network stack:

    My bet is on it lost its connection to the secret servers in ChinaMilwaukeePC.

    1 ben_lubar 1 Reply
  • ben_lubar

    @wft said in Linux server occasionally losing its network stack:

    For the love of whatever deity, why do your real hardware interfaces get renamed to „veth*”? Does docker then fuck them up?

    Ahem, could it also be that you have something real fucked up, like MAC address of one of your schmocker containers being the same as that of your physical interface?

    Those aren't real hardware interfaces. I think it somehow got confused by something the router sent it at 3 AM and then tried to use an alternate network card to reconnect and for some reason tried to use a virtual internal network for the entire system.

    1 wft 1 Reply
  • ben_lubar

    @pie_flavor said in Linux server occasionally losing its network stack:

    @Polygeekery said in Linux server occasionally losing its network stack:

    My bet is on it lost its connection to the secret servers in ChinaMilwaukeePC.

    This but unironically

    3
  • wft

    @ben_lubar they are not, I know. But you want to know why a veth interface gets renamed to eth0. Something makes it think it’s possible and desirable. Maybe you should unfuck your bridge configuration.

    4
  • M
    Impossible Mission - B

    3 ben_lubar 1 Reply
  • ben_lubar

    @masonwheeler said in Linux server occasionally losing its network stack:

    Yes. That's how I fixed it both times it's happened so far.

    1
  • C

    Does dmesg or kernel.log say anything relevant when the link goes down?

    1 ben_lubar 1 Reply
  • ben_lubar

    @Captain said in Linux server occasionally losing its network stack:

    Does dmesg or kernel.log say anything relevant when the link goes down?

    0_1540149591225_d3ef2b75-8eb0-41fa-b116-889393372c10-image.png

    dmesg -k contains no messages about the actual network card named enp6s0

    1 C 1 Reply
  • C

    @ben_lubar My bad. Look in the kern.log for the date/time of the incident. You might not find that interface mentioned, but you might find an error from the driver, underlying interface, PCI errors, etc.

    2 ben_lubar 1 Reply
  • ben_lubar

    @Captain said in Linux server occasionally losing its network stack:

    @ben_lubar My bad. Look in the kern.log for the date/time of the incident. You might not find that interface mentioned, but you might find an error from the driver, underlying interface, PCI errors, etc.

    kern.log is full of messages about Ubuntu's livepatch service not finding anything to apply and nothing else.

    1
Log in to reply