Systemd stops a service but does not restart it...

Rhywden

So, today I got a call that our Moodle server reported that it can't reach the Mysql database. So I log in, and check the status of the Mysql service. It's stopped. sudo service mysql start starts the service back up without a hitch.

So, off to the logs we go to determine the root cause. /var/log/mysql/error.log? Empty. Any other Mysql log? Similarly devoid of information.

The only thing I can find is some sparse entries in the syslog:

May  5 06:56:01 moodle CRON[18512]: (www-data) CMD (/usr/bin/php /var/www/html/moodle/admin/cli/cron.php >/dev/null)
May  5 06:56:16 moodle systemd[1]: Starting Daily apt upgrade and clean activities...
May  5 06:56:35 moodle systemd[1]: Reloading.
May  5 06:56:35 moodle systemd[1]: Starting Message of the Day...
May  5 06:56:35 moodle systemd[1]: Stopping MySQL Community Server...
[...]
May  5 06:56:37 moodle mysqld[2078]: 2020-05-05T06:56:37.143770Z 0 [Note] /usr/sbin/mysqld: Shutdown complete
May  5 06:56:37 moodle systemd[1]: Stopped MySQL Community Server.
May  5 06:56:37 moodle systemd[1]: Reloading.
May  5 06:56:42 moodle systemd[1]: message repeated 3 times: [ Reloading.]
May  5 06:56:46 moodle systemd[1]: Started Daily apt upgrade and clean activities.

And that's it. The service is enabled and I have no fucking clue where this is coming from - this is the only service subject to this random stop. And both RAM and discs are not even in spitting distance of being full.

Thankfully, Google searches like Systemd stops Mysql service are super helpful! /s

So, anyone who can tell me where to begin to look?

acrow

@Rhywden systemd has it's own ~~process~~daemon-starting system, which replaced SystemV. This new system lists dependencies. Basically, it may be that your application does not inform systemd of the dependency in its daemon declaration.

Let me see if I can find where those are now...

Edit:FTFM

acrow

@Rhywden This article explains how the system works, but does not tell where the files are, unfortunately:

Paul W. Frields / Nov 25, 2015

systemd: Unit dependencies and order - Fedora Magazine

Learn how the concepts of dependencies and order work in systemd -- and how they ultimately affect the time it takes to boot your system.

...Looks like it's in /etc/systemd/system ....according to:

Understanding Systemd Units and Unit Files | DigitalOcean

Increasingly, Linux distributions are adopting or planning to adopt the systemd init system. This powerful suite of software can manage many aspects of your …

It's been a while since I last touched these, so not right on top of my head.

Do you know where and what your service's unit-file is? You might want to read it. Maybe copy to here.

acrow

It may be that systemd just stops unneeded services now. If not, then stopping them for the duration of software updates is understandable. Not starting them due to lack of demand is understandable too, albeit this should be logged separately.

Rhywden

@acrow Yeah, but the thing is: The Mysql service starts automatically upon boot.

The more important question is why systemd shuts down the bloody service in the first place!

acrow

@Rhywden Well, he thing just before that in the log is the scheduled software update. Those restart a lot of things.

I'd look if MySql's unit-file has a "shutdown if unused" line. But the... umm... "Moodle" software should declare the dependency.

Rhywden

@acrow The unit file is almost identical to the one for the nginx service. And the mysql service was very much not "unused" at the time it shutdown.

Bloody stupid system that it does not tell you what actually initiated the shutdown.

Also, it's a database server. It should never be required to declare a dependency.

acrow

@Rhywden said in Systemd stops a service but does not restart it...:

@acrow The unit file is almost identical to the one for the nginx service.

Moodle's or MySql's file?
Also, I'm pretty sure that nginx does not depend on MySql.

And the mysql service was very much not "unused" at the time it shutdown.

How's systemd supposed to know that? Software doesn't exactly have a "busy now" flag, unless that API has been in since the OS was born. And even then idiots would ruin it for the rest of us.

Bloody stupid system that it does not tell you what actually initiated the shutdown.

The systemd logs can be made more verbose, I think. But don't ask me how...

Bottom line:
Does Moodle's unit-file have a "Requires=mysql" or similar line?

Edit: typos

Rhywden

@acrow It's a database. That bloody thing should never be shutdown because the access may very well come from remote. I will not use a "require" line because then I'll have the same problem again when I move the database to a server of its own.

If that would even solve anything.

Please also note that the line

Starting Daily apt upgrade and clean activities...

actually lies because it's not doing upgrades.

acrow

@Rhywden Ah. Moving to a different machine makes it... different. Although, you should then make sure that the Moodle unit-file has a "requires=networking" line.

In any case, can you post mysql's unit file? Let's see if we can spot a bad config line.

Rhywden

@acrow

# /lib/systemd/system/mysql.service
# MySQL systemd service file

[Unit]
Description=MySQL Community Server
After=network.target

[Install]
WantedBy=multi-user.target

[Service]
Type=forking
User=mysql
Group=mysql
PIDFile=/run/mysqld/mysqld.pid
PermissionsStartOnly=true
ExecStartPre=/usr/share/mysql/mysql-systemd-start pre
ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/run/mysqld/mysqld.pid
TimeoutSec=600
Restart=on-failure
RuntimeDirectory=mysqld
RuntimeDirectoryMode=755
LimitNOFILE=5000

There's nothing in there that says: "Please fuck everything up by randomly shutting down"

I also want to know where that shutdown command came from in the first place.

dkf

@Rhywden said in Systemd stops a service but does not restart it...:

Systemd

Rhywden

@dkf Not helping.

dkf

@Rhywden

acrow

@Rhywden Let's see...

Restart=on-failure seems suspect. You could change that to "always" and see if it helps.
Being shutdown by a software update is not exactly a failure.

Beyond that... it'll be beyond me too. I bet the answer is in this file. And if the problem started recently, in either apt-get's or systemd's release notes.
On the other hand, you're not the only one with this problem. MySql is widely used. So if an update borked it, an update will likely restore it. ...Maybe run a software update to see if the fix is already in the wild?

Rhywden

@acrow No, that's not a good thing because then it'll restart even if I myself shut it down manually. It's also not recommended.

acrow

@Rhywden That's fair. ...Was the shutdown always caused by a software update? As in, does the log always have that first? I'd do server updates manually, but you may not have the time...

Also, there's this:

PermissionsStartOnly alternative in systemd

I found PermissionsStartOnly setting in systemd is already deprecated according to systemd's NEWS file. PermissionsStartOnly= setting is deprecated (but is still supported for backwards compatibil...

So I think the default MySql unit file is going to be updated soon-ish. If it hasn't been already; you may have an old version. Apt-get does not replace settings files if they have been touched by the user (or anyone).

Rhywden

@acrow As I said, the only thing that line indicates is that it's doing apt-get update. It does not do upgrades.

dkf

@acrow I note in that linked answer that the restart increases the privilege with which the service is running (initially at least; well-behaved services drop such permissions if they get them). Nice subtle fuckup right there!

dkf

This seems relevant:

Automatically restart a service after unattended updates under systemd

I've added a service (Seafile, in this case) that I want to have running at all times to systemd with a service file. It works great, but every time the unattended updates run, the service gets shu...

There's also a great comment on the answer worthy of this site:

well I certainly didn't downvote you. Writing my own service files - which I clearly stated I did - isn't quite "administrative" "super user" stuff either. If people just want to swing around their e-p I wish they did it outside of honest, carefully written questions. Yes, I did consider other sites. Yes, I do consider writing my own service files part of my programming activities. Hence, after consideration, I posted on this site. What an absolute hostile environment here. Have an accept vote for +10 rep, that's all I can do. – Aaa Jan 30 '18 at 0:277

acrow

@dkf You need to either start as root or magic to reserve known ports, like port 80. Starting as root is the traditional way.

dkf

@acrow said in Systemd stops a service but does not restart it...:

You need to either start as root or magic to reserve known ports

Or to have the specific capability (typically to a particular port) granted to that user, if you've enabled such advanced security features.

Rhywden

So, I dug a bit further and it seems that the stupid system does do upgrades. Why did I think that it didn't? Because, well, you see, when I login I'm greeted with "7 updates available" or stuff like that.

Obviously the upgrade system does upgrades but not for everything.

Also, who in the fuck designed this system to do upgrades at random times? At least Windows pays lipservice to "We only update when not in use!" (mileage may vary), but this system seemingly does not give a single flying solitary fuck about whether the service is in use or not!

At least it's easily discoverable and configurable.

...

ach, sorry, don't know where that came from. Of course not!

Also please note that I explicitly chose the "Ubuntu 18.04 Server" variant.

acrow

@Rhywden said in Systemd stops a service but does not restart it...:

Obviously the upgrade system does upgrades but not for everything.

Also, who in the fuck designed this system to do upgrades at random times?

Security updates. It's assumed that any internet-connected machine wants to install those yesterday.

At least it's easily discoverable and configurable.

At least explicit command line commands for doing it are a quick google search away. And the google result does not redirect to a "we revamped the site page". Small victories, etc..

I explicitly chose the "Ubuntu 18.04 Server" variant.

If you want to install the GUI after all, just sudo apt-get install gnome3.

acrow

@dkf said in Systemd stops a service but does not restart it...:

advanced security features

I.e. magic.

Rhywden

@acrow said in Systemd stops a service but does not restart it...:

@Rhywden said in Systemd stops a service but does not restart it...:

Obviously the upgrade system does upgrades but not for everything.

Also, who in the fuck designed this system to do upgrades at random times?

Security updates. It's assumed that any internet-connected machine wants to install those yesterday.

Not if the application those updates are for is actually behind a firewall + NAT so not actually reachable from the internet. Plus the mysql server only listens on localhost currently.

powerlord

@acrow said in Systemd stops a service but does not restart it...:

@dkf You need to either start as root or magic to reserve known ports, like port 80. Starting as root is the traditional way.

It's actually because Ubuntu configures mysql to use a UNIX socket in /var/run; only root can create files in that directory. The default mysqld UNIX socket location is in /tmp

When mysqld is configured to use TCP sockets, it listens on port 3306, which is outside of the protected range.

Watson

@Rhywden said in Systemd stops a service but does not restart it...:

[Service]
Type=forking

I have very little to do with MySQL admin and a hazy memory of Fun with Upgrades, so this may be cargo-cult territory, but unless this is an old MySQL version Type=notify is recommended.

error

@Rhywden said in Systemd stops a service but does not restart it...:

actually behind a firewall + NAT so not actually reachable from the internet.

A firewall is one layer of security. Attackers can get onto your network via your thermostat or refrigerator, thanks to IoT.

Using that as an excuse to keep your servers unpatched is .

dkf

@error said in Systemd stops a service but does not restart it...:

@Rhywden said in Systemd stops a service but does not restart it...:

actually behind a firewall + NAT so not actually reachable from the internet.

A firewall is one layer of security. Attackers can get onto your network via your thermostat or refrigerator, thanks to IoT.

Using that as an excuse to keep your servers unpatched is .

The problem isn't now, not really. The problem is in a few years down the road when you've gone to work somewhere else and the new guy running the network doesn't realize that replacing that firewall+NAT box with some simple gateway is a bad idea.

Rhywden

@error I don't want to keep them unpatched. I just don't need them patched a nanosecond after the patch is out and I also don't need them patched during school hours while they're in active use. For some reason, the default setting was: "Oh, 9am is a good time to shut down and restart an essential service!"

error

@dkf said in Systemd stops a service but does not restart it...:

@error said in Systemd stops a service but does not restart it...:

@Rhywden said in Systemd stops a service but does not restart it...:

actually behind a firewall + NAT so not actually reachable from the internet.

A firewall is one layer of security. Attackers can get onto your network via your thermostat or refrigerator, thanks to IoT.

Using that as an excuse to keep your servers unpatched is .

The problem isn't now, not really. The problem is in a few years down the road when you've gone to work somewhere else and the new guy running the network doesn't realize that replacing that firewall+NAT box with some simple gateway is a bad idea.

Sounds like his problem.

HardwareGeek

@error SEP isn't a problem.

error

@HardwareGeek said in Systemd stops a service but does not restart it...:

@error SEP isn't a problem.

dcon

@Rhywden said in Systemd stops a service but does not restart it...:

"Oh, 9am is a good time to shut down and restart an essential service!"

Of course it is. When it's 9a GMT, it's still dark-o'clock in CA. That's all that matters.