Limiting needrestart in Ubuntu 24.04
We were in for a surprise when we upgraded our backup server from Ubuntu 20.04 LTS to 24.04 LTS: our backups started failing over the weekends. Here’s the story…
Our backup server at work runs Bacula with a PostgreSQL database and large Quantum tape library. Most clients are scheduled to run full backups to tape over the weekend and incremental backups to disk during the work week. So the weekend jobs are necessarily larger and longer-running than the weekday jobs.
When we first installed this server, it was running Ubuntu 20.04 LTS (focal). We skipped the upgrade to 22.04 LTS (jammy) because that release did not include a full suite of Bacula packages. (This was weird. I think it had to do with an OpenSSL mismatch, but my investigation was limited.)
I should note that over the life of the 20.04 installation, we discovered that it was necessary to exclude all the bacula and postgresql packages from software updates. Updates to any of them would cause jobs to fail. The Debian universe has an easy way to manage this:
apt-mark hold bacula-director bacula-fd bacula-sd postgresql-12
If Ubuntu released an important update to any of those packages, we had to shut down Bacula, remove the hold from the package(s) to be updated, update the packages, and restart Bacula. It’s a pain, but useful. With the packages marked with holds, we rarely had unexpected outages of the backup service.
As we started backing up newer servers, esp. those running Ubuntu 24.04 and Red Hat Enterprise Linux 9, we ran into a standard Bacula problem: Bacula server verion X can backup any client with a version less than or equal to X but not a client whose Bacula version is newer than X.
Ubuntu 20.04 included Bacula 9.4, but RHEL 9 includes Bacula 11.0 and Ubuntu 24.04 includes Bacula 13.0, so our backup service becoming obsolete.
Upgrading the server from 20.04 to 24.04 was mostly painless (our group has a decent amount of experience with Ubuntu double upgrades), and for a while all was well.
Until it wasn’t. Specifically, backups started failing over the weekends. The Bacula error messages not being able to contact the database server.
We had, as outlined above, placed holds on the bacula and postgresql packages so we knew they weren’t being upgraded without our knowledge.
After investigation, we discovered that in Ubuntu 24.04 a whole host of systemd-managed services were restarted just about any time there was a package update. Those restarts included the postgresql service, which caused the running bacula processes to lose contact with the database. Not good.
It appears the official way to manage the auto-restart list is via the needrestart application. It’s a complicated application, but the short version is that Ubuntu’s postgresql packages are built with lots of dependencies; if any of those dependencies was upgraded, needrestart would would restart PostgreSQL.
The answer was to create an exception file in /etc/needrestart/conf.d. It’s a snippet of Perl code. I used some existing files in that directory as my template because I never found any canonical documentation for this feature.
# /etc/needrestart/conf.d/postgresql.conf
# default to not restarting postgresql
$nrconf{override_rc}{qr(^postgresql.*\.service$)} = 0;
With that in place, PostgreSQL is no longer subject to unexpected restarts.