Rant : SystemD is too complex for Linux distributors
I could say too complex for Linux distributions, but it looks more like a lack of understanding of how SystemD works, and the fact that unit files get installed without asking leading to blocking issues.
The story
During the beginning of September, this server started to be
inaccessible. I guess it’s during this period. Turns out that I
checked my blog during August and it was still up !
Then holidays ended, I went back to work and… I checked my blog
back at the start of October.
Note that, I receive emails from the hosting company I use, if the
servers were to be “inaccessible” from their point of view.
But they only check their local network.
So, I tried to reach https://miouyouyou.fr … “Connection refused”….
Uh-oh…
Ok, let’s start a SSH shell ! “Connection refused”.
!!? What’s happening !? Was my server hijacked !? What the fuck !
I then tried to use my provider (Scaleway) “Web console” : Nothing.
(Spoiler alert : Turns out that Scaleway web console sucks)
Ugh… Okay… I got locked out from my server ? Is that some kind of “HA-HA ! You forgot to secure this part of your server ! pwned you, m0r0n !” ?
Let’s try a nmap -sS -v on my server !
Discovered open port 22/tcp
Wait, what !? What’s running on port 22 !? The SSH server is supposed to run on another port !
… I changed my client ~/.ssh/config to use port 22 instead of the
“configured” port. Then I retried to get a SSH Shell and…
Got a shell on my server !
… ?
Alright… ps auxww… nothing unusual…
Checked dmesg, checked journalctl… Nothing unusual !
Threw a tcpdump not port 22 for kicks… nothing unusual…
Maybe the machine wasn’t hacked ? I’m flairing some retarded system update now…
… Let’s see if something happens if we put the the system back together for the moment… I mean, I’m just hosting a static blog which content is available on a public git repository, which I can redeploy at any moment so, if it blows, I’ll order a new server unit.
docker container ls … Ok, the containers are down…
iptables -L … The firewall was reset ?
Fine, ran my script to restablish the firewall, deleted all the
containers, updated the docker images, deployed the new instances with
docker-compose AAAND, then, I tried to reach https://miouyouyou.fr
Success !
I got my blog back !
Updated the SSL certificates, the OCSP staples : Alright !
Then I tried to create a script to automate the blog updates
as much as possible, discovered that there’s some big differences
between the Hugo 0.46 on my machine and the Hugo 0.56 on the server,
which fucked up my templates real bad.
After a few hours, checking each Hugo release note, to understand
which update fucked up the templates, I pinpointed the issue,
updated the templates and Voila : My blog is restored !
Now, I can focus on the real issue !
WHY THE FUCK IS MY SSH SERVER LISTENING ON PORT 22 !
Checked /etc/ssh/sshd_service … It’s clearly written to listen
on another port.
I don’t get it… Either sshd is executed with specific instructions to not listen on this port or there’s a rogue ssh server !
root# ps auxww | grep ssh
root 2119 0.0 0.3 14272 7572 ? Ss 18:29 0:00 sshd: root@pts/0… ? sshd: root@pts/0 … ? Whaaat ?
Maybe there’s an environment variable fucking with the sshd server ?
root# ps auxwwe | grep sshd
root 2119 0.0 0.3 14272 7572 ? Ss 18:29 0:01 sshd: root@pts/0 =
root# systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:sshd(8)
man:sshd_config(5)DEAD !!!? WHO’S LAUNCHING THIS SERVER THEN ?
… Why don’t I see any real sshd server with ps auxwww ? Here,
it’s listing my connection as a server… ??? Whut ?
I can’t see anything related to sshd “as-is”.
Maybe it’s executed from the initrd file ? Hmm… This server is
certainly booted from the network so I’m not going to access the initrd
easily… (Well, turns out that I could actually, but didn’t know that
back then).
But even then, I should see it in the list…
Wait, if I run the ssh service with systemctl start ssh… and then
try to connect on the good port… Success !
WHHAAAAT !?
Then WHY !? Why systemd refused to execute this service on startup !?
… Maybe it’s not executed on startup ?
How do I check the services ran on startup again ?
root# systemctl -t service --state=active
UNIT LOAD ACTIVE SUB DESCRIPTION
blk-availability.service loaded active exited Availability of block devices
containerd.service loaded active running containerd container runtime
cron.service loaded active running Regular background program processing daemon
dbus.service loaded active running D-Bus System Message Bus
docker.service loaded active running Docker Application Container Engine
exim4.service loaded active running LSB: exim Mail Transport Agent
getty@tty1.service loaded active running Getty on tty1
getty@ttyAMA0.service loaded active running Getty on ttyAMA0
haveged.service loaded active running Entropy daemon using the HAVEGE algorithm
kmod-static-nodes.service loaded active exited Create list of required static device nodes for the current kernel
lvm2-monitor.service loaded active exited Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling
ntp.service loaded active running Network Time Service
polkit.service loaded active running Authorization Manager
rsyslog.service loaded active running System Logging Service
serial-getty@ttyAMA0.service loaded active running Serial Getty on ttyAMA0
ssh@0-10.X.Y.Z:22-A.B.C.D:39924.service loaded active running OpenBSD Secure Shell server per-connection daemon
sysstat.service loaded active exited Resets System Activity Data Collector
systemd-journal-flush.service loaded active exited Flush Journal to Persistent Storage
systemd-journald.service loaded active running Journal Service
systemd-logind.service loaded active running Login Service
systemd-modules-load.service loaded active exited Load Kernel Modules
systemd-networkd-wait-online.service loaded active exited Wait for Network to be Configured
systemd-networkd.service loaded active running Network Service
systemd-random-seed.service loaded active exited Load/Save Random Seed
systemd-remount-fs.service loaded active exited Remount Root and Kernel File Systems
systemd-resolved.service loaded active running Network Name Resolution
systemd-sysctl.service loaded active exited Apply Kernel Variables
systemd-sysusers.service loaded active exited Create System Users
systemd-tmpfiles-setup-dev.service loaded active exited Create Static Device Nodes in /dev
systemd-tmpfiles-setup.service loaded active exited Create Volatile Files and Directories
systemd-udev-trigger.service loaded active exited udev Coldplug all Devices
systemd-udevd.service loaded active running udev Kernel Device Manager
systemd-update-utmp.service loaded active exited Update UTMP about System Boot/Shutdown
systemd-user-sessions.service loaded active exited Permit User Sessions
ufw.service loaded active exited Uncomplicated firewall
unattended-upgrades.service loaded active running Unattended Upgrades Shutdown
user-runtime-dir@0.service loaded active exited User Runtime Directory /run/user/0
user@0.service loaded active running User Manager for UID 0 The IP were replaced by
10.X.Y.ZandA.B.C.Din this copy.
… What the fuck ? There’s a service executed for my IP, but there’s no ssh.service running the server ?
This shit is insane ! What the fuck is wrong with SystemD !?
No wait… maybe I’m blaming SystemD while it’s actually System-V generating issues.
root# grep sshd /etc/* -r
/etc/default/ssh:# Options to pass to sshd
/etc/init.d/ssh:# Provides: sshd
/etc/init.d/ssh:test -x /usr/sbin/sshd || exit 0
/etc/init.d/ssh:( /usr/sbin/sshd -\? 2>&1 | grep -q OpenSSH ) 2>/dev/null || exit 0
/etc/init.d/ssh: # forget it if we're trying to start, and /etc/ssh/sshd_not_to_be_run exists
/etc/init.d/ssh: if [ -e /etc/ssh/sshd_not_to_be_run ]; then
/etc/init.d/ssh: log_action_msg "OpenBSD Secure Shell server not in use (/etc/ssh/sshd_not_to_be_run)" || true
/etc/init.d/ssh: if [ ! -d /run/sshd ]; then
/etc/init.d/ssh: mkdir /run/sshd
/etc/init.d/ssh: chmod 0755 /run/sshd
/etc/init.d/ssh: if [ ! -e /etc/ssh/sshd_not_to_be_run ]; then
/etc/init.d/ssh: /usr/sbin/sshd $SSHD_OPTS -t || exit 1
/etc/init.d/ssh: log_daemon_msg "Starting OpenBSD Secure Shell server" "sshd" || true
/etc/init.d/ssh: if start-stop-daemon --start --quiet --oknodo --chuid 0:0 --pidfile /run/sshd.pid --exec /usr/sbin/sshd -- $SSHD_OPTS; then
/etc/init.d/ssh: log_daemon_msg "Stopping OpenBSD Secure Shell server" "sshd" || true
/etc/init.d/ssh: if start-stop-daemon --stop --quiet --oknodo --pidfile /run/sshd.pid --exec /usr/sbin/sshd; then
/etc/init.d/ssh: log_daemon_msg "Reloading OpenBSD Secure Shell server's configuration" "sshd" || true
/etc/init.d/ssh: if start-stop-daemon --stop --signal 1 --quiet --oknodo --pidfile /run/sshd.pid --exec /usr/sbin/sshd; then
/etc/init.d/ssh: log_daemon_msg "Restarting OpenBSD Secure Shell server" "sshd" || true
/etc/init.d/ssh: start-stop-daemon --stop --quiet --oknodo --retry 30 --pidfile /run/sshd.pid --exec /usr/sbin/sshd
/etc/init.d/ssh: if start-stop-daemon --start --quiet --oknodo --chuid 0:0 --pidfile /run/sshd.pid --exec /usr/sbin/sshd -- $SSHD_OPTS; then
/etc/init.d/ssh: log_daemon_msg "Restarting OpenBSD Secure Shell server" "sshd" || true
/etc/init.d/ssh: start-stop-daemon --stop --quiet --retry 30 --pidfile /run/sshd.pid --exec /usr/sbin/sshd || RET="$?"
/etc/init.d/ssh: if start-stop-daemon --start --quiet --oknodo --chuid 0:0 --pidfile /run/sshd.pid --exec /usr/sbin/sshd -- $SSHD_OPTS; then
/etc/init.d/ssh: status_of_proc -p /run/sshd.pid /usr/sbin/sshd sshd && exit 0 || exit $?
grep: /etc/motd: No such file or directory
/etc/pam.d/sshd:# access limits that are hard to express in sshd_config.
/etc/passwd:sshd:x:109:65534::/run/sshd:/usr/sbin/nologin
/etc/passwd-:sshd:x:109:65534::/run/sshd:/usr/sbin/nologin
/etc/shadow:sshd:*:17346:0:99999:7:::
/etc/shadow-:sshd:*:17346:0:99999:7:::
/etc/ssh/sshd_config:# See the sshd_config(5) manpage for details
/etc/ssh/sshd_config:# Use these options to restrict which interfaces/protocols sshd will bind to
/etc/ssh/sshd_config.ucf-dist:# $OpenBSD: sshd_config,v 1.103 2018/04/09 20:41:22 tj Exp $
/etc/ssh/sshd_config.ucf-dist:# This is the sshd server system-wide configuration file. See
/etc/ssh/sshd_config.ucf-dist:# sshd_config(5) for more information.
/etc/ssh/sshd_config.ucf-dist:# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin
/etc/ssh/sshd_config.ucf-dist:# The strategy used for options in the default sshd_config shipped with
/etc/ssh/sshd_config.ucf-dist:#PidFile /var/run/sshd.pid
/etc/systemd/system/scw-generate-ssh-keys.service:Before=sshd.service
/etc/systemd/system/scw-fetch-ssh-keys.service:Before=sshd.serviceHa ! /etc/init.d/ssh ! Maybe it’s that stupid service that’s
generating issues ! … Why is Debian mixing System-V with SystemD ?
Either go System-V or go SystemD… Don’t do both, it’s irritating…
Ok, let’s check if this script is actually run ! Let’s edit it and
add a echo MEOW > /tmp/stoopid after set -e.
root# /etc/init.d/ssh restart
[ ok ] Restarting ssh (via systemctl): ssh.service.
root# cat /tmp/stoopid
MEOW
root# systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2019-10-25 19:48:28 UTC; 13min ago
Docs: man:sshd(8)
man:sshd_config(5)
Process: 3914 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Main PID: 3915 (sshd)
Tasks: 1 (limit: 2377)
Memory: 1016.0K
CGroup: /system.slice/ssh.service
└─3915 /usr/sbin/sshd -D
Oct 25 19:48:27 myy-blargh systemd[1]: Starting OpenBSD Secure Shell server...
Oct 25 19:48:28 myy-blargh sshd[3915]: Server listening on 0.0.0.0 port N.
Oct 25 19:48:28 myy-blargh sshd[3915]: Server listening on :: port N.
Oct 25 19:48:28 myy-blargh systemd[1]: Started OpenBSD Secure Shell server.The actual port was replaced by
Nin this copy of the logs.
Hmm… If reload it, SystemD is considering the service loaded as well… Both systems are cooperating correctly then ?
Alright, let’s reboot and see if the file is present on reboot !
After reboot :
root# cat /tmp/stoopid
cat: /tmp/stoopid: No such file or directory… This script is not executed ?? … Maybe it’s the initrd after all ?
Turns out that Scaleway mount the initrd file in
/run/initramfs.
root# grep ssh /run/initramfs/*
Binary file /run/initramfs/bin/busybox matches
/run/initramfs/functions:start_sshd() {
/run/initramfs/functions: run mkdir -p /etc/dropbear /root/.ssh
/run/initramfs/functions: run chmod 700 /root/.ssh
/run/initramfs/functions: run sh -ec "scw-metadata --cached | grep 'SSH_PUBLIC_KEYS_.*_KEY' | cut -d'=' -f 2- | tr -d \' > /root/.ssh/authorized_keys"
/run/initramfs/functions: run sh -ec "scw-metadata --cached | grep 'TAGS_.*=AUTHORIZED_KEY' | cut -d'=' -f 3- | sed 's/_/\ /g' >> /root/.ssh/authorized_keys"
/run/initramfs/init:log_begin_msg "Checking metadata for debug sshd (dropbear)"
/run/initramfs/init: log_success_msg "Starting a debug sshd"
/run/initramfs/init: start_sshd
/run/initramfs/init: ewarn "You can connect to your server with 'scw' or 'ssh'"
/run/initramfs/init: ewarn " -- ssh root@${PUBLIC_IP_ADDRESS}"
/run/initramfs/init: ewarn "You can connect to your server with 'ssh'"
/run/initramfs/init: ewarn " -- ssh root@${PUBLIC_IP_ADDRESS}"
/run/initramfs/init:# Ensure sshd is killed if running
Binary file /run/initramfs/lib/aarch64-linux-gnu/libtinfo.so.5.9 matches
Binary file /run/initramfs/usr/sbin/dropbear matches
Binary file /run/initramfs/usr/bin/dropbearkey matchesOoooh, here you go ! Let’s edit /run/initramfs/init and check how
they # Ensure sshd is killed if running :
# Ensure sshd is killed if running
if [ "$(pgrep dropbear)" != "" ]; then
run killall dropbear
fiHmm :
root# ps auxww | grep drop
root 4016 0.0 0.0 5796 648 pts/0 S+ 20:07 0:00 grep dropOk… Nothing… I don’t get it…
Let’s check for rootkits, just in case.
root# apt install chkrootkit
root# chkrootkitNo problems detected…
I’m tired… I’m tired of this shit, there’s a fucking SSH server running on my machine, I have NO idea who’s spawning it !
Oh wait ! I forgot about lsof !
lsof -tNo, wrong one… Couldn’t they use netstat syntax ?
lsof -isystemd 1 root 52u IPv6 17371 0t0 TCP TCP *:22 (LISTEN)
O_O … …
…
O_O !
YOU’RE FUCKING KIDDING ME !? SystemD ITSELF IS LISTENING ON PORT 22 !? But the SSH service is dead ! HOW ?
A little search on the internet, got me this gem : https://bbs.archlinux.org/viewtopic.php?id=166582
“So, ive found out that sshd.socket was enabled and this was the cause”
…
root# find /etc -name "ssh*.socket"
/etc/systemd/system/sockets.target.wants/ssh.socket
root# cat /etc/systemd/system/sockets.target.wants/ssh.socket
[Unit]
Description=OpenBSD Secure Shell server socket
Before=ssh.service
Conflicts=ssh.service
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run
[Socket]
ListenStream=22
Accept=yes
[Install]
WantedBy=sockets.targetFUCK YOU ! FUCK THIS SHIT !
I AM DONE WITH SYSTEMD !
The real issue with SystemD
It’s OVERLY COMPLEX ! If you don’t remember all the commands and, most importantly, ALL THE WAYS SYSTEMD CAN START A SERVICE, YOU WILL NOT be able to understand what’s going on.
systemctl status ssh was indicating the status of the
ssh.service file, not the status or presence of ssh.socket.
Understand that I threw a journactl and searched for SSH and saw
sshd being started and receiving connections as sshd[PID], but not
why, nor how it was started ! And the logs were about sshd[PID]
not systemd[PID] !
So until you understand that SystemD can start services using socket connections (Overkill feature for simple servers), you will NEVER KNOW :
- Why is the SSH service not started
- Why is there a SSH server listening on another port than the one provided in the configuration
Understand that I had a running server until start of September, the server got rebooted for whatever reasons and THEN, POOF, the SSH server started to listen on port 22.
So, the real issue with SystemD is that it’s TOO COMPLEX FOR DISTRIBUTORS !.
It can do a ton of things and maybe, once you understand it
perfectly, you’ll be happy to use all the bells and whistles…
If you need them, of course…
However :
systemctl status sshjust shown thesshservice as DEAD !journalctlhad some “SSH connections” entries, but didn’t show why it started an SSH service !ps auxww | grep sshdmade it look like SSH servers are run “on the fly”.- MY ACTUAL SSHD CONFIGURATION WAS COMPLETELY IGNORED.
It’s those things combined which make me hate SystemD for server
management.
It’s UNRELIABLE !
If something breaks, I want AS MUCH INFORMATION AS POSSIBLE !
If SystemD start a sshd server, systemctl status ssh or
systemctl status sshd should show information about it !
I don’t give two shits about the file extension of the unit file
triggering the execution of “sshd” !
That said, the issue here is not only due to SystemD overcomplexity. It’s also the fact that some distributors thought :
“Hey ! Let’s put a ssh.socket in
/etc/systemd/system/sockets.target.wants/ , so that the user will
never know why his SSH server start ignoring /etc/ssh/sshd_config !
This will be so much fun !”
The fact is : even if I remove this .socket file, it might just
come back after an update ! And fuck up my system again !
Do you understand the issue here ? I’M LOSING CONTROL OF THE SYSTEM ! Because distributors started to ship with an OVERCOMPLEX INIT, and started using features of this init system without understanding the consequences !
If I put CONFIGURATION DIRECTIVES in a CONFIGURATION FILE, I don’t want them to be OVERRIDEN BY SOME RANDOM .whatever FILE used by my init system. If I want a different configuration, I’ll either edit the configuration file, or force the daemon to use another configuration file by editing the appropriate init file.
I’d appreciate if distributions using SystemD went with the “least amount of SystemD unit files” on server configurations.
Look at “Clear Linux”, there’s a way to use SystemD while making
things “Lean & Clean”.
Just check the /etc/ folder after installing Clear Linux :
It’s clean !
They don’t add tons of .service, .socket, .mount or
.whatever extension systemd reacts on !
No, they put a clean and lean /etc directory, with only the
strictly necessary files.
And it works !
That said, an Init system so complex do not interest me. I’ll
brush up my systemd-foo on Arch Linux, because I really need
it. But, still, I don’t give it a shit about SystemD on my
servers ATM.
If you run tons of microservices, maybe you do.
But me ? Nope. All my main public services run in docker
containers and the only daemons and configurations I care
about on my system are :
- The SSH server
- The firewall
- The Docker containers
I could manage this with a busybox ‘init’ file…
So I’ll start looking for anoter distro that I can deploy on
my server, and which doesn’t use SystemD.
And I can’t find one, I’ll look for one using SystemD with
the least amount of units files.
I’m done with Debian and SystemD.
And I’d like to be done playing detective to understand :
- Why my server is not responding anymore ?
- Why services are executing while ignoring their configurations ?
- How I can I avoid traps added by systemd updates ?
