Rant : QML Horrors

I decided to make a Stock Photo Selector with QT Widgets.
Then, after quickly understanding how the QT Widgets system is old and a pain to deal with, when dealing with web resources, I decided to go the QML route.

The thing with QML is that you can quickly put an interface in place, add some code and then… When you try to architecture the whole program, things get complicated.

The idea

So, the whole concept of the Stock Photo Selector is to test Stock Photos on websites quickly and efficiently. There’s two objectives :

  • Be able to test the Stock Photos on PC and mobiles
  • Be able to save the right Stock Photo in the right folder of the website when done.

The first part can be done with simple Javascript, the second part however, not so much.

So, I went like “How about I create a search engine app with QT. It will search the photo. Once I double-click a photo, it sends it to the browser through a WebSocket, which then put it on the right place. Then when I got the right image, I just click ‘Save on server’ in the app and boom, done !“.

Sure, there’s still parts that are not clearly decided yet.
But I don’t care, because I want to test if the whole WebSocket communication is possible, in the first place.

Dealing with QML

QT Creator is almost useless with QML. Seriously.

The only thing that “kind of matter” is the Designer.
The IDE can really kick in when adding C++ helpers. But that’s it. For the rest, the IDE is USELESS. It provides almost no useful information, and can sometimes even provide idiotic completion or add things that are not understood by the QML parser.

I won’t even talk about the Designer crashing here and there, when the interface gets “too complicated”, and how the categorization of the wigets (?) make no sense most of the time.

In terms of “Debugging”, we’re far from In-Browser Javascript or Electron (kind of the same thing).

First, QML is an UI descriptive language. You can describe your UI in a human-readable manner. Something like this :

Window {
    id: window
    visible: true
    width: 640
    height: 480
    title: qsTr("Hello World")
    MyyToolbar {
        id: toolbar
        x: 0
        anchors.bottom: parent.bottom

        searchButton.onClicked: {
            var http = new XMLHttpRequest()
            var url = "https://pixabay.com/api/?image_type=photo&q=" + query.text + "&key=" + config.pixabay_key;
            http.open("GET", url, true);
            http.onreadystatechange = function() {
              if (http.readyState === 4) {
                  if (http.status === 200) {
                  else {


However, trying to debug UI problems quickly become a pain in the ass, if you’re not accustomed to the weird QT Creator UI.

By default, the debugger is UNABLE to provide you a good view of the layout tree. If you click on “Pause” during the execution, you’ll see this :


When I saw this I was like “??? So… ? app maybe ? Let’s see…”

|  >-[d]
|  >-[parent]
|  v-[children]
|  |  >-[0]
|  |  >-[1]
|  |  >-[2]
|  |  >-[3]
|  |  >-[4]
|  |  >-[5]
|  |  >-[6]
|  |  >-[7]

Then I was like “Whaaat ?”

I then discovered on the debugging view, you have your main code on the center, the locals on the right, and the stack trace pane below. In the toolbar of the stack trace pane, you have :

  • a menu Debugger ▼ which basically helps you swap this pane place with another pane
  • a menu GDB for “NameOfYourProject” ▼ which dictate which debugger you want to choose.

Turns out that, here, you want to click on GDB for “NameOfYourProject” ▼, then select QML for “NameOfYourProject” in the dropdown menu.
THEN, you’ll be able to see something that looks like this :


If you expand the window list, you’ll see all the children of your view, with all its properties and such.

Which brings my first question :


I hate the whole “Where’s Wally” UI game. Having to specifically click ONE element of a pane which seems to only dispay stack traces (with thread selection, step by step debugging, assembler debugging, …), in order to see my UI elements is madness.
I kind of understand the mental process that went behind it “Well, to select which locals you want to display, you’ll have to select the right thread in the stack trace pane, so we’ll put the menu there so that they can switch between C++ locals and QML/Javascript locals” but… ugh… it’s still unintuitive.

I’d highly prefer a tool that’s specifically oriented towards UI debugging.
If you programmed with Android, you know what I’m talking about.
I’m talking about a panel where you can see all the currently instantiated UI elements.
Following the previous example, such a panel would look like this :

	⌞ MyyToolbar
		⌞ TextField
		⌞ ComboBox
		⌞ Button

Then when you click on an element, it selects it in the program and you can view its properties. Like its position, width, height, z-index, …

That kind of thing helps TREMENDOUSLY in trying to determine why something is not displayed correctly.

But well, with QT Creator, you don’t have this.

But wait, there’s more !

Dimensions hell

So, in my app, I was able to display the thumbnails of the Pixabay’s queries results in a Grid, and I was able to trigger the download of a decent resolution version of a photo when clicking on it…

“Wow, you were able to display images in a Grid and add an OnClickListener ! Must have been really hard ! /s”

Well, YES ! IT WAS !

Due to some to mystic positioning and size rules on the Image widget, I lost an entire day figuring out how to add an “Click” listener on a freaking Image widget !

By default, in my case, the Image widget would start with no width and height, because it changes based on the width and height of the thumbnail downloaded.

So, when adding a Mouse Area as a child, the MouseArea starts with a width and height of 0.
And even with anchors.fill, changing the width and height properties of Image will NOT change the properties of the MouseArea child. Because… REASONS !

Trying to put a height/width on the Image on start also didn’t work.

However ! When doing something like this :

Rectangle {
	Image { anchors.fill: parent }
	MouseArea { anchors.fill: parent }

It works !

Turns out that updating the Rectangle dimensions, when displaying the thumbnail in Image, makes the MouseArea redimension itself correctly and the whole things start working.

YAY ! I just lost an entire day !

Broken WebSocket and useless Javascript debugger

So, the pictures are displayed correctly (but are still going over the toolbar for reasons I don’t get !… ugh… z-index maybe ?), I can click on them :

Now, what’s remaining is downloading them and sending them through WebSocket.

The setup of the WebSocket server was impressively easy ! Almost no fuss at all. It just worked as intended.

So, I found a way to save the WebSocket object provided during the client connection. Then, when clicking on a thumbnail, I triggered an HTTP Request with a good old XMLHttpRequest object, and when receiving the response, I did :


On the browser side, I first tried to rush and do something like this :

if (message.constructor === Blob) {
	var reader = new FileReader();
	reader.onloadend = function() {
		var base64data = reader.result.substring("data:application/octet-stream;base64,".length);
		const mimed_data = "url('data:image/jpg;base64," + base64data + "')";
		document.querySelector("#wss_image").style.backgroundImage = mimed_data;

And… it didn’t work…

Hmm… I tried to copy the base64 to a file and convert it back, to see how it went.


Now, let me just say how most of today apps can’t deal with large block of text. KDE apps being on the worst offender side.

In Firefox, the UI was almost to a crawl when trying to select the large block of Base64 text representing the picture.

Seriously, I tried to save it in Kate : Nope. I tried to save it in KWrite : Nope. Then, I launched notepad through Wine. YES, seriously !


But it was sluggish as hell…

Then I brought emacs and did the operation in 3 seconds.

What’s the spec of the machine doing this already ? Hmm ?
Oh, it’s just an 8 core / 16 threads AMD Ryzen 7 2700X with 32GB of RAM and NVMe disks. So it’s normal if it can’t copy 2MB of text…

KDE apps are so well optimised.

Back on track

I saved the base64 block in a text file, decoded it with a simple Ruby script :

require "base64"
File.write("decoded.jpg", Base64.decode64(File.read("encoded_image.txt")))

Then downloaded the actual picture on Firefox, saved it and compared the size… ALMOST TWICE THE SIZE !?… WHAT ?


Can I see the headers ?

ef bf bd ef bf bd ef bf bd ef bf bd 00 10 4a 46 49 46

4a 46 49 46 is JFIF. But what’s that ef bf bd thing ?

Let’s compare it with a valid JPG header.

ff d8 ff e0 00 10 4a 46 49 46

… ??? Okay ??? Where’s the ff d8 ff sequence in the converted version ? Did it convert it ?

If I do a websearch for ef bf bd ef, maybe I’ll get a clue about what’s going on…

… It’s an UTF-8 to Latin-1 decoding issue … !!?




Wait, maybe that’s Firefox ? Maybe I didn’t setup the whole <meta charset="utf8"> correctly and it starts to convert data.

Let’s bring Wireshark, listen on lo, redo the whole operation;



For fuck sake, this reminds me the whole bullshit that reading files is on Windows. If you don’t open files in “binary” mode on Windows, lonely \x0d and \x0a are automatically converted into \x0d\x0a sequences, for archaic Mac OS and UNIX compatability reasons, botching up binary files to no end.

Let’s bring up the debugger

Back to the QML code, I put a breakpoint just below if (http.readyState === 4 && http.status === 200) { in the onreadystatechange callback of the ‘XMLHttpRequest’ used for the request.

Basically, it looks like this :

http.onreadystatechange = function() {
	if (http.readyState === 4 && http.status === 200) {
		console.debug("Got : " + picture.data["largeImageURL"]);

You’d be surprised to know that the size provided by http.response.length and http.responseText.length are identical and they also also seem wrong.
They’re down a few kilo-bytes compared to the original image…
I don’t know why.

But it’s way different from the size of the element sent via sendBinaryMessage which inflates the size by almost 2 times.

So I put a breakpoint, run the code with the debugger, the debugger stops and in the locals, all I can see is this :

|  ⌞onreadystatechange
	⌞ ...

So this is empty. http has one field onreadystatechange and… that’s it. onClicked is empty and mouse has fields corresponding the pointer properties when clicking the area.

The locals pane is USELESS to me, here.
Sure I can test a few things in the QML console, but that doesn’t help me that much.

Ok, how about Step into, maybe I can see the code of that sendBinaryMessage thing… it just jumped onto the next console.debug line…

How stupid this thing can be ?

Switching back to GDB for … shows nothing…

Ugh… so I have zero idea about :

  • Why did it do a conversion ?
  • How to prevent it ?

I LOVE to code with QML ?

The idea now is to add some C++ helpers, hoping that the C++ part is not AS broken AS the Javascript part.
Worse part, I’ll have to resort to an external WebSocket C/C++ library, use it in C++ helpers and hope that it does the trick.

Rant : SystemD is too complex for Linux distributors

I could say too complex for Linux distributions, but it looks more like a lack of understanding of how SystemD works, and the fact that unit files get installed without asking leading to blocking issues.

The story

During the beginning of September, this server started to be inaccessible. I guess it’s during this period. Turns out that I checked my blog during August and it was still up !
Then holidays ended, I went back to work and… I checked my blog back at the start of October.

Note that, I receive emails from the hosting company I use, if the servers were to be “inaccessible” from their point of view.
But they only check their local network.

So, I tried to reach https://miouyouyou.fr … “Connection refused”….
Ok, let’s start a SSH shell ! “Connection refused”.
!!? What’s happening !? Was my server hijacked !? What the fuck !

I then tried to use my provider (Scaleway) “Web console” : Nothing.
(Spoiler alert : Turns out that Scaleway web console sucks)

Ugh… Okay… I got locked out from my server ? Is that some kind of “HA-HA ! You forgot to secure this part of your server ! pwned you, m0r0n !” ?

Let’s try a nmap -sS -v on my server !

Discovered open port 22/tcp

Wait, what !? What’s running on port 22 !? The SSH server is supposed to run on another port !

… I changed my client ~/.ssh/config to use port 22 instead of the “configured” port. Then I retried to get a SSH Shell and…
Got a shell on my server !

… ?

Alright… ps auxww… nothing unusual…
Checked dmesg, checked journalctl… Nothing unusual !

Threw a tcpdump not port 22 for kicks… nothing unusual…

Maybe the machine wasn’t hacked ? I’m flairing some retarded system update now…

… Let’s see if something happens if we put the the system back together for the moment… I mean, I’m just hosting a static blog which content is available on a public git repository, which I can redeploy at any moment so, if it blows, I’ll order a new server unit.

docker container ls … Ok, the containers are down… iptables -L … The firewall was reset ?

Fine, ran my script to restablish the firewall, deleted all the containers, updated the docker images, deployed the new instances with docker-compose AAAND, then, I tried to reach https://miouyouyou.fr
Success !
I got my blog back !

Updated the SSL certificates, the OCSP staples : Alright !

Then I tried to create a script to automate the blog updates as much as possible, discovered that there’s some big differences between the Hugo 0.46 on my machine and the Hugo 0.56 on the server, which fucked up my templates real bad.
After a few hours, checking each Hugo release note, to understand which update fucked up the templates, I pinpointed the issue, updated the templates and Voila : My blog is restored !

Now, I can focus on the real issue !


Checked /etc/ssh/sshd_service … It’s clearly written to listen on another port.

I don’t get it… Either sshd is executed with specific instructions to not listen on this port or there’s a rogue ssh server !

root# ps auxww | grep ssh
root      2119  0.0  0.3  14272  7572 ?        Ss   18:29   0:00 sshd: root@pts/0

… ? sshd: root@pts/0 … ? Whaaat ?

Maybe there’s an environment variable fucking with the sshd server ?

root# ps auxwwe | grep sshd
root      2119  0.0  0.3  14272  7572 ?        Ss   18:29   0:01 sshd: root@pts/0                                                                                                                                                                                                                                          =
root# systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:sshd(8)


… Why don’t I see any real sshd server with ps auxwww ? Here, it’s listing my connection as a server… ??? Whut ?
I can’t see anything related to sshd “as-is”.

Maybe it’s executed from the initrd file ? Hmm… This server is certainly booted from the network so I’m not going to access the initrd easily… (Well, turns out that I could actually, but didn’t know that back then).
But even then, I should see it in the list…

Wait, if I run the ssh service with systemctl start ssh… and then try to connect on the good port… Success !


Then WHY !? Why systemd refused to execute this service on startup !?

… Maybe it’s not executed on startup ?

How do I check the services ran on startup again ?

root# systemctl -t service --state=active
UNIT                                               LOAD   ACTIVE SUB     DESCRIPTION                                                                  
blk-availability.service                           loaded active exited  Availability of block devices                                                
containerd.service                                 loaded active running containerd container runtime                                                 
cron.service                                       loaded active running Regular background program processing daemon                                 
dbus.service                                       loaded active running D-Bus System Message Bus                                                     
docker.service                                     loaded active running Docker Application Container Engine                                          
exim4.service                                      loaded active running LSB: exim Mail Transport Agent                                               
getty@tty1.service                                 loaded active running Getty on tty1                                                                
getty@ttyAMA0.service                              loaded active running Getty on ttyAMA0                                                             
haveged.service                                    loaded active running Entropy daemon using the HAVEGE algorithm                                    
kmod-static-nodes.service                          loaded active exited  Create list of required static device nodes for the current kernel           
lvm2-monitor.service                               loaded active exited  Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling
ntp.service                                        loaded active running Network Time Service                                                         
polkit.service                                     loaded active running Authorization Manager                                                        
rsyslog.service                                    loaded active running System Logging Service                                                       
serial-getty@ttyAMA0.service                       loaded active running Serial Getty on ttyAMA0                                                      
ssh@0-10.X.Y.Z:22-A.B.C.D:39924.service loaded active running OpenBSD Secure Shell server per-connection daemon                            
sysstat.service                                    loaded active exited  Resets System Activity Data Collector                                        
systemd-journal-flush.service                      loaded active exited  Flush Journal to Persistent Storage                                          
systemd-journald.service                           loaded active running Journal Service                                                              
systemd-logind.service                             loaded active running Login Service                                                                
systemd-modules-load.service                       loaded active exited  Load Kernel Modules                                                          
systemd-networkd-wait-online.service               loaded active exited  Wait for Network to be Configured                                            
systemd-networkd.service                           loaded active running Network Service                                                              
systemd-random-seed.service                        loaded active exited  Load/Save Random Seed                                                        
systemd-remount-fs.service                         loaded active exited  Remount Root and Kernel File Systems                                         
systemd-resolved.service                           loaded active running Network Name Resolution                                                      
systemd-sysctl.service                             loaded active exited  Apply Kernel Variables                                                       
systemd-sysusers.service                           loaded active exited  Create System Users                                                          
systemd-tmpfiles-setup-dev.service                 loaded active exited  Create Static Device Nodes in /dev                                           
systemd-tmpfiles-setup.service                     loaded active exited  Create Volatile Files and Directories                                        
systemd-udev-trigger.service                       loaded active exited  udev Coldplug all Devices                                                    
systemd-udevd.service                              loaded active running udev Kernel Device Manager                                                   
systemd-update-utmp.service                        loaded active exited  Update UTMP about System Boot/Shutdown                                       
systemd-user-sessions.service                      loaded active exited  Permit User Sessions                                                         
ufw.service                                        loaded active exited  Uncomplicated firewall                                                       
unattended-upgrades.service                        loaded active running Unattended Upgrades Shutdown                                                 
user-runtime-dir@0.service                         loaded active exited  User Runtime Directory /run/user/0                                           
user@0.service                                     loaded active running User Manager for UID 0     

The IP were replaced by 10.X.Y.Z and A.B.C.D in this copy.

… What the fuck ? There’s a service executed for my IP, but there’s no ssh.service running the server ?

This shit is insane ! What the fuck is wrong with SystemD !?

No wait… maybe I’m blaming SystemD while it’s actually System-V generating issues.

root# grep sshd /etc/* -r
/etc/default/ssh:# Options to pass to sshd
/etc/init.d/ssh:# Provides:             sshd
/etc/init.d/ssh:test -x /usr/sbin/sshd || exit 0
/etc/init.d/ssh:( /usr/sbin/sshd -\? 2>&1 | grep -q OpenSSH ) 2>/dev/null || exit 0
/etc/init.d/ssh:    # forget it if we're trying to start, and /etc/ssh/sshd_not_to_be_run exists
/etc/init.d/ssh:    if [ -e /etc/ssh/sshd_not_to_be_run ]; then 
/etc/init.d/ssh:            log_action_msg "OpenBSD Secure Shell server not in use (/etc/ssh/sshd_not_to_be_run)" || true
/etc/init.d/ssh:    if [ ! -d /run/sshd ]; then
/etc/init.d/ssh:        mkdir /run/sshd
/etc/init.d/ssh:        chmod 0755 /run/sshd
/etc/init.d/ssh:    if [ ! -e /etc/ssh/sshd_not_to_be_run ]; then
/etc/init.d/ssh:        /usr/sbin/sshd $SSHD_OPTS -t || exit 1
/etc/init.d/ssh:        log_daemon_msg "Starting OpenBSD Secure Shell server" "sshd" || true
/etc/init.d/ssh:        if start-stop-daemon --start --quiet --oknodo --chuid 0:0 --pidfile /run/sshd.pid --exec /usr/sbin/sshd -- $SSHD_OPTS; then
/etc/init.d/ssh:        log_daemon_msg "Stopping OpenBSD Secure Shell server" "sshd" || true
/etc/init.d/ssh:        if start-stop-daemon --stop --quiet --oknodo --pidfile /run/sshd.pid --exec /usr/sbin/sshd; then
/etc/init.d/ssh:        log_daemon_msg "Reloading OpenBSD Secure Shell server's configuration" "sshd" || true
/etc/init.d/ssh:        if start-stop-daemon --stop --signal 1 --quiet --oknodo --pidfile /run/sshd.pid --exec /usr/sbin/sshd; then
/etc/init.d/ssh:        log_daemon_msg "Restarting OpenBSD Secure Shell server" "sshd" || true
/etc/init.d/ssh:        start-stop-daemon --stop --quiet --oknodo --retry 30 --pidfile /run/sshd.pid --exec /usr/sbin/sshd
/etc/init.d/ssh:        if start-stop-daemon --start --quiet --oknodo --chuid 0:0 --pidfile /run/sshd.pid --exec /usr/sbin/sshd -- $SSHD_OPTS; then
/etc/init.d/ssh:        log_daemon_msg "Restarting OpenBSD Secure Shell server" "sshd" || true
/etc/init.d/ssh:        start-stop-daemon --stop --quiet --retry 30 --pidfile /run/sshd.pid --exec /usr/sbin/sshd || RET="$?"
/etc/init.d/ssh:                if start-stop-daemon --start --quiet --oknodo --chuid 0:0 --pidfile /run/sshd.pid --exec /usr/sbin/sshd -- $SSHD_OPTS; then
/etc/init.d/ssh:        status_of_proc -p /run/sshd.pid /usr/sbin/sshd sshd && exit 0 || exit $?
grep: /etc/motd: No such file or directory
/etc/pam.d/sshd:# access limits that are hard to express in sshd_config.
/etc/ssh/sshd_config:# See the sshd_config(5) manpage for details
/etc/ssh/sshd_config:# Use these options to restrict which interfaces/protocols sshd will bind to
/etc/ssh/sshd_config.ucf-dist:# $OpenBSD: sshd_config,v 1.103 2018/04/09 20:41:22 tj Exp $
/etc/ssh/sshd_config.ucf-dist:# This is the sshd server system-wide configuration file.  See
/etc/ssh/sshd_config.ucf-dist:# sshd_config(5) for more information.
/etc/ssh/sshd_config.ucf-dist:# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin
/etc/ssh/sshd_config.ucf-dist:# The strategy used for options in the default sshd_config shipped with
/etc/ssh/sshd_config.ucf-dist:#PidFile /var/run/sshd.pid

Ha ! /etc/init.d/ssh ! Maybe it’s that stupid service that’s generating issues ! … Why is Debian mixing System-V with SystemD ? Either go System-V or go SystemD… Don’t do both, it’s irritating…

Ok, let’s check if this script is actually run ! Let’s edit it and add a echo MEOW > /tmp/stoopid after set -e.

root# /etc/init.d/ssh restart
[ ok ] Restarting ssh (via systemctl): ssh.service.
root# cat /tmp/stoopid
root# systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2019-10-25 19:48:28 UTC; 13min ago
     Docs: man:sshd(8)
  Process: 3914 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
 Main PID: 3915 (sshd)
    Tasks: 1 (limit: 2377)
   Memory: 1016.0K
   CGroup: /system.slice/ssh.service
           └─3915 /usr/sbin/sshd -D

Oct 25 19:48:27 myy-blargh systemd[1]: Starting OpenBSD Secure Shell server...
Oct 25 19:48:28 myy-blargh sshd[3915]: Server listening on port N.
Oct 25 19:48:28 myy-blargh sshd[3915]: Server listening on :: port N.
Oct 25 19:48:28 myy-blargh systemd[1]: Started OpenBSD Secure Shell server.

The actual port was replaced by N in this copy of the logs.

Hmm… If reload it, SystemD is considering the service loaded as well… Both systems are cooperating correctly then ?

Alright, let’s reboot and see if the file is present on reboot !

After reboot :

root# cat /tmp/stoopid
cat: /tmp/stoopid: No such file or directory

… This script is not executed ?? … Maybe it’s the initrd after all ?

Turns out that Scaleway mount the initrd file in /run/initramfs.

root# grep ssh /run/initramfs/*
Binary file /run/initramfs/bin/busybox matches
/run/initramfs/functions:start_sshd() {
/run/initramfs/functions:    run mkdir -p /etc/dropbear /root/.ssh
/run/initramfs/functions:    run chmod 700 /root/.ssh
/run/initramfs/functions:    run sh -ec "scw-metadata --cached | grep 'SSH_PUBLIC_KEYS_.*_KEY' | cut -d'=' -f 2- | tr -d \' > /root/.ssh/authorized_keys"
/run/initramfs/functions:    run sh -ec "scw-metadata --cached | grep 'TAGS_.*=AUTHORIZED_KEY' | cut -d'=' -f 3- | sed 's/_/\ /g' >> /root/.ssh/authorized_keys"
/run/initramfs/init:log_begin_msg "Checking metadata for debug sshd (dropbear)"
/run/initramfs/init:    log_success_msg "Starting a debug sshd"
/run/initramfs/init:    start_sshd
/run/initramfs/init:    ewarn "You can connect to your server with 'scw' or 'ssh'"
/run/initramfs/init:    ewarn " -- ssh root@${PUBLIC_IP_ADDRESS}"
/run/initramfs/init:    ewarn "You can connect to your server with 'ssh'"
/run/initramfs/init:    ewarn " -- ssh root@${PUBLIC_IP_ADDRESS}"
/run/initramfs/init:# Ensure sshd is killed if running
Binary file /run/initramfs/lib/aarch64-linux-gnu/libtinfo.so.5.9 matches
Binary file /run/initramfs/usr/sbin/dropbear matches
Binary file /run/initramfs/usr/bin/dropbearkey matches

Ooooh, here you go ! Let’s edit /run/initramfs/init and check how they # Ensure sshd is killed if running :

# Ensure sshd is killed if running
if [ "$(pgrep dropbear)" != "" ]; then
    run killall dropbear

Hmm :

root# ps auxww | grep drop
root      4016  0.0  0.0   5796   648 pts/0    S+   20:07   0:00 grep drop

Ok… Nothing… I don’t get it…
Let’s check for rootkits, just in case.

root# apt install chkrootkit
root# chkrootkit

No problems detected…

I’m tired… I’m tired of this shit, there’s a fucking SSH server running on my machine, I have NO idea who’s spawning it !

Oh wait ! I forgot about lsof !

lsof -t

No, wrong one… Couldn’t they use netstat syntax ?

lsof -i
systemd      1            root   52u  IPv6  17371      0t0  TCP TCP *:22 (LISTEN)

O_O … …

O_O !

YOU’RE FUCKING KIDDING ME !? SystemD ITSELF IS LISTENING ON PORT 22 !? But the SSH service is dead ! HOW ?

A little search on the internet, got me this gem : https://bbs.archlinux.org/viewtopic.php?id=166582

“So, ive found out that sshd.socket was enabled and this was the cause”

root# find /etc -name "ssh*.socket"
root# cat /etc/systemd/system/sockets.target.wants/ssh.socket
Description=OpenBSD Secure Shell server socket




The real issue with SystemD

It’s OVERLY COMPLEX ! If you don’t remember all the commands and, most importantly, ALL THE WAYS SYSTEMD CAN START A SERVICE, YOU WILL NOT be able to understand what’s going on.

systemctl status ssh was indicating the status of the ssh.service file, not the status or presence of ssh.socket.

Understand that I threw a journactl and searched for SSH and saw sshd being started and receiving connections as sshd[PID], but not why, nor how it was started ! And the logs were about sshd[PID] not systemd[PID] !

So until you understand that SystemD can start services using socket connections (Overkill feature for simple servers), you will NEVER KNOW :

  • Why is the SSH service not started
  • Why is there a SSH server listening on another port than the one provided in the configuration

Understand that I had a running server until start of September, the server got rebooted for whatever reasons and THEN, POOF, the SSH server started to listen on port 22.

So, the real issue with SystemD is that it’s TOO COMPLEX FOR DISTRIBUTORS !.

It can do a ton of things and maybe, once you understand it perfectly, you’ll be happy to use all the bells and whistles…
If you need them, of course…

However :

  • systemctl status ssh just shown the ssh service as DEAD !
  • journalctl had some “SSH connections” entries, but didn’t show why it started an SSH service !
  • ps auxww | grep sshd made it look like SSH servers are run “on the fly”.

It’s those things combined which make me hate SystemD for server management.
If something breaks, I want AS MUCH INFORMATION AS POSSIBLE !

If SystemD start a sshd server, systemctl status ssh or systemctl status sshd should show information about it !
I don’t give two shits about the file extension of the unit file triggering the execution of “sshd” !

That said, the issue here is not only due to SystemD overcomplexity. It’s also the fact that some distributors thought :

“Hey ! Let’s put a ssh.socket in /etc/systemd/system/sockets.target.wants/ , so that the user will never know why his SSH server start ignoring /etc/ssh/sshd_config !
This will be so much fun !”

The fact is : even if I remove this .socket file, it might just come back after an update ! And fuck up my system again !

Do you understand the issue here ? I’M LOSING CONTROL OF THE SYSTEM ! Because distributors started to ship with an OVERCOMPLEX INIT, and started using features of this init system without understanding the consequences !

If I put CONFIGURATION DIRECTIVES in a CONFIGURATION FILE, I don’t want them to be OVERRIDEN BY SOME RANDOM .whatever FILE used by my init system. If I want a different configuration, I’ll either edit the configuration file, or force the daemon to use another configuration file by editing the appropriate init file.

I’d appreciate if distributions using SystemD went with the “least amount of SystemD unit files” on server configurations.

Look at “Clear Linux”, there’s a way to use SystemD while making things “Lean & Clean”.
Just check the /etc/ folder after installing Clear Linux :
It’s clean !
They don’t add tons of .service, .socket, .mount or .whatever extension systemd reacts on !
No, they put a clean and lean /etc directory, with only the strictly necessary files.
And it works !

That said, an Init system so complex do not interest me. I’ll brush up my systemd-foo on Arch Linux, because I really need it. But, still, I don’t give it a shit about SystemD on my servers ATM.
If you run tons of microservices, maybe you do.
But me ? Nope. All my main public services run in docker containers and the only daemons and configurations I care about on my system are :

  • The SSH server
  • The firewall
  • The Docker containers

I could manage this with a busybox ‘init’ file…

So I’ll start looking for anoter distro that I can deploy on my server, and which doesn’t use SystemD.
And I can’t find one, I’ll look for one using SystemD with the least amount of units files.

I’m done with Debian and SystemD.
And I’d like to be done playing detective to understand :

  • Why my server is not responding anymore ?
  • Why services are executing while ignoring their configurations ?
  • How I can I avoid traps added by systemd updates ?