linux

Synapse quick sands

The result

I’ve put a few scripts and a documentation on how to “quickly” prepare a Synapse server, with SSL management, a TURN server for quick and easy VOIP, a Load balancer and a hosted Riot client.

https://github.com/Miouyouyou/matrix-coturn-docker-setup

The reasons

Oh, a new toy !

End of December 2019, I started to have more free time for myself and, after some Gitlab debacles which went like “we’re changing our TOS, SIGN or GTFO !“, I started to think about starting to host more and more services myself.

Aaand, I also wanted to play with ActivityPub, Federated tools and play a bit with some Chat servers, see how the new “Chat servers” are doing now.

A few months ago, I already stumbled into Synapse, a MATRIX protocol server, which ought to combine the joys of “Federated servers” and chat protocols, while providing all the bells and whistles of current chat rooms (picture, file and video sharing, video-conferencing, bridges towards others protocols, …).

Then I started to read the documentation and went like “Ugh… Typical hipster project.”

“A Docker ? Oh yeah, we have one, you just need to setup various environment variables, run it once, edit the configuration file then relaunch it again, but with different variables this time, …”

And then I went like “Good… I’ll just try to install this, test it a little, and then document the installation in a more ‘friendly’ manner in order to let other people try this project. The installation documentation is terrible but The Riot client for this server looks very nice and promising, so let’s give it shot !”

This was a terrible idea.

WHAT THE !?

Can I add a user, please ?

First : The installation was a pain. The documentation is horribly done and I quicky discovered that, the main reason why they ask you to generate a configuration file with specific tools before hand, is because the main configuration file has some “keys”, that has various passwords and cryptographic uses…

Because, YEAH, I love to put my SSH keys in sshd_config !

I learned a few weeks ago, while fiddling with Synapse code and trying to put up my guide, scripts and Docker build scripts; that Synapse configuration file can actually be split, so the keys could be in a separate file loaded with “docker secrets”. But, yeah…

I went on with their default Docker setup first, which invoke the default configuration file generation script, because I didn’t want to go with the whole Ansible Playbook (I haven’t had the time to play with Ansible… One thing at a time…).

I rebuilt their Docker image from their Docker file because I had the good idea to run this on a cheap ARM server from Scaleway, started to run it and… it started ?

Good !

I remember hitting a few issues with my HAProxy setup, and the fact that you need to serve some .well-known/matrix/server that Synapse’s own webserver won’t serve nor generate by itself ! Because why would you serve files that pertain to the protocol related to your program, hmm ? Just force the admin to serve this on another web server, it’s much more fun !

I then used the Riot client on https://riot.im, configured it to ping my server and, after fixing a few issues in my DNS and HAProxy configuration, it worked !
GOOD !
Well kinda… It accepted that I try to connect to my own server but I forgot the most essential part : I didn’t add a user with a password to connect on my chat server !
So, let’s look at the main README.md !

It’s not written…

Wow…

Why would you go so far as creating a chat server that can do so much and forget the essential part “How to add a user”.

Now, I had this reflex because Riot actually asks for a user/password combo.
Thinking about it, IRC servers don’t need such things and work quite well ! Well, until 10000 people start using your chat server for WAREZ and “file sharing”, then it’s less fun suddenly.

So, yeah, I need to know how to add a user to my chat server !

Turned out that the repository with the Ansible playbook for installing Synapse has more useful documentations than the actual Synapse repository ! So, after searching for 30 minutes, I learned that you have to invoke this command, on the Synapse host :

register_new_matrix_user -c /etc/synapse/homeserver.yaml -u chat_user -p chat_password -a http://localhost:8008

Now, /etc/synapse/homeserver.yaml is just “a YAML file” containing “a registration_shared_secret: directive” that is used by your Synapse server.

Running that command with the wrong URL will lead to various hang and crashes, while failing to provide the configuration file will lead to a quick error message telling you that it cannot guess the registration_shared_secret, which seems to be used to encrypt the password (I guess ?).

Which lead to my main question :

Isn’t there an administration UI by now !? Do they do everything by hand !? Why develop a very extensive chat client while forgetting the basics : Managing your server !

Anyway, I connected to my chat server, created a room and…
“Well, it works, at last… I can chat with myself…”

Then I clicked on “Room discovery” and, due to the whole Federation thing, filled with dreams and rainbows, you can list the chat rooms of others serv… No, from “matrix.org”… And only matrix.org.
If you want to list chatrooms from other servers, you’ll have to know their addresses first…

Wow… So much for the “Federation”…

“It’s okay, let’s look at the chat rooms. WTF ? The first room has 40000 users ! Let’s jump in !”

…

That was a terrible idea.

Resources management ? What’s this ? Can you eat it ?

ONE SINGLE USER ! ON ONE SINGLE SERVER !
IS ALL IT TAKES TO DOWN A SYNAPSE MATRIX SERVER !

I, ALONE, ON MY OWN SERVER, was able to DOS it by joining ONE room !

That’s where my opinion for Synapse went from “The documentation is HORRENDOUS but at least the chat features are there” to “WTF !? Dumby The Clown connects to your server, joins a room and THE SERVER GOES OUT OF RESOURCES !? THIS IS INSANE !!!”

Some people will say “But, hey, it’s 40K users maaan !”, to which I’ll answer : “Let’s do the math, people”.

I understand that, now, most softwares are written like shit and if you don’t have the latest 4GHZ processor with 32GB RAM, you’ll feel that “it’s ok if it lags, or if you’re out of resources”, but you really have to understand the POWER you have with a single machine, and how little a chat consumes.

So, there’s 40K users. Let’s just say that for each individual message you consume 4KB of RAM for the message and 4KB of RAM for the metadata of this message. I’m ignoring files and pictures for a reason, they can be ignored until you deal with the messages first.

So that’s 8KB of RAM for every single message. It’s a chat room, most messages will be less than 150 characters, meaning that even with UTF-8 encoding, it’s still “overkill”.

Now let’s say that EVERY user in the 40K users chatroom send ONE message per second, you’ll have to deal with :
8192 bytes/message * 40 000 messages = 327 680 000 bytes for the messages. Roughly 330 MB (or roughly 312.5 MiB).

Then, let’s say that you use double buffers for every single user, because why not :
You’ll use 330 MB per buffer * 2 buffers -> 660 MB in total.

Understand that you’ll have to output 330 MB PER SECOND to the chat users, which is no-no with my connection. I’m “at best” at 2 MB/sec, that’s not going to cut it.
Also 40K messages per second is unreadable and will mostly put your browser on its knees.

BUT 660 MB is not THAT much on a system with 2 GB of RAM. You can still store them and, if messages are not sent quickly enough, you can either ditch the old ones by overwriting their buffers OR wait for the messages to be sent and ignore new messages for the moment.

So what’s the problem ? Synapse ate 2GB of ram in a few seconds and then got killed by the OOM killer ! All of this for ZERO message sent to my chat client !

See, if you eat more than 660 MB of RAM in this kind of setup, it’s already OVERKILL. Sure, you might deal with 400 MB for storing and sending multimedia content (videos are generally YouTube links with a thumbnail). Yet, even in that kind of unrealistic setup, 1GB of RAM is more than enough !
Just in case, I retried this with a PostgreSQL setup and 4GB of RAM : Almost same result ! I was able to see the users list but then OOM and boom !

So, yeah, after seeing this, I let it down for a few days, then went like “Let’s just make a documentation, say that the service is shit in its current state and call it a day… ?“.
Then I thought about it and tried to look at the documentation, see if there isn’t some others trick I could try.
Note that I tried the tricks in the README.md, under the section :
“Help!! Synapse is slow and eats all my RAM/CPU!”

Given how Synapse ate 4G of RAM with a simple PostgreSQL setup, I’m pretty convinced that these ‘tricks’ should be the default setup for Synapse. If your service is known to down a server with just a few individuals, maybe you should think about the default configuration.

But, yeah, I gave these tricks a try and these didn’t make that much of a difference. The whole SYNAPSE_CACHE_FACTOR just lower the time it takes for the server to go down while eating all the RAM, but still this made it possible to join less crowded rooms.
Which is where I found that most of these rooms are QUIET. With at most 4 people chatting… I mean 30 messages in a day, roughly !
Also most of “online users” were users bridged to Matrix from various other protocols, which gave a “we overinflated the number of users in our chat to make it cooler !” vibe.

In the end, the only thing I can say about this is “Nice chat client. It works nicely on a single server with my chat rooms setup… But the resource management is horrendous when you check up federated chat rooms !“.

Is there a point to federation ?

The main issue with Federation in Synapse is that I don’t see the others servers ! Riot is unable to list them directly, you have to configure the servers you want to see in the “Federated” list in its config.json (which is weird BTW, the CLIENT SHOULD ASK THE SERVER about which others servers it federates with !).

So on the basic setup, any user using “Riot” will :

connect to your server
list the chatrooms on matrix.org
join the crowded ones
down your server
never come back gain

Now, by looking for other parts of the documentation (yeah, you read it right), I found some configuration directives named “room_complexity” which allows you to tell your server “Hey, I won’t try to relay messages from rooms that are ‘too complex’“.

The root_complexity comes with a score that makes no sense. You can start with 1.0, which basically will make any client connecting on your server unable to join most of the rooms listed in matrix.org. There’s a few rooms at best you’ll be able to connect to, but they are not that common.

Still, where are the others servers ? How do I discover them ?
See, when you use that kind of federated services, you’re interested in these little servers with peculiar rooms (and straight weird shit too). However, here you can only see matrix.org rooms. If you’re looking for the addresses for these little servers, just search for some index of federated servers on the web…

To me, that’s the opposite of Mastodon and Fediverse, where you have your server, you start to federate with the “mainstreams” servers and already start seeing tons of messages coming from tons of small servers. Then you start browsing on these little servers, have a look at their federations, find some new federation streams that interest you and start adding these streams back to your server !

In Mastodon, Federation “just works”.
In Matrix Synapse, Federation “just sucks”.

That’s mostly due to the main client used for that, Riot, which is powerful but seems to be more and more tailored towards ‘matrix.org’ and their services, and less and less towards small federated servers.

I understand that they want to compete with Discord and Slack, in order to generate revenues and pay the developpers working on their software full-time.

But at the same time, I really feel that it’s missing the point. I still have to test “Rocket.chat”, but my biggest question is : Why would I use Matrix Synapse for something like a Discord server ? Just open up a Discord server, it’s quick, easy and gratis.
It’s clearly not “Open Source” nor Free Software but most users don’t give a shit about that. They care about features, usability and integration. And Discord is feature-full, easy (but not quite) to use, and is being MORE and MORE integrated in the Gaming community, to the point where some chat rooms in mobile games are directly connected to Discord servers.

Meanwhile, the whole “interconnected” and “big community” that federated MATRIX servers provide is a feature that big services like Discord or Slack cannot compete with.
But, for that, you’ll need a client that actually list all servers federated to Matrix.org and let the user discover the chat rooms of these servers.

But the server doesn’t exist !

I forgot… In the Matrix protocol, a room is the aggregation of messages sent by different elements from different federated servers.

See, take Mastodon and the federated stream. Take a client and filter the federated stream with a Hashtag. Make every message from a ‘chatroom’ send ‘toots’ with this Hashtag.
BAM ! You got a Matrix chatroom !

When you list the chatrooms in matrix.org, you actually list the all the public federated chatrooms from servers federating with matrix.org…
It just DOESN’T LOOK LIKE THIS. BUT IT IS.

Now, matrix chatrooms have all the bells and whistles (and video-conferencing too, you won’t get that with Mastodon).
Still, that means that your server doesn’t ‘host’ chatrooms per-se… AAAND that’s where it becomes VERY blurry. However, the documentation insists on how your chatroom isn’t really hosted by a single entity, yada yada yada.
So, MAYBE they’re not listing other servers with their chatrooms to enforce their points ?

Now, there are cases for Federation that I can clearly see from a corporate point of view.
Let’s say that you want to provide chat services, with your own chat servers scattered around the globe.
You want these chat rooms federated so that every user see the messages sent to every server.
All of that with minimum latency.

With a federated chat server, you could this quickly :

Remove federation with the matrix.org server.
Add the addresses of the different chat servers from around the world in the federated list (in homeserver.yaml).
Ensure low-latency (peered) communications between your servers
PROFIT !

The users will connect to the closest end-point, reducing lag, and the servers will relay messages sent to the chatrooms from any endpoint with low latencies !

However, for that, you’ll first need to be able to setup the chat server quickly and efficiently…
Also, that’s a very limited use for Federation, and is just a clustered server with ‘shards’ all over the place.

Back on the installation

Anyway, I wanted to finish this guide about “How to install a Synapse server” that was not as INSANE as the original Docker image from Synapse.

The reason why I called this post “Synapse Quick Sands” is because the more I tried to do this correctly, the more the Synapse project seemed to be fucked up for simple installations, the more I felt that it was a wrong decision to try generating such guides.
This gradually went from “Providing a nice and user-friendly documentation on how to quickly setup a working Synapse server could be a nice plus” to “I should just ditch this shit and forget about it ! This is taking WAY TOO MUCH TIME ! This project is made by insane people !“.

I won’t talk about the issues I got with CoTURN, in order to have a working video-conferencing setup… this project also took me a while to setup, mostly because the documentation is also written “by the main developers, for the main developers”.
Note that the point of TURN is to make the various endpoints, that would like to initiate a video-conference directly, setup their firewalls so that the communications are not blocked.

So, these two weeks, I put in place a ‘semi-automated’ Docker Compose setup that allows you to prepare and install a Synapse Matrix server with :

a TURN server (using CoTURN), for video-conferencing setup;
a PostgreSQL server (because Synapse default SQLite setup has terrible performances);
an HAProxy load balancer, to handle SSL connections, provide ALPN and provide some basic HTTP protections against bots;
a NGINX server hosting potential ACME challenges for Let’s Encrypt SSL certificates creations, and ready to host a Riot-web client.

The setup takes a few script calls to configure the various services, can help you setup SSL certificates with Let’s Encrypt, and should allow any admin with Docker skills to have a try at “Synapse”, “Riot” and all their joys.
Video-conferencing works ‘out-the-box’, as long as you setup your DNS servers correctly. Tried with my smartphone and my PC, one connected on a 3G network and one connected on my Wifi.
You can share files, pictures, links, …

Now, I remade parts of the Docker build image for nothing… After checking Synapse code to understand how the initial Docker image generated the configuration, and how I could modify the part concerning the database, which is setup for SQLite by default, I learned that the main YAML file could be split into multiple files…
So, yeah, if you want to enhance it, remove the overcomplex.sh reference in Synapse build file, use Synapse as the ENTRYPOINT for the Docker image, create and split an initial configuration into multiple YAML files in one folder, make the ‘docker-prepare.sh’ script generate the ‘macaroon’ and other secret keys independantly into a specific YAML file and this should be better already.

Still, I’m taking a break out of Synapse. It looks like a nice project, but it lacks SO MUCH basic things to make it useable, and making people want to host such servers, that I’ll just give up on it for now.

Quick notes

This a special note that list various tips that are too short for a single “light note”, since I only list two light notes per page.

Firefox

How to import ‘recent’ Seamonkey and Waterfox profiles in recent Firefox releases

Locate the folder of your Seamonkey or Waterfox profile.
On Linux systems, it’s in ~/.mozilla/seamonkey and ~/.waterfox respectively.
On Windows, a random guess would be %APPDATA%\Mozilla\Seamonkey\Profiles or %APPDATA%\Waterfox\Profiles.

Check that keys4.db and logins.json are present. If only keys3.db is present, your installation is too old and won’t be imported correctly.

If these files are present, copy the profile folder into ~/.mozilla/firefox (or %APPDATA%\Mozilla\Firefox\Profiles).
In the same folder, you should see a profiles.ini file. Back it up and edit it.

This should look like this :

[Install4F96D1932A9F858E]
Default=rp4qlak7.default-release
Locked=1

[Profile1]
Name=default-release
IsRelative=1
Path=rp4qlak7.default-release
Default=1

[Profile0]
Name=default
IsRelative=1
Path=9ruyutsw.default

[General]
StartWithLastProfile=1
Version=2

The ‘Path’ will be different on your installation.

Add a [ProfileN] section where N is the next Profile number.

For example, if :

the profile folder you just copied is named abcdef12345.default
you have [Profile0] and [Profile1] sections in your profiles.ini file

Then add the following

[Profile2]
Name=toimport
IsRelative=1
Path=abcdef12345.default

The Name= is not very important. You should just remember it as you’ll have to choose it during the migration wizard.

Once done, close every window of Firefox and then start firefox with the --migration argument :

firefox --migration

On Windows, you’ll have to open a Powershell in the installation folder of Firefox, located in Program Files.

This should open a ‘Migration windows’. From this point :

Select Firefox
Click on Next >
Select toimport
Click on Next >
Then click on Finish

toimport is the name set in the profiles.ini just before.

Firefox should open the main page by now.
Check the “Logins and passwords” to ensure that, at least, your passwords were imported successfully.
Then check your bookmarks.

If the name toimport didn’t appear, check that you didn’t mess up the profiles.ini configuration. If that’s not the case, maybe your profile is really too old for a direct Firefox import.

Docker

`standard_init_linux.go:207: exec user process caused "exec format error"`

Either :

you tried to run an image not built for your architecture;
the entry point script of your image does not have a valid # shebang on the first line.

View the content of a running container, or post-mortem

docker export containerid -o /path/to/file.tar.gz

/path/to/file.tar.gz will be a snapshot of the / directory of your container.

No network during builds, while using custom networks

One solution would be to use the ‘host’ network just for the build, by adding a network: host node below your build node.

Complete example :

version: '3.4'

services:
  matrix:
    image: myy/synapse:latest
    build:
      context: ./build-synapse
      network: host
    networks:
      myynet:
        ipv6_address: fc00::105

networks:
  myynet:
    external: true

In this example, the Dockerfile path is ./build-synapse/Dockerfile
Also, the network myynet is not used during the build

Let’s encrypt (`letsencrypt`)

`requests.exceptions.SSLError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443)`

The complete error could be something like :

Max retries exceeded with url: /directory (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))

Check your /etc/hosts if there’s a acme-v02.api.letsencrypt.org entry.

Then compare your result of :

$ nslookup
> acme-v02.api.letsencrypt.org

Between your current machine and another machine.
If the results are differents, you might to try using the same DNS server as your other machine, by :

Modifying /etc/resolv.conf for quick-testing and adding a nameserver W.X.Y.Z entry at the top.
Re-executing your certbot script.

If that works, configure your network to use the same DNS server, through the standard configuration files/utilities, since /etc/resolv.conf generally don’t persist after a reboot (or a DHCP lease renewal).

If that doesn’t work, try to use acme-v01.api.letsencrypt.org instead, by passing --server https://acme-v01.api.letsencrypt.org/directory to certbot.

Also make sure that your firewall isn’t blocking HTTPS egress connections, by checking up on https://google.com :

curl -L https://google.com

Systemd

I can connect to SSH but the port is wrong and the ssh.service is disabled ?

Turns out that some distributions install a ssh.socket in /etc/systemd/system/sockets.target.wants that, not only ignores your port configuration in /etc/ssh/sshd_config but also block ssh.service from running !

I’d like to say :
Just rm /etc/systemd/system/sockets.target.wants/ssh.socket
BUT ! It turns out that I got burned by this issue after a standard Debian unstable system update and a simple reboot… which means that you can easily be burned by this after a system update.

So… one alternative is to create a custom SSH service unit in /etc/systemd/system/ with a name like donttouchmyssh.service.
The content of /etc/systemd/system/donttouchmyssh.service like this :

[Unit]
Description=OpenBSD Secure Shell server
Documentation=man:sshd(8) man:sshd_config(5)
After=network.target auditd.service
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

[Service]
EnvironmentFile=-/etc/default/ssh
ExecStartPre=/usr/sbin/sshd -t
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/usr/sbin/sshd -t
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartPreventExitStatus=255
Type=notify
RuntimeDirectory=sshd
RuntimeDirectoryMode=0755

[Install]
WantedBy=multi-user.target

The unit is based on /lib/systemd/system/ssh.service, but ours don’t advertise itself as ssh.service or sshd.service, meaning that ssh.socket won’t block it.

Test it by running :

systemctl start donttouchmyssh

And then try to connect on the SSH port you configured in the server /etc/ssh/sshd_config file.

If you can connect correctly, set this service to execute on startup

systemctl enable donttouchmyssh

Reboot and retry to connect on the configured port.

If you can, then you can use your firewall to block port 22 connections !
Block these ports from a machine, or an interface, which you can access if your custom service fails to start after another system update.

If you cannot, check for errors using journalctl -u donttouchmyssh and systemctl status donttouchmyssh (with sudo if required).

Iptables

Save / Restore

If you don’t have a iptables rules save/restore service, here’s a quick one for IPv4 and IPv6 rules.

In these services, rules will be loaded and saved from /etc/iptables/v4-rules and /etc/iptables/v6-rules, so prepare the environment before like this :

mkdir /etc/iptables
iptables-save > /etc/iptables/v4-rules
ip6tables-save > /etc/iptables/v6-rules

After that, create two services for loading and restoring IPv4 and IPv6 rules. The services must be saved in /etc/systemd/system.
You can name them iptables.service and ip6tables.service for example, but the name is up to you.

/etc/systemd/system/iptables.service

[Unit]
After=network.service
Description=Iptables IPv4 rules save/load service

[Service]
Type=oneshot
ExecStart=/bin/sh -c "iptables-restore < /etc/iptables/v4-rules"
ExceStop=/bin/sh -c "iptables-save > /etc/iptables/v4-rules"
RemainAfterExit=yes

[Install]
WantedBy=default.target

/etc/systemd/system/ip6tables.service

[Unit]
After=network.service
Description=Iptables IPv6 rules save/load service

[Service]
Type=oneshot
ExecStart=/bin/sh -c "ip6tables-restore < /etc/iptables/v6-rules"
ExceStop=/bin/sh -c "ip6tables-save > /etc/iptables/v6-rules"
RemainAfterExit=yes

[Install]
WantedBy=default.target

Test the services using :

systemctl start iptables && systemctl start ip6tables

Then check your firewall rules, using iptables -L, iptables -t nat -L, ip6tables -L and ip6tables -t nat -L.

If the rules appear ok, enable the script on startup and reboot

systemctl enable iptables
systemctl enable ip6tables
reboot

On reboot check the rules again, just to be sure.

If the rules aren’t good

If the rules are not good, correct them and then save them again in /etc/iptables/v4-rules and /etc/iptables/v6-rules using iptables-save and ip6tables-save respectively, then restart the iptables and ip6tables services like this :

systemctl restart iptables
systemctl restart ip6tables

Check the content of the files in /etc/iptables before and after the systemctl restart if the problem persist, in order to understand what’s going on.

Why is it blocking ?

If you happen to wonder why some packets are blocked in INPUT or OUTPUT, one simple way to get some clues is to set up a catch-all LOG rule :

INPUT

iptables -A INPUT -j LOG --log-level 6

OUTPUT

iptables -A OUTPUT -j LOG --log-level 6

Other cases

You can also put your catch-all sentence in any rule-chain, to see if the rule-chain is traversed :

iptables -A YOUR_RULE_CHAIN -j LOG --log-level 6

If you’re afraid that some rules might send the traffic to other rule chains, you can setup the LOG rule as the first rule of your chain. This will break your rulechain temporarily though.

iptables -I YOUR_RULE_CHAIN -j LOG --log-level 6

To check the logs, you can use journalctl on SystemD systems :

journalctl -k

Or just check dmesg or /var/log/messages on other systems.

/var/log/messages requires a syslog daemon logging kernel messages to this filepath.

Matrix

Synapse

`(config for tls_private_key_path): No such file or directory` even though `no_tls: True` is added

So if you server cannot start, with the following error :

matrix_1  | 2019-10-24 20:27:50,458 - twisted - 171 - ERROR -  - Traceback (most recent call last):
matrix_1  | 2019-10-24 20:27:50,459 - twisted - 171 - ERROR -  -   File "/usr/local/lib/python3.7/site-packages/synapse/app/_base.py", line 263, in start
matrix_1  | 2019-10-24 20:27:50,459 - twisted - 171 - ERROR -  -     refresh_certificate(hs)
matrix_1  | 2019-10-24 20:27:50,460 - twisted - 171 - ERROR -  -   File "/usr/local/lib/python3.7/site-packages/synapse/app/_base.py", line 212, in refresh_certificate
matrix_1  | 2019-10-24 20:27:50,461 - twisted - 171 - ERROR -  -     hs.config.read_certificate_from_disk(require_cert_and_key=True)
matrix_1  | 2019-10-24 20:27:50,462 - twisted - 171 - ERROR -  -   File "/usr/local/lib/python3.7/site-packages/synapse/config/tls.py", line 221, in read_certificate_from_disk
matrix_1  | 2019-10-24 20:27:50,462 - twisted - 171 - ERROR -  -     self.tls_private_key = self.read_tls_private_key()
matrix_1  | 2019-10-24 20:27:50,463 - twisted - 171 - ERROR -  -   File "/usr/local/lib/python3.7/site-packages/synapse/config/tls.py", line 487, in read_tls_private_key
matrix_1  | 2019-10-24 20:27:50,465 - twisted - 171 - ERROR -  -     private_key_pem = self.read_file(private_key_path, "tls_private_key_path")
matrix_1  | 2019-10-24 20:27:50,467 - twisted - 171 - ERROR -  -   File "/usr/local/lib/python3.7/site-packages/synapse/config/_base.py", line 135, in read_file
matrix_1  | 2019-10-24 20:27:50,468 - twisted - 171 - ERROR -  -     cls.check_file(file_path, config_name)
matrix_1  | 2019-10-24 20:27:50,469 - twisted - 171 - ERROR -  -   File "/usr/local/lib/python3.7/site-packages/synapse/config/_base.py", line 117, in check_file
matrix_1  | 2019-10-24 20:27:50,469 - twisted - 171 - ERROR -  -     % (file_path, config_name, e.strerror)
matrix_1  | 2019-10-24 20:27:50,470 - twisted - 171 - ERROR -  - synapse.config._base.ConfigError: Error accessing file '/data/matrix.miouyouyou.fr.tls.key' (config for tls_private_key_path): No such file or directory

Even though you seem to have enable no_tls: true in your /data/homeserver.yaml, check the two following things :

The directory you are actually mounting has a file named homeserver.yaml.
While this seems dumb, you have to understand that :
If you setup the environment variable SYNAPSE_SERVER_NAME, this will generate a temporary configuration file that will overshadow your current configuration file.
I.e. If you set SYNAPSE_SERVER_NAME, the server won’t look for your /data/homeserver.yaml but use an auto-generated one instead !.

Migrate your SQLite DB to a PostgreSQL DB using Docker

First, ensure that your synapse Docker /data is file-system accessible volume, or ensure at least that you can access this directory to create the new configuration file, and do a database copy.

Here, I’ll assume that you’re in a directory where the sub-directory data is mounted as /data in your Synapse Docker instance.

Quick and dirty

Create a volume for the PostgreSQL data
```
docker volume create synapse_db_data
```

If you know what you’re doing and want to use a host directory for the database files, just replace following synapse_db_data references with /absolute/path/to/your/host/directory, like this : -v /absolute/path/to/your/host/directory:/var/lib/postgresql/data

Create a temporary environment variable for PostgreSQL configuration through Docker

cat <<EOF > postgres.env
POSTGRES_USER=your_db_user
# If you actually use Y0urP4ssw0rd as a password, you're an idiot
POSTGRES_PASSWORD=Y0urP4ssw0rd
POSTGRES_DB=your_db_name
EOF

Fire the PostgreSQL server

docker run --rm --env-file docker.env -v synapse_db_data:/var/lib/postgresql/data --name synapsedb -d postgres

If the instance doesn’t start, remove --rm, then type docker logs synapsedb to view the logs, and then docker container rm synapsedb to get rid of the stopped container else you won’t be able to recreate it

Copy and modify your Synapse server homeserver.yaml configuration to use PostgreSQL
```
cp data/homeserver.yaml data/homeserver-pgsql.yaml
```

Then modify the section that look like this :

## Database ##

database:
  # The database engine name
  name: "sqlite3"
  # Arguments to pass to the engine
  args:
    # Path to the database
    database: "/data/homeserver.db"

and make it look this :

database:
  # The database engine name
  name: "psycopg2"
  # Arguments to pass to the engine
  args:
    user: your_db_user
    password: Y0urP4ssw0rd
    database: your_db_name
    host: synapsedb

Stop your current Synapse server

docker stop your_synapse_container_tag_or_numeric_id

Copy your SQLite database just in case
```
cp data/homeserver.db{,.bak}
```
Run the migration script with a new synapse instance

Replace myy/synapse-image:latest-intel by the reference of your synapse image.

docker run --name synapse  -v $PWD/data:/data --entrypoint synapse_port_db -d myy/synapse-image:latest-intel --sqlite-database /data/homeserver.db --postgres-config /data/homeserver-postgres.yaml

If you haven’t build your own synapse-image, either use the official one, or build one using my guide.

Follow the migration
```
docker logs -f synapse
```
Shutdown your PostgreSQL instance
```
docker stop synapsedb
```
Launch the whole thing with a docker-compose.yml