apulse libpulse-simple.so: undefined symbol: pa_threaded_mainloop_new

Long story short

At the moment, compile my patched version of apulse like this :

git clone https://github.com/Miouyouyou/apulse
mkdir build
cd build
cmake ../apulse
cmake --build .
sudo make install
cd ..

After that, if that’s the first time you install them by hand, you can add the libraries to the dynamic linker search path, so that these libraries are found automatically, without having to use the apulse wrapper everytime :

echo "/usr/local/lib/apulse" >> /etc/ld.so.conf.d/apulse.conf

Note that this only compiles the 64-bits version of apulse.

I currently have no idea how to compile the 32-bits version with cmake.
UPDATE Maybe I should read the README.md before hand…
It’s clearly written how to compile the 32-bits version…
Ugh…

Here’s how to compile the 32-bits version (on Gentoo at least), assuming you’ve already cloned apulse’s git repository :

mkdir build32
cd build32
PKG_CONFIG_LIBDIR=/usr/lib32/pkgconfig CFLAGS=-m32 cmake -DAPULSEPATH=/usr/local/lib32/apulse ../apulse/
cmake --build .
sudo make install
cd ..

Redefining the PKG_CONFIG_LIBDIR is required to get rid of compilation errors like size of array '_GStaticAssertCompileTimeAssertion_0' is negative, when compiling for 32-bits versions with CFLAGS=-m32.

Again, if it’s the first time you install them by hand and don’t want to prefix apulse for invoking softwares using PulseAudio, here’s how you can add the apulse libraries to the standard dynamic linker search path :

echo "/usr/local/lib32/apulse" >> /etc/ld.so.conf.d/apulse.conf

This patched apulse version might solve issues like crashes after seeing this :

g_hash_table_lookup_node: assertion failed: (hash_table->ref_count > 0)

I don’t trust you that much

Fine !

Here’s the patch applied on the official repository. Here’s the same patch on Github gists.

And here’s how to apply it :

wget https://gist.githubusercontent.com/Miouyouyou/a6f460e03e046478f92e6a82d9e4dc79/raw/90bec9ad5424a4b6abf86a7b9923420eecb9a2f4/0001-stream-Check-the-key-before-invoking-g_hash_table_re.patch
git clone https://github.com/i-rinat/pulse
cd pulse
git am ../0001-stream-Check-the-key-before-invoking-g_hash_table_re.patch
cd ..

Then you can recompile and install using the standard CMake build procedure :

64 bits version

mkdir build
cd build
cmake ../apulse
cmake --build .
sudo make install
cd ..

32 bits version

mkdir build32
cd build32
PKG_CONFIG_LIBDIR=/usr/lib32/pkgconfig CFLAGS=-m32 cmake -DAPULSEPATH=/usr/local/lib32/apulse ../apulse/
cmake --build .
sudo make install
cd ..

Additional quicktips

If you have to use gdb on a Unity3D game, here’s a quicktip. Before calling run, disable some signal handlers with the following command :

handle SIGXCPU SIG33 SIG35 SIGPWR nostop noprint

This will make debugging way easier. Works also when debugging Mono/C# executables.

Short story long

I hit that bug while testing a Steam game named “Wizards of Legends” on my Gentoo box, however this bug came back with a lot of Unity3D games distributed on Itch.io .

Basically the game crashed with some “Unable to preload the following plugins” messages.
As always, these messages were RED HERRINGS ! I LOVE error messages that distract me from the real issue ! So much fun !

The game also wrote some logs located in ~/.config/unity3d/$COMPANYNAME/$GAMETITLE/Player.log.
The Player.log contained the following message :

undefined symbol: pa_threaded_mainloop_new

So I did a : nm -D /usr/lib/apulse/libpulse-simple.so.0, saw that U pa_threaded_mainloop_new was listed as an Unresolved symbol…
Meaning that it should load a library that will lead to resolving this symbol… however I can only guess it didn’t, given the error message.

So I then tried to look for this symbol in the other libraries provided by apulse.
nm -D /usr/lib/apulse/libpulse.so.0 returned 000000000000ec10 T pa_threaded_mainloop_new, which indicates that pa_threaded_mainloop_new is provided by libpulse.so.0.

So : I now know where the symbol is !

The next question is then :

Is libpulse.so.0 loaded correctly when loading libpulse-simple.so.0

Let’s have a look at the dynamic libraries chain-loaded by libpulse-simple.so.0 with readelf -d :

readelf -d /usr/lib/apulse/libpulse-simple.so.0 

Dynamic section at offset 0x3dc0 contains 29 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libglib-2.0.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [libpulse-simple.so.0]
 0x000000000000000c (INIT)               0x1220
 0x000000000000000d (FINI)               0x26ac
 0x0000000000000019 (INIT_ARRAY)         0x203db0
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x203db8
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x190
 0x0000000000000005 (STRTAB)             0x788
 0x0000000000000006 (SYMTAB)             0x200
 0x000000000000000a (STRSZ)              1165 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0x204000
 0x0000000000000002 (PLTRELSZ)           1056 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0xe00
 0x0000000000000007 (RELA)               0xd28
 0x0000000000000008 (RELASZ)             216 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffc (VERDEF)             0xc90
 0x000000006ffffffd (VERDEFNUM)          2
 0x000000006ffffffe (VERNEED)            0xcc8
 0x000000006fffffff (VERNEEDNUM)         2
 0x000000006ffffff0 (VERSYM)             0xc16
 0x000000006ffffff9 (RELACOUNT)          3
 0x0000000000000000 (NULL)               0x0

So… it chainloads libglib-2.0.so.0, libpthread.so.0 and libc.so.6 but… no libpulse.so.0.

… I’d love to add libpulse.so.0 to the list by hacking the binary, but that’s not going to happen…

In this case, the next sensible choice is to compile the latest version of the library.
Which reminds me how OLD and OUTDATED most of the main Gentoo ebuilds are…

If we can recompile the library, it should be possible to, at least, hack around to generate a fixed version of libpulse-simple.so.0.

Anyway, I went out to compile the official version like this :

git clone https://github.com/i-rinat/apulse
mkdir build
cmake ../apulse
cmake --build .
sudo make install
echo "/usr/local/lib/apulse" >> /etc/ld.so.conf.d/apulse.conf

This installed a new version of libpulse-simple.so.0 with the correct exports this time !

readelf -d /usr/local/lib/apulse/libpulse-simple.so.0 

Dynamic section at offset 0x3d80 contains 32 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libpulse.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libglib-2.0.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libasound.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [libpulse-simple.so.0]
 0x000000000000000c (INIT)               0x1220
 0x000000000000000d (FINI)               0x29e4
 0x0000000000000019 (INIT_ARRAY)         0x203d70
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x203d78
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x190
 0x0000000000000005 (STRTAB)             0x760
 0x0000000000000006 (SYMTAB)             0x1d8
 0x000000000000000a (STRSZ)              1191 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0x204000
 0x0000000000000002 (PLTRELSZ)           1056 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0xe00
 0x0000000000000007 (RELA)               0xd28
 0x0000000000000008 (RELASZ)             216 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffc (VERDEF)             0xc80
 0x000000006ffffffd (VERDEFNUM)          2
 0x000000006ffffffe (VERNEED)            0xcb8
 0x000000006fffffff (VERNEEDNUM)         3
 0x000000006ffffff0 (VERSYM)             0xc08
 0x000000006ffffff9 (RELACOUNT)          3
 0x0000000000000000 (NULL)               0x0

See how libpulse.so.0 is now correctly chainloaded ?

So I restarted the game AND… another crash…
Which involved using gdb to understand what went wrong this time.

If you have to use gdb on a Unity3D game, here’s a quicktip :
Before calling run, use this command to disable some signal handlers :

handle SIGXCPU SIG33 SIG35 SIGPWR nostop noprint

This will make debugging the binary easier. This works for any Mono/C# executable.

So I went like this :

cd "~/.local/share/Steam/steamapps/common/Wizard of Legend"
gdb ./WizardOfLegend.x86_64
(gdb) handle SIGXCPU SIG33 SIG35 SIGPWR nostop noprint
Signal        Stop      Print   Pass to program Description
SIGXCPU       No        No      Yes             CPU time limit exceeded
SIGPWR        No        No      Yes             Power fail/restart
SIG33         No        No      Yes             Real-time event 33
SIG35         No        No      Yes             Real-time event 35
(gdb) run
...
Thread 75 "WizardOfLegend." received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffee1ff3700 (LWP 11529)]
0x00007ffff6aab74b in raise () from /lib64/libc.so.6
(gdb) where
#0  0x00007ffff6aab74b in raise () from /lib64/libc.so.6
#1  0x00007ffff6a948cd in abort () from /lib64/libc.so.6
#2  0x00007fff8ae44a23 in ?? () from /usr/lib64/libglib-2.0.so.0
#3  0x00007fff8aea0f4a in g_assertion_message_expr () from /usr/lib64/libglib-2.0.so.0
#4  0x00007fff8ae65396 in ?? () from /usr/lib64/libglib-2.0.so.0
#5  0x00007fff740ea520 in pa_stream_unref () from /usr/local/lib/apulse/libpulse.so.0
#6  0x00007fff740e8283 in deh_stream_first_readwrite_callback () from /usr/local/lib/apulse/libpulse.so.0
#7  0x00007fff740e6ce8 in pa_mainloop_dispatch () from /usr/local/lib/apulse/libpulse.so.0
#8  0x00007fff740e6e27 in pa_mainloop_iterate () from /usr/local/lib/apulse/libpulse.so.0
#9  0x00007fff740e747e in pa_mainloop_run () from /usr/local/lib/apulse/libpulse.so.0
#10 0x00007fff740eabf4 in mainloop_thread () from /usr/local/lib/apulse/libpulse.so.0
#11 0x00007ffff79b71d8 in start_thread () from /lib64/libpthread.so.0
#12 0x00007ffff6b7d2cf in clone () from /lib64/libc.so.6
(gdb) quit

So, basically, something went wrong in pa_stream_unref AND it went wrong when calling a glib function !
In such situations, you can guess that the latest function was called with bogus arguments.

Alright, since I had to clone apulse git repository, I might as well see and edit the code of pa_stream_unref and try to debug this.

Let’s look at the code of pa_stream_unref :

APULSE_EXPORT
void
pa_stream_unref(pa_stream *s)
{
    trace_info_f("F %s s=%p\n", __func__, s);

    s->ref_cnt--;
    if (s->ref_cnt == 0) {
        g_hash_table_remove(s->c->streams_ht, GINT_TO_POINTER(s->idx));
        ringbuffer_free(s->rb);
        free(s->peek_buffer);
        free(s->write_buffer);
        free(s->name);
        free(s);
    }
}

The only glib function called is g_hash_table_remove so… I can only guess that it’s the one causing the crash.

So, let’s look on the internet for g_hash_table_remove and see how it is supposed to be called :

gboolean
g_hash_table_remove (GHashTable *hash_table,
                     gconstpointer key);

Alright…

Then let’s add some logs… I need to check the environment.

When you have to add some logs in someone else code, look around the code to see how the developper logs things generally. See if there’s a log_error, warn or, in this case, trace_error function.

Try the error logging functions first. If you go for debug logging functions, you might not see anything unless you enable some specific flags during compilation… Errors, though, tend to be displayed in every configuration.

Now, in order to understand what’s passed to the function, I added this before calling g_hash_table_remove :

		trace_error("s->c->streams_ht : %p - %d",
			s->c->streams_ht,
			s->idx);

Which led to these errors appearing the Player.log of the Unity3D game :

[apulse] [error] s->c->streams_ht : 0x3eed240 - 0
[apulse] [error] s->c->streams_ht : 0x3eed240 - 0

Ok…

Now, I want to know if the hash_table actually have something stored. Let’s check the documentation, see if there’s a way to get the size… or length… size !
g_hash_table_size : Returns the number of elements contained in the GHashTable.

guint
g_hash_table_size (GHashTable *hash_table);

Ok, let’s log the number of elements too

		trace_error("s->c->streams_ht : %p - (%u elements) %d",
			s->c->streams_ht,
			g_hash_table_size(s->c->streams_ht),
			s->idx);

Recompile, reinstall the apulse library, relaunch the game…

This led to new errors in the game’s Player.log :

[apulse] [error] s->c->streams_ht : 0x3d36640 (0 elements) - 0
[apulse] [error] s->c->streams_ht : 0x3d36640 (0 elements) - 0

Yeah, ok, the hash is empty so calling remove functions will only lead to issues.

Now, I could have searched for “why it’s empty and why this function is called with an empty hash table”. But, I went for the quick fix instead.

My first idead of the quick fix was :
Let’s check if the element to be deleted is actually stored in the hash table before trying to remove it.

So I tried using g_hash_table_lookup for that matter.

I modified the code like this :

		GHashTable * __restrict const streams_ht =
			s->c->streams_ht;
		void const * key = GINT_TO_POINTER(s->idx);
		if (g_hash_table_lookup(streams_ht, key))
			g_hash_table_remove(streams_ht, key);

This… led to another crash that required me to reuse gdb to catch the bug.
gdb returned this :

#0  0x00007ffff6aab74b in raise () from /lib64/libc.so.6
#1  0x00007ffff6a948cd in abort () from /lib64/libc.so.6
#2  0x00007fff8ae44a23 in ?? () from /usr/lib64/libglib-2.0.so.0
#3  0x00007fff8aea0f4a in g_assertion_message_expr () from /usr/lib64/libglib-2.0.so.0
#4  0x00007fff8ae65976 in g_hash_table_lookup () from /usr/lib64/libglib-2.0.so.0
#5  0x00007fff740ea531 in pa_stream_unref () from /usr/local/lib/apulse/libpulse.so.0
#6  0x00007fff740e8283 in deh_stream_first_readwrite_callback () from /usr/local/lib/apulse/libpulse.so.0
#7  0x00007fff740e6ce8 in pa_mainloop_dispatch () from /usr/local/lib/apulse/libpulse.so.0
#8  0x00007fff740e6e27 in pa_mainloop_iterate () from /usr/local/lib/apulse/libpulse.so.0
#9  0x00007fff740e747e in pa_mainloop_run () from /usr/local/lib/apulse/libpulse.so.0
#10 0x00007fff740eac1d in mainloop_thread () from /usr/local/lib/apulse/libpulse.so.0
#11 0x00007ffff79b71d8 in start_thread () from /lib64/libpthread.so.0
#12 0x00007ffff6b7d2cf in clone () from /lib64/libc.so.6

Oh, yeah, okay… g_hash_table_lookup also generated a crash…

Wait, in the previous logs, s->idx was 0 and this index is turned into a key using GINT_TO_POINTER…
0 converted into a pointer will most likely generate a NULL pointer, by definition, so here’s the new catch :

Let’s remove the element if the key is not 0 AND if the element is actually stored in the hash table.

        GHashTable * __restrict const streams_ht =
            s->c->streams_ht;
        void const * key = GINT_TO_POINTER(s->idx);
        if (key && g_hash_table_lookup(streams_ht, key))
            g_hash_table_remove(streams_ht, key);

This time IT WORKED ! The game launched and I was able to hear the music and sound effects !
YAAAY.

After that, I forked the apulse project, integrated this quick patch and then sent a pull request.