program startup fails with "Inconsistency detected by ld.so: dl-deps.c: 622: _dl_map_object_deps: Assertion `nlist > 1' failed!"

I've hit the following assert running glibc 2.13 on Linux x86_64:

Inconsistency detected by ld.so: dl-deps.c: 622: _dl_map_object_deps: Assertion `nlist > 1' failed!

This is caused by the following commit:
http://sourceware.org/git/?p=glibc.git;a=commit;h=968dad0ab1f367a087ff4ad503b511dd0c565adc

Reverting the commit "solves" the problem.

There are two ways to reproduce the problem on my machine.

1) Trying to start sage ( http://www.sagemath.org/ )
2) Changing outputs in mpd ( http://mpd.wikia.com )

Created attachment 5220
replace assert by if

Here's an ugly patch that fixes the issue for me.
Instead of "assert (nlist > 1);" wrap the whole while loop
with "if (nlist > 1)".

Created attachment 5229
Don't call _dl_sort_fini if there is only one object. (part 2)

Only call _dl_sort_fini in dl-close.c if there is more than one object.

This patch fixes:

gcc -nostdlib -nostartfiles -static -o /var/tmp/glibc-build/elf/tst-tls9-static -Wl,-O1,--hash-style=gnu,--as-needed /var/tmp/glibc-build/csu/crt1.
o /var/tmp/glibc-build/csu/crti.o `gcc --print-file-name=crtbegin.o` /var/tmp/glibc-build/elf/tst-tls9-static.o /var/tmp/glibc-build/dlfcn/libdl.a
/var/tmp/glibc-build/libc.a -lgcc -lgcc_eh /var/tmp/glibc-build/libc.a `gcc --print-file-name=crtend.o` /var/tmp/glibc-build/csu/crtn.o
/var/tmp/glibc-build/elf/tst-tls9-static.o: In function `do_test':
/var/tmp/glibc/elf/tst-tls9.c:16: warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc v
ersion used for linking
GCONV_PATH=/var/tmp/glibc-build/iconvdata LC_ALL=C LD_LIBRARY_PATH=/var/tmp/glibc-build/elf/:/var/tmp/glibc-build/:/var/tmp/glibc-build/dlfcn /var/t
mp/glibc-build/elf/tst-tls9-static > /var/tmp/glibc-build/elf/tst-tls9-static.out
tst-tls9-static: dl-fini.c:38: _dl_sort_fini: Assertion `nmaps > 1' failed.
Didn't expect signal from child: got `Aborted'
make[2]: *** [/var/tmp/glibc-build/elf/tst-tls9-static.out] Error 1
make[2]: Leaving directory `/var/tmp/glibc/elf'

I've also seen this on x86 with glibc 2.13 git trunk, where it effects the testsuite causing tst-cancel24, tst-chk4, tst-chk5, tst-chk6, tst-lfschk4, tst-lfschk5, and tst-lfschk6 to fail with "Inconsistency detected by ld.so: dl-deps.c: 622: _dl_map_object_deps: Assertion `nlist > 1' failed!".

Created attachment 5253
combined fixes

Here's a combined version of the fixes that doesn't mess with indenting and has the same effect.

The patches attached to this issue still do not address why they are required, and what's actually wrong with the code, or why the original commit is not correct. Until those questions are addressed it is unlikely that this patch will make it into glibc.

Can anyone answer "Why?"

http://sourceware.org/ml/libc-hacker/2011-02/msg00002.html

(In reply to comment #6)
> http://sourceware.org/ml/libc-hacker/2011-02/msg00002.html

Andreas,

Thanks, this helped pinpoint the defect in the sysroot I was using. Rather than getting an assert I get something useful now.

I patched glibc 2.13 git and recompiled it. I then compiled elfutils 0.152 and recompiled prelink 20101123.

Running the prelink testsuite, one test failed:

../src/prelink -c ./prelink.conf -C ./prelink.cache --ld-library-path=. --dynamic-linker=./ld-linux.so.2 -vm ./reloc11
../src/prelink: ./reloc11lib1.so: Could not parse `Inconsistency detected by ld.so: dl-deps.c: 626: _dl_map_object_deps: Assertion `nlist > 1' failed!

After that, running prelink, the usual "Could not find one of the dependencies" for missing dependencies.

I have no idea why the test failed.

If the prelink testsuite is failing then I recommend contacting the prelink developers and asking for their help in putting together a test case that reproduces the problem.

Until this issue has a reduced test case it's going to be difficult to determine what's wrong.

A reduced test case contains the *exact* steps to reproduce the problem, along with source, and pre-processed source.

It is not acceptable to post-process a binary and say that the post-processed binary fails. It could be a problem with the post-processing tool.

Could you please provide a reduced test case?

Thanks!

I don't know how to reproduce it without running prelink. Here, it can be reproduced with the following, extracted from the testsuite.

A file named reloc10lib4.c with just

int dummy = 24;

$ gcc -shared -O2 -nostdlib -fpic -o reloc11lib1.so reloc10lib4.c

$ prelink reloc11lib1.so
prelink: reloc11lib1.so: Could not parse `Inconsistency detected by ld.so: dl-deps.c: 626: _dl_map_object_deps: Assertion `nlist > 1' failed!'

if you `ldd` that file, do you get the same error ?

(In reply to comment #11)
> if you `ldd` that file, do you get the same error ?

Yes, so no need to have prelink.

Ulrich,

Could you please verify?

This reproduces for me on trunk and 2.13 on x86.

Steps to reproduce:

cat > test.c <<EOF
int dummy=64;
EOF
gcc -shared -fpic -o libtest.so test.c

Observed output:
~~~
LD_TRACE_LOADED_OBJECTS=1 /home/carlos/build/glibc/elf/ld.so ./libtest.so
Inconsistency detected by ld.so: dl-deps.c: 626: _dl_map_object_deps: Assertion `nlist > 1' failed!
~~~

Expected output (created using system ld.so based on 2.12):
~~~
LD_TRACE_LOADED_OBJECTS=1 /lib/ld-2.12.1.so ./libtest.so
        linux-gate.so.1 => (0x00961000)
        libc.so.6 => /lib/libc.so.6 (0x0026f000)
        /lib/ld-2.12.1.so (0x00cdf000)
~~~

I have not done any analysis other than reproducing the problem.

I used Andreas's patch which in any case is OK. If this isn't the entire fix, reopen.

Ulrich,
did you ever run Carlos testcase before closing this bug?
Apparently not, because the bug is still present even with Andreas'
patch applied.

Without Kees' patch (testcase from comment 10):
/libx32/ld-linux-x32.so.2 --list ./reloc11lib1.so
Inconsistency detected by ld.so: dl-deps.c: 626: _dl_map_object_deps: Assertion `nlist > 1' failed!

With Kees' patch:
/libx32/ld-linux-x32.so.2 --list ./reloc11lib1.so
        statically linked

It's your code that is calling these (bogus) assertions.
Could you at least think about it for a few minutes before closing this
bug again without a real fix in six weeks (which seems to be your average
latency lately)?.

> It's your code that is calling these (bogus) assertions.
> Could you at least think about it for a few minutes before closing this
> bug again without a real fix in six weeks (which seems to be your average
> latency lately)?.

Latest git pull (commit c6e13027abd4b9c2d694fb4d8b28ab8290ea5971) still not fixed. Applying both Octoploid's part 1 and 2 fixed Vmware 7.1.3 below:

Inconsistency detected by ld.so: dl-deps.c: 626: _dl_map_object_deps: Assertion `nlist > 1' failed!

Thanks,
Jeff

still happening with 2.13 ( GIT checkout of @2011-04-04,
'git checkout -f glibc-2.13' ) , for any shared object
for 32-bit sub-arch on x86_64, that is linked with '-nostdlib' :

1. 32-bit fails:
$ echo 'int r = 42;' > rc.c
$ gcc -m32 -shared -fPIC -o rc.so rc.c
$ /lib32/ld-2.13.so --list ./rc.so
        linux-gate.so.1 => (0xf7757000)
        libc.so.6 => /lib32/libc.so.6 (0xf75d6000)
        /lib32/ld-2.13.so (0xf7758000)
$ gcc -m32 -shared -fPIC -nostdlib -o rc.so rc.c
$ /lib32/ld-2.13.so --list ./rc.so
Inconsistency detected by ld.so: dl-deps.c: 622: _dl_map_object_deps: Assertion `nlist > 1' failed!

2. 64-bit (native arch) works:
$ gcc -shared -o rc.so -fPIC -nostdlib rc.c
$ ldd ./rc.so
        statically linked
$ gcc -shared -fPIC -o rc.so rc.c
$ /lib/ld-2.13.so --list ./rc.so
        linux-vdso.so.1 => (0x00007fff3a3ff000)
        libc.so.6 => /lib64/libc.so.6 (0x00007fad662f9000)
        /lib/ld-2.13.so (0x00007fad668b5000)

Not sure if this is pertinent here, but I suspect so -
what I'm seeing is :
  o no problem if ld.so has no 'DT_RPATH' / 'DT_RPATH_LINK'
  o problem if using ld.so with non-empty DT_RPATH / DT_RPATH_LINK

$ objdump -x /lib64/ld-2.13.so | grep RPATH

^- this bug does NOT happen when using this loader:

$ echo 'int i = 42; ' > rc.c
$ gcc -o rc.so -shared -nostlib -fPIC rc.c
$ /lib64/ld-2.13.so --list ./rc.so
        statically linked

$ objdump -x /lib32/ld-2.13.so | grep RPATH
  RPATH /mnt/sda3/gcc/./gcc/32/:/mnt/sda3/gcc/./gcc/:/mnt/sda3/gcc/x86_64-pc-linux-gnu/libgcc/32:/mnt/sda3/gcc/x86_64-pc-linux-gnu/libgcc:/usr/lib32:/lib32

^- bug only happens using this loader :

$ gcc -m32 -o rc.so -shared -nostdlib -fPIC rc.c
$ /lib32/ld-2.13.so --list ./rc.so
Inconsistency detected by ld.so: dl-deps.c: 622: _dl_map_object_deps: Assertion `nlist > 1' failed!

Now, in order to build any ld.so at all with an RPATH, you must be
somehow disabling this bit of code :
$ diff -U0 elf/dynamic-link.h~ elf/dynamic-link.h
--- elf/dynamic-link.h~ 2011-04-08 02:03:09.000000000 +0100
+++ elf/dynamic-link.h 2011-04-07 02:59:09.000000000 +0100
@@ -208,2 +208,3 @@
- assert (info[DT_RUNPATH] == NULL);
- assert (info[DT_RPATH] == NULL);
+ /* assert (info[DT_RUNPATH] == NULL);
+ assert (info[DT_RPATH] == NULL);
+ */

Now, I remember being roundly upbraded by Uli for attempting something so
crazy, with something along the lines of "an RPATH setting for ld.so
causes so many problems for the loader it never will be supported" - but
I've been building the 32-bit sub-architecture glibc (ONLY!) for x86_64
for years with these lines commented out, hoping to find (and maybe fix?)
any problems caused, and running 32-bit freeware such as mozilla plugins
against this 32-bit loader, with no problems (except this one!) found so
far. Yes, I can see how commenting out these lines would be crazy for
a NATIVE loader, but not for a sub-architecture loader such as 32-bit
on native 64-bit or vice versa.

There is nothing to do here. If you apply patches which you've told are not acceptable and those are creating problems, then it is up to you to fix the patches you apply.

that isnt what is happening here. people are running unmodified git trees and hitting this nlist error. some people might be testing patches in an attempt to address the original problem, but those patches arent the cause.

Sorry if my previous Comment #18 in any way deterred Ulrich from applying
something like Kee's patch of Comment #4, which makes perfect sense to
me and which is really needed to fix this bug - please, Ulrich, if
you haven't already done so, apply something like attachment #5253
of Comment #4.

What happened in my case was that I first compiled the unmodified glibc-2.13
in 32-bit mode, and noticed this problem, and others caused by the sub-arch
loader not having an RPATH . So I applied my usual "allow RPATH on ld.so"
patch and rebuilt, fixing some problems, but leaving this one, which occurs when
using any non-native-system ld.so (NOT /lib/ld-linux.so.1) with '-nostdlibs' .

typo in previous Comment #21: non-native-system ld.so (NOT /lib/ld-linux.so.1) with '-nostdlibs'
                 should be : non-native-system ld.so (NOT /lib/ld-linux.so.2) with '-nostdlib'

Anyway, I'm going to be compiling the latest glibc from GIT soon , after I
build the latest stable Linux kernel, and will test with / without patch
of attachment #5253 and post results . But Ulrich, please try :
  o build an installation from unmodified source in 32-bit mode
    with LIBDIR=/lib32
Then :
  $ echo 'int an_int = 1;' > t.c; gcc -m32 -o t.so -shared -nostdlib -fPIC t.c; ldd t.so
Inconsistency detected by ld.so: dl-deps.c: 622: _dl_map_object_deps: Assertion `nlist > 1' failed!

I've added a patch.

Backported this to 2.13 stable branch.

*** Bug 260998 has been marked as a duplicate of this bug. ***
Seen from the domain http://volichat.com
Page where seen: http://volichat.com/adult-chat-rooms
Marked for reference. Resolved as fixed @bugzilla.