Yet another post about dynamic lookup of shared libraries

This post is a quick reminder to self regarding specifics of RPATH, RUNPATH, LD_LIBRARY_PATH, LD_RUN_PATH and the lookup order.

Refresher (a very short one)

RPATH, RUNPATH - are entries in the ELF header, baked into the binary allowing the dynamic loader to lookup its shared dependencies.

RPATH/RUNPATH can be specified directly, using a linker option (-rpath):

g++ -Wl,-rpath=path/for/rpath

If missing in the command line, RPATH can be set using LD_RUN_PATH variable.

LD_LIBRARY_PATH are environment variables allowing to modify/extend dynamic loader’s set of search paths for shared dependencies during runtime.

Lookup order

When searching for a shared library, the dynamic loader will look in the following places:

• RPATH
• LD_LIBRARY_PATH
• RUNPATH
• /etc/ld.so.cache
• system paths (i.e. /lib, /lib64 etc)

This specific order differentiates RPATH and RUNPATH in only one way. RUNPATH can be overridden by LD_LIBRARY_PATH whilst RPATH cannot! Therefore, RPATH takes the highest precedence.

This became problematic as it wasn’t possible to override RPATH after the binary has been created without resorting to binary patching with tools like chrpath or patchelf. For that specific reason RUNPATH was introduced.

New dtags

RUNPATH is available if you specify --enable-new-dtags when linking. This enables the generation of so called “new tags”. Once enabled, requests to populate RPATH will in fact populate RUNPATH. Here’s the old way (g++ by default enables new dtags - they have to be disabled explicitly):

1
2
3

g++ main.cpp -L. -lmylib -Wl,-rpath='some/path/to/libs' -Wl,--disable-new-dtags
$ objdump -x a.out | grep PATH
  RPATH                some/path/to/libs

The default behaviour with new dtags enabled:

1
2
3

g++ main.cpp -L. -lmylib -Wl,-rpath='some/path/to/libs'
$ objdump -x a.out | grep PATH
  RUNPATH              some/path/to/libs

There’s one special case when the binary contains both RPATH and RUNPATH. If the latter is present, the former is ignored by the dynamic loader.

The implication of --enable-new-dtags is that effectively RPATH is superseded with RUNPATH. The former one is still supported only because of backwards compatibility concerns.

Debugging the lookup

It’s as simple as defining an extra environment variable

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

0 $ LD_DEBUG=libs ./a.out
      5233:     find library=libfoo.so [0]; searching
      5233:      search path=/home/tomasz/rpath_test/glibc-hwcaps/x86-64-v3:/home/tomasz/rpath_test/glibc-hwcaps/x86-64-v2:/home/tomasz/rpath_test              (RUNPATH f
rom file ./a.out)
      5233:       trying file=/home/tomasz/rpath_test/glibc-hwcaps/x86-64-v3/libfoo.so
      5233:       trying file=/home/tomasz/rpath_test/glibc-hwcaps/x86-64-v2/libfoo.so
      5233:       trying file=/home/tomasz/rpath_test/libfoo.so
      5233:
      5233:     find library=libstdc++.so.6 [0]; searching
      5233:      search path=/home/tomasz/rpath_test            (RUNPATH from file ./a.out)
      5233:       trying file=/home/tomasz/rpath_test/libstdc++.so.6
...

Pathname lookups

Aside of the above, there are some nuances to shared library lookup. One of them is described in man 8 ld.so:

When resolving shared object dependencies, the dynamic linker first inspects each dependency string to see if it contains a slash (this can occur if a shared object pathname containing slashes was specified at link time). If a slash is found, then the dependency string is interpreted as a (relative or absolute) pathname, and the shared object is loaded using that pathname.

Which means that exact path to the library can be baked the binary and RPATH/RUNPATH lookup is ignored for that particular dependency.

Here’s an example (notice the path for libfoo.so):

1
2
3
4
5
6
7
8
9

g++ -L. -l:libs/libfoo.so main.cpp
$ ldd ./a.out
        linux-vdso.so.1 (0x00007ffd0a9f9000)
        libs/libfoo.so (0x0000708a30374000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x0000708a30000000)
        libm.so.6 => /usr/lib/libm.so.6 (0x0000708a2ff11000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x0000708a30317000)
        libc.so.6 => /usr/lib/libc.so.6 (0x0000708a2fd20000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x0000708a30380000)

$ORIGIN

RPATH or RUNPATH can contain a substitute token. One of the most commonly used ones is $ORIGIN which, in runtime` is expanded to the directory containing the program or the shared library.

Meson support

Meson adds RPATH entries allowing for executables to run from the build directory. There entries are removed during installation. This is mentioned in the documentation.

RPATH can be customised for targets using build_rpath and install_rpath options (refer to e.g. executable documentation).

CMake support

Similarly as meson, CMake adds RPATH to binaries linking with shared libraries. Meson tends to use ‘$ORIGIN’. CMake likes to specify an absolute path instead.

In a similar fashion as meson, CMake provides two target properties BUILD_RPATH and INSTALL_RPATH. The former one meant to define paths to be used in the build tree and the latter defining paths to be used after installation. By default, INSTALL_RPATH is empty. These can be set like so:

1
2
3

set_target_properties(your_target_name PROPERTIES
    BUILD_RPATH "/path/to/lib1;/path/to/lib2;/path/to/lib3"
)

Conclusion

Dynamic linker/loader contains a lot of customisation mechanisms and it’s a topic on its own. Most of the crux of its behaviour is very well described in

man ld
man 8 ld.so