Linux ELF Dynamic Linking: How It Actually Works
The ELF Format: The Envelope That Wraps Everything
The Dynamic Linker: The Middleman You Never Saw
Going Deeper: What Happens With strace
The Section Header: The Binary's Table of Contents
PLT and GOT: The Magic Trick Behind Lazy Binding
The Errors That Taught Me This the Hard Way
Error 1: "No such file or directory" on a Binary That Exists
Error 2: Library Version Mismatch in Production
Error 3: LD_PRELOAD for Good and Evil
FAQ: Linux ELF and Dynamic Linking
What I'm Taking Away: The Product Dev Who Finally Went Down to the Metal There are exactly 127 syscalls that an empty Node.js process makes before executing a single line of your code. One hundred and twenty-seven. When I measured it with strace last week, I had to read the output twice, then close the terminal and go for a walk. I have 33 years of history with computers. I started on an Amiga at age 5, went through DOS, was running Linux servers at 18, and today I'm deploying on Railway with Next.js. And in all that time I never truly understood — in real detail — what happens between typing ./my-program and the program actually running. I dodged it. There was always something more urgent. A deploy. A production bug. A client. This week I forced myself to go down to the metal. Here's what I found. Let's start at the beginning. When you execute a binary on Linux, the kernel doesn't simply "start" your program. There's a chain of events that most product devs never see: There it is. dynamically linked. interpreter /lib64/ld-linux-x86-64.so.2. That's the dynamic linker, and it's the main character of this story. ELF stands for Executable and Linkable Format. It's basically a file format — like a ZIP but for executable code. Every Linux binary is an ELF file, and it has a very specific structure: The Entry point is the memory address where execution will begin. But — and this is what blew my mind — that code is not the first thing that runs. When the kernel sees that an ELF is "dynamically linked", it doesn't execute the entry point directly. It first executes the interpreter — which in practice is /lib64/ld-linux-x86-64.so.2, the dynamic linker. This process does, in order: All of that before your main() runs a single line. The tool that opened my eyes was strace. It intercepts every syscall a process makes: Twenty-five syscalls for "hello world". For Node.js it's 127. That makes sense once you understand that Node links against a ton of shared libraries — V8, libuv, OpenSSL. Here's the most elegant part of the whole system. When your program calls printf(), it has no idea at compile time what memory address that function will be at. The library can be anywhere. The solution is two structures: That's lazy binding — the dynamic linker only resolves a function the first time you call it. Elegant and efficient. This happened to me years ago and I "fixed" it without understanding it: The error isn't that the binary doesn't exist. It's that the interpreter doesn't exist. The dynamic linker specified in the ELF isn't on the system. This happened when I was copying binaries between distros with different layouts. I compiled on Ubuntu 22.04, deployed on Debian 10. Different glibc version. The real fix is to build in the same environment as production — which is basically why Docker exists. This connects directly to what I learned while optimizing performance in production — the build environment matters as much as the code itself. The Freestyle sandbox I analyzed a few days ago uses similar mechanisms — intercepting syscalls at the process level to isolate what an agent can do. What is an ELF file in Linux?
ELF (Executable and Linkable Format) is the standard format for executable binaries, shared libraries, and object files on Linux. It's basically a structured container that tells the kernel how to load and execute the code. Every modern Linux binary is an ELF — you can verify it with file /path/to/binary. What's the difference between static linking and dynamic linking?With static linking, all the libraries your program needs are copied into the binary at compile time. The result is a larger but completely self-contained binary. With dynamic linking, the binary only stores references to libraries, and the dynamic linker loads them at runtime. Dynamic linking is the default because it saves memory (multiple apps share the same libc code in RAM) and makes security updates easier. Why does a binary sometimes say "No such file or directory" even though it exists?It usually means the interpreter (dynamic linker) specified in the ELF doesn't exist on that system. You move a binary from Alpine (which uses musl libc) to Ubuntu (which uses glibc) and the path to the dynamic linker just isn't there. You can diagnose it with readelf -l your-binary | grep interpreter. What is LD_PRELOAD and why is it dangerous?LD_PRELOAD is an environment variable that tells the dynamic linker to load a specific library BEFORE anything else, including libc. This lets you intercept and replace system functions. It's useful for profiling and debugging, but dangerous because it can be used to inject malicious code. That's why setuid binaries ignore it. What is the vDSO (linux-vdso.so.1)?It's a virtual library that the kernel automatically maps into every process's memory space. It contains implementations of very frequent syscalls (like gettimeofday) that execute in user space without a real context switch to the kernel. That's why ldd shows it without a path — it's not a file on disk, it lives in the kernel. How does this affect Docker and containers?
A lot. Containers share the host kernel but have their own filesystem. If you build a binary in an image with glibc 2.35 and run it in a container with glibc 2.17, it will fail. That's why Docker images need to be consistent between build and runtime. It's also why Alpine-based images (musl libc) can have unexpected behavior with binaries compiled for glibc. Honestly, I'm a little embarrassed I dodged this for so long. I've worked with Linux since I was 18, administered servers, diagnosed network outages at 11pm with a room full of people waiting on me, and I never seriously asked what happens in those microseconds between ./program and the first line of code. The pivot I made in 2020 toward software development pushed me up the abstraction ladder — React, TypeScript, Next.js. Learning to think in components was hard when you'd spent years thinking in network packets. But going up doesn't mean the layers below disappear. They're still there. When I'm working on LLM inference at the edge or thinking about how to isolate code agents, understanding what happens at the process level matters. Abstractions are useful right up until they break — and when they break, you either go down to the metal yourself or you pay someone who does. My concrete recommendation: spend an afternoon with strace, ldd, and readelf. Not to become a systems programmer — just to understand the machine that runs your code every single day. The Amiga in 1994 had no dynamic linking — everything was static, everything was in ROM or on disk, and the system was what it was. In a way, that simplicity was more honest. Today we run on layers upon layers upon layers, and every now and then it's worth going down to see what it's all standing on. How many syscalls does your app make before running a single line of code? Measure it with strace -c ./your-binary and send me the number. I bet it surprises you. Templates let you quickly answer FAQs or store snippets for re-use. Are you sure you want to ? It will become hidden in your post, but will still be visible via the comment's permalink. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse
./my-program
# Let's see what type of file a binary actually is
file /usr/bin/node
# ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV),
# dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2
# Let's see what type of file a binary actually is
file /usr/bin/node
# ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV),
# dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2
# Let's see what type of file a binary actually is
file /usr/bin/node
# ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV),
# dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2
dynamically linked
interpreter /lib64/ld-linux-x86-64.so.2
# readelf shows you the guts of an ELF
readelf -h /usr/bin/ls # ELF Header:
# Magic: 7f 45 4c 46 02 01 01 00 ... <- "\x7fELF" — the format signature
# Class: ELF64
# Entry point address: 0x67d0 <- this is where YOUR code starts
# Start of program headers: 64 (bytes into file)
# Number of program headers: 13
# readelf shows you the guts of an ELF
readelf -h /usr/bin/ls # ELF Header:
# Magic: 7f 45 4c 46 02 01 01 00 ... <- "\x7fELF" — the format signature
# Class: ELF64
# Entry point address: 0x67d0 <- this is where YOUR code starts
# Start of program headers: 64 (bytes into file)
# Number of program headers: 13
# readelf shows you the guts of an ELF
readelf -h /usr/bin/ls # ELF Header:
# Magic: 7f 45 4c 46 02 01 01 00 ... <- "\x7fELF" — the format signature
# Class: ELF64
# Entry point address: 0x67d0 <- this is where YOUR code starts
# Start of program headers: 64 (bytes into file)
# Number of program headers: 13
Entry point
/lib64/ld-linux-x86-64.so.2
# Let's see what libraries a binary needs
ldd /usr/bin/node # linux-vdso.so.1 (0x00007ffd8c9f3000) <- virtual, lives in the kernel
# libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
# libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6
# libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6
# libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
# /lib64/ld-linux-x86-64.so.2 (0x00007f3a...) <- the dynamic linker itself
# Let's see what libraries a binary needs
ldd /usr/bin/node # linux-vdso.so.1 (0x00007ffd8c9f3000) <- virtual, lives in the kernel
# libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
# libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6
# libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6
# libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
# /lib64/ld-linux-x86-64.so.2 (0x00007f3a...) <- the dynamic linker itself
# Let's see what libraries a binary needs
ldd /usr/bin/node # linux-vdso.so.1 (0x00007ffd8c9f3000) <- virtual, lives in the kernel
# libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
# libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6
# libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6
# libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
# /lib64/ld-linux-x86-64.so.2 (0x00007f3a...) <- the dynamic linker itself
# Let's count syscalls in a minimal C program
cat > hello.c << 'EOF'
#include <stdio.h>
int main() { printf("hello\n"); return 0;
}
EOF gcc -o hello hello.c
strace -c ./hello # % time seconds usecs/call calls syscall
# 27.45 0.000156 31 5 mmap <- map memory
# 18.23 0.000104 20 5 mprotect <- protect memory regions
# 14.67 0.000083 83 1 munmap
# 9.44 0.000054 27 2 openat <- open .so files
# 8.92 0.000051 25 2 read
# ...
# Total calls before main(): ~25
# Let's count syscalls in a minimal C program
cat > hello.c << 'EOF'
#include <stdio.h>
int main() { printf("hello\n"); return 0;
}
EOF gcc -o hello hello.c
strace -c ./hello # % time seconds usecs/call calls syscall
# 27.45 0.000156 31 5 mmap <- map memory
# 18.23 0.000104 20 5 mprotect <- protect memory regions
# 14.67 0.000083 83 1 munmap
# 9.44 0.000054 27 2 openat <- open .so files
# 8.92 0.000051 25 2 read
# ...
# Total calls before main(): ~25
# Let's count syscalls in a minimal C program
cat > hello.c << 'EOF'
#include <stdio.h>
int main() { printf("hello\n"); return 0;
}
EOF gcc -o hello hello.c
strace -c ./hello # % time seconds usecs/call calls syscall
# 27.45 0.000156 31 5 mmap <- map memory
# 18.23 0.000104 20 5 mprotect <- protect memory regions
# 14.67 0.000083 83 1 munmap
# 9.44 0.000054 27 2 openat <- open .so files
# 8.92 0.000051 25 2 read
# ...
# Total calls before main(): ~25
# Let's look at an ELF's sections
readelf -S /usr/bin/ls | head -30 # [Nr] Name Type Address
# [ 0] NULL
# [ 1] .interp PROGBITS <- path to the dynamic linker
# [ 2] .note.gnu.build-i NOTE
# [ 3] .gnu.hash GNU_HASH <- hash table for symbol lookup
# [ 4] .dynsym DYNSYM <- dynamic symbol table
# [ 5] .dynstr STRSYM <- strings with function names
# [12] .plt PROGBITS <- Procedure Linkage Table
# [13] .text PROGBITS <- YOUR CODE is here
# [24] .got PROGBITS <- Global Offset Table
# [25] .got.plt PROGBITS <- GOT for PLT
# [26] .data PROGBITS <- initialized global variables
# [27] .bss NOBITS <- uninitialized global variables
# Let's look at an ELF's sections
readelf -S /usr/bin/ls | head -30 # [Nr] Name Type Address
# [ 0] NULL
# [ 1] .interp PROGBITS <- path to the dynamic linker
# [ 2] .note.gnu.build-i NOTE
# [ 3] .gnu.hash GNU_HASH <- hash table for symbol lookup
# [ 4] .dynsym DYNSYM <- dynamic symbol table
# [ 5] .dynstr STRSYM <- strings with function names
# [12] .plt PROGBITS <- Procedure Linkage Table
# [13] .text PROGBITS <- YOUR CODE is here
# [24] .got PROGBITS <- Global Offset Table
# [25] .got.plt PROGBITS <- GOT for PLT
# [26] .data PROGBITS <- initialized global variables
# [27] .bss NOBITS <- uninitialized global variables
# Let's look at an ELF's sections
readelf -S /usr/bin/ls | head -30 # [Nr] Name Type Address
# [ 0] NULL
# [ 1] .interp PROGBITS <- path to the dynamic linker
# [ 2] .note.gnu.build-i NOTE
# [ 3] .gnu.hash GNU_HASH <- hash table for symbol lookup
# [ 4] .dynsym DYNSYM <- dynamic symbol table
# [ 5] .dynstr STRSYM <- strings with function names
# [12] .plt PROGBITS <- Procedure Linkage Table
# [13] .text PROGBITS <- YOUR CODE is here
# [24] .got PROGBITS <- Global Offset Table
# [25] .got.plt PROGBITS <- GOT for PLT
# [26] .data PROGBITS <- initialized global variables
# [27] .bss NOBITS <- uninitialized global variables
# First call to printf — lazy binding in action
# 1. Jump to printf@PLT
# 2. PLT reads the GOT — still points to the dynamic linker
# 3. Dynamic linker resolves the real address of printf
# 4. Updates the GOT with the real address
# 5. Executes printf # Second call to printf — already resolved
# 1. Jump to printf@PLT
# 2. PLT reads the GOT — now points directly to printf
# 3. Executes printf (no dynamic linker involved) # You can watch this happen with:
LD_DEBUG=bindings ./hello 2>&1 | head -20
# binding file ./hello [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: # normal symbol `printf' [GLIBC_2.2.5]
# First call to printf — lazy binding in action
# 1. Jump to printf@PLT
# 2. PLT reads the GOT — still points to the dynamic linker
# 3. Dynamic linker resolves the real address of printf
# 4. Updates the GOT with the real address
# 5. Executes printf # Second call to printf — already resolved
# 1. Jump to printf@PLT
# 2. PLT reads the GOT — now points directly to printf
# 3. Executes printf (no dynamic linker involved) # You can watch this happen with:
LD_DEBUG=bindings ./hello 2>&1 | head -20
# binding file ./hello [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: # normal symbol `printf' [GLIBC_2.2.5]
# First call to printf — lazy binding in action
# 1. Jump to printf@PLT
# 2. PLT reads the GOT — still points to the dynamic linker
# 3. Dynamic linker resolves the real address of printf
# 4. Updates the GOT with the real address
# 5. Executes printf # Second call to printf — already resolved
# 1. Jump to printf@PLT
# 2. PLT reads the GOT — now points directly to printf
# 3. Executes printf (no dynamic linker involved) # You can watch this happen with:
LD_DEBUG=bindings ./hello 2>&1 | head -20
# binding file ./hello [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: # normal symbol `printf' [GLIBC_2.2.5]
./my-binary
# bash: ./my-binary: No such file or directory # But the file exists:
ls -la my-binary
# -rwxr-xr-x 1 juan juan 45231 Feb 20 14:32 my-binary
./my-binary
# bash: ./my-binary: No such file or directory # But the file exists:
ls -la my-binary
# -rwxr-xr-x 1 juan juan 45231 Feb 20 14:32 my-binary
./my-binary
# bash: ./my-binary: No such file or directory # But the file exists:
ls -la my-binary
# -rwxr-xr-x 1 juan juan 45231 Feb 20 14:32 my-binary
# Diagnosis:
readelf -l my-binary | grep interpreter
# [Requesting program interpreter: /lib/ld-musl-x86_64.so.1]
# ^ Compiled against musl libc, not glibc. Different distro.
# Diagnosis:
readelf -l my-binary | grep interpreter
# [Requesting program interpreter: /lib/ld-musl-x86_64.so.1]
# ^ Compiled against musl libc, not glibc. Different distro.
# Diagnosis:
readelf -l my-binary | grep interpreter
# [Requesting program interpreter: /lib/ld-musl-x86_64.so.1]
# ^ Compiled against musl libc, not glibc. Different distro.
./my-app
# ./my-app: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found
./my-app
# ./my-app: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found
./my-app
# ./my-app: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found
# Dockerfile that avoids this problem
FROM node:20-alpine AS builder
# Alpine uses musl, not glibc — watch out with native binaries FROM node:20-slim AS runner
# Debian slim, same glibc as most production environments
# Dockerfile that avoids this problem
FROM node:20-alpine AS builder
# Alpine uses musl, not glibc — watch out with native binaries FROM node:20-slim AS runner
# Debian slim, same glibc as most production environments
# Dockerfile that avoids this problem
FROM node:20-alpine AS builder
# Alpine uses musl, not glibc — watch out with native binaries FROM node:20-slim AS runner
# Debian slim, same glibc as most production environments
# LD_PRELOAD lets you inject a library BEFORE any other, including libc
# Use it carefully — it's powerful and dangerous # Legitimate example: use tcmalloc instead of the default allocator
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4 ./my-app # Debugging example: intercept function calls
# (basically how some agent sandboxes work)
# Related to what I explored in /blog/sandboxes-coding-agents-freestyle
# LD_PRELOAD lets you inject a library BEFORE any other, including libc
# Use it carefully — it's powerful and dangerous # Legitimate example: use tcmalloc instead of the default allocator
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4 ./my-app # Debugging example: intercept function calls
# (basically how some agent sandboxes work)
# Related to what I explored in /blog/sandboxes-coding-agents-freestyle
# LD_PRELOAD lets you inject a library BEFORE any other, including libc
# Use it carefully — it's powerful and dangerous # Legitimate example: use tcmalloc instead of the default allocator
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4 ./my-app # Debugging example: intercept function calls
# (basically how some agent sandboxes work)
# Related to what I explored in /blog/sandboxes-coding-agents-freestyle
file /path/to/binary
readelf -l your-binary | grep interpreter
gettimeofday
# Start here. Five minutes, on any Linux box:
strace -c ls /tmp 2>&1 # How many syscalls does ls make?
ldd $(which node) # What does Node depend on?
readelf -h $(which ls) # What's inside a binary?
file /bin/* # What types of ELFs live on your system?
# Start here. Five minutes, on any Linux box:
strace -c ls /tmp 2>&1 # How many syscalls does ls make?
ldd $(which node) # What does Node depend on?
readelf -h $(which ls) # What's inside a binary?
file /bin/* # What types of ELFs live on your system?
# Start here. Five minutes, on any Linux box:
strace -c ls /tmp 2>&1 # How many syscalls does ls make?
ldd $(which node) # What does Node depend on?
readelf -h $(which ls) # What's inside a binary?
file /bin/* # What types of ELFs live on your system?
strace -c ./your-binary - Load the ELF into memory — maps the file's segments
- Resolve dependencies — finds each .so the binary needs
- Perform relocation — patches memory addresses so everything fits together
- Run constructors — initialization code that runs before main()
- Hand control over to the real entry point - PLT (Procedure Linkage Table): intermediate code that jumps through the GOT
- GOT (Global Offset Table): a table of pointers to the real addresses
