Why This Matters
Crash Debugging Decision Map
Path A — You Have a Core File
A1 — Locate the Core File
A2 — Identify Which Binary Produced the Core
A3 — Check Whether the Binary Has Debug Symbols
A4 — Debugging With Symbols (Best Case)
A5 — Binary Is Stripped: Find the Symbol File
A5.1 — Extract Build ID
A5.2 — Locate Matching Symbol File
A6 — Debug Using Separate Symbol Files
A7 — No Symbol File: Reproduce With Debug Build
A8 — Map Raw Addresses Using an Unstripped Build
A9 — No Symbols, No Reproduction: Fallback Forensics
Inspect Registers
Inspect Instructions Around the Crash
Path B — No Core File Generated
B1 — Check Core Dump Settings
B2 — Enable Core Dumps
B3 — Set Core File Location
B4 — Fix Permissions
B5 — Test Core Dump Generation
B6 — Rerun the Crashed Application
Common Pitfalls
Quick Reference Table
Pro Tips
Conclusion You're on call. A production C++ service just crashed — no logs, no stack trace, just a dead process and maybe a core file. This guide gives you a clear, repeatable workflow to diagnose any crash, even when you're missing debug symbols or working with a stripped legacy binary. Whether you have a core file, a symbol file, an unstripped build, or nothing at all, you will always know the next step. Debugging crashes in legacy C++ systems is notoriously difficult because: This workflow eliminates guesswork and gives you a deterministic path from crash to root cause. If you find a core file, continue to A2.
If not, jump to Path B. Confirm that the core file belongs to the binary you intend to debug (path, build, version). At this point you usually have: In many production setups, the deployed binary is stripped, but symbol files are archived separately. You’ll get something like: Search your symbol store (example path): If your symbol file is myapp.dbg: Or recombine into a single unstripped binary: Now you can use the same commands as in A4. If you can rebuild and reproduce the crash: Run the debug build under the same conditions until it crashes and generates a new core file. Then debug that core with full symbols as in A4. If you cannot reproduce the crash (e.g., one‑off production incident), continue with A8 or A9. If you still have the original unstripped build (or can reconstruct it): This gives you the function and line number corresponding to the crash address. Even with no symbols and no way to reproduce, you can still extract useful information. You might see something like: If rdi is 0x0, you can infer a null pointer dereference, even without symbols. If ulimit -c is 0, core dumps are disabled for your shell or service. Temporary (current shell): Permanent (system‑wide): You may need to log out and back in, or restart services. Configure a directory for core files: This pattern includes: This ensures any process can write core files there. Create a small crash program: Verify that a core file appears in /var/core (or your configured directory). Now rerun the real application under the same conditions.When it crashes, it should generate a core file. Then return to Path A and continue from A2. This workflow covers every crash scenario — from “no core file” to “no symbols” to “full debug context.”
Bookmark it, share it with your team, and use it as your standard operating procedure for production crash analysis. Templates let you quickly answer FAQs or store snippets for re-use. Are you sure you want to ? It will become hidden in your post, but will still be visible via the comment's permalink. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse
$ CRASH | v
HAVE CORE FILE? |-- No --> Enable cores (Path B) → Reproduce crash | |-- Yes (Path A) | v HAVE DEBUG SYMBOLS? |-- Yes --> Debug now (A4) | |-- No | v HAVE SYMBOL FILE? |-- Yes --> Load with -s (A6) | |-- No | v CAN REPRODUCE WITH SYMBOLS? |-- Yes --> Rebuild with -g (A7) | |-- No | v HAVE ORIGINAL BUILD? |-- Yes --> Map addresses (A8) | |-- No --> Fallback analysis (A9)
CRASH | v
HAVE CORE FILE? |-- No --> Enable cores (Path B) → Reproduce crash | |-- Yes (Path A) | v HAVE DEBUG SYMBOLS? |-- Yes --> Debug now (A4) | |-- No | v HAVE SYMBOL FILE? |-- Yes --> Load with -s (A6) | |-- No | v CAN REPRODUCE WITH SYMBOLS? |-- Yes --> Rebuild with -g (A7) | |-- No | v HAVE ORIGINAL BUILD? |-- Yes --> Map addresses (A8) | |-- No --> Fallback analysis (A9)
CRASH | v
HAVE CORE FILE? |-- No --> Enable cores (Path B) → Reproduce crash | |-- Yes (Path A) | v HAVE DEBUG SYMBOLS? |-- Yes --> Debug now (A4) | |-- No | v HAVE SYMBOL FILE? |-- Yes --> Load with -s (A6) | |-- No | v CAN REPRODUCE WITH SYMBOLS? |-- Yes --> Rebuild with -g (A7) | |-- No | v HAVE ORIGINAL BUILD? |-- Yes --> Map addresses (A8) | |-- No --> Fallback analysis (A9)
ls -la core*
ls -la /var/core/
find / -name "core*" -type f 2>/dev/null
cat /proc/sys/kernel/core_pattern
ls -la core*
ls -la /var/core/
find / -name "core*" -type f 2>/dev/null
cat /proc/sys/kernel/core_pattern
ls -la core*
ls -la /var/core/
find / -name "core*" -type f 2>/dev/null
cat /proc/sys/kernel/core_pattern
file core
gdb -c core -batch -ex "info files"
file core
gdb -c core -batch -ex "info files"
file core
gdb -c core -batch -ex "info files"
file ./myapp
nm ./myapp | head
file ./myapp
nm ./myapp | head
file ./myapp
nm ./myapp | head
gdb ./myapp core
gdb ./myapp core
gdb ./myapp core
bt full
info threads
thread apply all bt
frame 0
info locals
print var
list
bt full
info threads
thread apply all bt
frame 0
info locals
print var
list
bt full
info threads
thread apply all bt
frame 0
info locals
print var
list
readelf -n ./myapp | grep "Build ID"
readelf -n ./myapp | grep "Build ID"
readelf -n ./myapp | grep "Build ID"
Build ID: 1234567890abcdef...
Build ID: 1234567890abcdef...
Build ID: 1234567890abcdef...
find /symbols -type f -exec grep -l "1234567890abcdef" {} \;
find /symbols -type f -exec grep -l "1234567890abcdef" {} \;
find /symbols -type f -exec grep -l "1234567890abcdef" {} \;
gdb -s myapp.dbg -e ./myapp -c core
gdb -s myapp.dbg -e ./myapp -c core
gdb -s myapp.dbg -e ./myapp -c core
eu-unstrip ./myapp myapp.dbg -o myapp.full
gdb ./myapp.full core
eu-unstrip ./myapp myapp.dbg -o myapp.full
gdb ./myapp.full core
eu-unstrip ./myapp myapp.dbg -o myapp.full
gdb ./myapp.full core
g++ -g -O0 -fno-omit-frame-pointer -o myapp_debug ...
g++ -g -O0 -fno-omit-frame-pointer -o myapp_debug ...
g++ -g -O0 -fno-omit-frame-pointer -o myapp_debug ...
gdb -c core -batch -ex "info registers" | grep rip
gdb -c core -batch -ex "info registers" | grep rip
gdb -c core -batch -ex "info registers" | grep rip
gdb -c core -batch -ex "info proc mappings"
gdb -c core -batch -ex "info proc mappings"
gdb -c core -batch -ex "info proc mappings"
offset = crash_address - base_address_of_binary
offset = crash_address - base_address_of_binary
offset = crash_address - base_address_of_binary
addr2line -e /path/to/unstripped/myapp -f 0xOFFSET
addr2line -e /path/to/unstripped/myapp -f 0xOFFSET
addr2line -e /path/to/unstripped/myapp -f 0xOFFSET
gdb -c core -batch -ex "info registers"
gdb -c core -batch -ex "info registers"
gdb -c core -batch -ex "info registers"
gdb -c core -batch -ex "x/20i $rip-40"
gdb -c core -batch -ex "x/20i $rip-40"
gdb -c core -batch -ex "x/20i $rip-40"
mov %rax,%rdi
test %rdi,%rdi
je <skip>
mov (%rdi),%rdx ← crash here
mov %rax,%rdi
test %rdi,%rdi
je <skip>
mov (%rdi),%rdx ← crash here
mov %rax,%rdi
test %rdi,%rdi
je <skip>
mov (%rdi),%rdx ← crash here
ulimit -c
cat /proc/sys/kernel/core_pattern
ulimit -c
cat /proc/sys/kernel/core_pattern
ulimit -c
cat /proc/sys/kernel/core_pattern
ulimit -c unlimited
ulimit -c unlimited
ulimit -c unlimited
echo "* soft core unlimited" | -weight: 600;">sudo tee -a /etc/security/limits.conf
echo "* hard core unlimited" | -weight: 600;">sudo tee -a /etc/security/limits.conf
echo "* soft core unlimited" | -weight: 600;">sudo tee -a /etc/security/limits.conf
echo "* hard core unlimited" | -weight: 600;">sudo tee -a /etc/security/limits.conf
echo "* soft core unlimited" | -weight: 600;">sudo tee -a /etc/security/limits.conf
echo "* hard core unlimited" | -weight: 600;">sudo tee -a /etc/security/limits.conf
echo "/var/core/core.%e.%p.%t" | -weight: 600;">sudo tee /proc/sys/kernel/core_pattern
echo "/var/core/core.%e.%p.%t" | -weight: 600;">sudo tee /proc/sys/kernel/core_pattern
echo "/var/core/core.%e.%p.%t" | -weight: 600;">sudo tee /proc/sys/kernel/core_pattern
-weight: 600;">sudo mkdir -p /var/core
-weight: 600;">sudo chmod 1777 /var/core
-weight: 600;">sudo mkdir -p /var/core
-weight: 600;">sudo chmod 1777 /var/core
-weight: 600;">sudo mkdir -p /var/core
-weight: 600;">sudo chmod 1777 /var/core
int main() { int* p = nullptr; *p = 42;
}
int main() { int* p = nullptr; *p = 42;
}
int main() { int* p = nullptr; *p = 42;
}
g++ -g crash_test.cpp -o crash_test
./crash_test
g++ -g crash_test.cpp -o crash_test
./crash_test
g++ -g crash_test.cpp -o crash_test
./crash_test
ulimit -c 0
-fomit-frame-pointer
ulimit -c unlimited
find / -name "core*"
file ./myapp
readelf -n ./myapp
gdb -s myapp.dbg -e myapp -c core
addr2line -e myapp -f 0xOFFSET
cat /proc/sys/kernel/core_pattern
-fno-omit-frame-pointer - Deployments often strip symbols
- Core dumps are disabled in production
- Build IDs don’t match
- ASLR shifts memory layouts
- Frame pointers are omitted
- Systemd overrides ulimit settings - If the binary is not stripped and you see symbol names → go to A4.
- If it is stripped → go to A5. - The crashing function and line
- The call stack
- Local variables and arguments - If you find a matching symbol file → go to A6.
- If not → go to A7. - Extract the crash address from the core: - Get the memory map of the process: - Compute the offset: - Map the offset to source: - Null pointers (rax, rdi, etc. equal to 0x0)
- Suspicious addresses in your binary’s range - %e — executable name
- %t — timestamp - Core dumps disabled in production (ulimit -c 0)
- Stripped binaries deployed without archiving symbol files
- Mismatched Build IDs between binary and symbol file
- ASLR causing incorrect address mapping when computing offsets
- Missing frame pointers (-fomit-frame-pointer) breaking backtraces
- Systemd or other -weight: 500;">service managers overriding ulimit
- Symbol files not stored or indexed by Build ID - Always compile with -g, then strip separately for release.
- Store symbol files indexed by Build ID in a central, backed‑up location.
- Use -fno-omit-frame-pointer for more reliable backtraces.
- Test core dump generation in a staging environment that mirrors production.
- Automate core collection and symbol archiving as part of your deployment pipeline.
