Arch Linux AUR Dependency Hell: Debugging the ICU 76 78 Migration

Fixing ICU Library Dependency Hell on Arch Linux: A Deep Dive into Shared Library Mismatches ## The Problem ## Understanding the Root Cause ## What is ICU? ## The Rolling Release Challenge ## Diagnostic Steps ## 1. Check Current ICU Version ## 2. Identify Package Origin ## 3. Attempt to Rebuild ## 4. Discover the Real Culprit ## 5. Repository Discovery ## The Solution ## Step 1: Remove Conflicting Packages ## Step 2: Rebuild Qt5WebEngine from AUR ## Step 3: Rebuild Notepadqq ## Step 4: Verification ## Key Lessons Learned ## 1. AUR Packages Don't Auto-Update with System Libraries ## 2. Partial Upgrades Are Dangerous ## 2. Rolling Release Requires Package Consistency ## 3. ABI Compatibility Matters ## 4. Dependency Chains Run Deep ## What NOT to Do ## ❌ Creating Symlinks ## ❌ Ignoring Update Errors ## Prevention Strategies ## 1. Regular Orphan Package Cleanup ## 2. Monitor AUR Package Build Failures ## 3. Use Binary Package Alternatives ## 4. Check Library Dependencies ## Conclusion I recently encountered a frustrating issue when trying to launch Notepadqq on my Arch-based system (EndeavourOS): This is a classic shared library dependency problem that's particularly common on rolling-release distributions like Arch Linux. ICU (International Components for Unicode) is a mature, widely used set of C/C++ libraries providing Unicode and globalization support. Many applications depend on it, including Qt's WebEngine component. The issue stemmed from a critical version mismatch between official Arch repositories and AUR packages: The core problem: Notepadqq hasn't been actively maintained since 2018. As an AUR package, it doesn't get automatically rebuilt when Arch updates system libraries like ICU. This created a desynchronization between: This cascading dependency mismatch is a common pitfall when mixing official packages with unmaintained AUR software. This confirmed I had ICU 78, while the application expected version 76. This confirmed notepadqq was installed, but being an AUR package (not from official repos), it wouldn't automatically rebuild when system libraries like ICU were updated. When I tried rebuilding Notepadqq from AUR: I hit a deeper issue: The problem wasn't just Notepadqq—Qt5WebEngine itself was compiled against the old ICU version. When trying to update qt5-webengine: Critical finding: qt5-webengine had been removed from official Arch repositories but remained installed on my system as an orphaned package. Total freed: 174.88 MiB This compiled qt5-webengine from source, linking it against the current ICU 78. With qt5-webengine now properly linked to ICU 78, Notepadqq compiled successfully. Success! The application now links against the correct ICU version. Critical insight: When Arch updates a core library like ICU, official packages are rebuilt automatically. AUR packages are not—they must be manually rebuilt by users or maintainers. Notepadqq was last updated in 2018, meaning: Even though I was updating my system 2-5 times daily, I hadn't rebooted in 7 days. However, rebooting wasn't the issue — the problem was binary incompatibility on disk. Arch's rolling release model means: The ICU version number in the shared library name (libicui18n.so.76 vs libicui18n.so.78) indicates an ABI break. You cannot simply symlink one version to another without risking crashes or undefined behavior. Recognizing an ABI Break How did I know this wasn't just a missing symlink issue? 1. Version in library filename: 2. Versioned symbols in linker errors: 3. Why symlinking would fail: Even if we created ln -s libicui18n.so.78 libicui18n.so.76: What appeared to be a simple Notepadqq issue actually required: While this might allow the program to start, it can cause: Regularly running pacman -Syu is good, but you must also: When a system library updates, rebuild affected AUR packages immediately: Important: Unmaintained AUR packages (like Notepadqq, last updated 2018) will always need manual rebuilds after system library updates. Consider: For stability-critical applications, consider -bin packages from AUR that bundle their dependencies, though this increases disk usage. After major library updates: This issue perfectly demonstrates why Arch Linux is considered an "advanced" distribution. The rolling release model provides cutting-edge packages but requires users to understand: The solution wasn't a quick fix but rather a systematic diagnosis and rebuild of the dependency chain. While frustrating in the moment, issues like this deepen understanding of how Linux systems work at a fundamental level. Have you encountered similar dependency issues on Arch? What's your approach to handling library version mismatches? Share your experiences in the comments! Templates let you quickly answer FAQs or store snippets for re-use. Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment's permalink. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse COMMAND_BLOCK: $ notepadqq /usr/bin/../lib/notepadqq/notepadqq-bin: error while loading shared libraries: libicui18n.so.76: cannot open shared object file: No such file or directory Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: $ notepadqq /usr/bin/../lib/notepadqq/notepadqq-bin: error while loading shared libraries: libicui18n.so.76: cannot open shared object file: No such file or directory COMMAND_BLOCK: $ notepadqq /usr/bin/../lib/notepadqq/notepadqq-bin: error while loading shared libraries: libicui18n.so.76: cannot open shared object file: No such file or directory COMMAND_BLOCK: $ ls /usr/lib/libicui18n.so* /usr/lib/libicui18n.so /usr/lib/libicui18n.so.78 /usr/lib/libicui18n.so.78.1 Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: $ ls /usr/lib/libicui18n.so* /usr/lib/libicui18n.so /usr/lib/libicui18n.so.78 /usr/lib/libicui18n.so.78.1 COMMAND_BLOCK: $ ls /usr/lib/libicui18n.so* /usr/lib/libicui18n.so /usr/lib/libicui18n.so.78 /usr/lib/libicui18n.so.78.1 COMMAND_BLOCK: $ pacman -Ss notepadqq $ # Returns empty line $ yay notepadqq $ 1 aur/notepadqq 2.0.0beta-3 (+1 0.32) (Installed) Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: $ pacman -Ss notepadqq $ # Returns empty line $ yay notepadqq $ 1 aur/notepadqq 2.0.0beta-3 (+1 0.32) (Installed) COMMAND_BLOCK: $ pacman -Ss notepadqq $ # Returns empty line $ yay notepadqq $ 1 aur/notepadqq 2.0.0beta-3 (+1 0.32) (Installed) COMMAND_BLOCK: $ yay -S notepadqq --rebuild Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: $ yay -S notepadqq --rebuild COMMAND_BLOCK: $ yay -S notepadqq --rebuild CODE_BLOCK: /usr/bin/ld: /usr/lib/libQt5WebEngineCore.so: undefined reference to `icu_76::Normalizer::normalize(...)` /usr/bin/ld: /usr/lib/libQt5WebEngineCore.so: undefined reference to `ucnv_open_76' Enter fullscreen mode Exit fullscreen mode CODE_BLOCK: /usr/bin/ld: /usr/lib/libQt5WebEngineCore.so: undefined reference to `icu_76::Normalizer::normalize(...)` /usr/bin/ld: /usr/lib/libQt5WebEngineCore.so: undefined reference to `ucnv_open_76' CODE_BLOCK: /usr/bin/ld: /usr/lib/libQt5WebEngineCore.so: undefined reference to `icu_76::Normalizer::normalize(...)` /usr/bin/ld: /usr/lib/libQt5WebEngineCore.so: undefined reference to `ucnv_open_76' COMMAND_BLOCK: $ pacman -Qo /usr/lib/libQt5WebEngineCore.so /usr/lib/libQt5WebEngineCore.so is owned by qt5-webengine 5.15.19-4 Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: $ pacman -Qo /usr/lib/libQt5WebEngineCore.so /usr/lib/libQt5WebEngineCore.so is owned by qt5-webengine 5.15.19-4 COMMAND_BLOCK: $ pacman -Qo /usr/lib/libQt5WebEngineCore.so /usr/lib/libQt5WebEngineCore.so is owned by qt5-webengine 5.15.19-4 COMMAND_BLOCK: $ sudo pacman -Syu qt5-webengine error: target not found: qt5-webengine Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: $ sudo pacman -Syu qt5-webengine error: target not found: qt5-webengine COMMAND_BLOCK: $ sudo pacman -Syu qt5-webengine error: target not found: qt5-webengine COMMAND_BLOCK: $ sudo pacman -Rns qt5-webengine notepadqq Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: $ sudo pacman -Rns qt5-webengine notepadqq COMMAND_BLOCK: $ sudo pacman -Rns qt5-webengine notepadqq COMMAND_BLOCK: $ yay -S qt5-webengine Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: $ yay -S qt5-webengine COMMAND_BLOCK: $ yay -S qt5-webengine COMMAND_BLOCK: $ yay -S notepadqq Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: $ yay -S notepadqq COMMAND_BLOCK: $ yay -S notepadqq COMMAND_BLOCK: $ ldd /usr/bin/notepadqq | grep icu libicui18n.so.78 => /usr/lib/libicui18n.so.78 libicuuc.so.78 => /usr/lib/libicuuc.so.78 Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: $ ldd /usr/bin/notepadqq | grep icu libicui18n.so.78 => /usr/lib/libicui18n.so.78 libicuuc.so.78 => /usr/lib/libicuuc.so.78 COMMAND_BLOCK: $ ldd /usr/bin/notepadqq | grep icu libicui18n.so.78 => /usr/lib/libicui18n.so.78 libicuuc.so.78 => /usr/lib/libicuuc.so.78 COMMAND_BLOCK: # DON'T DO THIS $ sudo ln -s /usr/lib/libicui18n.so.78 /usr/lib/libicui18n.so.76 Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: # DON'T DO THIS $ sudo ln -s /usr/lib/libicui18n.so.78 /usr/lib/libicui18n.so.76 COMMAND_BLOCK: # DON'T DO THIS $ sudo ln -s /usr/lib/libicui18n.so.78 /usr/lib/libicui18n.so.76 COMMAND_BLOCK: $ pacman -Qtdq | pacman -Rns - Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: $ pacman -Qtdq | pacman -Rns - COMMAND_BLOCK: $ pacman -Qtdq | pacman -Rns - COMMAND_BLOCK: $ yay -S --rebuild <package-name> Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: $ yay -S --rebuild <package-name> COMMAND_BLOCK: $ yay -S --rebuild <package-name> COMMAND_BLOCK: $ ldd /usr/bin/<application> | grep "not found" Enter fullscreen mode Exit fullscreen mode COMMAND_BLOCK: $ ldd /usr/bin/<application> | grep "not found" COMMAND_BLOCK: $ ldd /usr/bin/<application> | grep "not found" - Arch upstream had moved forward to ICU 78 in official repositories - Notepadqq (an AUR package) was still compiled against ICU 76 - Qt5WebEngine had been removed from official repos but remained on my system, still linked to ICU 76 - Official repository packages (ICU 78) - Outdated AUR binaries (expecting ICU 76) - Orphaned repository packages (Qt5WebEngine with ICU 76 linkage) - qt5-webengine (159.19 MiB) - notepadqq (5.89 MiB) - Related dependencies (qt5-location, qt5-webchannel, qt5-websockets) - It never received rebuilds for ICU 76, 77, or 78 - Each ICU update broke the package until manually rebuilt - This is a fundamental difference between official repos and AUR - Libraries like ICU update frequently - All dependent packages must be rebuilt or updated in sync - Orphaned packages from removed repositories can cause hidden issues - libicui18n.so.76 vs libicui18n.so.78 - The .76 and .78 are SONAME major versions - Different major versions = intentional ABI incompatibility - icu_76::Normalizer::normalize ← version baked into symbol name - ucnv_open_76 ← function name includes version - The library exposes icu_78:: symbols, but the binary needs icu_76:: - No amount of symlinking can resolve this mismatch - The binary would load, but call icu_76:: functions - The library only has icu_78:: functions - Result: immediate crash or undefined behavior - Diagnosing the ICU mismatch - Discovering the Qt5WebEngine dependency - Identifying the orphaned repository package - Rebuilding the entire dependency chain - Runtime crashes - Undefined behavior - Data corruption - Difficult-to-debug issues - Check for orphaned packages: pacman -Qtdq - Verify AUR packages after system library updates - Review failed transaction logs - Finding actively maintained alternatives - Adopting the AUR package yourself - Using AppImage/Flatpak versions that bundle dependencies - Shared library versioning (ABI compatibility) - Dependency chains - The relationship between official repositories and AUR - When to rebuild vs. reinstall packages - Distribution: EndeavourOS (Arch-based) - Package Manager: pacman + yay - Affected Versions: ICU 76 → 78, Qt5WebEngine 5.15.19-4