Score:3

curl dynamic linking problem in 21.10

cn flag

I recently got a Dell Precision 3450 with ubuntu 20.04, and upgraded to ubuntu 21.10 in two steps (20.04 -> 21.04 -> 21.10). This required upgrading the file /usr/lib/os-release by hand, as described here, probably because of customizations by Dell to the OEM install.

In any case, the system works fine now, with one exception. When I type curl in the terminal (which I installed after the second upgrade), I get the following error message.

curl: error while loading shared libraries: libldap_r-2.4.so.2: cannot open shared object file: No such file or directory

So it seems like curl is trying to load libldap in version 2.4, which is not installed. I asked a friend who also has ubuntu 21.10 and curl works fine for him with the same versions of curl (7.74.0-1.3ubuntu2) and libldap (2.5.6+dfsg-1~exp1ubuntu1).

I also tried to install an older version of libldap -- specifically the file libldap-2.4-2_2.4.57+dfsg-2ubuntu1_amd64.deb from the 21.04 repo -- but after doing that, curl gives me a new error message:

curl: symbol lookup error: curl: undefined symbol: curl_multi_poll, version URL_OPENSSL_4

so that doesn't help.

Does anybody have an idea what's going on?

I don't understand understand how dynamic linking works, and googling didn't really help. This site suggests to simply do sudo /sbin/ldconfig -v but that didn't work.

I don't know if the problem is specific to the OEM install of ubuntu and "forced" ugprade, but the surprising thing is that everything except curl seems to work.

EDIT:

Here is the output of ldd $(type -p curl) on the system where curl works.

 linux-vdso.so.1 (0x00007ffe043f1000)
libcurl.so.4 => /lib/x86_64-linux-gnu/libcurl.so.4 (0x00007fc870cbb000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc870c9f000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc870a77000)
libnghttp2.so.14 => /lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007fc870a49000)
libidn2.so.0 => /lib/x86_64-linux-gnu/libidn2.so.0 (0x00007fc870a28000)
librtmp.so.1 => /lib/x86_64-linux-gnu/librtmp.so.1 (0x00007fc870a09000)
libssh.so.4 => /lib/x86_64-linux-gnu/libssh.so.4 (0x00007fc87099a000)
libpsl.so.5 => /lib/x86_64-linux-gnu/libpsl.so.5 (0x00007fc870986000)
libssl.so.1.1 => /lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007fc8708ee000)
libcrypto.so.1.1 => /lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007fc870613000)
libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007fc8705c1000)
libldap-2.5.so.0 => /lib/x86_64-linux-gnu/libldap-2.5.so.0 (0x00007fc870562000)
liblber-2.5.so.0 => /lib/x86_64-linux-gnu/liblber-2.5.so.0 (0x00007fc87054f000)
libbrotlidec.so.1 => /lib/x86_64-linux-gnu/libbrotlidec.so.1 (0x00007fc870541000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc870db5000)
libunistring.so.2 => /lib/x86_64-linux-gnu/libunistring.so.2 (0x00007fc8703bf000)
libgnutls.so.30 => /lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007fc8701da000)
libhogweed.so.6 => /lib/x86_64-linux-gnu/libhogweed.so.6 (0x00007fc870192000)
libnettle.so.8 => /lib/x86_64-linux-gnu/libnettle.so.8 (0x00007fc87014a000)
libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007fc8700c7000)
libkrb5.so.3 => /lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007fc86fffe000)
libk5crypto.so.3 => /lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007fc86ffd1000)
libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007fc86ffcb000)
libkrb5support.so.0 => /lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007fc86ffbb000)
libsasl2.so.2 => /lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007fc86ffa0000)
libbrotlicommon.so.1 => /lib/x86_64-linux-gnu/libbrotlicommon.so.1 (0x00007fc86ff7d000)
libp11-kit.so.0 => /lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007fc86fe43000)
libtasn1.so.6 => /lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007fc86fe2d000)
libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007fc86fe24000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fc86fe0f000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fc86fe0a000)
libffi.so.8 => /lib/x86_64-linux-gnu/libffi.so.8 (0x00007fc86fdfe000)

And here's the output of ldd $(type -p curl) on the system where curl doesn't work.

linux-vdso.so.1 (0x00007fffa0f86000)
libcurl.so.4 => /usr/lib/dcaenabler/libcurl.so.4 (0x00007fc1b789f000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc1b7883000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc1b765b000)
libnghttp2.so.14 => /lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007fc1b762d000)
libidn2.so.0 => /lib/x86_64-linux-gnu/libidn2.so.0 (0x00007fc1b760c000)
librtmp.so.1 => /lib/x86_64-linux-gnu/librtmp.so.1 (0x00007fc1b75ed000)
libpsl.so.5 => /lib/x86_64-linux-gnu/libpsl.so.5 (0x00007fc1b75d7000)
libssl.so.1.1 => /usr/lib/dcaenabler/libssl.so.1.1 (0x00007fc1b734a000)
libcrypto.so.1.1 => /usr/lib/dcaenabler/libcrypto.so.1.1 (0x00007fc1b6e7f000)
libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007fc1b6e2d000)
libldap_r-2.4.so.2 => not found
liblber-2.4.so.2 => not found
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc1b6e26000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc1b7b74000)
libunistring.so.2 => /lib/x86_64-linux-gnu/libunistring.so.2 (0x00007fc1b6ca4000)
libgnutls.so.30 => /lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007fc1b6abf000)
libhogweed.so.6 => /lib/x86_64-linux-gnu/libhogweed.so.6 (0x00007fc1b6a77000)
libnettle.so.8 => /lib/x86_64-linux-gnu/libnettle.so.8 (0x00007fc1b6a31000)
libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007fc1b69ac000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fc1b69a7000)
libkrb5.so.3 => /lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007fc1b68de000)
libk5crypto.so.3 => /lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007fc1b68b1000)
libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007fc1b68ab000)
libkrb5support.so.0 => /lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007fc1b689d000)
libp11-kit.so.0 => /lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007fc1b6761000)
libtasn1.so.6 => /lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007fc1b674b000)
libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007fc1b6744000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fc1b672f000)
libffi.so.8 => /lib/x86_64-linux-gnu/libffi.so.8 (0x00007fc1b6723000)

Output of which curl on the problematic system:

/usr/bin/curl

Output of aptitude search ~o on problematic system:

i   dca-enabler                                                                            - Provides the DELL Client Agent Enabler software.                                                
i   dca-enabler-packages                                                                   - Provides the DELL Client Agent Enabler Dependencies.                                            
i   dell-canonical-logo                                                                    - Install the dell-canonical-logo in gnome-control-center.                                        
i   dell-super-key                                                                         - Disables the super key by default.                                                              
i   djvu2pdf                                                                               - Script to convert Djvu files to PDF files                                                       
i   fist                                                                                   - Factory Install System Test                                                                     
i A gir1.2-unity-5.0                                                                       - GObject introspection data for the Unity library                                                
i A linux-headers-5.10.0-1031-oem                                                          - Linux kernel headers for version 5.10.0 on 64 bit x86 SMP                                       
i A linux-headers-5.10.0-1050-oem                                                          - Linux kernel headers for version 5.10.0 on 64 bit x86 SMP                                       
i A linux-headers-oem-20.04b                                                               - OEM Linux kernel headers                                                                        
i A linux-image-5.10.0-1031-oem                                                            - Signed kernel image oem                                                                         
i A linux-image-5.10.0-1050-oem                                                            - Signed kernel image oem                                                                         
i A linux-image-oem-20.04b                                                                 - OEM Linux kernel image                                                                          
i A linux-modules-5.10.0-1031-oem                                                          - Linux kernel extra modules for version 5.10.0 on 64 bit x86 SMP                                 
i A linux-modules-5.10.0-1050-oem                                                          - Linux kernel extra modules for version 5.10.0 on 64 bit x86 SMP                                 
i   linux-oem-20.04b                                                                       - Complete OEM Linux kernel and headers                                                           
i A linux-oem-5.10-headers-5.10.0-1031                                                     - Header files related to Linux kernel version 5.10.0                                             
i A linux-oem-5.10-headers-5.10.0-1050                                                     - Header files related to Linux kernel version 5.10.0                                             
i A linux-oem-5.10-tools-5.10.0-1031                                                       - Linux kernel version specific tools for version 5.10.0-1031                                     
i   linux-tools-5.10.0-1031-oem                                                            - Linux kernel version specific tools for version 5.10.0-1031                                     
i   lsd                                                                                    - An ls command with a lot of pretty colors.                                                      
i   manage-distro-upgrade                                                                  - Disable upgrade for OEM.                                                                        
i A manage-estar-settings                                                                  - Energy Star related settings                                                                    
i A oem-fix-misc-cnl-tlp-estar-conf                                                        - customized configuration for tlp.                                                               
i   oem-ouagadougou-meta                                                                   - Meta package for the OEM mainstreams image.                                                     
i   oem-release                                                                            - Show oem release version                                                                        
i   oem-somerville-butterfree-meta                                                         - hardware support for Somerville Butterfree platform                                             
i   oem-somerville-factory-butterfree-meta                                                 - hardware support for Somerville Butterfree platform (factory)                                   
i   oem-somerville-factory-meta                                                            - hardware support for Somerville platform                                                        
i   oem-somerville-meta                                                                    - hardware support for Somerville platform                                                        
i A oem-somerville-partner-archive-keyring                                                 - Somerville project keyrings                                                                     
i   slack-desktop                                                                          - Slack Desktop                                                                                   
i   sosreport-oem                                                                          - OEM image releated sosreport plugins and utilities.                                             

aptitude search ~b outputs nothing -- no broken packages.

Here's the output of tree /usr/local/:

/usr/local/
├── bin
├── etc
├── games
├── include
├── lib
│   └── python3.9
│       └── dist-packages
├── man -> share/man
├── sbin
├── share
│   ├── ca-certificates
│   ├── fonts
│   ├── man
│   ├── sgml
│   │   ├── declaration
│   │   ├── dtd
│   │   ├── entities
│   │   ├── misc
│   │   └── stylesheet
│   ├── texmf
│   └── xml
│       ├── declaration
│       ├── entities
│       ├── misc
│       └── schema
├── src
└── stow

Output of dpkg -S /usr/lib/dcaenabler/libcurl.so.4 /usr/lib/dcaenabler/libssl.so.1.1 /usr/lib/dcaenabler/libcrypto.so.1.1:

dca-enabler-packages: /usr/lib/dcaenabler/libcurl.so.4
dca-enabler-packages: /usr/lib/dcaenabler/libssl.so.1.1
dca-enabler-packages: /usr/lib/dcaenabler/libcrypto.so.1.1

Most likely related: I just observed repeated lines of the form

10 Nov 2021 15:36:59    j-dell  env[10851]  /usr/sbin/dcae: error while loading shared libraries: libldap_r-2.4.so.2: cannot open shared object file: No such file or directory

in /var/log/syslog. If I run dcae on the command line I get the error message

dcae: error while loading shared libraries: libldap_r-2.4.so.2: cannot open shared object file: No such file or directory

just as with curl.

user313032 avatar
cn flag
@TBr: Ubuntu 21.10 comes with with libldap-2.5-0 which doesn't include the file in question (but on my friend's computer curl works nevertheless). That's why I installed libldap-2.4-2 from 21.04, which does include the file (in `/usr/lib/x86_64-linux-gnu/`). With the file present I get the second error message. I don't understand your suggestion.
user313032 avatar
cn flag
Thanks! Sorry I wasn't explicit about which package contained the file.
user313032 avatar
cn flag
That gives the same two error messages I already posted, depending on whether `libldap-2.4-2` is installed
waltinator avatar
it flag
Compare `ldd $(type -p curl)` on both systems. Use `dpkg -S filename` (read `man dpkg`) to see which package provides `filename`.
user313032 avatar
cn flag
@waltinator I added the outputs of the respective commands to my question. As I already said, on the system where curl doesn't work it's trying to load an older version of libldap. From the output it's also apparent that the same happens for liblber. So apparently the versions of the libraries are not hardcoded in the binaries? (The two binaries are the same, I diffed them to make sure).
user313032 avatar
cn flag
To find out which package a file comes from I usually use `apt-file search` instead of `dpkg -S` as I think the latter only searches through the files of installed packages. But for `libldap_r-2.4.so.2` both search commands fail since the file comes from the package `libldap-2.4-2` which is not included in Ubuntu 22.10.
user313032 avatar
cn flag
Same for `liblber-2.4.so.2`.
N0rbert avatar
zw flag
Output for `which curl` is also needed, it may be from `/usr/local/bin` , so self-compiled.
user313032 avatar
cn flag
@N0rbert: I added it above, it's /usr/bin/curl and not self-compiled. I double checked that the two binaries on the different systems are identical.
N0rbert avatar
zw flag
Installation of v2.4 is still possible by `wget http://archive.ubuntu.com/ubuntu/pool/main/o/openldap/libldap-2.4-2_2.4.57+dfsg-2ubuntu1_amd64.deb && sudo apt-get install --reinstall ./libldap-2.4-2_2.4.57+dfsg-2ubuntu1_amd64.deb` . Such strange things are usually happen when some local/obsolete package keep installed on upgraded system. Please install Aptitude and tree by `sudo apt-get install tree aptitude` and add output from broken system - `aptitude search ~o` with `tree /usr/local` to the question (if long - use pastebin).
user313032 avatar
cn flag
@N0rbert : I already tried installing v2.4, that leads to a different error message as explained above. I added the requested command outputs to the question. Thanks for your help!
N0rbert avatar
zw flag
I suspect that `dca-enabler` and maybe `dca-enabler-packages` are root of the problem. Please share the output of `dpkg -S /usr/lib/dcaenabler/libcurl.so.4 /usr/lib/dcaenabler/libssl.so.1.1 /usr/lib/dcaenabler/libcrypto.so.1.1` . Do you plan to use this Dell software?
user313032 avatar
cn flag
Added the requested information. Also observed relevant info in /var/log/syslog, see above. Regarding your final question: I don't really know what the dca stuff does. I ordered the system with ubuntu preinstalled since it seemed easier than installing myself, and also cheaper. I was also hoping for possibly better hardware support. Apart from that I'm not interested in any dell specific features.
user313032 avatar
cn flag
@N0rbert : are you suggestin that I deinstall the dca packages?
N0rbert avatar
zw flag
I would suggest to remove these Dell packages using `sudo apt-get purge --autoremove dca-enabler-packages dca-enabler` then reboot to check the changes.
user313032 avatar
cn flag
@N0rbert: Removing the packages solved the problem! Thanks so much for your help! If you add an answer I'll accept it.
Score:3
zw flag

Analysis of ldd /usr/bin/curl output shows that Dell packages cause this issue. To fix the system we need to remove them by

sudo apt-get purge --autoremove dca-enabler-packages dca-enabler

Note: tool for ldd output analysis is discussed in this Q&A. It helped me a lot to detect your problem programmatically.

user313032 avatar
cn flag
Thanks! In hindsight it's pretty clear how to identify the problematic packages by comparing the ldd outputs and observing e.g. that different versions of `libcurl.so.4` are loaded on the two machines. But I didn't see it myself until you pointed it out.
mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.