Score:0

Apt Downloading Files With Encoded Characters

th flag

I maintain a few Ubuntu servers that do not have access to the internet, and therefore cannot connect to any public repositories. To keep our instances updated, I run a script that basically only downloads the installed packages and any dependencies to keep the file transfer as small as possible (as opposed to downloading a mirror of the entire repo).

This has worked well for about 5-6 years (16.04LTS, 18.04LTS, 20.04LTS, and now 22.04LTS), but within the past 2 months I've noticed that some packages are being downloaded and saved to \var\cache\apt\archives with encoded characters in the file name.

As an example, I am seeing: vim_2%253a8.2.3995-1ubuntu2.5_amd64.deb and xauth_1%253a1.1-1build2_amd64.deb

This is causing issues when I load these packages to my local repository and run apt from the clients. The HTTP communication is unsuccessful for these packages, and I am forced to run apt --skip-broken and then install those packages separately.

While I'm aware I could change the filenames and create the repofile with the modified names, I'm hoping that there is an easier fix or a simple solution for what I'm doing wrong or what may have changed. Thank you for any assistance you may be able to provide!!

This is currently happening with Ubuntu 22.04LTS.

EDIT: I just reviewed downloaded files from November of 2022 before this was an issue, and I confirmed that the file names have always included those encoded characters...but it just appears that apt is handling these specific files differently within the past few months.

Additional info:

  • This is happening on at least two different networks (and two different local repositories using the same config).
  • This is happening with several packages with a % in the file path.

Update: Full output below.

root@vsite01-splsyslg:~# sudo apt update
Get:1 http://localrepo/2204  InRelease [1,912 B]
Get:2 http://localrepo/2204  Packages [285 kB]
Fetched 287 kB in 1s (445 kB/s)   
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
107 packages can be upgraded. Run 'apt list --upgradable' to see them.
W: http://localrepo/2204/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
root@vsite01-splsyslg:~# sudo apt upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
  linux-headers-5.15.0-75 linux-headers-5.15.0-75-generic linux-image-5.15.0-75-generic linux-modules-5.15.0-75-generic linux-modules-extra-5.15.0-75-generic
The following packages will be upgraded:
  apparmor apport apt apt-utils bind9-dnsutils bind9-host bind9-libs binutils binutils-common binutils-x86-64-linux-gnu ca-certificates cloud-init curl distro-info-data dpkg fwupd fwupd-signed git git-man grub-common grub2-common
  iptables libapparmor1 libapt-pkg6.0 libbinutils libcap2 libcap2-bin libctf-nobfd0 libctf0 libcurl3-gnutls libcurl4 libfreetype6 libfwupd2 libfwupdplugin5 libglib2.0-0 libglib2.0-bin libglib2.0-data libgssapi-krb5-2 libip4tc2
  libip6tc2 libk5crypto3 libkrb5-3 libkrb5support0 libldap-2.5-0 libldap-common libncurses6 libncursesw6 libnetplan0 libnss-systemd libnss3 libpam-cap libpam-systemd libperl5.34 libpython3.10 libpython3.10-minimal
  libpython3.10-stdlib libssh-4 libssl3 libsystemd0 libtinfo6 libudev1 libx11-6 libx11-data libxml2 libxtables12 linux-firmware linux-generic linux-headers-generic linux-image-generic mdadm ncurses-base ncurses-bin ncurses-term
  netplan.io open-vm-tools openssh-client openssh-server openssh-sftp-server openssl perl perl-base perl-modules-5.34 python3-apport python3-distupgrade python3-problem-report python3-requests python3-software-properties python3-tz
  python3.10 python3.10-minimal snapd software-properties-common sudo systemd systemd-sysv thermald tzdata ubuntu-advantage-tools ubuntu-drivers-common ubuntu-release-upgrader-core udev update-notifier-common vim vim-common
  vim-runtime vim-tiny xxd
107 upgraded, 5 newly installed, 0 to remove and 0 not upgraded.
Need to get 456 MB of archives.
After this operation, 624 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://localrepo/2204  libsystemd0 249.11-0ubuntu3.9 [318 kB]
Get:2 http://localrepo/2204  libnss-systemd 249.11-0ubuntu3.9 [133 kB]
Get:3 http://localrepo/2204  systemd-sysv 249.11-0ubuntu3.9 [10.5 kB]
Get:4 http://localrepo/2204  libpam-systemd 249.11-0ubuntu3.9 [203 kB]
Get:5 http://localrepo/2204  systemd 249.11-0ubuntu3.9 [4,581 kB]
Get:6 http://localrepo/2204  udev 249.11-0ubuntu3.9 [1,557 kB]
Get:7 http://localrepo/2204  libudev1 249.11-0ubuntu3.9 [77.1 kB]
Err:8 http://localrepo/2204  libcap2 1:2.44-1ubuntu0.22.04.1
  404  Not Found [IP: localrepo 80]
Get:9 http://localrepo/2204  libapparmor1 3.0.4-2ubuntu2.2 [39.2 kB]
Get:10 http://localrepo/2204  libxtables12 1.8.7-1ubuntu5.1 [31.2 kB]
Get:11 http://localrepo/2204  iptables 1.8.7-1ubuntu5.1 [455 kB]
Get:12 http://localrepo/2204  libip6tc2 1.8.7-1ubuntu5.1 [20.2 kB]
Get:13 http://localrepo/2204  libip4tc2 1.8.7-1ubuntu5.1 [19.8 kB]
Get:14 http://localrepo/2204  libssl3 3.0.2-0ubuntu1.10 [1,901 kB]
Get:15 http://localrepo/2204  libapt-pkg6.0 2.4.9 [906 kB]
Get:16 http://localrepo/2204  dpkg 1.21.1ubuntu2.2 [1,239 kB]
Get:17 http://localrepo/2204  ncurses-bin 6.3-2ubuntu0.1 [184 kB]
Get:18 http://localrepo/2204  libperl5.34 5.34.0-3ubuntu1.2 [4,818 kB]
Get:19 http://localrepo/2204  perl 5.34.0-3ubuntu1.2 [232 kB]
Get:20 http://localrepo/2204  perl-base 5.34.0-3ubuntu1.2 [1,759 kB]
Get:21 http://localrepo/2204  perl-modules-5.34 5.34.0-3ubuntu1.2 [2,977 kB]
Get:22 http://localrepo/2204  ncurses-base 6.3-2ubuntu0.1 [20.2 kB]
Get:23 http://localrepo/2204  apt 2.4.9 [1,379 kB]
Get:24 http://localrepo/2204  apt-utils 2.4.9 [211 kB]
Get:25 http://localrepo/2204  libglib2.0-data 2.72.4-0ubuntu2.2 [4,612 B]
Get:26 http://localrepo/2204  libglib2.0-bin 2.72.4-0ubuntu2.2 [80.9 kB]
Get:27 http://localrepo/2204  libglib2.0-0 2.72.4-0ubuntu2.2 [1,463 kB]
Get:28 http://localrepo/2204  libxml2 2.9.13+dfsg-1ubuntu0.3 [763 kB]
Err:29 http://localrepo/2204  open-vm-tools 2:12.1.5-3~ubuntu0.22.04.1
  404  Not Found [IP: localrepo 80]
Get:30 http://localrepo/2204  libk5crypto3 1.19.2-2ubuntu0.2 [86.3 kB]
Get:31 http://localrepo/2204  libkrb5support0 1.19.2-2ubuntu0.2 [32.3 kB]
Get:32 http://localrepo/2204  libkrb5-3 1.19.2-2ubuntu0.2 [357 kB]
Get:33 http://localrepo/2204  libgssapi-krb5-2 1.19.2-2ubuntu0.2 [145 kB]
Get:34 http://localrepo/2204  libldap-2.5-0 2.5.14+dfsg-0ubuntu0.22.04.2 [183 kB]
Get:35 http://localrepo/2204  libssh-4 0.9.6-2ubuntu0.22.04.1 [185 kB]
Get:36 http://localrepo/2204  libcurl3-gnutls 7.81.0-1ubuntu1.10 [285 kB]
Err:37 http://localrepo/2204  git-man 1:2.34.1-1ubuntu1.9
  404  Not Found [IP: localrepo 80]
Err:38 http://localrepo/2204  git 1:2.34.1-1ubuntu1.9
  404  Not Found [IP: localrepo 80]
Err:39 http://localrepo/2204  openssh-sftp-server 1:8.9p1-3ubuntu0.1
  404  Not Found [IP: localrepo 80]
Err:40 http://localrepo/2204  openssh-server 1:8.9p1-3ubuntu0.1
  404  Not Found [IP: localrepo 80]
Err:41 http://localrepo/2204  openssh-client 1:8.9p1-3ubuntu0.1
  404  Not Found [IP: localrepo 80]
Get:42 http://localrepo/2204  libpython3.10 3.10.6-1~22.04.2ubuntu1.1 [1,955 kB]
Get:43 http://localrepo/2204  python3.10 3.10.6-1~22.04.2ubuntu1.1 [497 kB]
Get:44 http://localrepo/2204  libpython3.10-stdlib 3.10.6-1~22.04.2ubuntu1.1 [1,832 kB]
Get:45 http://localrepo/2204  python3.10-minimal 3.10.6-1~22.04.2ubuntu1.1 [2,262 kB]
Get:46 http://localrepo/2204  libpython3.10-minimal 3.10.6-1~22.04.2ubuntu1.1 [810 kB]
Get:47 http://localrepo/2204  libncurses6 6.3-2ubuntu0.1 [111 kB]
Get:48 http://localrepo/2204  libncursesw6 6.3-2ubuntu0.1 [147 kB]
Get:49 http://localrepo/2204  libtinfo6 6.3-2ubuntu0.1 [105 kB]
Err:50 http://localrepo/2204  ubuntu-drivers-common 1:0.9.6.2~0.22.04.3
  404  Not Found [IP: localrepo 80]
Get:51 http://localrepo/2204  update-notifier-common 3.192.54.6 [185 kB]
Get:52 http://localrepo/2204  tzdata 2023c-0ubuntu0.22.04.2 [349 kB]
Err:53 http://localrepo/2204  libpam-cap 1:2.44-1ubuntu0.22.04.1
  404  Not Found [IP: localrepo 80]
Err:54 http://localrepo/2204  libcap2-bin 1:2.44-1ubuntu0.22.04.1
  404  Not Found [IP: localrepo 80]
Get:55 http://localrepo/2204  ubuntu-advantage-tools 27.14.4~22.04 [171 kB]
Err:56 http://localrepo/2204  vim 2:8.2.3995-1ubuntu2.8
  404  Not Found [IP: localrepo 80]
Err:57 http://localrepo/2204  vim-tiny 2:8.2.3995-1ubuntu2.8
  404  Not Found [IP: localrepo 80]
Err:58 http://localrepo/2204  vim-runtime 2:8.2.3995-1ubuntu2.8
  404  Not Found [IP: localrepo 80]
Err:59 http://localrepo/2204  xxd 2:8.2.3995-1ubuntu2.8
  404  Not Found [IP: localrepo 80]
Err:60 http://localrepo/2204  vim-common 2:8.2.3995-1ubuntu2.8
  404  Not Found [IP: localrepo 80]
Err:61 http://localrepo/2204  bind9-dnsutils 1:9.18.12-0ubuntu0.22.04.1
  404  Not Found [IP: localrepo 80]
Err:62 http://localrepo/2204  bind9-host 1:9.18.12-0ubuntu0.22.04.1
  404  Not Found [IP: localrepo 80]
Err:63 http://localrepo/2204  bind9-libs 1:9.18.12-0ubuntu0.22.04.1
  404  Not Found [IP: localrepo 80]
Get:64 http://localrepo/2204  openssl 3.0.2-0ubuntu1.10 [1,183 kB]
Get:65 http://localrepo/2204  ca-certificates 20230311ubuntu0.22.04.1 [155 kB]
Get:66 http://localrepo/2204  ncurses-term 6.3-2ubuntu0.1 [267 kB]
Get:67 http://localrepo/2204  apparmor 3.0.4-2ubuntu2.2 [595 kB]
Get:68 http://localrepo/2204  grub2-common 2.06-2ubuntu7.2 [652 kB]
Get:69 http://localrepo/2204  libfreetype6 2.11.1+dfsg-1ubuntu0.2 [389 kB]
Get:70 http://localrepo/2204  grub-common 2.06-2ubuntu7.2 [2,214 kB]
Get:71 http://localrepo/2204  python3-problem-report 2.20.11-0ubuntu82.5 [11.1 kB]
Get:72 http://localrepo/2204  python3-apport 2.20.11-0ubuntu82.5 [88.9 kB]
Get:73 http://localrepo/2204  apport 2.20.11-0ubuntu82.5 [133 kB]
Get:74 http://localrepo/2204  libctf0 2.38-4ubuntu2.2 [103 kB]
Get:75 http://localrepo/2204  libctf-nobfd0 2.38-4ubuntu2.2 [107 kB]
Get:76 http://localrepo/2204  binutils-x86-64-linux-gnu 2.38-4ubuntu2.2 [2,328 kB]
Get:77 http://localrepo/2204  libbinutils 2.38-4ubuntu2.2 [660 kB]
Get:78 http://localrepo/2204  binutils 2.38-4ubuntu2.2 [3,186 B]
Get:79 http://localrepo/2204  binutils-common 2.38-4ubuntu2.2 [222 kB]
Get:80 http://localrepo/2204  netplan.io 0.105-0ubuntu2~22.04.3 [92.9 kB]
Get:81 http://localrepo/2204  libnetplan0 0.105-0ubuntu2~22.04.3 [103 kB]
Get:82 http://localrepo/2204  python3-requests 2.25.1+dfsg-2ubuntu0.1 [48.8 kB]
Get:83 http://localrepo/2204  cloud-init 23.1.2-0ubuntu0~22.04.1 [531 kB]
Get:84 http://localrepo/2204  curl 7.81.0-1ubuntu1.10 [194 kB]
Get:85 http://localrepo/2204  libcurl4 7.81.0-1ubuntu1.10 [290 kB]
Get:86 http://localrepo/2204  distro-info-data 0.52ubuntu0.4 [4,986 B]
Get:87 http://localrepo/2204  fwupd 1.7.9-1~22.04.3 [2,562 kB]
Get:88 http://localrepo/2204  libfwupdplugin5 1.7.9-1~22.04.3 [191 kB]
Get:89 http://localrepo/2204  libfwupd2 1.7.9-1~22.04.3 [106 kB]
Get:90 http://localrepo/2204  fwupd-signed 1.51.1~22.04.1+1.4-0ubuntu0.1 [30.3 kB]
Get:91 http://localrepo/2204  libldap-common 2.5.14+dfsg-0ubuntu0.22.04.2 [15.8 kB]
Err:92 http://localrepo/2204  libnss3 2:3.68.2-0ubuntu1.2
  404  Not Found [IP: localrepo 80]
Err:93 http://localrepo/2204  libx11-data 2:1.7.5-1ubuntu0.2
  404  Not Found [IP: localrepo 80]
Err:94 http://localrepo/2204  libx11-6 2:1.7.5-1ubuntu0.2
  404  Not Found [IP: localrepo 80]
Get:95 http://localrepo/2204  linux-firmware 20220329.git681281e4-0ubuntu3.13 [249 MB]
Get:96 http://localrepo/2204  linux-modules-5.15.0-75-generic 5.15.0-75.82 [23.2 MB]                                                                                                                                                
Get:97 http://localrepo/2204  linux-image-5.15.0-75-generic 5.15.0-75.82 [11.4 MB]                                                                                                                                                  
Get:98 http://localrepo/2204  linux-modules-extra-5.15.0-75-generic 5.15.0-75.82 [64.2 MB]                                                                                                                                          
Get:99 http://localrepo/2204  linux-generic 5.15.0.75.73 [1,700 B]                                                                                                                                                                  
Get:100 http://localrepo/2204  linux-image-generic 5.15.0.75.73 [2,510 B]                                                                                                                                                           
Get:101 http://localrepo/2204  linux-headers-5.15.0-75 5.15.0-75.82 [12.3 MB]                                                                                                                                                       
Get:102 http://localrepo/2204  linux-headers-5.15.0-75-generic 5.15.0-75.82 [2,904 kB]                                                                                                                                              
Get:103 http://localrepo/2204  linux-headers-generic 5.15.0.75.73 [2,358 B]                                                                                                                                                         
Get:104 http://localrepo/2204  mdadm 4.2-0ubuntu2 [464 kB]                                                                                                                                                                          
Err:105 http://localrepo/2204  ubuntu-release-upgrader-core 1:22.04.16                                                                                                                                                              
  404  Not Found [IP: localrepo 80]
Err:106 http://localrepo/2204  python3-distupgrade 1:22.04.16                                                                                                                                                                       
  404  Not Found [IP: localrepo 80]
Get:107 http://localrepo/2204  software-properties-common 0.99.22.7 [14.1 kB]                                                                                                                                                       
Get:108 http://localrepo/2204  python3-software-properties 0.99.22.7 [28.8 kB]                                                                                                                                                      
Get:109 http://localrepo/2204  python3-tz 2022.1-1ubuntu0.22.04.1 [30.7 kB]                                                                                                                                                         
Get:110 http://localrepo/2204  snapd 2.58+22.04.1 [23.8 MB]                                                                                                                                                                         
Get:111 http://localrepo/2204  sudo 1.9.9-1ubuntu2.4 [821 kB]                                                                                                                                                                       
Get:112 http://localrepo/2204  thermald 2.4.9-1ubuntu0.2 [212 kB]                                                                                                                                                                   
Fetched 437 MB in 9s (49.6 MB/s)                                                                                                                                                                                                        
E: Failed to fetch http://localrepo/2204/./libcap2_1%253a2.44-1ubuntu0.22.04.1_amd64.deb  404  Not Found [IP: localrepo 80]
E: Failed to fetch http://localrepo/2204/./open-vm-tools_2%253a12.1.5-3%7eubuntu0.22.04.1_amd64.deb  404  Not Found [IP: localrepo 80]
E: Failed to fetch http://localrepo/2204/./git-man_1%253a2.34.1-1ubuntu1.9_all.deb  404  Not Found [IP: localrepo 80]
E: Failed to fetch http://localrepo/2204/./git_1%253a2.34.1-1ubuntu1.9_amd64.deb  404  Not Found [IP: localrepo 80]
E: Failed to fetch http://localrepo/2204/./openssh-sftp-server_1%253a8.9p1-3ubuntu0.1_amd64.deb  404  Not Found [IP: localrepo 80]
E: Failed to fetch http://localrepo/2204/./openssh-server_1%253a8.9p1-3ubuntu0.1_amd64.deb  404  Not Found [IP: localrepo 80]
E: Failed to fetch http://localrepo/2204/./openssh-client_1%253a8.9p1-3ubuntu0.1_amd64.deb  404  Not Found [IP: localrepo 80]
E: Failed to fetch http://localrepo/2204/./ubuntu-drivers-common_1%253a0.9.6.2%7e0.22.04.3_amd64.deb  404  Not Found [IP: localrepo 80]
E: Failed to fetch http://localrepo/2204/./libpam-cap_1%253a2.44-1ubuntu0.22.04.1_amd64.deb  404  Not Found [IP: localrepo 80]
E: Failed to fetch http://localrepo/2204/./libcap2-bin_1%253a2.44-1ubuntu0.22.04.1_amd64.deb  404  Not Found [IP: localrepo 80]
E: Failed to fetch http://localrepo/2204/./vim_2%253a8.2.3995-1ubuntu2.8_amd64.deb  404  Not Found [IP: localrepo 80]
E: Failed to fetch http://localrepo/2204/./vim-tiny_2%253a8.2.3995-1ubuntu2.8_amd64.deb  404  Not Found [IP: localrepo 80]
E: Failed to fetch http://localrepo/2204/./vim-runtime_2%253a8.2.3995-1ubuntu2.8_all.deb  404  Not Found [IP: localrepo 80]
E: Failed to fetch http://localrepo/2204/./xxd_2%253a8.2.3995-1ubuntu2.8_amd64.deb  404  Not Found [IP: localrepo 80]
E: Failed to fetch http://localrepo/2204/./vim-common_2%253a8.2.3995-1ubuntu2.8_all.deb  404  Not Found [IP: localrepo 80]
E: Failed to fetch http://localrepo/2204/./bind9-dnsutils_1%253a9.18.12-0ubuntu0.22.04.1_amd64.deb  404  Not Found [IP: localrepo 80]
E: Failed to fetch http://localrepo/2204/./bind9-host_1%253a9.18.12-0ubuntu0.22.04.1_amd64.deb  404  Not Found [IP: localrepo 80]
E: Failed to fetch http://localrepo/2204/./bind9-libs_1%253a9.18.12-0ubuntu0.22.04.1_amd64.deb  404  Not Found [IP: localrepo 80]
E: Failed to fetch http://localrepo/2204/./libnss3_2%253a3.68.2-0ubuntu1.2_amd64.deb  404  Not Found [IP: localrepo 80]
E: Failed to fetch http://localrepo/2204/./libx11-data_2%253a1.7.5-1ubuntu0.2_all.deb  404  Not Found [IP: localrepo 80]
E: Failed to fetch http://localrepo/2204/./libx11-6_2%253a1.7.5-1ubuntu0.2_amd64.deb  404  Not Found [IP: localrepo 80]
E: Failed to fetch http://localrepo/2204/./ubuntu-release-upgrader-core_1%253a22.04.16_all.deb  404  Not Found [IP: localrepo 80]
E: Failed to fetch http://localrepo/2204/./python3-distupgrade_1%253a22.04.16_all.deb  404  Not Found [IP: localrepo 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
muru avatar
us flag
What issues? What's the actual command you run, and the actual error you get?
dcdb avatar
th flag
Hey @muru the command is `apt install *insert list of packages to download* --reinstall --download-only`. This part executes fine, but it's when I run `apt upgrade` on the client servers to my local repo do i get errors saying that the file cannot be found, which i've traced back to an http transfer issue with the %encoded characters in the file path.
muru avatar
us flag
What was the exact error? How did you do this tracing back?
dcdb avatar
th flag
I apologize for not having the exact error in front of me, but the error implied that the package was unavailable. For example, it would say that `xauth_1%253a1.1-1build2_amd64.deb` did not exist on the server,, despite it being present, the permissions were set equal to every other package, and no other issues noticeable.
dcdb avatar
th flag
@muru I included the full output above!
muru avatar
us flag
Hmm, which web server are you using for `http://localrepo`? What if you put the files on disk somewhere and use a `file://` repo instead?
dcdb avatar
th flag
It's a local apache server. It's worked fine up to this point. As you can see, it downloads many of the packages, except for the ones that have the encoded characters. Unfamiliar with the file://repo. Would that just direct apt, locally, to a folder and require me to do this on each server?
muru avatar
us flag
Yes, it's to a local folder, but you can use NFS or SMB to get that folder over the network. What does the Apache log say for these file accesses?
dcdb avatar
th flag
Unsure, haven't thought to check that since I can see the issue is caused only by the encoded characters, and the output above has given me enough info to figure out that's the root cause. I'll check the apache logs next time I am at that site. Thank you.
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.