Score:2

Creating a customized Ubuntu Live USB with read-only filesystem squashfs greater than 4GB

br flag

TLDR

How do I create a bootable Ubuntu Live USB that has a compressed filesystem.squashfs of size greater than 4GB, specifically 4,294,967,295 bytes, the limit of a file for drives formatted in FAT32.

More Detail

I'm trying to create a customized bootable Ubuntu Live USB that does not persist data, but am running into limitations:

What I've Tried

  • Cubic which is fantastic, but once I'm done customizing the image the filesystem.squashfs can be > 4GB in size, even when using XZ compression
  • Using this script to build the filesystem.squashfs from a currently running installation (preferred) using mksquashfs, but I run into the same filesize limit for FAT32 even when using XZ compression

Desired Outcome

  • I want a Live Ubuntu USB that does not persist between sessions
  • I want to be done with this filesize limit without having to worry about trimming packages or removing installed packaged (I've already ran things like apt clean/autoclean etc)
  • I want to customize and add as many packages as I want, either through some application like Cubic or from some live-running Ubuntu installation without worrying about the squashed FS filesize

To Summarize

  • Is it possible to split the filesystem.squashfs into multiple files that casper's menuentry linux reference to the squashed FS will understand?
  • Are there alternatives to read-only filesystems other than mksquashfs that are compatible with grub/casper?
  • Is this possibly a chance to write my own package to solve this problem, or would this be too much of an undertaking?
Score:2
br flag

I was able to figure this out. UEFI cannot read from anything except FAT, but casper can. Here's instructions for how to create a Bootable Live USB with a squash FS > 4GB. This method does NOT destroy the whole USB unlike other Bootable USB creation methods.

Environment

  • OS: Ubuntu 22.04
  • ISO Creator: Cubic
  • Source Image: Latest 22.04 Desktop downloaded from ubuntu.com

Bootable USB Partitioning

  • Partition 1

    • Format: FAT32
    • Size: ~5GB
    • Flags: boot
  • Partition 2

    • Format: ext4
    • Size: 15GB+
  • The rest of the USB can be partitioned as you like without removing any data during installation.

  • Be sure to take note of the device mounts for your partitions, for example /dev/sda1 (FAT32), /dev/sda2 (ext4).

Instructions

  1. Create your customized ISO using Cubic.

  2. Follow instructions until you get to the Options page.

This is where we tell casper to load the squashed FS from a different location than the FAT32 boot partition.

First, we want to reference the UUID of the ext4 partition instead of it's device ID (/dev/sda2 in my case) since the device ID can change.

You can find the UUID of the ext4 partition you created using these instructions: https://linuxhint.com/uuid_storage_devices_linux/. Since my ext4 partition is /dev/sda2, I can do this to get my ext4 UUID:

$ ll /dev/disk/by-uuid | grep sda2
lrwxrwxrwx 1 root root  10 Oct 10 20:38 727cac18-044b-4504-87f1-a5aefa774bdb -> ../../sda2

and this shows that my ext4 partition's UUID is 727cac18-044b-4504-87f1-a5aefa774bdb.

Modify the grub config in the Boot tab. You'll need to modify the first menuentry like so:

menuentry "Ubuntu RAM" {
    set gfxpayload=keep
    linux   /casper/vmlinuz boot=casper live-media=/dev/disk/by-uuid/727cac18-044b-4504-87f1-a5aefa774bdb toram file=/cdrom/preseed/ubuntu.seed quiet splash noprompt --- 
    initrd  /casper/initrd.gz
}

Be sure to change /dev/disk/by-uuid/<UUID> to the correct ext4 partition UUID you found during this step!

Casper boot option documentation can be found here.

We've added the following boot options

  • toram (note: remove this boot option if you do not have the RAM to hold the entire squashfs filesystem)
  • noprompt
  • live-media=DEVICE

We've removed the following boot options

  • maybe-ubiquity
  1. When prompted for compression method on the Compression Page, feel free to choose lz4 as this is fastest and filesystem.squashfs file size no longer matters.

  2. Generate your custom ISO.

  3. Extract your custom ISO to somewhere like Documents.

  4. Copy all files from the extracted ISO except for casper/filesystem* to the FAT32 partition. casper/vmlinuz and casper/initrd.gz must be on the FAT32 boot partition.

You might see some warning of inability to copy symlinks because FAT32 doesn't support them; this is OK and you can skip these files while maintaining usability of the bootable USB.

  1. Create a directory named casper in your ext4 partition and copy all extracted casper/filesystem* files to this directory.

Once finished, the casper directory on your FAT32 partition should look like this:

$ ls -1 casper/
initrd.lz  
vmlinuz

And the casper directory on your ext4 partition should look like this

$ ls -1 casper/
filesystem.manifest
filesystem.manifest-minimal-remove
filesystem.manifest-remove
filesystem.size
filesystem.squashfs

You've now created a UEFI bootable USB that uses a separate storage partition for its read-only filesystem via casper's live-media boot option.

Alexei Martianov avatar
cn flag
Glad you figured that out. I simply splitted file system when rebuilding iso file. I did it after reading casper scripts and seeing IIRC it works (mounts via overlayfs as root) with all files in casper folder with squashfs extension.
Alexei Martianov avatar
cn flag
Initially I was confused you talk about FAT as afaik ISO file is some CD/DVD format ISO9660. After reading carefully I see you extract ISO to FAT. Well at least afaik you are incorrect stating UEFI only boot fat, no it can boot ISO9660 too! However with ISO9660 I seem to hit that limit too around 4.2Gb which you mentioned. BTW I've only recently done that (squashfs split) and now testing stability of such system.
Score:2
cn flag

AFAIK as far as I've seen casper scripts from initramfs file they make new root via overlayfs from all files ending with .squashfs located in casper folder on ISO.

So I simply splitted file system when rebuilding ISO file and it works! I've only recently done that (squashfs split) and now testing stability of such system.

The way OP solved the issue is interesting too, split's benefit is that with ISO file one can use tools such as Ventoy to have multiple systems on one USB stick.

P.S. Disclaimer: try and see for your ISO file, if does not work out maybe on ISO you try to customize casper scripts are different.

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.