If you are worried about someone gaining physical access to your device, you are 100% correct that someone with physical access can do whatever they want with the device, including accessing all your data and manipulating files without your knowledge.
If you do not want someone in this position to be able to access your data or manipulate your installation without your knowledge, the only thing you can do is use full disk encryption.
Full disk encryption is an option when you install Ubuntu for the first time. It will encrypt your entire root file system and will require authentication on boot. Your security in this case is only as secure as your authentication method, so your password should be strong, unique, and secret.
Even with full disk encryption, someone with access to your device can still perform tasks like wiping the hard drive or messing with the bootloader which needs to be unencrypted.
Even when using full disk encryption, if you had previously used the device without encryption, an attacker could also access information from the previous installation since files are usually not deleted when the space is marked as available. The protection against this would be to secure wipe the hard drive before installing Ubuntu with full disk encryption. Many SSDs can secure erase quickly and easily but mechanical hard drives must be completely overwritten, which can take a very long time.