I don't understand that
If your system is infected, there is no guarantee that you read the real file contents, as it is stored on the disk. It can be that the file is encrypted by ransomware. When you request an application to read it, it calls operating system. If ransomware infected the system, it will read the encrypted contents, decrypt it, and provide to the OS and thus to your application. As long as you use an infected system you cannot know what is the real contents on the disk.
The only reliable way to detect encryption is to read files using other system. Boot from USB stick, create hashes of the files, repeat it time to time, e.g. daily or weekly. Of course, it differs from your desire to detect changes immediately.
wouldn't it be impossible to hash, say, a file that the user has
securely encrypted?
You can hash any file. Only you know if you have encrypted the file or not. For the operating and for ransomware there is no difference: Any file is just a set of bytes. If you encrypt the file, compute hash, write file to the disk, then read it back, you will get exactly what you have written (your encrypted file). But you will not know if before saving to the disk it was encrypted by ransomware and after reading was decrypted.
that hash values, for a file, may vary according to the timing of
encryption with a public key
- Not the hash of the plain file, but the encryption result may vary. As a consequence, for different files you will get different hashes.
- Encryption results of the same file may vary, but not because of timing. For instance, you can launch AES GCM encryption of the same file with the same password on 100 parallel threads on the same computer at the same time, all of them will produce different results. But when decrypted, they all will produce the same original file.
