For any sort of password-based key derivation function or password hash, you need the salt to prevent guessing the same password across many different users or accounts. In the typical case, verifying the password is by comparing the output of the KDF; in this case, verifying the password is by verifying the MAC or AEAD on the encrypted data encrypted by that user.
Is it slightly more computationally difficult to do so? Yes. Is it impossible? No. You do definitely need to use a salt here, because typically people pick bad passwords and salt makes attacking them substantially more difficult. The only time you don't is if you assign people random cryptographically secure passwords with sufficient entropy (such as tokens) and they use a password manager or other secret store to store them.
However, if you're using the same user account to encrypt many files, you can make this a little more complex to be a lot faster. You can use Argon2 with a per-user salt to generate a 512-bit output. Then, you can use a per-file salt and the Argon2 output (in place of the secret) as the input to HKDF to generate the key and nonce for AES-GCM to encrypt the data. HKDF is very fast, so you can generate the key and nonce for many, many files very quickly.
You could also use something like NaCl's crypto_box
. Generate a public-private key pair. Use a per-user salt and password in Argon2i to generate a key and nonce to encrypt the private key with AES-GCM (or whatever secure algorithm). Then, you can encrypt your files with crypto_box
using the public key, and you only need to use Argon2 to decrypt the private key during decryption.