How to protect a non-extractable Secret Key in indexedDB?

I have encrypted data in AES-GCM with the crypto API. The initialisation vector is then added to the data, forming a unique encrypted string stored in local-storage. Finally, the secret key is stored in IndexedDB.

Since the secret key is non-extractable I though it was secure enough for most use-cases. To my understanding, an attacker would have to rob both the local storage and the indexed db, find the initialisation vector inside the data, convert it to a buffer array and then directly perform the decryption in the browser before sending the data back to his server. Indeed, it seems the non-extractable nature of the secret key means that he would not be able to send the raw secret key to his server and thus not being able to complete the decryption.

But I've been told I was very wrong, and that my strategy was actually barely more secure than letting all the data directly readable in local storage.

So, how could I improve this workflow? Is it really that insecure? Would it be possible to encrypt and decrypt the secret key thanks a unique password provided by the user? The password would be in a .env file and thus never exposed to an attacker. How would you do it?

Here is how the secret key is generated so far:

crypto.subtle.generateKey(
      { name: "AES-GCM", length: 256 },
      false,
      ["encrypt", "decrypt"]
    );

Thanks for your help!