"access to kms is not allowed" error returned when trying to fetch secret from AWS Secrets Manager

An IAM user in our aws account is trying to fetch a particular secret from Secrets Manager via aws cli but they cannot do that although they should have required permissions:

aws secretsmanager get-secret-value --secret-id "config/my/secret"

This fails with error access to kms is not allowed. We use DefaultEncryptionKey for encrypting the secret and the key policy (managed by AWS) looks sensible to me:

    "Statement": [
    {
        "Sid": "Allow access through AWS Secrets Manager for all principals in the account that are authorized to use AWS Secrets Manager",
        "Effect": "Allow",
        "Principal": {
            "AWS": "*"
        },
        "Action": [
            "kms:Encrypt",
            "kms:Decrypt",
            "kms:ReEncrypt*",
            "kms:CreateGrant",
            "kms:DescribeKey"
        ],
        "Resource": "*",
        "Condition": {
            "StringEquals": {
                "kms:CallerAccount": "<REDACTED>",
                "kms:ViaService": "secretsmanager.eu-west-1.amazonaws.com"
            }
        }
    },
    {
        "Sid": "Allow access through AWS Secrets Manager for all principals in the account that are authorized to use AWS Secrets Manager",
        "Effect": "Allow",
        "Principal": {
            "AWS": "*"
        },
        "Action": "kms:GenerateDataKey*",
        "Resource": "*",
        "Condition": {
            "StringEquals": {
                "kms:CallerAccount": "<redacted>"
            },
            "StringLike": {
                "kms:ViaService": "secretsmanager.*.amazonaws.com"
            }
        }
    }

In Permissions for the KMS key they say you need

• kms:GenerateDataKey
• kms:Decrypt

Those are be provided by the key policy.

The user has attached IAM policy to give them permissions to Secrets Manager via API as described here: https://curiousprogrammer.net/posts/2022-02-16-aws-secrets-manager-least-privilege#_2_create_the_iam_policy

I looked in Cloudtrail and didn't found anything useful, just an "unknown error" message:

"eventSource": "secretsmanager.amazonaws.com",
"eventName": "GetSecretValue",
"awsRegion": "eu-west-1",
"userAgent": "aws-cli/2.4.18 Python/3.9.10 Darwin/21.2.0 source/arm64 prompt/off command/secretsmanager.get-secret-value",
"errorCode": "InternalFailure",
"errorMessage": "An unknown error occurred",
"requestParameters": {
  "secretId": "config/my/secret"
},

How can I figure out what's going on and what additional permissions are needed?

This was caused by our Enforce MFA policy This part in particular:

{
        "Sid": "DenyAllExceptListedIfNoMFA",
        "Effect": "Deny",
        "NotAction": [
            "iam:CreateVirtualMFADevice",
            "iam:EnableMFADevice",
            "iam:GetUser",
            "iam:ListMFADevices",
            "iam:ListVirtualMFADevices",
            "iam:ResyncMFADevice",
            "sts:GetSessionToken"
        ],
        "Resource": "*",
        "Condition": {
            "BoolIfExists": {"aws:MultiFactorAuthPresent": "false"}
        }
    }

Since aws cli doesn't normally use MFA, the policy was applied and the "Deny" rule overrode the otherwise working permissions stated in the Key policy.

The solution was to use temporary session tokens: https://aws.amazon.com/premiumsupport/knowledge-center/authenticate-mfa-cli/

$ aws sts get-session-token --serial-number arn-of-the-mfa-device --token-code code-from-token

I talked more about this on my blog: https://curiousprogrammer.net/posts/2022-02-16-aws-secrets-manager-least-privilege#_update_2022_02_22_the_perils_of_mfa