Podman fails pushing image to AWS ECR repository with: Error: writing blob: Post

I have issues trying to push an image I created with podman to an ECR container I have.

The issue doesn't seem to be related to permissions or authentication and I am struggling to find similar issues online. I am also seeing the tagged image so the issue seems to be related to the push only.

I am getting 3 of the below errors until the 'podman push ..' commands eventually fail.

[ec2-user@ip-10-0-151-137 ~]$ grep PRETTY /etc/os-release 
PRETTY_NAME="Red Hat Enterprise Linux 9.1 (Plow)"
[ec2-user@ip-10-0-151-137 ~]$ podman version 
Client:       Podman Engine
Version:      4.2.0
API Version:  4.2.0
Go Version:   go1.18.4
Built:        Wed Oct 26 18:30:14 2022
OS/Arch:      linux/amd64
[ec2-user@ip-10-0-151-137 ~]$ podman images | grep git-pod.*06
localhost/git-pod                                                  06          95d2df2c0ae0  2 days ago         310 MB
[ec2-user@ip-10-0-151-137 ~]$ 

[ec2-user@ip-10-0-151-137 ~]$ aws ecr get-login-password --region eu-central-1 | podman login --username AWS --password-stdin ACCOUNTID.dkr.ecr.eu-central-1.amazonaws.com
Login Succeeded!
[ec2-user@ip-10-0-151-137 ~]$ podman tag git-pod:06 ACCOUNTID.dkr.ecr.eu-central-1.amazonaws.com/karlito96/git-pod:06
[ec2-user@ip-10-0-151-137 ~]$ podman push ACCOUNTID.dkr.ecr.eu-central-1.amazonaws.com/karlito96/git-pod:06
Getting image source signatures
Copying blob f1f8e45a0453 [--------------------------------------] 8.0b / 18.5KiB
Copying blob 741a880671a3 [--------------------------------------] 8.0b / 31.5KiB
Copying blob ce9d08ae8f99 [--------------------------------------] 8.0b / 209.2MiB
Copying blob 6a7d463177c0 [--------------------------------------] 8.0b / 4.5KiB
Copying blob 5d8a1e83af01 [--------------------------------------] 8.0b / 5.5KiB
Copying blob 9b2fc3057618 [--------------------------------------] 8.0b / 86.2MiB
Copying blob 7aeca7e32e2f [--------------------------------------] 8.0b / 5.5KiB
Copying blob 5a56fd9df592 [--------------------------------------] 8.0b / 2.5KiB
Copying blob 82588404a590 [--------------------------------------] 8.0b / 6.0KiB
Copying blob 7b4bd06c8069 [--------------------------------------] 8.0b / 8.5KiB
WARN[0006] Failed, retrying in 1s ... (1/3). Error: writing blob: Post "https://ACCOUNTID.dkr.ecr.eu-central-1.amazonaws.com/v2/karlito96/git-pod/blobs/uploads/": EOF 
Getting image source signatures
Copying blob ce9d08ae8f99 [--------------------------------------] 8.0b / 209.2MiB
Copying blob 9b2fc3057618 [--------------------------------------] 8.0b / 86.2MiB
Copying blob 741a880671a3 [--------------------------------------] 8.0b / 31.5KiB
Copying blob 5d8a1e83af01 [--------------------------------------] 8.0b / 5.5KiB
Copying blob f1f8e45a0453 [--------------------------------------] 8.0b / 18.5KiB
Copying blob 6a7d463177c0 [--------------------------------------] 8.0b / 4.5KiB
Copying blob 7aeca7e32e2f [--------------------------------------] 8.0b / 5.5KiB
Copying blob 5a56fd9df592 [--------------------------------------] 8.0b / 2.5KiB
Copying blob 82588404a590 [--------------------------------------] 8.0b / 6.0KiB
Copying blob 7b4bd06c8069 [--------------------------------------] 8.0b / 8.5KiB
WARN[0012] Failed, retrying in 1s ... (2/3). Error: writing blob: Post "https://ACCOUNTID.dkr.ecr.eu-central-1.amazonaws.com/v2/karlito96/git-pod/blobs/uploads/": EOF 
Getting image source signatures
Copying blob 9b2fc3057618 [--------------------------------------] 8.0b / 86.2MiB
Copying blob ce9d08ae8f99 [--------------------------------------] 8.0b / 209.2MiB
Copying blob 741a880671a3 [--------------------------------------] 8.0b / 31.5KiB
Copying blob 5d8a1e83af01 [--------------------------------------] 8.0b / 5.5KiB
Copying blob f1f8e45a0453 [--------------------------------------] 8.0b / 18.5KiB
Copying blob 6a7d463177c0 [--------------------------------------] 8.0b / 4.5KiB
Copying blob 7aeca7e32e2f [--------------------------------------] 8.0b / 5.5KiB
Copying blob 5a56fd9df592 [--------------------------------------] 8.0b / 2.5KiB
Copying blob 82588404a590 [--------------------------------------] 8.0b / 6.0KiB
Copying blob 7b4bd06c8069 [--------------------------------------] 8.0b / 8.5KiB
WARN[0017] Failed, retrying in 1s ... (3/3). Error: writing blob: Post "https://ACCOUNTID.dkr.ecr.eu-central-1.amazonaws.com/v2/karlito96/git-pod/blobs/uploads/": EOF 
Getting image source signatures
Copying blob 741a880671a3 [--------------------------------------] 8.0b / 31.5KiB
Copying blob 6a7d463177c0 [--------------------------------------] 8.0b / 4.5KiB
Copying blob 9b2fc3057618 [--------------------------------------] 8.0b / 86.2MiB
Copying blob f1f8e45a0453 [--------------------------------------] 8.0b / 18.5KiB
Copying blob ce9d08ae8f99 [--------------------------------------] 8.0b / 209.2MiB
Copying blob 5d8a1e83af01 [--------------------------------------] 8.0b / 5.5KiB
Copying blob 7aeca7e32e2f [--------------------------------------] 8.0b / 5.5KiB
Copying blob 5a56fd9df592 [--------------------------------------] 8.0b / 2.5KiB
Copying blob 82588404a590 [--------------------------------------] 8.0b / 6.0KiB
Copying blob 7b4bd06c8069 [--------------------------------------] 8.0b / 8.5KiB
Error: writing blob: Post "https://ACCOUNTID.dkr.ecr.eu-central-1.amazonaws.com/v2/karlito96/git-pod/blobs/uploads/": EOF
[ec2-user@ip-10-0-151-137 ~]$ podman images | grep git-pod.*06
ACCOUNTID.dkr.ecr.eu-central-1.amazonaws.com/karlito96/git-pod  06          95d2df2c0ae0  2 days ago         310 MB
localhost/git-pod                                                  06          95d2df2c0ae0  2 days ago         310 MB
[ec2-user@ip-10-0-151-137 ~]$ 

Thanks

The push command was failing as the specified repository never existed.

ECR repository name: karlito96 Podman image:tag: git-pod:06

In both my tag and push commands the ECR path I chose didn't exist: ACCOUNTID.dkr.ecr.eu-central-1.amazonaws.com/karlito96/git-pod:06

It seems everything after the ..amazon.com/ and before the : is considered as the repository name. In the above example this would be: karlito96/git-pod which doesn't exist.

Resolution:

There are two ways one can choose from to work around this:

1. Tag your local image onto the repository with the repository name, with the earlier used ECR repo and image this would look like:

    $ podman tag git-pod:06 ACCOUNTID.dkr.ecr.eu-central-1.amazonaws.com/karlito96:latest
    $ podman push ACCOUNTID.dkr.ecr.eu-central-1.amazonaws.com/karlito96:latest
   

2. Create a repository with the name of your image, and then push to that directly:

    $ podman tag git-pod:06 ACCOUNTID.dkr.ecr.eu-central-1.amazonaws.com/git-pod:06
    $ podman push ACCOUNTID.dkr.ecr.eu-central-1.amazonaws.com/git-pod:06