Docker execution slower on EC2 host

Sam Smart

11/28/22, 12:30 PM

We have recently created a docker image to run the Terraspace tool for executing our CI/CD in GitLab pipelines. The container uses ruby:3.0.2-alpine for it's upstream, for reference.

The problem we're facing is that it is extremely slow when executed on EC2 (m5.large), here are some timings that include running the tool inside the docker image and natively on the host. The docker timings are performed inside the image after it has already been downloaded.

EC2 Docker	EC2	Local Docker	Local
real 5m33.403s	real 0m44.799s	real 1m40.842s	real 0m39.626s
user 0m11.150s	user 0m26.531s	user 0m24.736s	user 0m10.913s
sys 0m1.437s	sys 0m3.276s	sys 0m13.846s	sys 0m4.580

The slower execution inside docker is also true for the standard terraspace image boltops/terraspace.

It doesn't appear to be a resource usage issue as there are plenty of resources left on the host during execution

CONTAINER ID   NAME            CPU %     MEM USAGE / LIMIT    MEM %     NET I/O          BLOCK I/O     PIDS
9b05765250b8   quirky_spence   0.54%     1.21GiB / 7.583GiB   15.95%    300MB / 3.43MB   0B / 1.08MB   3

Here is the docker machine info:

lient:
 Context:    default
 Debug Mode: false

Server:
 Containers: 4
  Running: 0
  Paused: 0
  Stopped: 4
 Images: 3
 Server Version: 20.10.4
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc version: %runc_commit
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.14.238-182.422.amzn2.x86_64
 Operating System: Amazon Linux 2
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 7.583GiB
 Name: ip-10-0-4-227
 ID: FUYW:PCXQ:5ZFW:4SMP:YIG4:RNBH:HCMH:6R53:NHS2:HTJO:VAKM:5QFB
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://docker-proxy/
 Live Restore Enabled: false

Any help would be appreciated on this.

126

0 + 4

linux

amazon-ec2

docker

Michael Hampton

11/28/22, 12:31 PM

Check your DNS.

Sam Smart

11/28/22, 12:40 PM

Just checked, DNS is fine. No issues resolving from within the container.

Michael Hampton

11/28/22, 12:43 PM

Interesting. Well, the numbers say that it's spending a lot of time in the container _doing nothing_, i.e. waiting for something, where this time is not really being consumed when run outside the container. Usually this is a DNS problem though it could just be a general connectivity issue, or something else entirely that the app is spending its time waiting for.

rvs

11/29/22, 8:48 AM

I/O would be my next guess.

Elon Musk

I sit in a Tesla and translated this thread with Ai:

EN: Docker execution slower on EC2 host

TH: การดำเนินการนักเทียบท่าช้าลงบนโฮสต์ EC2

RO: Execuția Docker mai lentă pe gazda EC2

RU: Выполнение Docker медленнее на хосте EC2

VI: Docker thực thi chậm hơn trên máy chủ EC2

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.