Just lately our server started experiencing increased CPU usage by the php(Symfony) and mysql processes. For quite some time we been trying to find the cause and we found out that we have big amount of TCP retransmissions in our docker network:
Firstly we read about some people having issues with php-alpine version docker images, we migrated to debian but the problem is still occuring. Nextly, we tried lowering/changing MTU, still no effect.
Right now we found another interesting thing, when we are capturing the traffic of communication between two docker containers there is no retransmissions and the packet length is equal to 1520. But when we do analyze whole traffic on the machine the length is way higher and the retransmissions occurs.
I'm attaching our config files, please let me know if something more is required.
Our docker-compose file looks as follows:
version: '3'
volumes:
database: {}
logs: {}
services:
mysql:
container_name: foxy_mysql
image: mysql:5.7
ports:
- 3306:3306
volumes:
- database:/var/lib/mysql
environment:
MYSQL_ROOT_PASSWORD: ${DATABASE_ROOT_PASSWORD}
MYSQL_DATABASE: ${DATABASE_NAME}
MYSQL_USER: ${DATABASE_USERNAME}
MYSQL_PASSWORD: ${DATABASE_PASSWORD}
restart: on-failure
networks:
- foxy
assets:
container_name: foxy_assets
build:
context: .
dockerfile: docker/dev/assets/Dockerfile
args:
SYMFONY_ENV: ${SYMFONY_ENV}
volumes:
- .:/var/www/symfony
depends_on:
- php
php:
container_name: foxy_php-fpm
image: foxy/php-fpm:latest
build:
context: .
dockerfile: docker/dev/php/Dockerfile
ports:
- ${PHP_PORT}:9000
extra_hosts:
- ${APP_DOMAIN}:${NETWORK_GATEWAY}
volumes:
- .:/var/www/symfony
- ./logs:/var/www/symfony/var/logs:cached
restart: on-failure
networks:
- foxy
nginx:
container_name: foxy_nginx
image: foxy/nginx:latest
build:
context: .
dockerfile: docker/dev/nginx/Dockerfile
args:
PHP_PORT: ${PHP_PORT}
APP_NAME: foxy
environment:
APP_NAME: foxy
ports:
- ${NGINX_PORT}:80
depends_on:
- assets
volumes:
- .:/var/www/symfony
- ./logs:/var/log/nginx:cached
restart: on-failure
networks:
- foxy
networks:
foxy:
ipam:
config:
- subnet: ${NETWORK_SUBNET}
gateway: ${NETWORK_GATEWAY}
driver_opts:
com.docker.network.driver.mtu: 1520
PHP Dockerfile:
FROM php:7.4-fpm
RUN apt update
#RUN apt upgrade
RUN curl --insecure https://getcomposer.org/download/1.10.1/composer.phar -o /usr/bin/composer && chmod +x /usr/bin/composer
RUN deluser www-data && adduser -uid 1000 www-data
ARG APCU_VERSION=5.1.22
ARG APCU_BC_VERSION=1.0.5
# Install build dependencies
RUN apt-get install -y --no-install-recommends \
$PHPIZE_DEPS \
...
libpcre3-dev \
# Install additional stuff needed for modules
&& apt install -y \
libzip-dev \
...
libssh2-1-dev \
libc-client-dev libkrb5-dev \
git \
...
fonts-liberation \
# Instal PHP extensions \
&& rm -r /var/lib/apt/lists/* \
&& docker-php-ext-install -j"$(getconf _NPROCESSORS_ONLN)" \
soap \
zip
# Install PECL extensions
RUN pecl install apcu-$APCU_VERSION \
&& docker-php-ext-enable apcu --ini-name 20-apcu.ini \
&& pecl install apcu_bc-$APCU_BC_VERSION \
&& docker-php-ext-enable apc --ini-name 21-apc.ini \
&& docker-php-ext-configure imap --with-kerberos --with-imap-ssl \
&& docker-php-ext-install imap
COPY docker/dev/php/symfony.ini /usr/local/etc/php/conf.d/
COPY docker/dev/php/symfony.ini /etc/php7/cli/conf.d/
COPY docker/dev/php/symfony.pool.conf /etc/php7/php-fpm.d/
WORKDIR /var/www/symfony
COPY docker/dev/php/entrypoint.sh /usr/bin/entrypoint.sh
RUN ln -s /etc/init.d/php-fpm7 /usr/bin/php-fpm7 \
&& chmod +x /usr/bin/entrypoint.sh
Clean up
RUN apk del .build-dependencies \
&& docker-php-source delete \
&& rm -rf /tmp/* /var/cache/apk/*
CMD ["entrypoint.sh"]
entrypoint.sh:
#!/bin/sh
deluser www-data
#addgroup -g 1000 www-data
adduser -uid 1000 www-data
# -G www-data -g 'Linux User named' -s /bin/sh -D www-data
dir='/var/www/symfony/web' && ls -a $dir | grep -v 'uploads' | sed 1d | sed 1d | while read r; do chown -R www-data:www-data $dir/$r; done
su www-data <<USER
APP_ENV=dev composer install --no-dev --optimize-autoloader --apcu-autoloader --no-interaction --no-progress
php bin/console fos:js-routing:dump
php bin/console bazinga:js-translation:dump
php bin/console doctrine:migrations:migrate --no-interaction
php bin/console sylius:rbac:initialize
rm -rf var/cache/*
rm -rf var/logs/*
rm -rf var/sessions/*
chown -R www-data:www-data var/*
php bin/console cache:warmup --env=dev
USER
php-fpm -F
Nginx Dockerfile:
FROM nginx:stable
#FOR SSL GENERATE
RUN apt install openssl;
ARG PHP_PORT
COPY docker/dev/nginx/nginx.conf /etc/nginx/
COPY docker/dev/nginx/custom-errors.conf /etc/nginx/
COPY docker/dev/nginx/symfony.conf /etc/nginx/conf.d/
COPY docker/dev/nginx/error_pages/* /usr/share/nginx/html/
RUN echo "upstream php-upstream { server php:${PHP_PORT}; }" > /etc/nginx/conf.d/upstream.conf
# ensure www-data user exists
RUN set -x ; \
# addgroup -g 1000 -S www-data ; \
deluser www-data && \
adduser -uid 1000 www-data && exit 0 ; exit 1
WORKDIR /var/www/symfony
COPY web /var/www/symfony/web
COPY .env /var/www/symfony
COPY docker/dev/nginx/error_pages/* /usr/share/nginx/html/
COPY docker/dev/nginx/entrypoint.sh /usr/bin/entrypoint.sh
RUN chmod -R 755 /usr/share/nginx/html
RUN chown -R nginx:nginx /usr/share/nginx/html
RUN chmod +x /usr/bin/entrypoint.sh
CMD ["entrypoint.sh"]
EXPOSE 80
EXPOSE 443
entrypoint.sh for nginx:
#!/bin/sh
. /var/www/symfony/.env
mkdir /etc/nginx/certs
chown -R www-data:www-data /etc/nginx/certs
openssl req -x509 -nodes -days 365 -subj "/C=CA/ST=QC/O=Company, Inc./CN=$APP_DOMAIN" -addext "subjectAltName=DNS:$APP_DOMAIN" -newkey rsa:2048 -keyout /etc/nginx/certs/nginx-selfsigned.key -out /etc/nginx/certs/nginx-selfsigned.crt
sed -i -e "s~#APP_DOMAIN~$APP_DOMAIN~g" /etc/nginx/conf.d/symfony.conf
nginx
Thank you :)
