We have a deployment system that borrows from the immutable servers pattern. That is, we bake AMIs with our application on in a way than when they are booted, the application runs and can be added to a ALB pool.
(The reason for this is somewhat complex, mainly because we have a lot of infrequently changed read-heavy data in the application, and we need to add capacity quickly. Let's assume this architecture is fixed, even if there are other ways to do this!)
We don't want our running instances to be 100% immutable, in a couple of ways:
Change based on a property of the instance
We use gunicorn to serve our WSGI application. Gunicorn suggests 2-4 x $(NUM_CORES), but this changes depending on the instance we spin up.
Change based on the AWS account / SSM values / some external factor
An AMI might find itself spun up in a staging or prod account. It's nice to have staging AMIs being the same as the prod ones, but we don't want staging AMIs to connect to prod databases / external services.
Rather than focusing on the details of those two examples (I know for example I can set WEB_CONCURRENCY for gunicorn), I am trying to work out what the "best" way of performing these sort of runtime configurations of AMIs.
The options as I see them:
- Use AWS CodeDeploy run at start up, configure CodeDeploy with the config changes I'm after
- Use the instance user data to set up the environment vars, have a boot-time script that changes things
- Use some other automation like ansible
Whatever system I use, I would like it to work with auto scaling, meaning I can add instances in to a ALB pool and have them configure themselves correctly.
I've hunted for suggestions on this, but the AWS examples in particular don't seem to join the dots for me. I'm sure there's a commonly used solution for this, maybe even an AWS provided one. What am I missing?
