I have an autoscaling group which is scaled up/down based on proprietary logic.
We use the boto3 client to perform such scaling activities when required.
By default our EC2 instances have a maximum execution lifecycle of 60 seconds. When spot instances are requested to be terminated it is very important that these are immediately detached from their load balancer.
When AWS reclaims their Spot instances, this is handled through ec2-spot-termination-handler lambdas to detach them from their respective load balancer: this helps us stop processing new tasks in those soon-to-be-dead EC2 instances, and let them complete their lifecycle.
Reading the boto3 documentation gives insights on how we can stop and terminate a spot instances, but not on how we can tell AWS to gracefully reclaim them:
ids = ['instance-id-1', 'instance-id-2', ...]
# Boto 2.x
ec2_connection.stop_instances(instance_ids=ids)
ec2_connection.terminate_instances(instance_ids=ids)
# Boto3
ec2.instances.filter(InstanceIds=ids).stop()
ec2.instances.filter(InstanceIds=ids).terminate()
How can we achieve the desired functionality which would fire a termination notification and further telling AWS to reclaim its spot instances within two minutes, programmatically?