Join Log in
Score:1
Rohini avatar Server

Scaling based on number of processes running on ec2 instance

gr flag
Rohini

I have a use case where multiple python scripts will run in parallel on ec2. They read an SQS queue and perform operations such as download from S3, parse files, split, process, and update database. The scripts run every 5 min. If SQS is empty script does nothing and exits. If SQS has message, it gets processed by the script which may run for over an hour. So the run durations vary.

In auto scaling I see there is option to scale based on average cpu utilization but the randomness with which cpu is utilized makes it not a good indicator when to scale(e.g. the python script will take more cpu while doing disk io operations. But other times it takes memory but no cpu)

Is there a way to scale based on number of python processes running on the ec2 instance? So that when number of parallel invocations reach 10, the instances scale up?

Further, is there a way to specify that scale down should NOT happen for an ec2 instance if it has at least one python process running inside it(even if that one process is taking <5% cpu), because the scale down would terminate an ongoing process and it would be better if the scale down can happen after the python process completes

There is no load balancer in the picture.

33
0 + 2
Rohini avatar
gr flag
Rohini
I looked at the idea of triggering lambda with SQS trigger for every message that SQS receives. Which would theoretically mean as the number of messages increase, more function invocations happen so it scales. But https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html specifies timeout for any lambda as 15min. This won't fit my use case as the python processing may go well over an hour
0
Reply
Oscar De León avatar
la flag
Oscar De León
What about controlling the Autoscaling from inside the instance? It seems to me you have some kind of control mechanism for the script so maybe use that to scale out when needed and when there's no longer need for an instance to run you can scale back in. This past July they added a feature to select which instance to terminate when scaling in. Maybe it helps https://aws.amazon.com/about-aws/whats-new/2021/07/amazon-ec2-auto-scaling-now-lets-you-control-which-instances-to-terminate-on-scale-in/
0
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.