Hosting of inconsistent workloads in Azure

In our company we have a lot of algorithms which need to process large datasets. The time to run these algorithms differs from a few minutes to hours. They also need to be run ad-hoc from multiple times in a week to once a month. We would like to trigger these algorithms with an event such as a file upload in Azure blob storage or and API call.

To solve this I started looking into queued processing of tasks in Azure. At first I thought that Azure functions might be a good solution, because it’s pay as you use. The problem with this is that they are not meant for long running operations. Thus I started looking elsewhere. I found two pretty good alternatives namely Azure WebJobs or Jobs in Azure Kubernetes Services. The problem with both of them is that they still need an active server even though nothing is running on them. This could be quite expensive for tasks that only need to be run once a month.

My question is thus: does there exist a solution in Azure for hosting long running jobs without needing a dedicated server running 24-7?

Separate the job submission component from the workers that process them. Perhaps the submit is implemented as with Azure Functions and Queues, uploading the inputs to storage somewhere and making a pointer to that. Part of the function is to scale the workers up from zero, perhaps via VM Scale Sets.

May be some development work, if not using an existing job scheduling system.

Or, spend the cost of running Kubernetes all the time. Yes, AKS requires running at least one node. May be worthwhile, especially if it also runs other container applications.

Perhaps another solution to what John have suggested would be to use Azure Spot VMs: https://azure.microsoft.com/en-us/services/virtual-machines/spot/