Dynamic Job Scaling

The loader function model on the rigor platform is loosely based on the age old map-reduce mentality. A loader function gives you the opportunity to parallelise tasks while automatically scaling them to the number of inputs you have.

Load and Run Model

In this example, we check for any reports that could be available and then distribute the computation of each into a separate job. By not specifying a jobs key in the loader return object, the platform will automatically scale the number of jobs to the number of inputs (up to a maximum of 100).

import requests
import os
 
def loader():
 
    response = requests.get('https://mydomain.com/api/v1/reports')
    data = response.json()
 
    return {
        'runner_inputs': data['reports_ready'],
    }
 
def runner(args):
    print('Reports processor started')
 
    report_to_process = args[int(os.environ['TASK_INDEX'])]
 
    print('I am processing report: ' + report_to_process)

{
"name": "Parallel Reports Processor",
"runFile" "prepared_reports_fetch_parallel.py",
"tasks": 1,
"timeout": 200,
"cpus": 1,
"memory": 4,
"retries": 3
}