Running Locally and Development

Developing locally is not difficult at all! In fact, running the pipelines locally is nearly as simple as just executing them.

Simple Runner (no loader function)

If you have a simple runner, you don't have to do anything special, just run it as a script and it will execute in the same fashion as on the rigor platform!

> python ./my_runner.py

Loader Pipeline

It is slightly more complicated for running the loader and parallelised runner tasks. The closest imitation to how rigor handles the spinning up of parallelised tasks is by making use of the following script. This script will execute the loader function and then spin up the number of runners specified in the runner_inputs list.

Use it as follows. Keeping in mind the rigor_executor script is in the same directory as the pipeline script.

> python ./rigor_executor.py ./test_loader_pipeline

The dev script (don't use this anywhere but locally!):

 
import importlib
import subprocess
import sys
import os
 
runner_name = sys.argv[1]
 
job = importlib.import_module(runner_name)
 
result = job.loader()
 
number_of_runners = len(result.get('runner_inputs', []))
 
if result.get('jobs', None):
    number_of_runners = len(result['jobs'])
 
running_processes = []
for task_index in range(number_of_runners):
 
    env = os.environ.copy()
    env['TASK_INDEX'] = f'{task_index}'
    task_process = subprocess.Popen(
        [
            'python',
            '-c',
            f"import importlib; job = importlib.import_module('{runner_name}'); job.runner({result.get('runner_inputs')})"
        ], env=env)
    running_processes.append(task_process)