Single Job Data Fetch Pipeline

This is probably the most common and simple pipeline you'll want to build. Thankfully, its easy with the rigor platform!

hourly_orders_data_fetch.py

import requests
import pandas as pd
 
def runner():
 
    last_updated = requests.get('https://api.mybusiness.com/analytics/orders/last_updated/').json()
 
    response = requests.get('https://api.mybusiness.com/orders?last_updated=' + last_updated)
 
    if not response.status_code == 200:
        raise Exception('Failed to fetch orders data')
 
    orders_data = response.json()
 
    # You probably want to do a whole lot more here, but you get the idea
    orders_df = pd.DataFrame(orders_data)
 
    try:
        orders_df.to_csv(f's3://raw/ecom-orders/raw-orders.csv', index=False)
        print('Successfully fetched orders data')
    except:
        raise Exception('Failed to save orders data')

Recipes Dynamic Job Scaling