Async Functionality

Unfortunately Cerebrium doesn’t “properly” support async functionality however please see below how you can implement something similar using Cortex. Please let our team know you would like the ability to use async functionality and your use case so we can add it to our roadmap.

The main reason Cortex, doesn’t support async functionality is because our predict function is executed synchronously. This means that you can use async functionality throughout your code however, when it gets to the predict functionality, it needs to be executed synchronously.

For example you can implement the following below:

from asyncio import (
    new_event_loop,
    set_event_loop,
    create_task,
    gather,
    run,
)

def predict(item, run_id, logger):
    loop = new_event_loop()
    set_event_loop(loop)
    first_model = loop.create_task(predict_first_model())
    second_model = loop.create_task(predict_second_model())
    tasks = gather(first_model, second_model)
    results = loop.run_until_complete(tasks)

    return results

Essentially what we are doing above is creating an event loop which is responsible for executing coroutines and scheduling callbacks. We then run two separate async functions on the same loop since we would like both these tasks to finish. If this is not the case, you can create multiple different loops. We then use the ‘run_until_complete’ function to wait until both functions have returned. Lastly we return the results from the two predict functions.

The above code converts asynchronous code to run synchronously.