1. Cerebrium
  2. Advanced Functionality

Deploying model ensembles

With Cerebrium you are also able to deploy a sequence of models simply by specifying a list of tuples of model types and model files. Currently this functionality is supported across all model types, with the caveat that we assume the models are evaluated in the order they are supplied. For example, if you have a PyTorch model that takes in a 2D image and outputs a 1D vector, and you have a XGB model that takes in a 1D vector and outputs a single class prediction, you can deploy them as a sequence of models by supplying the following to the deploy function:

model_flow = [(model_type.TORCH, 'torch.pt'), (model_type.XGBOOST_CLASSIFIER, 'xgb.json')]
endpoint = deploy(model_flow, 'my-flow', "<API_KEY>")

We are currently working on a more robust way to specify the input and output shapes of your models or supply ways to specify pre-processing functions, so that you can deploy more complex flows. If you have any suggestions or feedback, please reach out to us.

Post-Processing Functions

In many cases, you may want to post-process the data in your single or multi-model flow before returning it to the next flow stage or to the user. To do this, you can supply a post-processing function to your flow. This function will be applied to the output of your model before it is returned to the next flow stage. The function must take in a single argument, which is the output of your model, and return a single value, which is the post-processed output of your model.

For example, if you have a model that outputs a 1D vector of probabilities, you may want to return the argmax. You can do this by supplying the following function to the deploy function with the last flow stage:

def post_process(result):
    import numpy as np
    return np.argmax(result, axis=1)
model_flow = [(model_type.TORCH, 'torch.pt'), (model_type.XGBOOST_CLASSIFIER, 'xgb.json', post_process)]
endpoint = deploy(model_flow, 'my-flow', "<API_KEY>")

You may use functions inside your post-processor from the following libraries:

  • numpy
  • scikit-learn
  • torch We are expanding this list of libraries, so if you have a specific library you would like to use, please let us know!

Note that any imports you need to do in your post-processing function must be done inside the function. This is because the function is serialized and sent to the Cerebrium servers, and the imports will not be available on the server.