Cerebrium offers file management through a 50GB persistent volume that’s available to all applications in a project. This storage mounts at /persistent-storage and helps store model weights and files efficiently across deployments.

Including Files in Deployments

The cerebrium.toml configuration file controls which files become part of the application:

[cerebrium]
include = [
    "src/*.py",         # Python files in src
    "config/*.json",    # JSON files in config
    "requirements.txt"  # Specific files
]

exclude = [
    "tests/*",          # Skip test files
    "*.log"            # Skip logs
]

Files included in deployments must be under 2GB each, with deployments working best for files under 1GB. Larger files should use persistent storage instead.

Managing Persistent Storage

The CLI provides three commands for working with persistent storage:

  1. Upload files with cerebrium cp:
# Upload to root directory
cerebrium cp src_file_name.txt

# Upload to specific location
cerebrium cp src_file_name.txt dest_file_name.txt

# Upload entire directory
cerebrium cp dir_name sub_folder/
  1. List files with cerebrium ls:
# View root contents
cerebrium ls

# View specific folder
cerebrium ls sub_folder/
  1. Remove files with cerebrium rm:
# Remove a file
cerebrium rm file_name.txt

# Remove a directory
cerebrium rm sub_folder/

Using Stored Files

Here’s how to work with files in persistent storage:

import os
import torch

# Load a model from persistent storage
file_path = "/persistent-storage/segment-anything/sam_vit_h_4b8939.pth"
model = torch.jit.load(file_path)

Should you require additional storage capacity, please reach out to us through support.