Using GPUs

GPUs provide specialized computing power that dramatically accelerates computational workloads. While originally designed for graphics rendering, modern GPUs excel at parallel processing tasks, making them essential for a wide range of applications.

Applications deployed on Cerebrium can access GPU computing power without managing complex infrastructure. The platform supports sophisticated AI models, large-scale data processing, and GPU-accelerated applications through simple configuration.

​

Specifying GPUs

The GPU configuration in the cerebrium.toml is handled through the [cerebrium.hardware] section, where you can specify both the type (using the compute parameter) and quantity of GPUs (gpu_count) for your app. We address additional deployment configurations and GPU scaling considerations in more detail in the sections below.

​

Available GPUs

The platform offers a range of GPUs to match various computational needs and budgets, from cost-effective development options to high-end enterprise hardware.

​

Multi-GPU Configuration

Multiple GPUs can enhance application performance through parallel processing and increased memory capacity.

​

Use Cases

Multiple GPUs become essential when:

• Models exceed single GPU memory capacity
• Workloads require parallel inference processing
• Applications need distributed training capabilities
• Production environments demand high availability

​

Configuration

Multiple GPUs are configured in the cerebrium.toml file:

[cerebrium.hardware]
compute = "AMPERE_A100_80GB"
gpu_count = 4        # Number of GPUs needed
cpu = 8
memory = 128.0

​

Selecting the Right GPU

Selecting appropriate hardware requires balancing performance requirements with resource efficiency. GPU selection involves calculating VRAM usage based on model parameters and input requirements. This is particularly important for:

• LLMs and transformer architectures: Account for attention processes and positional encoding.
• CNNs: Consider filter numbers and input sizes.
• Batch processing: Factor in concurrent processing requirements.

​

VRAM Requirement Calculation

Base VRAM requirements can be calculated using:

modelVRAM = numParams × numBytesPerDataType

Common data types:

• FP32 (32-bit floating point): 4 bytes
• FP16 (16-bit floating point): 2 bytes
• INT8 (8-bit quantization): 1 byte

Example calculations:

1. 7B parameter model with FP32:

modelVRAM = 7B × 4 bytes = 28GB

2. Same model with INT8 quantization:

modelVRAM = 7B × 1 byte = 7GB

​

Safety Buffer

A 1.5× VRAM buffer should be added to account for runtime memory requirements, attention mechanisms, intermediate computations, and input variations:

recommendedVRAM = modelVRAM × 1.5

Using the 7B parameter example:

• FP32: 28GB × 1.5 = 42GB (requires A100_40GB or higher)
• INT8: 7GB × 1.5 = 10.5GB (T4 or higher is sufficient)

The Pricing Calculator provides detailed cost comparisons for different GPU configurations.

For custom GPU configurations, contact Cerebrium support.