The artificial intelligence (AI) revolution is here, and for many companies, that means leveraging the cloud to access powerful graphics processing unit (GPU) resources.

Renting GPU instances can be a game-changer, but the reality is often more complex than the marketing promises. The cost of these rentals is significant, and ensuring you’re actually getting the performance you’re paying for requires careful scrutiny and asking the right questions.

Here are some ways to evaluate your AI cloud provider and maximize your investment.

The Performance Pitfalls: Where a Cloud Provider Can Fall Short

It’s easy to get caught up in the specs of the GPUs themselves — NVIDIA for example has several performance leaders on its portfolio — but raw power is only part of the equation.  The surrounding infrastructure is just as critical. Here are key areas where cloud providers can underdeliver:

1. Network Bottlenecks: The Silent Performance Killer

AI workloads, especially training large models, demand ultra-fast networking. A seemingly impressive GPU can sit idle waiting for data if the network is oversubscribed.

The Problem: Oversubscription means the provider has allocated more bandwidth than physically available. During peak times, your job will be competing with others, leading to dramatically reduced performance.

The Question: “What is your network oversubscription ratio at peak times? What guarantees can you offer regarding consistent bandwidth availability for my instances?”

Don’t settle for a generic answer. Demand specific metrics and performance guarantees. And don’t just ask, test it (more on that later).

2. Storage Struggles: Slow Storage Cripples Training Runs

A fast GPU is useless if it’s constantly waiting for data from storage. Slow storage not only impacts training speed but can also lead to checkpointing failures, potentially losing valuable progress.

The Problem: Cloud providers might advertise impressive storage speeds, but those numbers often don’t reflect real-world, under-load performance.

The Question: “What is the sustained storage throughput under load that I can expect for my GPU instances? Can you provide metrics or allow me to conduct performance tests to verify?”

3. Cooling Compromises: Throttling and Hidden Performance Degradation

Often overlooked, cooling is crucial. If GPUs overheat, they throttle their performance to prevent damage. This throttling can significantly reduce their performance without you even knowing it.

The Problem: Providers rarely highlight their cooling infrastructure. You might be paying for peak GPU performance, but secretly only getting half the speed due to overheating.

The Question: “What cooling solutions do you employ to ensure consistent GPU performance? Do you monitor GPU temperatures and proactively address potential throttling issues? What transparency do you provide regarding potential throttling events?”

4. Resource Reality: Are You Getting a Full GPU?

The pricing model is often based on a “GPU instance,” but are you truly getting a dedicated GPU, or a slice of one that’s being shared with other users?

The Problem: Some providers quietly slice up GPUs or overschedule resources, reducing the actual performance. This is often masked by the overall “availability” of instances.

The Question: “Am I getting a full, dedicated GPU for my instance? If not, what is the sharing ratio and what guarantees can you provide regarding consistent performance despite resource sharing?”

5. Power Protection: Redundancy and Reliability

Power outages, whether a full floor outage or a simple power supply failure, can halt your critical AI workloads. Redundancy is key.

The Problem: Unexpected downtime can disrupt training runs, delay project timelines and cost you money.

The Question: “What is your power redundancy setup (N+1, 2N)? Do you have 24/7 on-site support with skilled personnel and spare parts to address power-related issues?”

6. Egress Expense: Watch Out for Data Transfer Fees

Egress fees, the charges for moving data out of the cloud, can quickly become a significant, and often unexpected, budget killer.

The Problem: Some providers charge exorbitant fees to move your datasets in and out, effectively locking you into their platform.

The Question: “What are your egress fees for data transfer? Are there any volume discounts or alternative pricing models available? What are the costs of transferring data in and out of the service?”

7. Software Stack Sufficiency: Drivers, Storage and Kernels Matter

AI isn’t just about hardware; it’s also about the software stack. Outdated NVIDIA drivers, slow storage stacks or unoptimized kernels can severely impact performance.

The Problem: A shiny new GPU is only as good as the software that drives it. If the provider isn’t actively tuning their infrastructure and keeping their software up-to-date, you’re leaving performance on the table.

The Question: “How frequently do you update your NVIDIA drivers and other software components? Are you actively tuning your infrastructure to optimize performance for AI workloads? Can you provide information about your storage stack and kernel configurations?”

Verifying Performance: Beyond the Marketing Claims

Asking the right questions is essential, but it’s only the first step. You need to test everything.

• GPU Performance: Run benchmarks specific to your workload. Don’t rely solely on synthetic benchmarks.
  Storage Throughput: Measure sustained read and write speeds under realistic load conditions.
  Network Throughput: Test network bandwidth between your instances and your local environment.

Remember that testing provides a snapshot in time. Ongoing monitoring and analysis are necessary to ensure consistent performance.

Service Level Agreements: Holding Providers Accountable

Service Level Agreements (SLAs) are crucial for defining expectations and holding providers accountable.

Key Questions: “What is your SLA for GPU replacement in case of failure? Do you have spare parts on-site to expedite repairs? What are the penalties for failing to meet the SLA terms?”

Demand Transparency and Verify Performance

Investing in AI infrastructure is a significant decision. Don’t rely solely on marketing materials. Ask the tough questions, demand transparency and meticulously test performance. By taking a proactive approach, you can ensure that your cloud provider delivers the performance you’re paying for and unlocks the true potential of your AI initiatives.