As more companies pour money into building their artificial intelligence (AI) infrastructure from ground up, resource sharing is offering a new path to ROI.

Multi-tenancy, the architecture that allows resources to be dynamically split between different tenants, has rose to prominence with the cloud, making it easier for one provider to serve multiple customers concurrently.

But although companies are eager to do it, managing a multi-tenant environment in the AI era is harder than you might think.

“AI workloads are very demanding, so the entire stack requires to be improved starting from power, cooling, compute hardware, compute software to networking hardware,” noted Alex Saroyan, CEO and cofounder of Netris, maker of a network automation, abstraction and multi-tenancy software.

But interestingly, this stack does not need to be as divergent as the companies adopting them. “Although organizations that are running AI infrastructures are coming from different verticals – we work with neo clouds, infrastructure as a service providers, PaaS, sovereign AI operators, telcos – these organizations have different business models but from a technical perspective, what they are building is very similar.”

Based out of Santa Clara, Calif, Netris was founded in 2018.

“In 2018, AI wasn’t as big as today,” remembered Saroyan, “But some of the technologies that were being used for cloud providers in networking [then] are also relevant and applicable to AI.”

While both the hardware and software aspects of AI networking are exceedingly complex, multi-tenancy adds still more complexity to that.

AI network infrastructure operators have a few tricks up their sleeves to overcome the headaches. The first of them is physical segmentation of the network which involves physically moving apart hardware to create small sections or subnets.

“That provides maximum security. There is no way to break in from one cluster into another if they are physically separated, but the problem is that it takes a lot of time,” Saroyan noted.

Another part of the problem is that physical segregation is hard to scale , leading to idling of GPUs.

To work around that, operators take to manually configuring the infrastructure to make it sharable across the network of tenants. “That does the job, but the problem with that is…it takes sometimes weeks to configure hundreds of switches, and the AI infrastructure has lots of switches.”

There is a high risk of human error involved with manually configuring a network that is as large and complex as this. The smallest misconfiguration can potentially break the network for other tenants using it, impacting their training workloads, Saroyan pointed out.

To bypass that, many operators resort to building in-house automation solutions, but Saroyan argues those are “not great solutions”.

“The big problem with building automation inhouse is that the organization that is building [it] learns from their own errors versus a specialized product that is learning from the data coming from lots of lots of customers… They are risky, they take a lot of time, and not a solution for cloud providers.”

The alternative is API or software-driven isolation using Kubernetes and containers. Many operators tend to prefer this option for the instantaneous result but it fails to enforce isolation on the networking layer and that opens up security holes in the fabric, Saroyan pointed out.

“It takes just one container vulnerability to jeopardize the reputation of a cloud provider,” he said.

A solution that provides security at the level comparable to physical isolation, while also delivering it at software speed would make a big difference. Netris’s “cloud-provider-grade network automation” solution aims to do just that.

The Netris network automation, abstraction and multi-tenancy software brings together the concept of VPC, cloud networking functions like VPC peering, elastic load balancer and internet/NAT gateways, and network management, onto a single plane.

“VPC is the unit of isolation and when AI infrastructure operator is building cloud, they need this same abstraction model that works in the cloud,” Saroyan explained, while showcasing the Netris platform at the AI Infrastructure Field Day event in late April.

The Netris VPC allows operators to securely manage resources within a logically separated virtual network. They can create new VPCs, edit existing ones and delete those that are not in use, dynamically.

Available on top of this is a range of ready cloud networking functions – VPC peering to connect VPCs to upstream networks, internet gateways and load balancers – which make building and operating the infrastructure much simpler.

“Inference workloads are used by users on the internet; think ChatGPT and their customers. So if you’re running inference loads in the AI cluster, you need some sort of load balancer which will take traffic from the internet and balance between these different machines like in the cloud…If you are a cloud provider, you need methods to provide these constructs yourself,” he said.

The third piece of the puzzle is network management. “Let’s not forget that all these beautiful clouds are running on the physical network, and there are network engineers who need to take care of the network, deploy the fabric, upgrade and downgrade it, perform maintenance, learn if something is not good with the health of the fabric. So, for that you need a system that does your fabric management.”

If these three components came from different providers, the tools could cancel each other out attempting to do overlapping things.

“If more than one system is trying to make the same changes to the front or back-end network, it creates risks for logical disconnect and errors,” Saroyan added.

Netris, being a single tool, uses the same data which eliminates the risk of conflicts.

Saroyan informed that up to 60% of its customers use InfiniBand for their backend fabric. So to avoid friction, Netris works through NVIDIA UFM controller for InfiniBand management. “Like I said, two systems should not edit one system. We don’t want to break our own rules. So we talk to UFM, and UFM talks to InfiniBand switches”.

For ethernet however, Netris serves as the full management layer. “From a customer perspective, there is a single pane of glass that takes care of entire stack, but behind the scenes, we manage ethernet and we manage InfiniBand through UFM,” he told.

However, it is important to remember that no one vendor can solve all the challenges that come with building and managing the AI infrastructure, which makes it a smarter approach to find the right vendor with the right solution for each facet.

As Netris says, “One company cannot solve all these pieces of the puzzle. Even NVIDIA being this amazing large organization, they don’t have all the components. That’s why it creates opportunity for partners like us and others.”