Get started on Modelplane Docs

Build the platform

This is the platform team’s side of Modelplane. You set up the gateway that fronts your models, give the control plane cloud credentials, and register your first GPU cluster: a hardware profile published as an InferenceClass and an InferenceCluster that offers it.

In the next step, the ML team will create a model deployment that schedules against this capacity without knowing which cluster it runs on.

Prerequisites

An AWS account with permissions to create EKS clusters, VPCs, and IAM roles
AWS access key ID and secret access key

A GCP service account JSON key, granted these roles on the project:

Deploying a model

Now that the platform is provisioned, the ML team can declare what a model needs with a ModelDeployment. Describe the hardware requirements and the scheduler schedules against the capacity the platform team published.

Create a deployment

Create a namespace for the model:

kubectl create namespace ml-team

The device selector matches against the capacity declared in the InferenceClass, not the pod’s resource requests. Any L4 node satisfies >= 20Gi, so this deployment runs on the cluster you just added:

Scale the platform

You have one small-GPU cluster with a running model. In this guide, you’ll grow the fleet with larger-GPU capacity so the ML team has more to schedule against.

Provisioning takes about 10 to 15 minutes.

Register more clusters

Scale the model

A ModelService can front more than one ModelDeployment. Here you add a second deployment, pinned to a different region, and point the same service at both. The endpoint you already curled stays the same. Behind it, traffic now load-balances across two regions.

graph LR
subgraph fleet ["Fleet"]
IC1["cluster-a\nsmall GPU"]
IC2["cluster-b\nlarger GPU"]
end
subgraph ml ["ML team"]
MD1["ModelDeployment\nqwen-demo"]
MD2["ModelDeployment\nqwen-west\ntargets cluster-b"]
MS["ModelService qwen\nmodel: ml-team/qwen"]
end
IC1 --> MD1
IC2 --> MD2
MD1 --> MS
MD2 --> MS

Add a second deployment

The new deployment targets a larger-GPU cluster you added in the last step. On EKS, GKE, and AKS it pins to a second region with a clusterSelector. On Nebius, which runs one region per project, the capacity selector alone routes it to the H100 tier:

Clean up

Delete the model resources, the gateway, the clusters, and finally the control plane.

Delete model resources

Delete model resources before clusters. A cluster refuses deletion while anything still runs on it. Foreground cascading deletion holds each resource until what it composed on the clusters is gone, so a cluster isn’t released while that’s still being removed:

kubectl delete md --all -n ml-team --cascade=foreground
kubectl delete ms --all -n ml-team --cascade=foreground

Delete the gateway

Delete the gateway before its cluster. The InferenceGateway runs a load balancer on the cluster it names; deleting it while that cluster is still up lets the load balancer be removed, rather than leaking it when the cluster goes. Foreground deletion holds the gateway until its objects on the cluster are deleted: