A 20 GB Model, Two T4s, and Not a Single H100: GPUs for Confidential Containers 

We worked through the entire path of a GPU workload step by step: instance provisioning, image build, node configuration, and model serving. Blockers at every stage were fixed directly in the product.

The key architectural decision concerned passing GPUs into podVMs. A 20 GB model does not fit into the memory of a single T4 (16 GB). Teaching the adaptor to pass multiple GPUs into one VM would have required significant changes to a security-critical component. So we split the model across two podVMs, each with a single GPU, using KubeRay and tensor parallelism.

Splitting a model across multiple GPUs is a common task in LLM deployment, but here it had to be solved inside podVMs. This turned a platform limitation into an orchestration problem with a proven solution. Distributed serving also gives the platform a path for models that need more resources than a single GPU can provide.

Production workloads with hardware-backed confidentiality guarantees require confidential instances with Hopper- or Blackwell-class GPUs. The N1 + T4 test environment does not provide these guarantees: it lets you verify how the platform stack works, but not memory protection.

Solution Components

Modified Cloud API Adaptor: creates podVMs on N1 + NVIDIA T4 instances for low-cost development and functional testing.

Custom podVM image built with mkosi: NVIDIA drivers, libraries, and initialization services are baked into the image instead of relying on the GPU Operator.

KubeRay and tensor parallelism: the model is split across multiple podVMs, each with a single GPU.

Self-healing containerd DaemonSet: monitors the containerd configuration and restores it after GKE node upgrades.

Upstream PR to Cloud API Adaptor: the ability to create podVMs without public IP addresses.

Approaches That Didn't Work

NVIDIA GPU Operator. The operator works only with regular Kubernetes nodes and cannot see podVMs. This is an architectural trait, not a bug that can be fixed. So the drivers had to be built directly into the image, and the image build became the core of the work.

Multiple GPUs in one VM. The model could have been placed on a VM with several GPUs, but the adaptor did not support passing multiple GPUs into a single podVM. So we used the KubeRay-based distributed serving described above.

Manual containerd patch. The fix held only until the first GKE node upgrade, after which the failure came back. So we dropped the one-time installation step: the containerd configuration is now continuously maintained by a DaemonSet.

Working without documentation. The podVM image build process wasn't documented anywhere, and every attempt started from scratch. That's why the build guide became a deliverable in its own right.

The Hardest Part: A podVM Image as Part of the Security Boundary

On a regular Kubernetes node, installing GPU drivers is a routine operation. In a production configuration, the image becomes the foundation of a confidential VM, so everything baked into it ends up inside the trust boundary. That meant the build had to be more than functional: it had to be reproducible, minimal, and auditable.

NVIDIA drivers, libraries, and initialization services were placed into the image via a skeleton filesystem. This is a set of directories and files that mkosi copies into the image during the build. As a result, the services start correctly without external configuration management. There were no existing examples of such a build, so it took far more iterations than similar work on a regular node.

The second challenge was making changes persist in managed Kubernetes. The provider manages the nodes, there is no persistent host access, and upgrades reset the local configuration. So we turned the containerd settings into a continuously maintained state rather than a one-off change.

Custom Solutions

Cloud API Adaptor fork: creates smaller, cheaper instance types based on a predefined configuration. This allows GPU pods in podVMs to run on N1 with NVIDIA T4. 

podVM build with mkosi: drivers and their initialization services are added to the image via a skeleton filesystem.

Containerd Remediation DaemonSet: deployed via a label selector during installation. In a watch loop, it tracks the state of each worker node and reapplies the required containerd configuration if it has changed. The installer automatically detects the environments where the DaemonSet is needed.

Upstream PR: a change to Cloud API Adaptor that allows creating VMs without public IP addresses was accepted into the open-source project.

podVM image build guide: common pitfalls and recommended approaches for engineers who will build their own images.

Tech Stack

Core: Kubernetes, GKE (Google Cloud), Confidential Containers, Cloud API Adaptor, podVM, containerd, DaemonSets, Helm, Docker/OCI registries, NVIDIA Drivers & Container Toolkit, NVIDIA T4 / N1 instances, KubeRay, Ray, Mistral, Go.

Specialized: mkosi (podVM image builds), skeleton filesystem injection (baking drivers and services into the image), tensor parallelism across single-GPU podVMs, label-selected DaemonSet reconciler (continuous reconciliation of node configuration).