Perform multihost inference using Pathways

Multihost inference is a method of running model inference that distributes the model across multiple accelerators hosts. This enables the inference of large models that don't fit on a single host. Pathways can be deployed for both batch and real time multihost inference use cases.

Before you begin

Make sure you have:

Run Batch inference using JetStream

JetStream is a throughput and memory-optimized engine for large language model (LLM) inference on XLA devices, primarily Tensor Processing Units (TPUs) written in JAX.

You can use a prebuilt JetStream Docker image to run a batch inference workload, as shown in the following YAML. This container is built from the OSS JetStream project. For more information about MaxText-JetStream flags, see JetStream MaxText server flags. The following example uses Trillium chips (v6e-16) to load the Llama3.1-405b int8 checkpoint and perform inference over it. This example assumes you already have a GKE cluster with at least one v6e-16 nodepool inside it.

Start model server and Pathways

  1. Get credentials to the cluster and add them to your local kubectl context. 
     gcloud container clusters get-credentials $CLUSTER \  --zone=$ZONE \  --project=$PROJECT \  && kubectl config set-context --current --namespace=default
  2. Deploy LeaderWorkerSet (LWS) API. 
     VERSION=v0.4.0  kubectl apply --server-side -f "https://github.com/kubernetes-sigs/lws/releases/download/${VERSION}/manifests.yaml"
  3. Copy and paste the following YAML into a file named pathways-job.yaml: This YAML has been optimized for the v6e-16 slice shape. For more information on how to convert a Meta checkpoint into a JAX compatible checkpoint, follow the checkpoint creation guide in Creating inference checkpoints. As an example, instructions for Llama3.1-405B are provided here Checkpoint conversion for Llama3.1-405B. 
     apiVersion: leaderworkerset.x-k8s.io/v1  kind: LeaderWorkerSet  metadata:  name: jetstream-pathways  annotations:  leaderworkerset.sigs.k8s.io/exclusive-topology: cloud.google.com/gke-nodepool  spec:  replicas: 1  leaderWorkerTemplate:  leaderTemplate:  metadata:  labels:   app: jetstream-pathways  spec:  nodeSelector:  cloud.google.com/gke-tpu-accelerator: TPU_ACCELERATOR_TYPE # Example: tpu-v6e-slice  cloud.google.com/gke-tpu-topology: TPU_TOPOLOGY # Example: 4x4  tolerations:  - key: "google.com/tpu"  operator: "Exists"  effect: "NoSchedule"  containers:  - name: pathways-proxy  image: us-docker.pkg.dev/cloud-tpu-v2-images/pathways/proxy_server:latest  args:  - --resource_manager_address=$(LWS_LEADER_ADDRESS):38677  - --server_port=38681  - --gcs_scratch_location=gs://cloud-pathways-staging/tmp  imagePullPolicy: Always  ports:  - containerPort: 38681  - name: pathways-rm  env:   - name: HOST_ADDRESS  value: "$(LWS_LEADER_ADDRESS)"  - name: TPU_SKIP_MDS_QUERY  value: "true"  image: us-docker.pkg.dev/cloud-tpu-v2-images/pathways/server:latest  args:  - --server_port=38677  - --gcs_scratch_location=gs://cloud-pathways-staging/tmp  - --node_type=resource_manager  - --instance_count=1  - --instance_type=tpuv6e:TPU_TOPOLOGY # Example: 4x4  - --temporary_flags_for_debugging=temporary_flag_for_debugging_worker_expected_tpu_chip_config=megachip_tccontrol  imagePullPolicy: Always  ports:  - containerPort: 38677  - name: jax-tpu  image: us-docker.pkg.dev/cloud-tpu-images/inference/jetstream-pathways:v0.2.0  # Optimized settings used to serve Llama3.1-405b.  args:  - MaxText/configs/v5e/inference/llama3_405b_v5e-64.yml  - model_name=llama3.1-405b  - load_parameters_path=GCS_CHECKPOINT_PATH  - max_prefill_predict_length=1024  - max_target_length=2048  - async_checkpointing=false  - steps=1  - ici_fsdp_parallelism=1  - ici_autoregressive_parallelism=2  - ici_tensor_parallelism=8  - scan_layers=false  - weight_dtype=bfloat16  - per_device_batch_size=10  - enable_single_controller=true  - quantization=int8  - quantize_kvcache=true  - checkpoint_is_quantized=true  - enable_model_warmup=true  imagePullPolicy: Always  ports:   - containerPort: 9000  startupProbe:  httpGet:  path: /healthcheck  port: 8000  scheme: HTTP  periodSeconds: 1  initialDelaySeconds: 900  failureThreshold: 10000  livenessProbe:  httpGet:  path: /healthcheck  port: 8000  scheme: HTTP  periodSeconds: 60  failureThreshold: 10  readinessProbe:  httpGet:  path: /healthcheck  port: 8000  scheme: HTTP  periodSeconds: 60  failureThreshold: 10  - name: jetstream-http  image: us-docker.pkg.dev/cloud-tpu-images/inference/jetstream-http:v0.2.3  imagePullPolicy: Always  ports:  - containerPort: 8000  # The size variable defines the number of worker nodes to be created.  # It must be equal to the number of hosts + 1 (for the leader node).  size: 5  workerTemplate:  spec:  nodeSelector:  cloud.google.com/gke-tpu-accelerator: TPU_ACCELERATOR_TYPE # Example: tpu-v6e-slice  cloud.google.com/gke-tpu-topology: TPU_TOPOLOGY # Example: 4x4  tolerations:  - key: "google.com/tpu"  operator: "Exists"  effect: "NoSchedule"  containers:  - name: worker  args:  - --server_port=38679  - --resource_manager_address=$(LWS_LEADER_ADDRESS):38677  - --gcs_scratch_location=gs://cloud-pathways-staging/tmp  image: us-docker.pkg.dev/cloud-tpu-v2-images/pathways/server:latest  imagePullPolicy: Always  ports:  - containerPort: 38679  resources:  limits:  google.com/tpu: "4"  ---   apiVersion: v1  kind: Service  metadata:  name: jetstream-svc  spec:  selector:  app: jetstream-pathways  ports:  - protocol: TCP  name: jetstream-http  port: 8000  targetPort: 8000
    Replace the following:
    • TPU_ACCELERATOR_TYPE: The TPU accelerator type. For example, tpu-v6e-slice.
    • TPU_TOPOLOGY: The TPU topology. For example, 2x4.
    • GCS_CHECKPOINT_PATH: The GCS path to the checkpoint.
    Apply this YAML. Wait for the PathwaysJob to be scheduled. Once scheduled, the model server may take some time to restore the checkpoint. For the 405B model, this takes ~7 minutes.
  4. Look at the Kubernetes logs to see if the JetStream model server is ready: The workload was named `jetstream-pathways` in the previous YAML, and `0` is the head node. 
     kubectl logs -f jetstream-pathways-0 -c jax-tpu
    The output is similar to the following which indicates the JetStream model server is ready to serve requests: 
     2025-03-02 02:15:07,682 - JetstreamLogger - INFO - Initializing the driver with 1 prefill engines and 1 generate engines in interleaved mode  2025-03-02 02:15:07,683 - JetstreamLogger - INFO - Spinning up prefill thread 0.  2025-03-02 02:15:07,683 - JetstreamLogger - INFO - Spinning up transfer thread 0.  2025-03-02 02:15:07,684 - JetstreamLogger - INFO - Spinning up generate thread 0.  2025-03-02 02:15:07,684 - JetstreamLogger - INFO - Spinning up detokenize thread 0.  2025-03-02 02:15:07,685 - JetstreamLogger - INFO - Driver initialized.  ...  ...  ...  INFO: Started server process [7]  INFO: Waiting for application startup.  INFO: Application startup complete.  INFO: Uvicorn running on http://0.0.0.0:9999 (Press CTRL+C to quit)

Connect to the model server

You can access the JetStream Pathways deployment using GKE's ClusterIP service. The ClusterIP service is only reachable from within the cluster. Therefore, to access the service from outside the cluster, you must first establish a port-forwarding session by running the following command:

kubectl port-forward pod/${HEAD_POD} 8000:8000

Verify that you can access the JetStream HTTP server by opening a new terminal and running the following command:

curl --request POST \ --header "Content-type: application/json" \ -s \ localhost:8000/generate \ --data \ '{  "prompt": "What are the top 5 programming languages",  "max_tokens": 200 }'

The initial request can take several seconds to complete due to model warmup. The output should be similar to the following:

{  "response": " for web development?\nThe top 5 programming languages for web development are:\n1. **JavaScript**: JavaScript is the most popular language for web development, used by over 90% of websites for client-side scripting. It's also popular for server-side programming with technologies like Node.js.\n2. **HTML/CSS**: HTML (Hypertext Markup Language) and CSS (Cascading Style Sheets) are not programming languages, but are essential for building websites. HTML is used for structuring content, while CSS is used for styling and layout.\n3. **Python**: Python is a popular language for web development, especially with frameworks like Django and Flask. It's known for its simplicity, flexibility, and large community of developers.\n4. **Java**: Java is a popular language for building enterprise-level web applications, especially with frameworks like Spring and Hibernate. It's known for its platform independence, strong security features, and large community of developers.\n5. **PHP**: PHP is a mature language for web" }

Disaggregated inference

Disaggregated serving is a technique for running large language models (LLMs) that separates the prefill and decode stages into different processes, potentially on different machines. This allows for better utilization of resources and can lead to improvements in performance and efficiency, especially for large models.

  • Prefill: this stage processes the input prompt and generates an intermediate representation (like a key-value cache). It's often compute intensive.
  • Decode: this stage generates the output tokens, one by one, using the prefill representation. It is typically memory-bandwidth bound.

By separating these stages, disaggregated serving allows for prefill and decode to run in parallel, improving throughput and latency.

To enable disaggregated serving, modify the following YAML to utilize two v6e-8 slices: one for prefill and the other for generate. Before proceeding, ensure your GKE cluster has at least two node pools configured with this v6e-8 topology. For optimal performance, specific XLA flags have been configured.

Create a llama2-70b checkpoint following the same process as llama3.1-405b checkpoint creation, detailed in the previous section.

  1. To launch the JetStream server in disaggregated mode using Pathways, copy and paste the following YAML into a file named pathways-job.yaml: 
    apiVersion: leaderworkerset.x-k8s.io/v1 kind: LeaderWorkerSet metadata:  name: jetstream-pathways  annotations:  leaderworkerset.sigs.k8s.io/subgroup-exclusive-topology: cloud.google.com/gke-nodepool spec:  replicas: 1  leaderWorkerTemplate:  subGroupPolicy:  subGroupSize: 2  leaderTemplate:  metadata:  labels:  app: jetstream-pathways  spec:  nodeSelector:  cloud.google.com/gke-tpu-accelerator: TPU_ACCELERATOR_TYPE # Example: tpu-v6e-slice  cloud.google.com/gke-tpu-topology: TPU_TOPOLOGY # Example: 2x4  tolerations:  - key: "google.com/tpu"  operator: "Exists"  effect: "NoSchedule"  containers:  - name: pathways-proxy  image: us-docker.pkg.dev/cloud-tpu-v2-images/pathways/proxy_server:latest  args:  - --resource_manager_address=$(LWS_LEADER_ADDRESS):38677  - --server_port=38681  - --gcs_scratch_location=gs://cloud-pathways-staging/tmp  imagePullPolicy: Always  ports:  - containerPort: 38681  - name: pathways-rm  env:   - name: HOST_ADDRESS  value: "$(LWS_LEADER_ADDRESS)"  - name: TPU_SKIP_MDS_QUERY  value: "true"  image: us-docker.pkg.dev/cloud-tpu-v2-images/pathways/server:latest  args:  - --server_port=38677  - --gcs_scratch_location=gs://cloud-pathways-staging/tmp  - --node_type=resource_manager  - --instance_count=2  - --instance_type=tpuv6e:TPU_TOPOLOGY # Example: 2x4  imagePullPolicy: Always  ports:  - containerPort: 38677  - name: jax-tpu  image: us-docker.pkg.dev/cloud-tpu-images/inference/jetstream-pathways:v0.2.0  # Optimized settings used to serve Llama2-70b.  args:  - MaxText/configs/base.yml  - tokenizer_path=assets/tokenizer.llama2  - load_parameters_path=GCS_CHECKPOINT_PATH  - max_prefill_predict_length=1024  - max_target_length=2048  - model_name=llama2-70b  - ici_fsdp_parallelism=1  - ici_autoregressive_parallelism=1  - ici_tensor_parallelism=-1  - scan_layers=false  - weight_dtype=bfloat16  - per_device_batch_size=1  - checkpoint_is_quantized=true   - quantization=int8  - quantize_kvcache=true  - compute_axis_order=0,2,1,3  - ar_cache_axis_order=0,2,1,3  - stack_prefill_result_cache=True  # Specify disaggregated mode to run Jetstream  - inference_server=ExperimentalMaxtextDisaggregatedServer_8  - inference_benchmark_test=True  - enable_model_warmup=True  env:  - name: LOG_LEVEL  value: "INFO"  imagePullPolicy: Always  securityContext:  capabilities:  add: ["SYS_PTRACE", "NET_ADMIN", "SYS_TIME"]  ports:   - containerPort: 9000  startupProbe:  httpGet:  path: /healthcheck  port: 8000  scheme: HTTP  periodSeconds: 1  initialDelaySeconds: 240  failureThreshold: 10000  livenessProbe:  httpGet:  path: /healthcheck  port: 8000  scheme: HTTP  periodSeconds: 60  failureThreshold: 100  readinessProbe:  httpGet:  path: /healthcheck  port: 8000  scheme: HTTP  periodSeconds: 60  failureThreshold: 100  - name: jetstream-http  image: us-docker.pkg.dev/cloud-tpu-images/inference/jetstream-http:v0.2.3  imagePullPolicy: Always  ports:  - containerPort: 8000  # The size variable defines the number of worker nodes to be created.  # It must be equal to the number of hosts + 1 (for the leader node).  size: 5  workerTemplate:  spec:  nodeSelector:  cloud.google.com/gke-tpu-accelerator: TPU_ACCELERATOR_TYPE # Example: tpu-v6e-slice  cloud.google.com/gke-tpu-topology: TPU_TOPOLOGY # Example: 2x4  containers:  - name: worker  args:  - --server_port=38679  - --resource_manager_address=$(LWS_LEADER_ADDRESS):38677  - --gcs_scratch_location=gs://cloud-pathways-staging/tmp  image: us-docker.pkg.dev/cloud-tpu-v2-images/pathways/server:latest  imagePullPolicy: Always  ports:  - containerPort: 38679  resources:  limits:  google.com/tpu: "4" ---  apiVersion: v1 kind: Service metadata:  name: jetstream-svc spec:  selector:  app: jetstream-pathways  ports:  - protocol: TCP  name: jetstream-http  port: 8000  targetPort: 8000
    Replace the following:
    • TPU_ACCELERATOR_TYPE: The TPU accelerator type. For example, tpu-v6e-slice.
    • TPU_TOPOLOGY: The TPU topology. For example, 2x4.
    • GCS_CHECKPOINT_PATH: The GCS path to the checkpoint.
  2. Apply this YAML, the model server will take some time to restore the checkpoint. For the 70B model, this may take ~2 minutes. 
     kubectl apply -f pathways-job.yaml
  3. Look at the Kubernetes logs to see if the JetStream model server is ready: 
     kubectl logs -f jetstream-pathways-0 -c jax-tpu
    You will see output similar to the following which indicates the JetStream model server is ready to serve requests: 
     2025-03-02 02:15:07,682 - JetstreamLogger - INFO - Initializing the driver with 1 prefill engines and 1 generate engines in interleaved mode  2025-03-02 02:15:07,683 - JetstreamLogger - INFO - Spinning up prefill thread 0.  2025-03-02 02:15:07,683 - JetstreamLogger - INFO - Spinning up transfer thread 0.  2025-03-02 02:15:07,684 - JetstreamLogger - INFO - Spinning up generate thread 0.  2025-03-02 02:15:07,684 - JetstreamLogger - INFO - Spinning up detokenize thread 0.  2025-03-02 02:15:07,685 - JetstreamLogger - INFO - Driver initialized.  ...  ...  ...  INFO: Started server process [7]  INFO: Waiting for application startup.  INFO: Application startup complete.  INFO: Uvicorn running on http://0.0.0.0:9999 (Press CTRL+C to quit)

Connect to the model server

You can access the JetStream Pathways deployment through the GKE's ClusterIP service. The ClusterIP service is only reachable from within the cluster. Therefore, to access the service from outside the cluster, establish a port-forwarding session by running the following command:

kubectl port-forward pod/${HEAD_POD} 8000:8000

Verify that you can access the JetStream HTTP server by opening a new terminal and running the following command:

curl --request POST \ --header "Content-type: application/json" \ -s \ localhost:8000/generate \ --data \ '{  "prompt": "What are the top 5 programming languages",  "max_tokens": 200 }'

The initial request can take several seconds to complete due to model warmup. The output should be similar to the following:

{  "response": " used in software development?\nThe top 5 programming languages used in software development are:\n\n1. Java: Java is a popular programming language used for developing enterprise-level applications, Android apps, and web applications. Its platform independence and ability to run on any device that has a Java Virtual Machine (JVM) installed make it a favorite among developers.\n2. Python: Python is a versatile language that is widely used in software development, data analysis, artificial intelligence, and machine learning. Its simplicity, readability, and ease of use make it a popular choice among developers.\n3. JavaScript: JavaScript is a widely used programming language for web development, allowing developers to create interactive client-side functionality for web applications. It is also used for server-side programming, desktop and mobile application development, and game development.\n4. C++: C++ is a high-performance programming language used for developing operating systems, games, and other high-performance applications." }

What's next