Run NCCL on custom GKE clusters that use A4 or A3 Ultra

This page describes how to run NCCL/gIB tests on provisioned clusters that use GPUDirect RDMA. It describes tests for the following scenarios:

Test on two nodes

Run the two node test:

A4

  1. To deploy a NCCL test workload of two test Pods that are running on two A4 nodes, apply one of the following manifests:

    • For an Autopilot cluster:

      kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/refs/heads/master/gpudirect-rdma/nccl-test-a4-autopilot.yaml

    • For a Standard cluster:

      kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/refs/heads/master/gpudirect-rdma/nccl-test-a4.yaml

  2. Check if the Pods are scheduled to and running on some nodes:

    kubectl get pods nccl-test-host-1 nccl-test-host-2

    If the two Pods have the Running status, you can proceed to the next step. For nodes that are provisioned by flex-start, it might take a few minutes before the nodes are created and the Pods are scheduled on those nodes.

  3. Trigger a NCCL all-gather test for the nodes:

    kubectl exec nccl-test-host-1 -it -- /usr/local/gib/scripts/run_nccl_tests.sh -t all_gather -b 1K -e 8G nccl-host-1 nccl-host-2

    The output should be similar to the following:

    # size count type redop root time algbw busbw #wrong time algbw busbw #wrong # (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) 1024 16 float none -1 48.17 0.02 0.02 0 47.21 0.02 0.02 0 2048 32 float none -1 47.23 0.04 0.04 0 47.17 0.04 0.04 0 4096 64 float none -1 47.43 0.09 0.08 0 47.48 0.09 0.08 0 8192 128 float none -1 47.93 0.17 0.16 0 47.98 0.17 0.16 0 16384 256 float none -1 48.90 0.34 0.31 0 48.75 0.34 0.32 0 32768 512 float none -1 50.10 0.65 0.61 0 49.59 0.66 0.62 0 65536 1024 float none -1 51.70 1.27 1.19 0 51.66 1.27 1.19 0 131072 2048 float none -1 52.23 2.51 2.35 0 55.60 2.36 2.21 0 262144 4096 float none -1 53.89 4.86 4.56 0 53.39 4.91 4.60 0 524288 8192 float none -1 56.80 9.23 8.65 0 57.66 9.09 8.52 0 1048576 16384 float none -1 87.85 11.94 11.19 0 88.47 11.85 11.11 0 2097152 32768 float none -1 92.52 22.67 21.25 0 93.22 22.50 21.09 0 4194304 65536 float none -1 97.41 43.06 40.37 0 96.15 43.62 40.90 0 8388608 131072 float none -1 110.0 76.27 71.51 0 110.9 75.66 70.93 0 16777216 262144 float none -1 141.3 118.77 111.35 0 140.7 119.27 111.81 0 33554432 524288 float none -1 203.2 165.14 154.82 0 202.3 165.90 155.53 0 67108864 1048576 float none -1 303.3 221.25 207.42 0 301.9 222.27 208.38 0 134217728 2097152 float none -1 513.2 261.56 245.21 0 509.3 263.56 247.08 0 268435456 4194304 float none -1 842.4 318.64 298.72 0 832.3 322.54 302.38 0 536870912 8388608 float none -1 1511.8 355.12 332.92 0 1502.5 357.31 334.98 0 1073741824 16777216 float none -1 2976.7 360.72 338.17 0 2923.2 367.32 344.36 0 2147483648 33554432 float none -1 5888.9 364.66 341.87 0 5766.2 372.43 349.15 0 4294967296 67108864 float none -1 11722 366.39 343.49 0 11457 374.88 351.45 0 8589934592 134217728 float none -1 23379 367.43 344.46 0 22818 376.45 352.92 0 # Out of bounds values : 0 OK # Avg bus bandwidth : 120.845

A3 Ultra

  1. To deploy a NCCL test workload of two test Pods that are running on two A3 Ultra nodes, apply one of the following manifests:

    • For an Autopilot cluster:

      kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/refs/heads/master/gpudirect-rdma/nccl-test-autopilot.yaml

    • For a Standard cluster:

      kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/refs/heads/master/gpudirect-rdma/nccl-test.yaml

  2. Check if the Pods are scheduled to and running on some nodes:

    kubectl get pods nccl-test-host-1 nccl-test-host-2

    If the two Pods have the Running status, you can proceed to the next step. For nodes that are provisioned by flex-start, it might take a few minutes before the nodes are created and the Pods are scheduled on those nodes.

  3. Trigger a NCCL all-gather test for the nodes:

    kubectl exec nccl-test-host-1 -it -- /usr/local/gib/scripts/run_nccl_tests.sh -t all_gather -b 1K -e 8G nccl-host-1 nccl-host-2

    The output should be similar to the following:

    # size count type redop root time algbw busbw #wrong time algbw busbw #wrong # (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) 1024 16 float none -1 56.00 0.02 0.02 0 55.59 0.02 0.02 0 2048 32 float none -1 55.79 0.04 0.03 0 55.57 0.04 0.03 0 4096 64 float none -1 56.29 0.07 0.07 0 57.35 0.07 0.07 0 8192 128 float none -1 56.44 0.15 0.14 0 56.32 0.15 0.14 0 16384 256 float none -1 57.57 0.28 0.27 0 57.60 0.28 0.27 0 32768 512 float none -1 57.92 0.57 0.53 0 59.35 0.55 0.52 0 65536 1024 float none -1 59.92 1.09 1.03 0 60.15 1.09 1.02 0 131072 2048 float none -1 59.21 2.21 2.08 0 61.82 2.12 1.99 0 262144 4096 float none -1 63.58 4.12 3.87 0 63.34 4.14 3.88 0 524288 8192 float none -1 64.89 8.08 7.57 0 65.09 8.06 7.55 0 1048576 16384 float none -1 80.90 12.96 12.15 0 77.49 13.53 12.69 0 2097152 32768 float none -1 80.22 26.14 24.51 0 79.88 26.25 24.61 0 4194304 65536 float none -1 82.86 50.62 47.45 0 82.47 50.86 47.68 0 8388608 131072 float none -1 95.83 87.53 82.06 0 93.27 89.94 84.32 0 16777216 262144 float none -1 122.8 136.58 128.04 0 121.7 137.86 129.24 0 33554432 524288 float none -1 180.6 185.75 174.14 0 179.2 187.19 175.49 0 67108864 1048576 float none -1 279.7 239.90 224.90 0 277.0 242.26 227.12 0 134217728 2097152 float none -1 507.5 264.46 247.93 0 485.1 276.66 259.37 0 268435456 4194304 float none -1 866.3 309.88 290.51 0 864.0 310.70 291.28 0 536870912 8388608 float none -1 1576.1 340.62 319.33 0 1558.2 344.54 323.01 0 1073741824 16777216 float none -1 3096.6 346.75 325.08 0 3047.5 352.33 330.31 0 2147483648 33554432 float none -1 6148.0 349.30 327.47 0 6034.3 355.88 333.64 0 4294967296 67108864 float none -1 12226 351.29 329.33 0 12000 357.92 335.55 0 8589934592 134217728 float none -1 24391 352.17 330.16 0 23920 359.11 336.67 0 # Out of bounds values : 0 OK # Avg bus bandwidth : 120.94

Test with Topology Aware Scheduling (TAS)

If you have more than two nodes, we recommend using the following test, which uses TAS. Follow the steps in the next sections to prepare and run the test on your cluster.

Set up your cluster with Jobset and the TAS plugin

  1. Install JobSet.

  2. Install the TAS plugin:

    1. Clone the container-engine-accelerators git repository:

      cd ~ git clone https://github.com/GoogleCloudPlatform/container-engine-accelerators.git

    2. Apply the TAS plugin:

      cd container-engine-accelerators/gke-topology-scheduler kubectl create configmap topology-scheduler-scripts --namespace kube-system --from-file=schedule-daemon.py=schedule-daemon.py --from-file=label-nodes-daemon.py=label-nodes-daemon.py kubectl apply -f service-account.yaml kubectl apply -f schedule-daemon.yaml kubectl apply -f label-nodes-daemon.yaml

Deploy a NCCL test workload with TAS

A4

  1. Create the following nccl-jobset-test.yaml manifest:

    apiVersion: jobset.x-k8s.io/v1alpha2 kind: JobSet metadata:  # The name `nccl-ag` is used for an NCCL all-gather test.  name: nccl-ag spec:  ttlSecondsAfterFinished: 1200  suspend: False  network:  enableDNSHostnames: true  replicatedJobs:  - name: worker  template:  spec:  parallelism: NUM_NODES  completions: NUM_NODES  template:  metadata:  annotations:  networking.gke.io/default-interface: 'eth0'  networking.gke.io/interfaces: |  [  {"interfaceName":"eth0","network":"default"},  {"interfaceName":"eth2","network":"rdma-0"},  {"interfaceName":"eth3","network":"rdma-1"},  {"interfaceName":"eth4","network":"rdma-2"},  {"interfaceName":"eth5","network":"rdma-3"},  {"interfaceName":"eth6","network":"rdma-4"},  {"interfaceName":"eth7","network":"rdma-5"},  {"interfaceName":"eth8","network":"rdma-6"},  {"interfaceName":"eth9","network":"rdma-7"}  ]  spec:  activeDeadlineSeconds: 3600  restartPolicy: Never  nodeSelector:  cloud.google.com/gke-accelerator: nvidia-b200  tolerations:  - key: cloud.google.com/gke-queued  effect: NoSchedule  value: "true"  - key: "nvidia.com/gpu"  operator: "Exists"  effect: "NoSchedule"  setHostnameAsFQDN: true  volumes:  - name: gib  hostPath:  path: /home/kubernetes/bin/gib  - name: nvidia  hostPath:  path: /home/kubernetes/bin/nvidia  - name: lib64  hostPath:  path: /lib64  - name: shared-memory  emptyDir:  medium: "Memory"  sizeLimit: 250Gi  schedulingGates:  - name: "gke.io/topology-aware-auto-nccl-test"  containers:  - name: nccl-test  stdin: true  tty: true  image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic:v1.0.6  env:  - name: MY_NODE_NAME  valueFrom:  fieldRef:  fieldPath: spec.nodeName  - name: OMPI_ALLOW_RUN_AS_ROOT  value: "1"  - name: OMPI_ALLOW_RUN_AS_ROOT_CONFIRM  value: "1"  - name: N_NODES  value: "NUM_NODES"  - name: LD_LIBRARY_PATH  value: /usr/local/nvidia/lib64  command:  - bash  - -c  - |  set -x  echo "Starting workload container on ${MY_NODE_NAME} for $N_NODES benchmark"  # Install ping  apt update -y  apt install -y iputils-ping  # Start sshd  /scripts/container_entry.sh daemon &  # Get helper variables to form all hostnames  export POSTFIX=$(hostname | cut -d . -f 2-)  export WORKERS_BASENAME=$(hostname | cut -d . -f 1 | rev | cut -d - -f 2- | rev )  export NODE_RANK=$JOB_COMPLETION_INDEX  # For every worker, wait till online and add to hostfile  for i in `seq 0 $(($N_NODES-1))`; do  OTHER=${WORKERS_BASENAME}-${i}.${POSTFIX}  until ssh -p 222 -o StrictHostKeyChecking=no $OTHER hostname; do  echo Waiting for ${OTHER}...  sleep 10  done  echo ${OTHER} port=222 slots=8 | tee -a /tmp/hostfile;  done  cat /tmp/hostfile  # Launch from head node  if [[ "${NODE_RANK}" -eq "0" ]]; then  # World Level = 0x0, Rail Aligned = 0x7  export NCCL_TESTS_SPLIT_MASK="0x0";  # Force use of libnccl-gib  export NCCL_NET=gIB  # Set all the correct libnccl-gib environment variables  source /usr/local/gib/scripts/set_nccl_env.sh  # Get all relevant NCCL / env vars to pass to all workers  ENV_VARS=$(echo ${!NCCL*} ${!OMPI*} LD_LIBRARY_PATH PATH | sed 's/ / -x /g')  mpirun --hostfile /tmp/hostfile \  -x $ENV_VARS \  -mca plm_rsh_no_tree_spawn 1 \  --mca mtl ^ofi \  --mca orte_keep_fqdn_hostnames 1 \  --mca btl self,tcp \  --mca btl_tcp_if_include eth0 \  --bind-to none \  --mca plm_rsh_agent "ssh -q -o LogLevel=ERROR -o StrictHostKeyChecking=no -p 222" \  /third_party/nccl-tests/build/all_gather_perf -b 1K -e 8G -f 2 -g 1 -w 5 --iters 100 -c 1  else  while ping -c 1 ${WORKERS_BASENAME}-0.${POSTFIX}; do  sleep 5  done  fi  exit 0  volumeMounts:  - name: nvidia  mountPath: /usr/local/nvidia  - name: gib  mountPath: /usr/local/gib  - name: shared-memory  mountPath: /dev/shm  resources:  limits:  nvidia.com/gpu: 8  requests:  nvidia.com/gpu: 8  restartPolicy: Never

    Replace NUM_NODES with the number of nodes in the node pool.

    Make sure that you understand the following about this manifest:

    • The JobSet is a headless Service with the same name as the JobSet name, in this case, nccl-ag.
    • The gke.io/topology-aware-auto-nccl-test scheduling gate is used to verify the Pods are scheduled for colocation.
    • The parallelism and completions fields are both set to the number of nodes that you want to use to run the NCCL test.

  2. Apply the manifest:

    kubectl apply -f nccl-jobset-test.yaml

  3. Confirm that the workload is admitted:

    kubectl get jobsets

    The output is similar to the following:

    NAME RESTARTS COMPLETED AGE nccl-ag 3s

  4. Confirm that the workload is in the Completed state:

    kubectl get pods

    The output is similar to the following:

    NAME READY STATUS RESTARTS AGE nccl-ag-worker-0-0-n9s6j 0/1 Completed 0 9m34s nccl-ag-worker-0-1-rsf7r 0/1 Completed 0 9m34s ...

  5. The logs of the Pod with the pattern nccl-ag-worker-0-0-.* contain the results of the test.

    Fetch the logs for this Pod:

     kubectl logs $(kubectl get pods -o go-template='{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' | grep nccl-ag-worker-0-0)

    The output should be similar to the following:

    # size count type redop root time algbw busbw #wrong time algbw busbw #wrong # (B) (elements) (us) (GB/s) (GB/s) (us) ∂ç (GB/s) (GB/s) 1024 16 float none -1 54.07 0.02 0.02 0 55.80 0.02 0.02 0 2048 32 float none -1 55.46 0.04 0.03 0 55.31 0.04 0.03 0 4096 64 float none -1 55.59 0.07 0.07 0 55.38 0.07 0.07 0 8192 128 float none -1 56.05 0.15 0.14 0 55.92 0.15 0.14 0 16384 256 float none -1 57.08 0.29 0.27 0 57.75 0.28 0.27 0 32768 512 float none -1 57.49 0.57 0.53 0 57.22 0.57 0.54 0 65536 1024 float none -1 59.20 1.11 1.04 0 59.20 1.11 1.04 0 131072 2048 float none -1 59.58 2.20 2.06 0 63.57 2.06 1.93 0 262144 4096 float none -1 63.87 4.10 3.85 0 63.61 4.12 3.86 0 524288 8192 float none -1 64.83 8.09 7.58 0 64.40 8.14 7.63 0 1048576 16384 float none -1 79.74 13.15 12.33 0 76.66 13.68 12.82 0 2097152 32768 float none -1 78.41 26.74 25.07 0 79.05 26.53 24.87 0 4194304 65536 float none -1 83.21 50.41 47.26 0 81.25 51.62 48.39 0 8388608 131072 float none -1 94.35 88.91 83.35 0 99.07 84.68 79.38 0 16777216 262144 float none -1 122.9 136.55 128.02 0 121.7 137.83 129.21 0 33554432 524288 float none -1 184.2 182.19 170.80 0 178.1 188.38 176.60 0 67108864 1048576 float none -1 294.7 227.75 213.51 0 277.7 241.62 226.52 0 134217728 2097152 float none -1 495.4 270.94 254.00 0 488.8 274.60 257.43 0 268435456 4194304 float none -1 877.5 305.92 286.80 0 861.3 311.65 292.17 0 536870912 8388608 float none -1 1589.8 337.71 316.60 0 1576.2 340.61 319.33 0 1073741824 16777216 float none -1 3105.7 345.74 324.13 0 3069.2 349.85 327.98 0 2147483648 33554432 float none -1 6161.7 348.52 326.74 0 6070.7 353.75 331.64 0 4294967296 67108864 float none -1 12305 349.03 327.22 0 12053 356.35 334.08 0 8589934592 134217728 float none -1 24489 350.77 328.85 0 23991 358.05 335.67 0 # Out of bounds values : 0 OK # Avg bus bandwidth : 120.248

A3 Ultra

  1. Create the following nccl-jobset-test.yaml manifest:

    apiVersion: jobset.x-k8s.io/v1alpha2 kind: JobSet metadata:  # The name `nccl-ag` is used for an NCCL all-gather test.  name: nccl-ag spec:  ttlSecondsAfterFinished: 1200  suspend: False  network:  enableDNSHostnames: true  replicatedJobs:  - name: worker  template:  spec:  parallelism: NUM_NODES  completions: NUM_NODES  template:  metadata:  annotations:  networking.gke.io/default-interface: 'eth0'  networking.gke.io/interfaces: |  [  {"interfaceName":"eth0","network":"default"},  {"interfaceName":"eth2","network":"rdma-0"},  {"interfaceName":"eth3","network":"rdma-1"},  {"interfaceName":"eth4","network":"rdma-2"},  {"interfaceName":"eth5","network":"rdma-3"},  {"interfaceName":"eth6","network":"rdma-4"},  {"interfaceName":"eth7","network":"rdma-5"},  {"interfaceName":"eth8","network":"rdma-6"},  {"interfaceName":"eth9","network":"rdma-7"}  ]  spec:  activeDeadlineSeconds: 3600  restartPolicy: Never  nodeSelector:  cloud.google.com/gke-accelerator: nvidia-h200-141gb  tolerations:  - key: cloud.google.com/gke-queued  effect: NoSchedule  value: "true"  - key: "nvidia.com/gpu"  operator: "Exists"  effect: "NoSchedule"  setHostnameAsFQDN: true  volumes:  - name: gib  hostPath:  path: /home/kubernetes/bin/gib  - name: nvidia  hostPath:  path: /home/kubernetes/bin/nvidia  - name: lib64  hostPath:  path: /lib64  - name: shared-memory  emptyDir:  medium: "Memory"  sizeLimit: 250Gi  schedulingGates:  - name: "gke.io/topology-aware-auto-nccl-test"  containers:  - name: nccl-test  stdin: true  tty: true  image: us-docker.pkg.dev/gce-ai-infra/gpudirect-gib/nccl-plugin-gib-diagnostic:v1.0.6  securityContext:  privileged: true  env:  - name: MY_NODE_NAME  valueFrom:  fieldRef:  fieldPath: spec.nodeName  - name: OMPI_ALLOW_RUN_AS_ROOT  value: "1"  - name: OMPI_ALLOW_RUN_AS_ROOT_CONFIRM  value: "1"  - name: N_NODES  value: "NUM_NODES"  - name: LD_LIBRARY_PATH  value: /usr/local/nvidia/lib64  command:  - bash  - -c  - |  set -x  echo "Starting workload container on ${MY_NODE_NAME} for $N_NODES benchmark"  # Install ping  apt update -y  apt install -y iputils-ping  # Start sshd  /scripts/container_entry.sh daemon &  # Get helper variables to form all hostnames  export POSTFIX=$(hostname | cut -d . -f 2-)  export WORKERS_BASENAME=$(hostname | cut -d . -f 1 | rev | cut -d - -f 2- | rev )  export NODE_RANK=$JOB_COMPLETION_INDEX  # For every worker, wait till online and add to hostfile  for i in `seq 0 $(($N_NODES-1))`; do  OTHER=${WORKERS_BASENAME}-${i}.${POSTFIX}  until ssh -p 222 -o StrictHostKeyChecking=no $OTHER hostname; do  echo Waiting for ${OTHER}...  sleep 10  done  echo ${OTHER} port=222 slots=8 | tee -a /tmp/hostfile;  done  cat /tmp/hostfile  # Launch from head node  if [[ "${NODE_RANK}" -eq "0" ]]; then  # World Level = 0x0, Rail Aligned = 0x7  export NCCL_TESTS_SPLIT_MASK="0x0";  # Force use of libnccl-gib  export NCCL_NET=gIB  # Set all the correct libnccl-gib environment variables  source /usr/local/gib/scripts/set_nccl_env.sh  # Get all relevant NCCL / env vars to pass to all workers  ENV_VARS=$(echo ${!NCCL*} ${!OMPI*} LD_LIBRARY_PATH PATH | sed 's/ / -x /g')  mpirun --hostfile /tmp/hostfile \  -x $ENV_VARS \  -mca plm_rsh_no_tree_spawn 1 \  --mca orte_keep_fqdn_hostnames 1 \  --mca btl self,tcp \  --mca btl_tcp_if_include eth0 \  --bind-to none \  --mca plm_rsh_agent "ssh -q -o LogLevel=ERROR -o StrictHostKeyChecking=no -p 222" \  /third_party/nccl-tests/build/all_gather_perf -b 1K -e 8G -f 2 -g 1 -w 5 --iters 100 -c 1  else  while ping -c 1 ${WORKERS_BASENAME}-0.${POSTFIX}; do  sleep 5  done  fi  exit 0  volumeMounts:  - name: nvidia  mountPath: /usr/local/nvidia  - name: gib  mountPath: /usr/local/gib  - name: shared-memory  mountPath: /dev/shm  resources:  limits:  nvidia.com/gpu: 8  requests:  nvidia.com/gpu: 8  restartPolicy: Never

    Replace NUM_NODES with the number of nodes in the node pool.

    Make sure that you understand the following about this manifest:

    • The JobSet is a Headless Service with the same name as the JobSet name, in this case, nccl-ag.
    • The gke.io/topology-aware-auto-nccl-test scheduling gate is used to verify the Pods are scheduled for colocation.
    • The parallelism and completions fields are both set to the number of nodes that you want to use to run the NCCL test.

  2. Apply the manifest:

    kubectl apply -f nccl-jobset-test.yaml

  3. Confirm that the workload is admitted:

    kubectl get jobsets

    The output is similar to the following:

    NAME RESTARTS COMPLETED AGE nccl-ag 3s

  4. Confirm that the workload is in the Completed state:

    kubectl get pods

    The output is similar to the following:

    NAME READY STATUS RESTARTS AGE nccl-ag-worker-0-0-n9s6j 0/1 Completed 0 9m34s nccl-ag-worker-0-1-rsf7r 0/1 Completed 0 9m34s ...

  5. The logs of the Pod with the pattern nccl-ag-worker-0-0-.* contain the results of the test.

    Fetch the logs for this Pod:

     kubectl logs $(kubectl get pods -o go-template='{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' | grep nccl-ag-worker-0-0)

    The output should be similar to the following:

     # size count type redop root time algbw busbw #wrong time algbw busbw #wrong # (B) (elements) (us) (GB/s) (GB/s) (us) ∂ç (GB/s) (GB/s) 1024 16 float none -1 54.07 0.02 0.02 0 55.80 0.02 0.02 0 2048 32 float none -1 55.46 0.04 0.03 0 55.31 0.04 0.03 0 4096 64 float none -1 55.59 0.07 0.07 0 55.38 0.07 0.07 0 8192 128 float none -1 56.05 0.15 0.14 0 55.92 0.15 0.14 0 16384 256 float none -1 57.08 0.29 0.27 0 57.75 0.28 0.27 0 32768 512 float none -1 57.49 0.57 0.53 0 57.22 0.57 0.54 0 65536 1024 float none -1 59.20 1.11 1.04 0 59.20 1.11 1.04 0 131072 2048 float none -1 59.58 2.20 2.06 0 63.57 2.06 1.93 0 262144 4096 float none -1 63.87 4.10 3.85 0 63.61 4.12 3.86 0 524288 8192 float none -1 64.83 8.09 7.58 0 64.40 8.14 7.63 0 1048576 16384 float none -1 79.74 13.15 12.33 0 76.66 13.68 12.82 0 2097152 32768 float none -1 78.41 26.74 25.07 0 79.05 26.53 24.87 0 4194304 65536 float none -1 83.21 50.41 47.26 0 81.25 51.62 48.39 0 8388608 131072 float none -1 94.35 88.91 83.35 0 99.07 84.68 79.38 0 16777216 262144 float none -1 122.9 136.55 128.02 0 121.7 137.83 129.21 0 33554432 524288 float none -1 184.2 182.19 170.80 0 178.1 188.38 176.60 0 67108864 1048576 float none -1 294.7 227.75 213.51 0 277.7 241.62 226.52 0 134217728 2097152 float none -1 495.4 270.94 254.00 0 488.8 274.60 257.43 0 268435456 4194304 float none -1 877.5 305.92 286.80 0 861.3 311.65 292.17 0 536870912 8388608 float none -1 1589.8 337.71 316.60 0 1576.2 340.61 319.33 0 1073741824 16777216 float none -1 3105.7 345.74 324.13 0 3069.2 349.85 327.98 0 2147483648 33554432 float none -1 6161.7 348.52 326.74 0 6070.7 353.75 331.64 0 4294967296 67108864 float none -1 12305 349.03 327.22 0 12053 356.35 334.08 0 8589934592 134217728 float none -1 24489 350.77 328.85 0 23991 358.05 335.67 0 # Out of bounds values : 0 OK # Avg bus bandwidth : 120.248 ```

What's next