Insufficient CPU

Symptom

When starting up, the telemetry pods go in and out of CrashLoopBackoff state. This can cause periodic gaps in your metrics or graphs as the pods restart. You could also see discrepancies with analytics data as some sections of data are missing.

Error messages

When you use kubectl to view the pod states, you will see one or more metric pods in the CrashLoopBackoff state. Refer to the following command:

kubectl get pods -n APIGEE_NAMESPACE

Where APIGEE_NAMESPACE is the Kubernetes namespace for your Apigee hybrid components. For more information, see Create the apigee namespace.

Sample Output

NAME READY STATUS RESTARTS AGE apigee-metrics-default-telemetry-proxy-1104-hvwoo-zlmlw 0/1 CrashLoopBackoff 10 10m apigee-metrics-adapter-apigee-telemetry-1104-7fyff-tts65 0/1 CrashLoopBackoff 10 10m apigee-metrics-default-telemetry-proxy-1104-hvwoo-zlmlw 0/1 FailedScheduling 0 12m

Common diagnosis steps

1. Check the events for issues with telemetry pods with the following command:

   kubectl -n apigee get event 

   Sample Output

   LAST SEEN TYPE REASON OBJECT MESSAGE 53m Normal SuccessfulCreate job/apigee-cassandra-schema-val-jghunt-20250709-0820206-29251940 Created pod: apigee-cassandra-schema-val-jghunt-20250709-0820206-292519fkt7j 53m Normal Completed job/apigee-cassandra-schema-val-jghunt-20250709-0820206-29251940 Job completed 43m Normal SuccessfulCreate job/apigee-cassandra-schema-val-jghunt-20250709-0820206-29251950 Created pod: apigee-cassandra-schema-val-jghunt-20250709-0820206-292519l87m8 43m Normal Completed job/apigee-cassandra-schema-val-jghunt-20250709-0820206-29251950 Job completed 33m Normal SuccessfulCreate job/apigee-cassandra-schema-val-jghunt-20250709-0820206-29251960 Created pod: apigee-cassandra-schema-val-jghunt-20250709-0820206-29251962ncc 

2. You can also check the events of telemetry pods with a CrashLoopBackOff state using the following command:

   kubectl -n apigee describe POD_NAME

   Where POD_NAME is the name of the pod that is in a CrashLoopBackOff state.

   Sample Output

    apigee-metrics-apigee-telemetry-app-1101-qc36n-dxzrv 

3. You can also check the cpu status of the pods with the following command:

   kubectl -n apigee get hpa | grep unknown

   Sample Output

   apigee-metrics-apigee-telemetry-app-1101-qc36n-dxzrv ReplicaSet/apigee-metrics-apigee-telemetry-app-1101-qc36n-dxzrv /80% 2 10 2 8h 

Possible causes

Cause

cpu is not mentioned in the overrides.yaml file, so cpu gets an undefined value.

Diagnosis

Check your overrides.yaml file to see if both cpu values are defined for metrics.app.resources.requests.cpu and metrics.app.resources.limits.cpu.

Resolution

If cpu settings are missing in your overrides.yaml file for metrics, provide both cpu values in the overrides.yaml file.

1. Add the following configuration under the metrics section in your overrides.yaml file:

   metrics:  app: # The apigee-prometheus-app container in the "app" pod  resources:  requests:  memory: 512Mi # Default value: 512Mi  cpu: 500m # Default value: 500m  limits:  memory: 2Gi # default: 1Gi  cpu: 500m # Default value: 500m  

2. Apply changes using the following command:

   helm upgrade ENV_RELEASE_NAME apigee-env/ \ --install \ --namespace APIGEE_NAMESPACE \ --set env=ENV_NAME \ -f OVERRIDES_FILE

• Where ENV_RELEASE_NAME is a unique name used to track installation and upgrade of the apigee-env chart. While it's typically the same as the ENV_NAME, it must be different if your environment has the same name as your environment group. For example, if both are named dev, you would use dev-env-release and dev-envgroup-release to distinguish them.

• Where APIGEE_NAMESPACE is the Kubernetes namespace for your Apigee hybrid components. For more information, see Create the apigee namespace.

• Where ENV_NAME is the name you used when you created the environment in the UI.

• Where OVERRIDES_FILE is the overrides.yaml file that is used during upgrades or install.

Must gather diagnostic information

If the problem persists even after following the above instructions, gather the following diagnostic information and then contact Google Cloud Customer Care:

1. The overrides.yaml file.
2. The output from the Apigee hybrid must-gather script.