12

we updated our google app engine health checks from the legacy version to the new version using and now our deployments are failing. Nothing else on the project has changed. We tested the default settings and then extended checks just in case.

This is the error: ERROR: (gcloud.app.deploy) Error Response: [4] Your deployment has failed to become healthy in the allotted time and therefore was rolled back. If you believe this was an error, try adjusting the 'app_start_timeout_sec' setting in the 'readiness_check' section.

This is our app.yaml:

liveness_check: check_interval_sec: 120 timeout_sec: 40 failure_threshold: 5 success_threshold: 5 initial_delay_sec: 500 readiness_check: check_interval_sec: 120 timeout_sec: 40 failure_threshold: 5 success_threshold: 5 app_start_timeout_sec: 1500

Unfortunately, no matter the configuration, both the readiness and liveness checks are throwing 404s.

What could be causing the problem? and how can we debug this? Is it possible to rollback to the legacy health checks?

Improve this question
4
  • Update: The updated health checks are causing the problem. We just changed our app.yaml to use legacy health checks and we were able to deploy without a hitch. Commented Sep 9, 2017 at 5:05
  • How do you go back to using legacy healthchecks? The command gcloud beta app update --split-health-checks only goes one way. Commented Sep 25, 2017 at 18:57
  • For now (until 2018-09-30) you can roll back to the old healthchecks by running gcloud app update --no-split-health-checks and then redeploying. Beware that old healthchecks will be going away at the end of September 2018 though. Source: cloud.google.com/appengine/docs/flexible/python/… Commented Dec 20, 2017 at 9:57
  • similar issue with new deployment! the option to revert to legacy health checks doesn't exist for new deployments and it seems health checks are a persistent issue - raised it here again: stackoverflow.com/questions/62735687/… Commented Jul 5, 2020 at 0:26

3 Answers 3

Reset to default
7

This is usually caused when the application is still reading from the legacy health check flags and/or deploying the app using gcloud app deploy without enabling the updated health checks first. You can check this by:

1- Making sure the legacy health_check flag does not exist on your app.yaml.

2- Run gcloud beta app describe to see whether splitHealthChecks flag is set to true under featureSettings.

By default, HTTP requests from updated health checks are not forwarded to your application container. If you want to extend health checks to your application, then specify a path for liveness checks or readiness checks.

You can then enable updated health checks by using gcloud beta app update --split-health-checks --project [your-project-id]. See this public issue tracker or this article about Updated Health Checks about for more details.

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

What if you have two apps under the same project? I have an app.yaml that runs on GAE classic and a vm.yaml that runs on flexible. How does gcloud beta app update affect those?
@speedplane just specify the regular, old health checks in your app.yaml. That's what I did to make it work. health_check: enable_health_check: True check_interval_sec: 5 timeout_sec: 4 unhealthy_threshold: 2 healthy_threshold: 2
The split-health-checks setting is project-wide. It doesn't apply to App Engine Standard though, only Flexible Environment. cloud.google.com/appengine/docs/flexible/python/…
Hi Kenworth, what do you mean by "By default, HTTP requests from updated health checks are not forwarded to your application container. If you want to extend health checks to your application, then specify a path for liveness checks or readiness checks.", how do I "specify a path for liveness checks or readiness checks" ?
6

In my case, I solved this issue by manually increasing memory allocation?

resources: cpu: 1 memory_gb: 2 disk_size_gb: 10

Found this solution in a google forum: https://groups.google.com/forum/#!topic/google-appengine/Po_-SkC5DOE

Improve this answer

1 Comment

After spending an hours or so playing with health check settings and digging through the log files, this solution worked instantly.
0

For those of you who want to migrate to the default settings for splitted health checks, follow these steps:

1) Remove health_check, liveness_check and readiness_check sections from your app.yaml file

2) Deploy to a newer version, This is important. So, for example, if your current version is production, change it to something else like prod in the command gcloud app deploy --version [new-version-name]

Improve this answer

2 Comments

if above doesn't work, run this command as well gcloud app update --split-health-checks --project [YOUR_PROJECT_ID]
Most people will get here because it doesn't work with the default settings

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.