8

I setup my CD with following commands to deploy container to gce:

gcloud compute instances stop my-awesome-app gcloud compute instances update my-awesome-app --no-shielded-integrity-monitoring gcloud beta compute instances update-container my-awesome-app --container-image=docker.io/my-docker-hub-user/my-awesome-app:${IMAGE_TAG} gcloud compute instances start my-awesome-app 

The instance will then stop, replace container image and start.

If I create new instance with the latest image, everything works fine.

If I deploy with my CD pipeline, the instance ALWAYS hang forever (after receiving start event, no new logging is coming), NEVER pulls the new image, and NEVER goes up and running.

Anyone can tell me what's the problem?

3 Answers 3

19

I don't start and stop my instance to update the container. I just:

gcloud compute instances update-container $instanceName2 --zone $instanceZone --container-image $registry:$BUILD_ID 

But before pulling a new image I always do a docker prune to avoid my disk getting filled with images:

gcloud compute ssh $instanceName2 --zone $instanceZone --command 'docker system prune -f -a' 

For me, a better way was to use a template pointing to my repository, and from that template to build a Managed-Instance-group. This way you can have zero downtime deployments and still have only one machine up 99% of the time. Details

Sign up to request clarification or add additional context in comments.

Comments

2

I hope the document [1] is helpful for you from where you could have steps for updating a container on a VM in Google Cloud.

When you want to update a VM running a container, Compute Engine performs below two steps:

  1. Updates container declaration on the instance. Compute Engine stores the updated container declaration in instance metadata under the gce-container-declaration metadata key.
  2. Stops and restarts the instance to actuate the updated configuration, if the instance is running. If the instance is stopped, updates the container declaration and keeps the instance stopped. The VM instance downloads the new image and launches the container on VM start.

Please follow 1 to 5 steps as the document [1], Compute Engine saves the changes and restarts the instance automatically to make the update. After the VM restarts, it downloads the new image and starts the container with the updated configuration.

[1] https://cloud.google.com/compute/docs/containers/deploying-containers#updating_a_container_on_a_vm_instance

Comments

2

Thanks for above answers!

I finally figured out my own way. 1. Every deploy will create a new managed instance with container. 2. Then my reserved IP address is pointed to the new instance. (So I don't need to update DNS) 3. Finally, shutdown the old deployment when possible.

The downside is obviously there will be down time during deploying, but I think the deploy pipeline is good enough for my current project.

# install dependencies apt-get --yes install jq # Create new instance NEW_INSTANCE="$INSTANCE_NAME_PREFIX-build-$CIRCLE_BUILD_NUM" gcloud beta compute instances create-with-container "$NEW_INSTANCE" \ --machine-type=f1-micro \ --container-image=docker.io/xxx/yyy:$IMAGE_TAG \ --container-env="ANSIBLE_VAULT_PASSWORD=${ANSIBLE_VAULT_PASSWORD}" \ --tags http-server,https-server \ --zone ${GOOGLE_COMPUTE_ZONE} IP_ADDRESS=$(gcloud compute addresses list --filter="name=('$RESERVED_IP_NAME')" --format=json | jq '.[0].address' --raw-output) OLD_INSTANCE=$(gcloud compute instances list --filter="EXTERNAL_IP=('$IP_ADDRESS')" --zones asia-east1-a --format=json | jq '.[0].name' --raw-output) # Remove ephemeral IP address from new instance echo "Removing ephemeral IP address from instance: $NEW_INSTANCE" gcloud compute instances delete-access-config "$NEW_INSTANCE" \ --access-config-name "external-nat" \ --zone ${GOOGLE_COMPUTE_ZONE} # Remove reserved IP address from old instance # Ignore error if there is no access config present if [ "$OLD_INSTANCE" != "null" ]; then echo "Removing reserved IP address from instance: $OLD_INSTANCE" gcloud compute instances delete-access-config "$OLD_INSTANCE" \ --access-config-name "external-nat" \ --zone ${GOOGLE_COMPUTE_ZONE} || true fi # Assign reserved IP address to new instance gcloud compute instances add-access-config "$NEW_INSTANCE" \ --access-config-name "external-nat" --address "$IP_ADDRESS" \ --zone ${GOOGLE_COMPUTE_ZONE} # Shutdown old instance if [ "$OLD_INSTANCE" != "null" ]; then gcloud compute instances stop "$OLD_INSTANCE" --zone ${GOOGLE_COMPUTE_ZONE} fi 

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.