volume leaks when PVC is deleted when associated PV is in the terminating state #546
Description
Steps
- Create PVC and wait for it to get bound.
# kubectl create -f pvc.yaml persistentvolumeclaim/example-vanilla-block-pvc created # kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE example-vanilla-block-pvc Bound pvc-6791fdd4-5fad-438e-a7fb-16410363e3da 5Gi RWO example-vanilla-block-sc 19s # kubectl get pv pvc-6791fdd4-5fad-438e-a7fb-16410363e3da NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-6791fdd4-5fad-438e-a7fb-16410363e3da 5Gi RWO Delete Bound default/example-vanilla-block-pvc example-vanilla-block-sc 23s - Delete PV.
# kubectl delete pv pvc-6791fdd4-5fad-438e-a7fb-16410363e3da persistentvolume "pvc-6791fdd4-5fad-438e-a7fb-16410363e3da" deleted ^C PV can not be deleted, so kubectl delete pv command is not giving up terminal prompt. So I typed ctl+c to exit.
Checked PV status. It went into terminating state.
# kubectl get pv pvc-6791fdd4-5fad-438e-a7fb-16410363e3da NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-6791fdd4-5fad-438e-a7fb-16410363e3da 5Gi RWO Delete Terminating default/example-vanilla-block-pvc example-vanilla-block-sc 2m23s - When PV is in the terminating state, delete PVC.
# kubectl delete pvc example-vanilla-block-pvc persistentvolumeclaim "example-vanilla-block-pvc" deleted - PV and PVC are both deleted from the system.
# kubectl get pv pvc-6791fdd4-5fad-438e-a7fb-16410363e3da Error from server (NotFound): persistentvolumes "pvc-6791fdd4-5fad-438e-a7fb-16410363e3da" not found The controller is not getting calls to delete the volume from the datastore. This results into leaking volume on the datastore.
Above workflow is executed using very latest external-provisioner - v2.1.0. This issue is also present on prior release of the external provisioner.
Filing this bug to discuss about what fix we can make to prevent this orphan volume on the system.
This issue was also discussed here - #195 (comment)