VM Snapshot: Prevent vm snapshots being indefinitely stuck in Expunging state on deletion failure by Pearl1594 · Pull Request #4898 · apache/cloudstack

Pearl1594 · 2021-04-07T10:29:01Z

Description

Fixes #4201

This PR addresses the issue of a vm snapshot being indefinitely stuck is Expunging state in case deletion fails.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Simulated an exception at the hypervisor end while deleting a VM snapshot and verified that the vm snapshot goes back to Ready state.

…tate on deletion failure

Pearl1594 · 2021-04-07T10:29:17Z

@blueorangutan package

Pearl1594 · 2021-04-07T11:26:45Z

@blueorangutan package

blueorangutan · 2021-04-07T11:27:03Z

@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-04-07T12:05:12Z

Packaging result: ✔️ centos7 ✔️ centos8 ✔️ debian. SL-JID 350

Pearl1594 · 2021-04-07T12:08:52Z

@blueorangutan test

blueorangutan · 2021-04-07T12:10:03Z

@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

DaanHoogland

one question

DaanHoogland · 2021-04-07T15:12:23Z

-        } catch (OperationTimedoutException e) {
-            throw new CloudRuntimeException("Delete vm snapshot " + vmSnapshot.getName() + " of vm " + userVm.getInstanceName() + " failed due to " + e.getMessage());
-        } catch (AgentUnavailableException e) {
+        } catch (OperationTimedoutException | AgentUnavailableException e) {


blueorangutan · 2021-04-07T22:31:39Z

Trillian test result (tid-373)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 35559 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4898-t373-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Smoke tests completed. 85 look OK, 2 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestKubernetesCluster>:teardown	`Error`	78.16	test_kubernetes_clusters.py
test_01_migrate_VM_and_root_volume	`Error`	69.16	test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks	`Error`	53.09	test_vm_life_cycle.py

Pearl1594 · 2021-04-08T06:28:58Z

@blueorangutan package

blueorangutan · 2021-04-08T06:30:03Z

@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-04-08T07:04:23Z

Packaging result: ✔️ centos7 ✔️ centos8 ✔️ debian. SL-JID 360

yadvr

Didn't test but lgtm

yadvr · 2021-04-09T06:24:39Z

@blueorangutan test centos7 vmware-67u3

blueorangutan · 2021-04-09T06:25:02Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + vmware-67u3) has been kicked to run smoke tests

harikrishna-patnala

Testes with the PR changes. Observed VM snapshots in error state upon expunge failure and later I could delete the snapshot which is in error state.
LGTM

yadvr · 2021-04-10T07:57:58Z

@blueorangutan test centos7 vmware-67u3

blueorangutan · 2021-04-10T07:59:03Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + vmware-67u3) has been kicked to run smoke tests

blueorangutan · 2021-04-10T18:01:48Z

Trillian test result (tid-401)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
Total time taken: 35087 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4898-t401-vmware-67u3.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Smoke tests completed. 85 look OK, 2 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_deploy_kubernetes_cluster	`Failure`	87.37	test_kubernetes_clusters.py
test_02_invalid_upgrade_kubernetes_cluster	`Failure`	53.55	test_kubernetes_clusters.py
test_03_deploy_and_upgrade_kubernetes_cluster	`Failure`	82.88	test_kubernetes_clusters.py
test_04_deploy_and_scale_kubernetes_cluster	`Failure`	53.26	test_kubernetes_clusters.py
test_05_delete_kubernetes_cluster	`Failure`	55.24	test_kubernetes_clusters.py
test_07_deploy_kubernetes_ha_cluster	`Failure`	56.31	test_kubernetes_clusters.py
test_08_deploy_and_upgrade_kubernetes_ha_cluster	`Failure`	68.55	test_kubernetes_clusters.py
test_09_delete_kubernetes_ha_cluster	`Failure`	56.27	test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardown	`Error`	100.73	test_kubernetes_clusters.py
ContextSuite context=TestVAppsVM>:setup	`Error`	79.84	test_vm_life_cycle.py

VM Snapshot: Prevent snapshot indefinitely being stuck in Expunging s…

ae770af

…tate on deletion failure

Pearl1594 requested review from DaanHoogland, shwstppr and yadvr April 7, 2021 10:29

Pearl1594 changed the title ~~VM Snapshot: Prevent snapshot indefinitely being stuck in Expunging state on deletion failure~~ VM Snapshot: Prevent vm snapshots being indefinitely stuck in Expunging state on deletion failure Apr 7, 2021

yadvr added this to the 4.15.1.0 milestone Apr 7, 2021

DaanHoogland reviewed Apr 7, 2021

View reviewed changes

Change state transition on vm snap deletion failure

06bf5f9

yadvr approved these changes Apr 9, 2021

View reviewed changes

Pearl1594 marked this pull request as ready for review April 9, 2021 07:01

yadvr requested review from DaanHoogland and harikrishna-patnala April 9, 2021 07:53

DaanHoogland approved these changes Apr 9, 2021

View reviewed changes

harikrishna-patnala approved these changes Apr 9, 2021

View reviewed changes

shwstppr approved these changes Apr 9, 2021

View reviewed changes

apache deleted a comment from blueorangutan Apr 10, 2021

yadvr mentioned this pull request Apr 10, 2021

VM snapshot deletion failure may put it indefinitely in expunging state #4201

Closed

yadvr merged commit a64ad9d into apache:4.15 Apr 12, 2021

Uh oh!

Conversation

Pearl1594 commented Apr 7, 2021

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

Pearl1594 commented Apr 7, 2021

Uh oh!

Pearl1594 commented Apr 7, 2021

Uh oh!

blueorangutan commented Apr 7, 2021

Uh oh!

blueorangutan commented Apr 7, 2021

Uh oh!

Pearl1594 commented Apr 7, 2021

Uh oh!

blueorangutan commented Apr 7, 2021

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

DaanHoogland Apr 7, 2021

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Apr 7, 2021

Uh oh!

Pearl1594 commented Apr 8, 2021

Uh oh!

blueorangutan commented Apr 8, 2021

Uh oh!

blueorangutan commented Apr 8, 2021

Uh oh!

yadvr left a comment

Choose a reason for hiding this comment

Uh oh!

yadvr commented Apr 9, 2021

Uh oh!

blueorangutan commented Apr 9, 2021

Uh oh!

harikrishna-patnala left a comment

Choose a reason for hiding this comment

Uh oh!

yadvr commented Apr 10, 2021

Uh oh!

blueorangutan commented Apr 10, 2021

Uh oh!

blueorangutan commented Apr 10, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants