server: fix finding pools for volume migration by shwstppr · Pull Request #4693 · apache/cloudstack

shwstppr · 2021-02-15T12:42:20Z

Description

While finding pools for volume migration list following compatible storages:

all zone-wide storages of the same hypervisor.
when the volume is attached to a VM, then all storages from the same cluster as that of VM.
for detached volume, all storages that belong to clusters of the same hypervisor.

Fixes #4692
Fixes #4400

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Using cmk and UI.
Tested with a multi-cluster VMware env.
Storage Pools,

(localcloud) SBCM5> > list storagepools filter=id,name,scope,clustername,tags,hypervisor
{
  "count": 8,
  "storagepool": [
    {
      "hypervisor": "VMware",
      "id": "53a4a81b-0271-3989-94d1-a00708fd9c32",
      "name": "ps8",
      "scope": "ZONE",
      "tags": "tag1"
    },
    {
      "hypervisor": "VMware",
      "id": "17899b5b-1838-3e8d-b0fe-76cab34093b1",
      "name": "ps7",
      "scope": "ZONE"
    },
    {
      "clustername": "10.10.3.184/Trillian/p1-c3",
      "id": "98a0e79c-f879-36e2-b9ce-230f66e88698",
      "name": "ps6",
      "scope": "CLUSTER",
      "tags": "tag1"
    },
    {
      "clustername": "10.10.3.184/Trillian/p1-c3",
      "id": "b9aa5fb3-ce0a-316c-a2bb-3dbf905082d4",
      "name": "ps5",
      "scope": "CLUSTER"
    },
    {
      "clustername": "10.10.3.184/Trillian/p1-c2",
      "id": "79f5ea41-e6dd-3f9d-acca-10267cf7b796",
      "name": "ps4",
      "scope": "CLUSTER",
      "tags": "tag1"
    },
    {
      "clustername": "10.10.3.184/Trillian/p1-c2",
      "id": "0a74f7c2-63f6-374f-b5df-bc6304d0ff58",
      "name": "ps2",
      "scope": "CLUSTER"
    },
    {
      "clustername": "p1-c1",
      "id": "e777166e-7dbe-340d-977d-a51d5349b5f1",
      "name": "ps2",
      "scope": "CLUSTER",
      "tags": "tag1"
    },
    {
      "clustername": "p1-c1",
      "id": "b2888e3e-b1e2-36d9-9936-861bc6901d3f",
      "name": "ps1",
      "scope": "CLUSTER"
    }
  ]
}

Disk offerings used for data disks,

(localcloud) SBCM5> > list diskofferings id=bc71e51d-b1fc-4311-87d6-23b366274038 filter=id,name,tags
{
  "count": 1,
  "diskoffering": [
    {
      "id": "bc71e51d-b1fc-4311-87d6-23b366274038",
      "name": "Tagged",
      "tags": "tag1"
    }
  ]
}
(localcloud) SBCM5> > list diskofferings id=310f053a-1c13-47fe-8590-ef3dd1f74e87 filter=id,name,tags
{
  "count": 1,
  "diskoffering": [
    {
      "id": "310f053a-1c13-47fe-8590-ef3dd1f74e87",
      "name": "Small"
    }
  ]
}

Disks tested,

Detached volume with tagged offering on a zone-wide pool
Detached volume with untagged offering on cluster scope pool
Volume attached to a VM with tagged offering on a zone-wide pool
Volume attached to a VM with untagged offering on a cluster scope pool
Root volume of a VM with untagged compute offering on a cluster scope pool
Detached volume with tagged offering on a cluster scope pool

(localcloud) SBCM5> > list volumes ids=73d242e1-f8fb-4447-8c78-a1e10d106694,7af032e8-a2b3-49bf-b280-45884a0aed90,4f7e80f1-f3b7-4907-9437-0cc02a14d615,65f5248e-dd27-42c8-83e2-bc1e0b781d27,db7e952a-507f-45f8-b113-724f0f4cc9fc,e665c777-7339-4604-9535-2564c23abca8 filter=id,name,virtualmachineid,vmname,diskofferingid,diskofferingname,storageid,storage,hypervisor
{
  "count": 6,
  "volume": [
    {
      "diskofferingid": "bc71e51d-b1fc-4311-87d6-23b366274038",
      "diskofferingname": "Tagged",
      "hypervisor": "VMware",
      "id": "db7e952a-507f-45f8-b113-724f0f4cc9fc",
      "name": "DATA-5-3",
      "storage": "ps8",
      "storageid": "53a4a81b-0271-3989-94d1-a00708fd9c32"
    },
    {
      "diskofferingid": "310f053a-1c13-47fe-8590-ef3dd1f74e87",
      "diskofferingname": "Small",
      "hypervisor": "VMware",
      "id": "65f5248e-dd27-42c8-83e2-bc1e0b781d27",
      "name": "DATA-5-2",
      "storage": "ps5",
      "storageid": "b9aa5fb3-ce0a-316c-a2bb-3dbf905082d4"
    },
    {
      "diskofferingid": "bc71e51d-b1fc-4311-87d6-23b366274038",
      "diskofferingname": "Tagged",
      "hypervisor": "VMware",
      "id": "4f7e80f1-f3b7-4907-9437-0cc02a14d615",
      "name": "DATA-5-1",
      "storage": "ps8",
      "storageid": "53a4a81b-0271-3989-94d1-a00708fd9c32",
      "virtualmachineid": "c893649d-cc11-43f6-81a2-a96a1c0cb9a8",
      "vmname": "t1"
    },
    {
      "diskofferingid": "310f053a-1c13-47fe-8590-ef3dd1f74e87",
      "diskofferingname": "Small",
      "hypervisor": "VMware",
      "id": "7af032e8-a2b3-49bf-b280-45884a0aed90",
      "name": "DATA-5",
      "storage": "ps5",
      "storageid": "b9aa5fb3-ce0a-316c-a2bb-3dbf905082d4",
      "virtualmachineid": "c893649d-cc11-43f6-81a2-a96a1c0cb9a8",
      "vmname": "t1"
    },
    {
      "hypervisor": "VMware",
      "id": "73d242e1-f8fb-4447-8c78-a1e10d106694",
      "name": "ROOT-5",
      "storage": "ps5",
      "storageid": "b9aa5fb3-ce0a-316c-a2bb-3dbf905082d4",
      "virtualmachineid": "c893649d-cc11-43f6-81a2-a96a1c0cb9a8",
      "vmname": "t1"
    },
    {
      "diskofferingid": "bc71e51d-b1fc-4311-87d6-23b366274038",
      "diskofferingname": "Tagged",
      "hypervisor": "VMware",
      "id": "e665c777-7339-4604-9535-2564c23abca8",
      "name": "DATA-5-4",
      "storage": "ps6",
      "storageid": "98a0e79c-f879-36e2-b9ce-230f66e88698"
    }
  ]
}

findStoragePoolForMigration results in above order:

(localcloud) SBCM5> > find storagepoolsformigration id=db7e952a-507f-45f8-b113-724f0f4cc9fc filter=id,name,scope,clustername,tags,hypervisor,suitableformigration
{
  "count": 7,
  "storagepool": [
    {
      "clustername": "p1-c1",
      "id": "e777166e-7dbe-340d-977d-a51d5349b5f1",
      "name": "ps2",
      "scope": "CLUSTER",
      "suitableformigration": true,
      "tags": "tag1"
    },
    {
      "clustername": "10.10.3.184/Trillian/p1-c2",
      "id": "79f5ea41-e6dd-3f9d-acca-10267cf7b796",
      "name": "ps4",
      "scope": "CLUSTER",
      "suitableformigration": true,
      "tags": "tag1"
    },
    {
      "clustername": "10.10.3.184/Trillian/p1-c3",
      "id": "98a0e79c-f879-36e2-b9ce-230f66e88698",
      "name": "ps6",
      "scope": "CLUSTER",
      "suitableformigration": true,
      "tags": "tag1"
    },
    {
      "hypervisor": "VMware",
      "id": "17899b5b-1838-3e8d-b0fe-76cab34093b1",
      "name": "ps7",
      "scope": "ZONE",
      "suitableformigration": false
    },
    {
      "clustername": "p1-c1",
      "id": "b2888e3e-b1e2-36d9-9936-861bc6901d3f",
      "name": "ps1",
      "scope": "CLUSTER",
      "suitableformigration": false
    },
    {
      "clustername": "10.10.3.184/Trillian/p1-c2",
      "id": "0a74f7c2-63f6-374f-b5df-bc6304d0ff58",
      "name": "ps2",
      "scope": "CLUSTER",
      "suitableformigration": false
    },
    {
      "clustername": "10.10.3.184/Trillian/p1-c3",
      "id": "b9aa5fb3-ce0a-316c-a2bb-3dbf905082d4",
      "name": "ps5",
      "scope": "CLUSTER",
      "suitableformigration": false
    }
  ]
}

(localcloud) SBCM5> > find storagepoolsformigration id=65f5248e-dd27-42c8-83e2-bc1e0b781d27 filter=id,name,scope,clustername,tags,hypervisor,suitableformigration
{
  "count": 7,
  "storagepool": [
    {
      "clustername": "p1-c1",
      "id": "b2888e3e-b1e2-36d9-9936-861bc6901d3f",
      "name": "ps1",
      "scope": "CLUSTER",
      "suitableformigration": true
    },
    {
      "clustername": "p1-c1",
      "id": "e777166e-7dbe-340d-977d-a51d5349b5f1",
      "name": "ps2",
      "scope": "CLUSTER",
      "suitableformigration": true,
      "tags": "tag1"
    },
    {
      "clustername": "10.10.3.184/Trillian/p1-c2",
      "id": "0a74f7c2-63f6-374f-b5df-bc6304d0ff58",
      "name": "ps2",
      "scope": "CLUSTER",
      "suitableformigration": true
    },
    {
      "clustername": "10.10.3.184/Trillian/p1-c2",
      "id": "79f5ea41-e6dd-3f9d-acca-10267cf7b796",
      "name": "ps4",
      "scope": "CLUSTER",
      "suitableformigration": true,
      "tags": "tag1"
    },
    {
      "clustername": "10.10.3.184/Trillian/p1-c3",
      "id": "98a0e79c-f879-36e2-b9ce-230f66e88698",
      "name": "ps6",
      "scope": "CLUSTER",
      "suitableformigration": true,
      "tags": "tag1"
    },
    {
      "hypervisor": "VMware",
      "id": "17899b5b-1838-3e8d-b0fe-76cab34093b1",
      "name": "ps7",
      "scope": "ZONE",
      "suitableformigration": true
    },
    {
      "hypervisor": "VMware",
      "id": "53a4a81b-0271-3989-94d1-a00708fd9c32",
      "name": "ps8",
      "scope": "ZONE",
      "suitableformigration": true,
      "tags": "tag1"
    }
  ]
}

(localcloud) SBCM5> > find storagepoolsformigration id=4f7e80f1-f3b7-4907-9437-0cc02a14d615 filter=id,name,scope,clustername,tags,hypervisor,suitableformigration
{
  "count": 3,
  "storagepool": [
    {
      "clustername": "10.10.3.184/Trillian/p1-c3",
      "id": "98a0e79c-f879-36e2-b9ce-230f66e88698",
      "name": "ps6",
      "scope": "CLUSTER",
      "suitableformigration": true,
      "tags": "tag1"
    },
    {
      "hypervisor": "VMware",
      "id": "17899b5b-1838-3e8d-b0fe-76cab34093b1",
      "name": "ps7",
      "scope": "ZONE",
      "suitableformigration": false
    },
    {
      "clustername": "10.10.3.184/Trillian/p1-c3",
      "id": "b9aa5fb3-ce0a-316c-a2bb-3dbf905082d4",
      "name": "ps5",
      "scope": "CLUSTER",
      "suitableformigration": false
    }
  ]
}

(localcloud) SBCM5> > find storagepoolsformigration id=7af032e8-a2b3-49bf-b280-45884a0aed90 filter=id,name,scope,clustername,tags,hypervisor,suitableformigration
{
  "count": 3,
  "storagepool": [
    {
      "clustername": "10.10.3.184/Trillian/p1-c3",
      "id": "98a0e79c-f879-36e2-b9ce-230f66e88698",
      "name": "ps6",
      "scope": "CLUSTER",
      "suitableformigration": true,
      "tags": "tag1"
    },
    {
      "hypervisor": "VMware",
      "id": "17899b5b-1838-3e8d-b0fe-76cab34093b1",
      "name": "ps7",
      "scope": "ZONE",
      "suitableformigration": true
    },
    {
      "hypervisor": "VMware",
      "id": "53a4a81b-0271-3989-94d1-a00708fd9c32",
      "name": "ps8",
      "scope": "ZONE",
      "suitableformigration": true,
      "tags": "tag1"
    }
  ]
}

(localcloud) SBCM5> > find storagepoolsformigration id=73d242e1-f8fb-4447-8c78-a1e10d106694 filter=id,name,scope,clustername,tags,hypervisor,suitableformigration
{
  "count": 3,
  "storagepool": [
    {
      "clustername": "10.10.3.184/Trillian/p1-c3",
      "id": "98a0e79c-f879-36e2-b9ce-230f66e88698",
      "name": "ps6",
      "scope": "CLUSTER",
      "suitableformigration": true,
      "tags": "tag1"
    },
    {
      "hypervisor": "VMware",
      "id": "17899b5b-1838-3e8d-b0fe-76cab34093b1",
      "name": "ps7",
      "scope": "ZONE",
      "suitableformigration": true
    },
    {
      "hypervisor": "VMware",
      "id": "53a4a81b-0271-3989-94d1-a00708fd9c32",
      "name": "ps8",
      "scope": "ZONE",
      "suitableformigration": true,
      "tags": "tag1"
    }
  ]
}

(localcloud) SBCM5> > find storagepoolsformigration id=e665c777-7339-4604-9535-2564c23abca8 filter=id,name,scope,clustername,tags,hypervisor,suitableformigration
{
  "count": 7,
  "storagepool": [
    {
      "clustername": "p1-c1",
      "id": "e777166e-7dbe-340d-977d-a51d5349b5f1",
      "name": "ps2",
      "scope": "CLUSTER",
      "suitableformigration": true,
      "tags": "tag1"
    },
    {
      "clustername": "10.10.3.184/Trillian/p1-c2",
      "id": "79f5ea41-e6dd-3f9d-acca-10267cf7b796",
      "name": "ps4",
      "scope": "CLUSTER",
      "suitableformigration": true,
      "tags": "tag1"
    },
    {
      "hypervisor": "VMware",
      "id": "53a4a81b-0271-3989-94d1-a00708fd9c32",
      "name": "ps8",
      "scope": "ZONE",
      "suitableformigration": true,
      "tags": "tag1"
    },
    {
      "hypervisor": "VMware",
      "id": "17899b5b-1838-3e8d-b0fe-76cab34093b1",
      "name": "ps7",
      "scope": "ZONE",
      "suitableformigration": false
    },
    {
      "clustername": "p1-c1",
      "id": "b2888e3e-b1e2-36d9-9936-861bc6901d3f",
      "name": "ps1",
      "scope": "CLUSTER",
      "suitableformigration": false
    },
    {
      "clustername": "10.10.3.184/Trillian/p1-c2",
      "id": "0a74f7c2-63f6-374f-b5df-bc6304d0ff58",
      "name": "ps2",
      "scope": "CLUSTER",
      "suitableformigration": false
    },
    {
      "clustername": "10.10.3.184/Trillian/p1-c3",
      "id": "b9aa5fb3-ce0a-316c-a2bb-3dbf905082d4",
      "name": "ps5",
      "scope": "CLUSTER",
      "suitableformigration": false
    }
  ]
}

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

blueorangutan · 2021-02-15T13:24:39Z

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2709

Refactor allocateToPool method to allow bypassing storage type check. While trying to to find pools for volume migration suitable pools can be found bypassing storage type check. Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

blueorangutan · 2021-02-15T21:16:56Z

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2712

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

blueorangutan · 2021-02-16T11:28:37Z

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2714

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr · 2021-02-17T12:56:32Z

@blueorangutan package

blueorangutan · 2021-02-17T12:57:42Z

@shwstppr a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2021-02-17T13:36:02Z

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2724

shwstppr · 2021-02-18T03:52:59Z

@blueorangutan test

blueorangutan · 2021-02-18T03:53:44Z

@shwstppr a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2021-02-18T13:07:58Z

Trillian test result (tid-3567)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 31544 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4693-t3567-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Smoke tests completed. 85 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_04_rvpc_privategw_static_routes	`Failure`	235.79	test_privategw_acl.py

blueorangutan · 2021-02-23T16:03:46Z

Packaging result: ✔️ centos7 ✔️ centos8 ✔️ debian. SL-JID 35

blueorangutan · 2021-02-24T04:36:25Z

Packaging result: ✔️ centos7 ✔️ centos8 ✔️ debian. SL-JID 36

yadvr · 2021-02-24T11:00:01Z

@blueorangutan test centos7 vmware-67u3

blueorangutan · 2021-02-24T11:01:17Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + vmware-67u3) has been kicked to run smoke tests

DaanHoogland

clgtm, javadox should make sense

DaanHoogland · 2021-02-24T12:28:08Z

     * @param ExcludeList
     *            avoid
     * @param int returnUpTo (use -1 to return all possible pools)
+     * @param boolean bypassStorageTypeCheck


why add javadoc without description? please add a short sentence on how to use it, or remove it.

@DaanHoogland done

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

blueorangutan · 2021-02-24T23:29:46Z

Trillian test result (tid-3614)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
Total time taken: 43017 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4693-t3614-vmware-67u3.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 85 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_03_deploy_and_upgrade_kubernetes_cluster	`Failure`	785.15	test_kubernetes_clusters.py

server: fix finding pools for migration for volume

387fd0f

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr added this to the 4.15.1.0 milestone Feb 15, 2021

fix pool suitability for storage allocators

71da59f

Refactor allocateToPool method to allow bypassing storage type check. While trying to to find pools for volume migration suitable pools can be found bypassing storage type check. Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

pass podid to allocators when available

6d712b7

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr added 2 commits February 16, 2021 23:15

fix

ec7ac23

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

fix

46c8cf3

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

apache deleted a comment from blueorangutan Feb 16, 2021

fix for detached volumes

f336e5f

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr added component:migration type:bug labels Feb 18, 2021

shwstppr self-assigned this Feb 18, 2021

shwstppr marked this pull request as ready for review February 22, 2021 09:53

apache deleted a comment from blueorangutan Feb 23, 2021

yadvr requested review from DaanHoogland and harikrishna-patnala February 23, 2021 10:53

apache deleted a comment from blueorangutan Feb 23, 2021

DaanHoogland requested changes Feb 24, 2021

View reviewed changes

javadoc comment

5fceecc

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

DaanHoogland approved these changes Feb 24, 2021

View reviewed changes

yadvr approved these changes Feb 25, 2021

View reviewed changes

yadvr merged commit 88337bd into apache:4.15 Feb 25, 2021

Conversation

shwstppr commented Feb 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

blueorangutan commented Feb 15, 2021

Uh oh!

blueorangutan commented Feb 15, 2021

Uh oh!

blueorangutan commented Feb 16, 2021

Uh oh!

shwstppr commented Feb 17, 2021

Uh oh!

blueorangutan commented Feb 17, 2021

Uh oh!

blueorangutan commented Feb 17, 2021

Uh oh!

shwstppr commented Feb 18, 2021

Uh oh!

blueorangutan commented Feb 18, 2021

Uh oh!

blueorangutan commented Feb 18, 2021

Uh oh!

blueorangutan commented Feb 23, 2021

Uh oh!

blueorangutan commented Feb 24, 2021

Uh oh!

yadvr commented Feb 24, 2021

Uh oh!

blueorangutan commented Feb 24, 2021

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland Feb 24, 2021

Choose a reason for hiding this comment

Uh oh!

shwstppr Feb 24, 2021

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Feb 24, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

shwstppr commented Feb 15, 2021 •

edited

Loading