Migration guide for new cloud storage open

Important Notice: Ongoing Cloud Storage Migration

We would like to inform you that the following projects are currently affected due to ongoing migration to the new cloud storage. If you have not initiated the migration process, your virtual machines will be permanently deleted.

Affected Projects:

Ended Projects (Still Running Servers):

We are happy to announce the availability of a new storage system for users of the Swedish Science Cloud EAST-1 and UPPMAX Cloud. The new storage system, Alburnus, will replace the old storage system, Aspius. Alburnus offers enhanced storage capacity and improved performance. Having undergone internal and user assisted testing since late summer, we are now prepared to provide you with the necessary information to migrate your existing infrastructure to Alburnus.

Alburnus is the Default Storage System for New Volumes and Images since January 19, 2024

If you have only created volumes and images after January 19th, 2024, you do not need to take any action.

Alburnus has been designated as the default storage system for all new volumes and images since January 19th, 2024. No action is required for users to access Alburnus. However, existing volumes and images created on Aspius must be migrated to Alburnus. This guide will outline the required steps to facilitate the migration of your volumes and images.

Verifying Storage System for Images

If you are solely using the public images in EAST-1, you can skip this section, since all public images have been migrated to Alburnus. However, if you have uploaded custom images, some action may be necessary. In Horizon, you can view your images under Compute > Images. Alternatively, using the OpenStack CLI, you can list your images with the command:

$ openstack image list

You can determine the storage system used by an image by clicking on the image in Horizon and inspecting the stores field. If it only mentions rbd, it indicates usage of the old storage system, necessitating action. Conversely, if it mentions alburnus, the image already resides on the new storage system, requiring no further action. Using the OpenStack CLI, you can view the image store by running:

$ openstack image show -f json -c properties <image_id> | jq

The presence of 'metadata': {'store': 'alburnus'} indicates that the image already resides on the new storage system, requiring no action.

Explanation:

Migrating Images to Alburnus

Migrating images can be performed by users by re-uploading the image, as the default image store is set to Alburnus. If you do not have a local copy of your image, please contact support at support@uppmax.uu.se, and we can attempt an image migration using the following command. Unfortunately, this command is not available for regular users in the OpenStack version used at EAST-1:

$ openstack image import --store alburnus --method copy-image <image_id>

Downloading image

You can download an image using glance

$ pip install python-glanceclient
$ . openrc
$ glance image-download --progress <image_id> --file <image_name>

Uploading image

Ensure your image is in the raw format before uploading. If you need to convert from .qcow to raw you can run:

$ qemu-img convert -f qcow2 -O raw <image_name>.qcow2 <image_name>.raw

You can now upload your raw image:

$ openstack image create --disk-format raw --public --file ./<image_name>.raw --min-disk 20 <image_name>

Verifying Storage System for Volumes

By default, when creating a VM, a volume is also created. You can identify the volume associated with your VM by running openstack server show <VM ID> and extracting the volume ID. Subsequently, using the volume ID, you can determine the volume type:

$ openstack volume show -c type

Explanation:

Migrating Volumes

Unfortunately, it is not possible to migrate volumes using Horizon.

Option 1. Migrate your volumes using the cinder command-line client, typically included when installing the OpenStack CLI.

Option 2. If retaining your VM is unnecessary but saving your data is crucial, you can attach a secondary volume from Alburnus and manually copy using standard tools.

Option 1. Migrate your volumes using cinder

Assuming you wish to migrate the following volume from Aspius to Alburnus:

$ openstack volume list
+--------------------------------------+-----------------------------+-----------+------+-------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+-----------------------------+-----------+------+-------------+
| 61e56496-f68c-41f8-803d-eaef18d6bcca | linusn_test_11 | available | 1 | |
+--------------------------------------+-----------------------------+-----------+------+-------------+

Migrate the volume using cinder retype --migration policy on-demand:

$ cinder retype --migration-policy on-demand <volume_id> rbd_alburnus

Explanation:

Certain restrictions apply to the types of volumes that can be migrated.

Restrictions:

Check the status of the retype operation:

$ openstack volume show <volume_id> -c id -c name -c status -c type
+--------+--------------------------------------+
| Field | Value |
+--------+--------------------------------------+
| id | <volume_id> |
| name | <volume_name> |
| status | available |
| type | rbd_alburnus |
+--------+--------------------------------------+

Issues may arise with existing snapshots.

Option 2. Attach a Secondary Volume and Manually Copy Files

This can be accomplished entirely in Horizon and from your VM.

  1. Create a new volume under Volumes > Volumes and select “Create Volume”. In the ensuing form, ensure that Type is set to rbd_alburnus (default setting).

  2. Navigate to Compute > Instances, select the rightmost drop-down list for your VM, and choose “Attach Volume”. Select the volume created in step 1.

  3. SSH into your VM and mount the new volume.

Identify the attached disk, e.g., vdb.

$ ssh <username>@<VM_IP>
$ dmesg | tail
[12247422.208063] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000
[12247422.209276] pci 0000:00:07.0: reg 0x10: [io  0x0000-0x007f]
[12247422.209452] pci 0000:00:07.0: reg 0x14: [mem 0x00000000-0x00000fff]
[12247422.210782] pci 0000:00:07.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit pref]
[12247422.216930] pci 0000:00:07.0: BAR 4: assigned [mem 0x100000000-0x100003fff 64bit pref]
[12247422.217049] pci 0000:00:07.0: BAR 1: assigned [mem 0x40000000-0x40000fff]
[12247422.217086] pci 0000:00:07.0: BAR 0: assigned [io  0x1000-0x107f]
[12247422.217236] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003)
[12247422.257706] virtio_blk virtio4: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)

Create and mount a filesystem.

$ sudo mkfs.ext4 /dev/vdb
mke2fs 1.45.5 (07-Jan-2020)
Discarding device blocks: done                            
Creating filesystem with 262144 4k blocks and 65536 inodes
Filesystem UUID: 435a5eaa-8427-4146-926f-60764c525e6e
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

$ sudo mount /dev/vdb

You are now ready to copy data.

$ rsync -ahvz <source> /mnt/

Tip: Calculate checksums to confirm the copy was successful.

Questions

Should you have any questions, feel free to contact us directly at support@uppmax.uu.se, as this pertains solely to the EAST-1 region hosted by UPPMAX.

Affected systems: UPPMAX Cloud
Written by Support Team on