Adding Another BtrFS Storage Pool To An Existing LXD Installation

The following post will likely be part of series documenting moving the containers running on the default Btrfs storage pool used by my LXD installation, to another newly created Btrfs storage pool within the same LXD installation. My LXD installation was installed via Snap and runs on CentOS 7.

Reference:
https://lxd.readthedocs.io/en/latest/storage/
https://lxd.readthedocs.io/en/latest/instances/#type-disk

System Paramenters:
Distribution: CentOS 7.7
Snap: 3.17
Kernel: 4.12.8-1.el7.elrepo.x86_64

Introduction:

At one time I had an LVM partition for miscellaneous data content on the boot drive of my server. Realising the data laying there was no longer  really needed, I decided to re-purpose the lvm partition for something else. Also it was an excuse to experiment more with Btrfs in less complicated setups.

Currently I have a 6 x 1 TB hard drives arranged in a 3.7TB Btrfs RAID6 Array, which has worked almost non-stop for over two years, but alas I digress.

Right, currently I have my existing LXD installation using a sub-volume on a btrfs storage pool on a 2 TB hard drive, on which I also store a lot of multimedia files that I manage and stream via Plex. Lately I've been considering moving the containers on that storage drive, onto the boot drive with the 'spare' lvm partition recently formatted with Btrfs. That way I could use media drive for just media.


My spare Btrfs formatted LVM partition is mounted at /extra.

Create a new Btrfs sub-volume under /extra:

As root:

# btrfs subvolume create /extra/lxdpool2
Create subvolume '/extra/lxdpool2'

I'm going to mount this sub-volume at sub-directory under /var/snap/lxd/common/lxd/storage-pools/  with the options suggested by the lxd storage documentation

Create a sub-directory

# mkdir -p /var/snap/lxd/common/lxd/storage-pools/lxdpool2


Modify /etc/fstab to ensure pool is mounted at boot up

# vim /etc/fstab

UUID="uid of btrfs fs at /extra"  /var/snap/lxd/common/lxd/storage-pools/lxdpool2  subvolid=258,noatime,nofail,user_subvol_rm_allowed 0 0

Mount btrfs subvolume we'll using as our new storage pool.

# mount -t btrfs -o subvol=/extra/lxdpool2,user_subvol_rm_allowed /var/snap/lxd/common/lxd/storage-pools/lxdpool2

Add the new storage pool to LXD installation:

# lxc storage create lxdpool2 btrfs source=/var/snap/lxd/common/lxd/storage-pools/lxdpool2
Storage pool lxdpool2 created


View a list known LXD storage pools

# lxc storage list



Let's view the folder structure lxd created for the new pool under our btrfs subvolume at /extra/lxdpool2

# btrfs subvolume list /extra
...
ID 258 gen 968 top level 5 path annex2pool
ID 259 gen 964 top level 258 path annex2pool/containers
ID 260 gen 965 top level 258 path annex2pool/containers-snapshots
ID 261 gen 966 top level 258 path annex2pool/images
ID 262 gen 967 top level 258 path annex2pool/custom
ID 263 gen 968 top level 258 path annex2pool/custom-snapshots

Okay, we've a added a new pool to LXD. However, we want to use this new pool for our new and existing containers

First, let's create a copy of the existing default profile, except we're gonna change the storage pool information to match our new pool

# lxc profile copy default lxdpool2-profile

# lxc profile show lxdpool2-profile
config: {}
description: Default LXD profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: storage
    type: disk
name: lxdpool2-profile
used_by: []


We'll be calling to the disk device to be used in lxdpool2-profile newroot
which will be mounted at / in any container launched with profile and backed by our new pool: lxdpool2

To modify our new profile with this new information:

# lxc profile device remove lxdpool2-profile root
Device root removed from lxdpool2-profile

Let's confirm that device was removed from profile
# lxc profile show lxdpool2-profile
config: {}
description: Default LXD profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
name: lxdpool2-profile
used_by: []

Adding new disk called newroot for profile lxdpool2-profile

# lxc profile device add lxdpool2-profile newroot disk path=/ pool=lxdpool2
Device newroot added to lxdpool2-profile

View updated profile

# lxc profile show lxdpool2-profile
config: {}
description: Default LXD profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  newroot:
    path: /
    pool: lxdpool2
    type: disk
name: lxdpool2-profile
used_by: []


Let's see if we can launch a  container with new profile characteristics on the new storage pool

# lxc launch  images:centos/7 c1 -p lxdpool2-profile
Creating c1
Starting c1

# lxc list ^c1 -c ns4tP
+------+---------------+------------------------+------------------+--------------------+
| NAME |  STATE    |        IPV4                 |    TYPE           |     PROFILES     |
+------+---------+------------------------------+-----------------+---------------------+
| c1     | RUNNING  | 10.86.20.140 (eth0) | PERSISTENT | lxdpool2-profile |
+------+---------------+------------------------+-----------------+---------------------+


Show settings for lxcpool2-profile

# lxc profile show lxdpool2-profile
config: {}
description: Default LXD profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  newroot:
    path: /
    pool: lxdpool2
    type: disk
name: lxdpool2-profile
used_by:
- /1.0/containers/c1

Let's see if are containers are created in correct pools:

# lxc storage show lxdpool2config:
  source: /var/snap/lxd/common/lxd/storage-pools/lxdpool2
  volatile.initial_source: /var/snap/lxd/common/lxd/storage-pools/lxdpool2
description: ""
name: lxdpool2
driver: btrfs
used_by:
- /1.0/containers/c1
- /1.0/images/c2f43057c3597c3362475e70ae65cbaa52e49f90725cdf056d987c399373a41f
- /1.0/profiles/lxdpool2-profile
status: Created
locations:
- none

We can check the filesystem manually just to be sure.

# ls /var/snap/lxd/common/lxd/storage-pools/lxdpool2/containers
c1

# ls /var/snap/lxd/common/lxd/storage-pools/lxdpool2/images
c2f43057c3597c3362475e70ae65cbaa52e49f90725cdf056d987c399373a41f

Well we can launch containers under the new profile and storage area from the online images: repo


Let's see if we can launch containers under the new pool from images stored on the old default pool. I have an image of gitea installed under alpine v3.9, so we'll use that one.

# lxc launch gitea-al39-image c2-gitea -p lxdpool2-profile
Creating c2-gitea
Starting c2-gitea

# lxc list ^c2 -c ns4tP
+------------+-------------+-----------------------+------------------+--------------------+
|   NAME   |  STATE     |        IPV4                |    TYPE           |     PROFILES      |
+------------+-------------+-----------------------+------------------+---------------------+
| c2-gitea   | RUNNING | 10.86.20.35 (eth0) | PERSISTENT | lxdpool2-profile  |
+------------+-------------+-----------------------+------------------+---------------------+


Well that's that for now. Next time we're gonna try to move existing images and containers residing on storage pool default over to storage pool lxdpool2