Linux LVM review

Notes that will eventually become the whitepaper

Notes

  1. Linux lvm much bigger than originally anticipated.

  2. -v option is verbose; multiple -v‘s increase the verbosity

    to the point that, at least on the lvdisplay, it’s challenging the xterm scroll length.

  3. lvmdiskscan used to scan for eligible disks

# lvmdiskscan -l
  WARNING: only considering LVM devices
  /dev/sda2            [      19.51 GiB] LVM physical volume
  /dev/sdb1            [      10.00 GiB] LVM physical volume
  /dev/sdc1            [      10.00 GiB] LVM physical volume
  0 LVM physical volume whole disks
  3 LVM physical volumes

if using whole disks, which seems smarter, on the whole:
# lvmdiskscan -l
  WARNING: only considering LVM devices
  /dev/sda2      [      19.51 GiB] LVM physical volume
  /dev/sdb       [      10.00 GiB] LVM physical volume
  /dev/sdc       [      10.00 GiB] LVM physical volume
  2 LVM physical volume whole disks
  1 LVM physical volume
  1. Have to say, linux is trying to steal my thunder. Commands available that I wrote scripts to do on HP...
# pvscan
  PV /dev/sdb    VG vgmirror   lvm2 [10.00 GiB / 10.00 GiB free]
  PV /dev/sdc    VG vgmirror   lvm2 [10.00 GiB / 10.00 GiB free]
  PV /dev/sda2   VG vg00       lvm2 [19.51 GiB / 0    free]
  Total: 3 [39.50 GiB] / in use: 3 [39.50 GiB] / in no VG: 0 [0   ]
  1. vgexport/import is a different too. much cleaner, easier than HPUX. no map files, it imports the lv names whereas hpux would simply call them lvol1 lvol2, etc, if no mapfile was used.
  2. vgs, lvs, and pvs commands. Interesting output control capabilities
  3. vgremove ${vg} to remove a volume group, unlike the HP vgexport
  4. fucking awesome! Splitting and rejoining vgs!
# pvs
  PV         VG       Fmt  Attr PSize  PFree
  /dev/sda2  vg00     lvm2 a--  19.51g     0
  /dev/sdb   vgmirror lvm2 a--  10.00g  6.00g
  /dev/sdc   vgmirror lvm2 a--  10.00g 10.00g
# vgsplit vgmirror vgsplit /dev/sdb
  Logical volume "mirror" must be inactive
# mount | grep mirror
# lvchange -a n vgmirror/mirror
# vgsplit vgmirror vgsplit /dev/sdb
  New volume group "vgsplit" successfully split from "vgmirror"
# pvs
  PV         VG       Fmt  Attr PSize  PFree
  /dev/sda2  vg00     lvm2 a--  19.51g     0
  /dev/sdb   vgsplit  lvm2 a--  10.00g  6.00g
  /dev/sdc   vgmirror lvm2 a--  10.00g 10.00g
# vgchange -a n vgsplit
  0 logical volume(s) in volume group "vgsplit" now active
# vgchange -a n vgmirror
  0 logical volume(s) in volume group "vgmirror" now active
# vgmerge vgmirror vgsplit
  Volume group "vgsplit" successfully merged into "vgmirror"
# vgs
  VG       #PV #LV #SN Attr   VSize  VFree
  vg00       1   2   0 wz--n- 19.51g     0
  vgmirror   2   1   0 wz--n- 19.99g 15.99g
  1. Creating mirrors: some good points. a. Log files:

    1. mirroring creates a log file on disk unless --mirrorlog core option is used.
    2. Log files, by default, go on disks other than the ones being mirrored - makes sense. Might have to put them on the same disks, though, (for instance, mirroring the OS). Can do that by specifying: --alloc anywhere
    1. If creating mirrors > 1.5TB, update the region size: lvcreate -L 2T -R 2 -n ... use factors of 2 every time you up the space: ``lvcreate -L 3T -R 4 -n ... ``
    2. Another useful feature. You can watch the mirroring taking place:
 # lvdisplay vgmirror/mirror
   --- Logical volume ---
   LV Path                /dev/vgmirror/mirror
   LV Name                mirror
   VG Name                vgmirror
   LV UUID                xmcKu2-sYgV-CPbm-W6s6-FPcu-9Mwm-i7j8YF
   LV Write Access        read/write
   LV Creation host, time guest, 2013-03-22 08:40:11 -0500
   LV Status              NOT available
   LV Size                4.00 GiB
   Current LE             1024
   Segments               1
   Allocation             inherit
   Read ahead sectors     auto
# lvconvert -m 1 vgmirror/mirror
  Conversion starts after activation.
# vgchange -a y vgmirror
  1 logical volume(s) in volume group "vgmirror" now active
# lvs
  LV     VG       Attr     LSize  Pool Origin Data%  Move Log         Copy%  Convert
  root   vg00     -wi-ao-- 17.54g
  swap   vg00     -wi-ao--  1.97g
  mirror vgmirror mwi-a-m-  4.00g                         mirror_mlog   9.67
# lvs vgmirror/mirror
  LV     VG       Attr     LSize Pool Origin Data%  Move Log         Copy%  Convert
  mirror vgmirror mwi-a-m- 4.00g                         mirror_mlog  13.57
# lvs vgmirror/mirror
  LV     VG       Attr     LSize Pool Origin Data%  Move Log         Copy%  Convert
  mirror vgmirror mwi-a-m- 4.00g                         mirror_mlog  15.04
# lvs vgmirror/mirror
  LV     VG       Attr     LSize Pool Origin Data%  Move Log         Copy%  Convert
  mirror vgmirror mwi-a-m- 4.00g                         mirror_mlog  62.30
  1. Initial steps: a. Assign disks, scan, get them visible to the OS. b. pvcreate, vgcreate, lvcreate:
# pvscan
  PV /dev/sda2   VG vg00   lvm2 [19.51 GiB / 0    free]
  Total: 1 [19.51 GiB] / in use: 1 [19.51 GiB] / in no VG: 0 [0   ]
# pvcreate /dev/sdb /dev/sdc
  Writing physical volume data to disk "/dev/sdb"
  Physical volume "/dev/sdb" successfully created
  Writing physical volume data to disk "/dev/sdc"
  Physical volume "/dev/sdc" successfully created
# vgcreate -s 4m vgmirror /dev/sd[bc]
  Volume group "vgmirror" successfully created
# pvs
  PV         VG       Fmt  Attr PSize  PFree
  /dev/sda2  vg00     lvm2 a--  19.51g     0
  /dev/sdb   vgmirror lvm2 a--  10.00g 10.00g
  /dev/sdc   vgmirror lvm2 a--  10.00g 10.00g
# lvcreate -L 4g -m 1 -n mirror vgmirror
  Logical volume "mirror" created
# lvs vgmirror
  LV     VG       Attr     LSize Pool Origin Data%  Move Log         Copy%  Convert
  mirror vgmirror mwi-a-m- 4.00g                         mirror_mlog  13.77
# lvs vgmirror
  LV     VG       Attr     LSize Pool Origin Data%  Move Log         Copy%  Convert
  mirror vgmirror mwi-a-m- 4.00g                         mirror_mlog  81.05
# lvs vgmirror
  LV     VG       Attr     LSize Pool Origin Data%  Move Log         Copy%  Convert
  mirror vgmirror mwi-a-m- 4.00g                         mirror_mlog 100.00
# mkfs.ext4 /dev/vgmirror/mirror
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
[[snip]]
# mkdir -p -m 755 /mnt/mirror
# mount /dev/vgmirror/mirror /mnt/mirror
# mount /dev/vgmirror/mirror /mnt/mirror
# echo "this is a test. this is only a test." > /mnt/mirror/README
# lvconvert --splitmirrors 1 --name copy vgmirror/mirror
  Logical volume mirror converted.
# lvs -o +devices vgmirror
  LV     VG       Attr     LSize Pool Origin Data%  Move Log Copy%  Convert Devices
  copy   vgmirror -wi-a--- 4.00g                                            /dev/sdc(0)
  mirror vgmirror -wi-ao-- 4.00g                                            /dev/sdb(0)
# lvchange -a n vgmirror/copy
# vgsplit vgmirror vgcopy /dev/sdc
  New volume group "vgcopy" successfully split from "vgmirror"
# lvs -o +devices
  LV     VG       Attr     LSize  Pool Origin Data%  Move Log Copy%  Convert Devices
  root   vg00     -wi-ao-- 17.54g                                            /dev/sda2(0)
  swap   vg00     -wi-ao--  1.97g                                            /dev/sda2(4490)
  copy   vgcopy   -wi-----  4.00g                                            /dev/sdc(0)
  mirror vgmirror -wi-ao--  4.00g                                            /dev/sdb(0)
# mkdir /mnt/mirror1
# mount vgcopy/copy /mnt/mirror1
mount: you must specify the filesystem type
# mount /dev/vgcopy/copy /mnt/mirror1
# md5sum /mnt/mirror/README /mnt/mirror1/README
a8f852b96dffa74d7135ff35c10677f6  /mnt/mirror/README
a8f852b96dffa74d7135ff35c10677f6  /mnt/mirror1/README
  1. OK, now let’s put it back:
# umount /mnt/mirror1
# vgchange -a n vgcopy
  0 logical volume(s) in volume group "vgcopy" now active
# lvconvert -m 1 vgmirror/mirror
  vgmirror/mirror: Converted: 0.0%
^C
# lvs vgmirror
  LV     VG       Attr     LSize Pool Origin Data%  Move Log         Copy%  Convert
  mirror vgmirror mwi-aom- 4.00g                         mirror_mlog  76.17
# lvs vgmirror
  LV     VG       Attr     LSize Pool Origin Data%  Move Log         Copy%  Convert
  mirror vgmirror mwi-aom- 4.00g                         mirror_mlog 100.00
  1. So, now, we blow away a device to simulate a failure.
# virsh detach-disk guest hdd --persistent
# run on vmhost, need to figure out how to ID correct dev w/o reading xml file directly
# vgreduce vgmirror --removemissing --force
  Couldn't find device with uuid mjpoEn-pZGd-ocew-oR6N-2E71-0eVX-3fKg6Y.
  Wrote out consistent volume group vgmirror
# lvs
  LV     VG       Attr     LSize  Pool Origin Data%  Move Log Copy%  Convert
  root   vg00     -wi-ao-- 17.54g
  swap   vg00     -wi-ao--  1.97g
  mirror vgmirror -wi-----  4.00g
# lvdisplay /dev/vgmirror/mirror
  --- Logical volume ---
  LV Path                /dev/vgmirror/mirror
  LV Name                mirror
  VG Name                vgmirror
  LV UUID                0OpUgK-HSPc-eIA7-5DcG-ALnE-dY95-NPOrGu
  LV Write Access        read/write
  LV Creation host, time guest, 2013-03-22 11:51:41 -0500
  LV Status              NOT available
  LV Size                4.00 GiB
  Current LE             1024
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto

# lvchange -a y vgmirror/mirror
# mount /dev/vgmirror/mirror /mnt/mirror
# md5sum /mnt/mirror/README
a8f852b96dffa74d7135ff35c10677f6  /mnt/mirror/README
  1. Adding a new disk:
# virsh vol-create-as default guest-3.img 10g
# virsh attach-disk guest /var/lib/libvirt/images/guest\-3.img hdd --persistent
# virsh start guest
# pvcreate /dev/sdc
  Writing physical volume data to disk "/dev/sdc"
  Physical volume "/dev/sdc" successfully created
# vgextend vgmirror /dev/sdc
  Volume group "vgmirror" successfully extended
# lvconvert -m 1 vgmirror/mirror
# lvs vgmirror
  LV     VG       Attr     LSize Pool Origin Data%  Move Log         Copy%  Convert
  mirror vgmirror mwi-a-m- 4.00g                         mirror_mlog 100.00

Software RAID review

  1. Reset the environment using the script
  2. Partition the disks, set partitiion type to fd (linux raid)
fdisk -uc /dev/sdc << eof
n
p
1
<ret>
<ret>
t
fd
w
eof
  1. Create the raid partition:
# mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
 Continue creating array? y
 mdadm: Defaulting to version 1.2 metadata
 mdadm: array /dev/md0 started.
  1. Status obtained in two ways: a. /proc/mdstat
# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdc1[1] sdb1[0]
      10483640 blocks super 1.2 [2/2] [UU]
      [>....................]  resync =  3.8% (407424/10483640) finish=13.6min speed=12320K/sec
unused devices: <none>
# grep resync /proc/mdstat
      [>....................]  resync =  4.8% (504192/10483640) finish=13.9min speed=11920K/sec

b.  ``mdadm --detail /dev/md0``
# mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Sun Mar 24 11:01:46 2013
     Raid Level : raid1
     Array Size : 10483640 (10.00 GiB 10.74 GB)
  Used Dev Size : 10483640 (10.00 GiB 10.74 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Sun Mar 24 11:01:46 2013
          State : clean, resyncing
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

  Resync Status : 6% complete

           Name : guest:0  (local to host guest)
           UUID : 7eeb50f9:cec710b8:7d491890:3310a328
         Events : 0

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1
  1. Standard lvm pvcreate/vgcreate/lvcreate
  2. Update mdadm.conf:
# mdadm --detail --scan | tee /etc/mdadm.conf
ARRAY /dev/md/guest:0 metadata=1.2 name=guest:0 UUID=7eeb50f9:cec710b8:7d491890:3310a328
  1. Reboot/verify filesystem is available.
  2. Manual failure of a drive:
# mdadm -f /dev/md0 /dev/sdb1
mdadm: set /dev/sdb1 faulty in /dev/md0
# mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Sun Mar 24 11:01:46 2013
     Raid Level : raid1
     Array Size : 10483640 (10.00 GiB 10.74 GB)
  Used Dev Size : 10483640 (10.00 GiB 10.74 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent
    Update Time : Sun Mar 24 11:59:01 2013
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 1
  Spare Devices : 0
           Name : guest:0  (local to host guest)
           UUID : 7eeb50f9:cec710b8:7d491890:3310a328
         Events : 20
    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       33        1      active sync   /dev/sdc1

       0       8       17        -      faulty spare   /dev/sdb1
# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdb1[0](F) sdc1[1]
      10483640 blocks super 1.2 [2/1] [_U]
  1. Remove and replace the drive:
# mdadm -r /dev/md0 /dev/sdb1
mdadm: hot removed /dev/sdb1 from /dev/md0
# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdc1[1]
      10483640 blocks super 1.2 [2/1] [_U]

unused devices: <none>
# mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Sun Mar 24 11:01:46 2013
     Raid Level : raid1
     Array Size : 10483640 (10.00 GiB 10.74 GB)
  Used Dev Size : 10483640 (10.00 GiB 10.74 GB)
   Raid Devices : 2
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Sun Mar 24 12:04:39 2013
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : guest:0  (local to host guest)
           UUID : 7eeb50f9:cec710b8:7d491890:3310a328
         Events : 23

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       33        1      active sync   /dev/sdc1
# mdadm --add /dev/md0 /dev/sdb1
mdadm: added /dev/sdb1
# mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Sun Mar 24 11:01:46 2013
     Raid Level : raid1
     Array Size : 10483640 (10.00 GiB 10.74 GB)
  Used Dev Size : 10483640 (10.00 GiB 10.74 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Sun Mar 24 12:05:37 2013
          State : clean, degraded, recovering
 Active Devices : 1
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 1

 Rebuild Status : 0% complete

           Name : guest:0  (local to host guest)
           UUID : 7eeb50f9:cec710b8:7d491890:3310a328
         Events : 29

    Number   Major   Minor   RaidDevice State
       2       8       17        0      spare rebuilding   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1
  1. Dead drive:
    1. virsh destroy guest; virsh detach-disk guest hdc –persistent; virsh vol-delete ${file} default; virsh start guest
    2. error conditions:
# mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Sun Mar 24 11:01:46 2013
     Raid Level : raid1
     Array Size : 10483640 (10.00 GiB 10.74 GB)
  Used Dev Size : 10483640 (10.00 GiB 10.74 GB)
   **Raid Devices : 2**
  **Total Devices : 1**
    Persistence : Superblock is persistent

    Update Time : Sun Mar 24 13:25:50 2013
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : guest:0  (local to host guest)
           UUID : 7eeb50f9:cec710b8:7d491890:3310a328
         Events : 48

    Number   Major   Minor   RaidDevice State
       2       8       17        0      active sync   /dev/sdb1
       1       0        0        1      removed
# cat /proc/mdstat
Personalities : [raid1]
md0 : active (auto-read-only) raid1 sdb1[2]
      10483640 blocks super 1.2 [2/1] [U_]
  1. Replace/repair: