A Linux storage management cheat-sheet

This article outlines, with step-by-step examples, how to carry out the most fundamental Linux storage management tasks: partitioning a disk, creating a filesystem, setting up swap space, managing a logical volume management (LVM) storage pool, and configuring a RAID mirror. I tested all examples on Ubuntu 12.10 but, because only command-line operations are described, the same procedures should work on most relatively modern Linux installations.

This article is written for Linux administrators and experienced desktop users who have some experience with command-line system administration.

1 Partitioning, formatting, and mounting a new disk

When you buy an internal hard disk, it will typically be supplied completely blank, or partitioned with one large FAT32 or exFAT (DOS/Windows) filesystem; either is likely to be unsuitable.

Identifying a disk

Most administrative operations on disks require the disk device entry in the /dev directory. The device mapper automatically assigns these names according to a set of rules, based on the bus number, the disk's position on the bus, the partition number, and the disk type. The disk type is becoming increasingly redundant as an indicator, as most types of disk are now managed to some extent by the kernel's SCSI support, and are therefore named /dev/sdXXX, where 'sd' stands for 'SCSI disk'.

It can therefore be a bit awkward to identify a specific disk or partition in a system with a number of disks. If a disk is formatted with a filesystem, then one useful approach is simply to write a number or name on the disk with a permanent marker, and create a file in the root directory of that disk called NAME.txt, where 'NAME' is replaced with the identifier written on the disk. If the disk isn't formatted with a filesystem, then you'll need to cross reference information from Linux with what you know of the disk.

The lshw utility will display a list of installed disks and their sizes:

root@Ubuntu-1:~# lshw -class disk
  *-disk:0  
       description: SCSI Disk
       physical id: 0.0.0
       bus info: scsi@2:0.0.0
       logical name: dev/sda
       size: 20GiB (21GB)
       capabilities: partitioned partitioned:dos
       configuration: sectorsize=512 signature=000dc6f7
            ...etc

If you have more than one disk of the same size, then you could use fdisk to report the geometry of the disk (which may be printed on the disk itself):

root@Ubuntu-1:~# fdisk -l dev/sda

Disk /dev/sda: 21.5 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors ...etc

Because fdisk -l also dumps the partition table, it is useful for indentifying a specific partion on a known disk.

Another useful utility is hwinfo, which may not be installed by default, but should be available in the standard repositories. hwinfo –disk reports the same basic information as lshw, but also includes the vendor name and model. Partitioning a disk This example demonstrates how to use the fdisk utility to partition a 20Gb SCSI or SATA disk into a filesystem partition of 15Gb, and a swap partition (see section 2) of 5Gb. The disk device is /dev/sdb, because it is the device number 2 on the SCSI bus. Note that partitioning a disk that already contains filesystems will most likely make it unusable, so exercise caution. For clarity, user input is highlighted in bold text in the transcript.

root@Ubuntu-1:~# fdisk /dev/sdb

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x9bdb67f9.

Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

This is the normal output when the disk is completely blank. Just to be sure, print the current partition table:

Command (m for help): p

Disk /dev/sdb: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x9bdb67f9

   Device Boot      Start         End      Blocks   Id  System

No partitions, so that's OK.

Now create the 15Gb partition:

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-41943039, default 2048):
Using default value 2048
.Last sector, +sectors or +size{K,M,G} (2048-41943039, default 41943039): +15G

Check that the partition has been defined correctly:

Command (m for help): p

Disk /dev/sdb: 21.5 GB, 21474836480 bytes 255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x9bdb67f9 Device Boot Start End Blocks Id System /dev/sdb1 2048 31459327 15728640 83 Linux

Note that the partition has been created with a default ID of 83, that is, a general Linux partition. Now create the 5Gb swap partition.

Command (m for help): n
Partition type:
   p   primary (1 primary, 0 extended, 3 free)
   e   extended
Select (default p): p
Partition number (1-4, default 2): 
First sector (31459328-41943039, default 31459328):
Using default value 31459328
Last sector, +sectors or +size{K,M,G} (31459328-41943039, default 41943039):
Using default value 41943039

In this case, there's no need to enter the partition size, because the default is to use the remainder of the disk.

Note that this partition is also created with partition ID 0x83, but there is a specific ID defined for swap: 0x82. To change the partition type, use the t command in fdisk. In practice, these type IDs are of little practical significance, but they can assist when you're looking at a disk you formatted two years ago, and are trying to work out what the various partitions contain.

Finally, write out the partition table:

Command (m for help): w

The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.

Once the disk has been successfully partitioned, you'll see two new entries in /dev, one for each partition. Since the whole disk in this is example was sdb, the two new partition entries are sdb1 and sdb2.

Creating a filesystem

Partitioning the disk does not create any filesystem on it, and it cannot be used to hold files. A number of applications do expect to work with raw (that is, unformatted) partitions but, in most cases, you'll want to create a filesystem.

Modern Linux distributions support a bewildering array of filesystem types, all with their own advantages and disadvantages. However, most seem to use the ext4 filesystem type by default, and that's probably as good a choice as any for general use. For critical server applications, there are many tuning options that can be applied when a filesystem is created; for desktop purposes, the defaults are likely to be reasonable. Some of these tuning values can be changed after the filesystem has been created (using tune2fs, for example).

To create a default ext4 filesystem on a partition couldn't be simpler:

root@Ubuntu-1:~# mkfs.ext4 /dev/sdb1 -L data
mke2fs 1.42.5 (29-Jul-2012)
Filesystem label=data
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
983040 inodes, 3932160 blocks
… etc
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

The -L switch specifies the volume name. Strictly speaking, this is optional; but you might be glad later if you picked a meaningful name when setting things up.

To check the filesystem, we can mount it. For example:

root@Ubuntu-1:~# mount /dev/sdb1 /mnt/tmp/
root@Ubuntu-1:~# df -k /mnt/tmp/
Filesystem     1K-blocks   Used Available Use% Mounted on
/dev/sdb1       15481840 169456  14525952   2% /mnt/tmp

The partition has a total size of about 15 million, one kilobyte blocks (which is about 15Gb). Note that it is already about 2% full; this unusable capacity is taken up by the various indexes and other infrastructure.

Notice that we didn't need to specify the filesystem type in the mount command – it is worked out automatically by specific 'magic numbers' in the partition header. If mount complains that you need to specify the filesystem type, it probably won't help to do so – most likely the filesystem is broken in some way.

Defining a permanent mount point for the partition

The use of the mount utility is not persistent – it will last only until the next reboot. Conventionally, permanent mounts are specified in the file /etc/fstab.

In this example, the newly-created 15Gb partition, whose device is /dev/sdb1, needs to be mounted on the directory /data. We therefore need to add a line to fstab like this:

/dev/sdb1 /data auto defaults 0 0

Note that the directory /data is not created by the mount process – you need to create it explicitly before trying to mount on it.

/dev/sdb1 is the device, and /data is the place it will be mounted. auto instructs the mounter to determine the filesystem type automatically – you could say 'ext4' here and shorten your boot time by a millisecond or two. defaults means to apply the default mount options – some alternatives to the defaults will be discussed later. The first '0' field is of only historical significance – it controls whether the filesystem is to be dumped to a backup device using the dump utility. The second '0' indicates that the filesystem will not be checked at boot time; if you want it to be checked, a value of '2' would be appropriate, meaning that it should be checked after the root filesystem. Creating a FAT32 filesystem for sharing with Windows If you're running a dual-boot Windows/Linux system, or if you're formatting an external disk to share with a Windows system, you may prefer to format the disk or partition as FAT32. In principle, modern Windows versions can work with Linux ext2 filesystems, and modern Linux versions can work with Windows NTFS filesystems. There's just one problem: the Windows and Linux security models are completely incompatible. What this means in practice is that you could spend so much time fiddling with access control lists and the like, that it's just not worth the effort. The FAT32 system has no access control at all, so that particular problem doesn't arise.

root@Ubuntu-1:~# mkfs.vfat -F 32 /dev/sdb2
mkfs.vfat 3.0.13 (30 Jun 2012)

'-F 32' instructs mkfs to use 32-bit values for the index table.

Because the FAT32 filesystem has no access control, Linux cannot assign file or group ownership to files. The entire filesystem will be owned by the user and group that performed the mount. If the filesystem is mounted by an entry in /etc/fstab, then that user and group will be root, which may well be very inconvenient. In this case, it's probably better to prevent the filesystem being mounted at boot, and allow it to be mounted as an ordinary user. All the files will belong to the user who mounts it. To do that, we need an entry in fstab like this:

/dev/sdb2 /data auto rw,user,noauto 0 0

'rw' means mount in read/write mode; user means allow any user to mount the partition; noauto means not to mount at boot time.

To check this, log in as an unprivileged user, and create a file:

kevin@Ubuntu-1:~$ mount /data
kevin@Ubuntu-1:~$ touch /data/myfile
kevin@Ubuntu-1:~$ ls -l /data/myfile
-rwxr-xr-x 1 kevin kevin 0 Dec  6 00:19 /data/myfile

Note that the file appears to be owned by the user who created it. This is an illusion – the user who mounted the filesystem will own all files on it. If a different user mounts the same filesystem, that user will appear to own the files. This is a consequence of working with a filesytem that has no access control.

There are many other useful mount options that can be used in fstab. For example, you can specify that the filesystem is mounted at boot time, but is owned by a specific user. If you're running a desktop system that has, in practice, only one user, this can be a more useable approach than requiring the user to mount the filesystem at the command line. 'man fstab' should give all the details.

2 Managing swap space

'Swap' is disk storage allocated to back up RAM. For a desktop computer, allocating swap to a total size of twice the physical RAM is often a good starting point. Depending on your distribution and how you ran the installer, you might have no swap at all. The Ubuntu installer allocates by default a swap volume of about the same size as RAM. If you find yourself running out of memory, you have essentially three ways to increase memory, other than the obvious one of fitting more RAM.

1. Create swap on an unused disk partition
2. Create one or more swap files
3. Increase the size of an existing swap volume

Increasing the size of an existing swap partition is usually only possible for volumes managed by LVM; this option will be described in more detail in section 3.

Creating a swap partition

Section 1 described how to use fdisk to assign a swap partition to a disk. Following on from that example, if the partition is identified as /dev/sdb2, we can do the following to add it as a swap partition.

root@Ubuntu-1:~# mkswap /dev/sdb2
Setting up swapspace version 1, size = 5241852 KiB
no label, UUID=af07d467-a18e-4a78-bf0c-9d2007850371
root@Ubuntu-1:~# swapon -a /dev/sdb2

To check the swap allocations:

root@Ubuntu-1:~# swapon -s
Filename                                Type            Size    Used   Priority
/dev/mapper/ubuntu-swap_1               partition       1044476 188     -1
/dev/sdb2                               partition       5241852 0       -2

Note that ubuntu-swap_1 is the 1Gb swap partition created by the Ubuntu installer. It has a higher priority that the new partition, and will therefore be used in preference to it. Ubuntu-swap_1 is not a 'real' disk partition; it is a LVM logical volume (about which, more later).

Creating a swap file

In general, swapping is fastest to a swap partition, and the use of swap files should really be seen as a poor alternative, to be used when there is no possibility of creating a dedicated partition. It should also be noted that Linux likes to use a swap partition for hibernation; storing hibernation data in anything other than a partition is rather experimental, if it is available at all.

The following example creates a swap file /tmp/swap of size 1Gb, and adds it to the swap space.

root@Ubuntu-1:~# dd if=/dev/zero of=/tmp/swap count=1000 bs=1M
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 6.82553 s, 154 MB/s
root@Ubuntu-1:~# mkswap /tmp/swap
Setting up swapspace version 1, size = 1023996 KiB
no label, UUID=5cc4ff26-36ef-411a-94be-58f0b20626ba
root@Ubuntu-1:~# swapon /tmp/swap

Using dd to copy a thousand, one megabyte blocks of zeros is a generally-applicable way to create a large empty file to serve as the swap file. On some filesystem you mind find the fallocate utility a much faster way to achieve the same result; but it won't make the actual swapping any faster. Making swap space changes persistent As with filesystems, changes made by swapon are not persistent; to have swap added at boot time, create a line in /etc/fstab like this:

/dev/sdb2 none swap defaults 0 0

The word none here informs the mounter that this is not a filesystem to mount.

3 Using Logical Volume Management to organize disk storage in a flexible way

The Linux filesystem is organized as one or more storage volumes (disks or disk partitions), mounted at particular points in a tree. As a user, you see a simple tree structure of files and directories and, unless you go looking for it, you won't necessarily be aware which physical disk a particular file or directory resides on. Contrast this with the 'drive letter' scheme user by Windows: it's clear at all times where your file is located, because the drive letter identifies the storage device.

Conventionally, there are separate volumes for the root filesystem, the home (user) filesystem, and for swap space. Some distributions use a separate volume for temporary files. This is the same organizational scheme that Unix has used, with slight variations from vendor to vendor, for thirty years; it works well in many applications, particular in server and multi-user installations. The use of fixed-size volumes for specific functions ensures that one application or user cannot easily consume enough space to compromise the overall system stability. You can create a specific volume, for example, for database log files, knowing that even if the database misbehaves, its log files won't grow to take over the entire system.

On the desktop, however, the use of fixed-size, specific volumes can be a real nuisance, particularly if you aren't in a position to plan in detail exactly how your system will be used at installation time. Consequently, some Linux installers default to creating one huge volume on one specific, selected disk. This is analogous to the way the Windows installer works – by default the entire system is installed on one disk or partition, with no internal subdivision, which ends up getting called 'drive C:'. Doing this, of course, negates the advantage that the use of separate volumes provides.

The use of Logical Volume Management (LVM), which is supported by all the major distributions, provides a compromise between the traditional Unix scheme, where volumes must be planned and sized at installation time, and the 'one big disk' scheme. With LVM, volumes are still of fixed size, but the size can be changed (to some extent) after installation. In an LVM scheme if you run out of disk space in one volume, and you can't steal space from another volume, you can add a new disk, and absorb it transparently into the storage scheme.

WIth logical volume management, one or more physical disks (or partitions) are grouped together into a volume group, which serves as a pool of storage. The pool can be expanded by adding new physical disks. From the volume group, logical volumes are assigned, each taking a defined amount of storage from the pool. Logical volumes look like ordinary disk partitions, and can be formatted with filesystems, or used as swap space, just as a real disk partitions can. The significant difference between logical volumes and traditional disk partitions is that, with some restrictions, logical volumes can be resized after creation.

Preparing a physical disk for use with LVM

You can add to an LVM logical volume (storage pool) either a whole disk or a partition. There are reasons to choose one or the other, but the decision will be transparent to the resulting logical volumes, the places that you actually keep files and directories.

This example, and those that follow in this section, are based on the following pair of two physical disks:

/dev/sdb: 20Gb
/dev/sdc: 30Gb

Neither disk is partitioned, or formatted with a filesystem.

To prepare the disks, use the pvcreate utility. You can apply pvcreate to multiple disks if necessary.

root@Ubuntu-1:~# pvcreate /dev/sdb /dev/sdc
  Writing physical volume data to disk "/dev/sdb"
  Physical volume "/dev/sdb" successfully created
  Writing physical volume data to disk "/dev/sdc"
  Physical volume "/dev/sdc" successfully created

Listing physical disks known to the volume manager

The pvdisplay utility shows the details of a specific disk, or lists the disks known to the volume manager. To display details of a specific device:

root@Ubuntu-1:~# pvdisplay /dev/sdb
  "/dev/sdb" is a new physical volume of "20.00 GiB"
  --- NEW Physical volume ---
  PV Name               /dev/sdb
  VG Name
  PV Size               20.00 GiB
  Allocatable           NO
  PE Size               0
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               3fOWlI-zICg-8tuG-wGy9-xyNN-IaeW-H6QZ3U

To get a brief list of all known disks, use the -s switch without specifying a disk device:

root@Ubuntu-1:~# pvdisplay -s
  Device "/dev/sda5" has a capacity of 20.00 MiB
  Device "/dev/sdb" has a capacity of 20.00 GiB
  Device "/dev/sdc" has a capacity of 30.00 GiB

The device /dev/sda5 was not defined in this example – this is the original system partition created by the Ubuntu installer. Adding physical disks to a volume group (storage pool) Now we'll add the 20Gb disk and the 30Gb disk to create a storage pool of 50Gb.

root@Ubuntu-1:~# vgcreate mypool /dev/sdb /dev/sdc
  Volume group "mypool" successfully created

The name mypool is arbitrary, but will be used to manage the storage pool later.

You don't have to create the full storage pool at one time, and you can add storage to it at any time using vgextend.

Checking the size and status of a volume group (storage pool)

Using the name of the storage pool, we can find out how much storage is available, and how much has been allocated to form logical volumes:

root@Ubuntu-1:~# vgdisplay mypool
  --- Volume group ---
  VG Name               mypool
  System ID
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  1
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               49.99 GiB
  PE Size               4.00 MiB
  Total PE              12798
  Alloc PE / Size       0 / 0
  Free  PE / Size       12798 / 49.99 GiB
  VG UUID               PJ6hvL-eNMW-5jUr-rnci-DVWP-ePqv-X0wFhf

This storage pool is of total size 49.99 GiB (50 Gb, essentially), and none has been allocated. Since this is a newly created volume group, composed of a 20Gb and 30Gb disk, that is to be expected.

Allocating a filesystem from a volume group (storage pool)

Now we'll create a 40Gb (approx) ext4 filesystem by carving it out of the storage pool called mypool. First define the logical volume and give it a name:

root@Ubuntu-1:~# lvcreate --size 40G --name mybigvolume mypool
  Logical volume "mybigvolume" created

The name mybigvolume is used by the kernel to form the name of the device in /dev. This name is comprised of the pool name and the logical volume name. In this case it will be /dev/mapper/mypool-mybigvolume.

Now create the filesystem:

root@Ubuntu-1:~# mkfs.ext4 /dev/mapper/mypool-mybigvolume
mke2fs 1.42.5 (29-Jul-2012)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
...etc
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

Let's check that the filesystem is usable, but mounting it on /mnt/tmp:

root@Ubuntu-1:~# mount /dev/mapper/mypool-mybigvolume /mnt/tmp/
root@Ubuntu-1:~# df -k /mnt/tmp/
Filesystem                     1K-blocks   Used Available Use% Mounted on
/dev/mapper/mypool-mybigvolume  41284928 180104  39007672   1% /mnt/tmp

You can define a persistent mount point for this new volume by editing /etc/fstab, as explained in section 1.

It's worth reflecting for a moment on what has happened here. We've created a filesystem of size 40Gb, out of two smaller disks. Using lvdisplay will confirm this:

root@Ubuntu-1:~# vgdisplay mypool
  --- Volume group ---
  VG Name               mypool
  ...etc
  VG Size               49.99 GiB
  PE Size               4.00 MiB
  Total PE              12798
  Alloc PE / Size       10240 / 40.00 GiB
  Free  PE / Size       2558 / 9.99 GiB
  ...etc

That is, out of a total size of 49.99 GiB, 40 GiB is allocated, and 9.99Gb remains free. It's important to note that spanning filesystems across disks like this is sometimes useful, even necessary; but it doubles the risk of losing a filesystem because of a disk failure. As always, some sort of backup strategy can mitigate the loss, but building you logical volumes out of RAID mirrors might be a more productive approach. Of course, the use of RAID is not limited to LVM setups – you can create a RAID mirror out of two ordinary disks and construct a filesystem on top, as explained in section 4.

Expanding a filesystem under LVM control

It's important to understand that a logical volume can always be expanded and contracted, subject to there being enough storage in the pool, but that doesn't necessarily mean that the filesystem itself will expand and contract. Fortunately, Linux ext filesystems are reasonably well able to tolerate size changes.

Let's increase the 40Gb filesystem on /dev/mapper/mypool-mybigvolume to 45Gb (unmounting it first, if it is mounted). Start by increasing the size of the logical volume:

root@Ubuntu-1:~# lvextend --size 45G /dev/mapper/mypool-mybigvolume
  Extending logical volume mybigvolume to 45.00 GiB
  Logical volume mybigvolume successfully

The resize the filesystem itself:

root@Ubuntu-1:~# resize2fs /dev/mapper/mypool-mybigvolume
resize2fs 1.42.5 (29-Jul-2012)
Resizing the filesystem on /dev/mapper/mypool-mybigvolume to 11796480 (4k) blocks.
The filesystem on /dev/mapper/mypool-mybigvolume is now 11796480 blocks long.

resize2fs might insist that you check the filesystem with fsck first. This is not necessarily cause for alarm.

Not all filesystem types are resizable. Moreover, while extending an ext filesystem is generally a safe thing to do, shrinking it not be. Shrinking a filesystem is certainly something that should only be considered when your sure you have a sound, up-to-date backup.

Closing remarks on LVM

The examples above have used new, blank disks to create a new volume group storage pool, but the same principles apply to the volumes created by the installer to contain the root and home filesystems, and the swap volume. You can, in principle, use lvextend to increase the amount of swap or the size of the root filesystem, although you might need to add more storage to the pool first to be able to do this.

Logical volume management in Linux is much more complex and flexible than this overview might suggest. You can, for example, create snapshots of LVM volumes, or specify exactly how the volume is to be carved out of the pool. These options are all documented in the relevant man pages for the various utilities.

4 Software RAID

RAID is a technique for creating one logical volume out of a number of physical disks; as such, its function overlaps with the Logical Volume Management discussed in section 3. However, in practice, LVM is typically used to provide more flexible storage management, while RAID is more often used to improve throughput or reliability.

RAID is perhaps of more interest to server administrators than Linux desktop users. However, software raid is sufficiently simple to set up, that it can readily be used on desktop computers to create a mirror for data security.

Setting up a RAID mirror for reliability

In this section, the RAID array will be a mirror of two identical disks of capacity 20Gb. These appears as devices /dev/sdb and /dev/sdc.

Depending on how your system was installed, you might need to load the kernel module for the type of RAID array you are creating before running for the first time. For a mirror (RAID level 1), this would be:

# modprobe raid1

On subsequent reboots, that should be taken care of automatically by the kernel.

If you're starting with new, empty disks, the first step is to create the RAID array by associating the disk devices with the meta-device. The meta-device is the entry in /dev that represents the composite disk, and on which the filesystem will be constructed. The entry in /dev will not exist at this point – it is created by the kernel when the RAID array is enabled.

Like most RAID operations, this initial association is done using the mdadm utility:

root@Ubuntu-1:~# mdadm --create --verbose /dev/md0 --level=mirror --raid-devices=2 /dev/sdb  /dev/sdc
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.

Thus use of mdadm is persistent – it will survive a reboot. However, on Ubuntu in particular, it is helpful to specify the RAID array properties in /etc/mdadm/mdadm.conf, and to update the system's initial RAM filesystem. These two steps together ensure that the name for the meta-device you used when creating the array – /dev/md0 – continues to be used after a reboot.

root@Ubuntu-1:~# mdadm --detail --scan >> /etc/mdadm/mdadm.conf
root@Ubuntu-1:~# update-initramfs -u

At the time of writing, it remains somewhat unclear whether the need to use update-initramfs constitutes a bug in the Ubuntu install or not – in any case, I have not found it necessary in Fedora.

Check the status of a RAID mirror

You can check the status of a RAID array at any time, using /proc/mdstat:

root@Ubuntu-1:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [rid10]
md0 : active raid1 sdc[1] sdb[0]
      20955008 blocks super 1.2 [2/2] [UU]
unused devices:

Here you can see the two disk devices in the array, and [UU] indicating that both are in the Up state.

Creating a filesystem on a RAID mirror

To use the array, you'll need to create a filesystem on it:

root@Ubuntu-1:~# mkfs.ext4 dev/md0 mke2fs 1.42.5 (29-Jul-2012) ...etc Creating journal (32768 blocks): done Writing superblocks and filesystem accounting information: done

root@Ubuntu-1:~# mount /dev/md0 /mnt/tmp/ root@Ubuntu-1:~# df /mnt/tmp/ Filesystem 1K-blocks Used Available Use% Mounted on /dev/md0 20625916 176064 19402104 1% /mnt/tmp

As always, you can create a permanent mount by editing /etc/fstab, as described in section 1. Rather than creating a single filesystem, you could also take advantage of Logical Volume Management, and add the new meta-device to a logical volume (storage pool), as described in section1.3.

Testing the RAID mirror

You can remove one of the disks from the mirrored array programmatically, even simulating a favour. However, my own view is that there's no substitute for physically pulling the disk from the machine. After powering off and pulling the disk at /dev/sdc, /proc/mdstat shows the following:

root@Ubuntu-1:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active (auto-read-only) raid1 sdb[0]
      20955008 blocks super 1.2 [2/1] [U_]

The status is now [U_], that is one Up disk and one disk missing. However, the filesystem should still have been mounted correctly, and been perfectly usable.

Replacing the failed disk

A RAID mirror will run in degraded mode indefinitely – so long as the other disk does not fail. Before that happens, we need to replace the faulty disk, and allow it to synchronize with the good one. The new disk, in this example, is on the same place in the bus, and so is still referred to as /dev/sdc as the failed (removed) one was. But as this is an empty disk, it will not become active until it has been synchronized.

To install the new disk in the RAID mirror, use mdadm –add:

root@Ubuntu-1:~# mdadm --add /dev/md0 /dev/sdc
mdadm: added /dev/sdc
root@Ubuntu-1:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdc[2] sdb[0]
      20955008 blocks super 1.2 [2/1] [U_]
      [=>...................]  recovery =  8.5% (1800064/20955008) finish=1.4min speed=225008K/sec

unused devices: none

Notice that, because I was quick in looking at mdstat, I see the RAID mirror still in the [U_] state, and a synchronization progress indicator. Depending on the size of the disk and how it is installed, this synchronization will take between a few seconds and a few hours. The disk can be used during the synchronization, but this will slow the process down. At the end of synchronization, you'll see the status return to [UU], and thus you're back to normal running. Removing a disk from the RAID mirror elegantly Pulling the disk is a reasonable way to simulate a failure, if you don't want to risk, say, taking a sledgehammer to one of the disks. However, if need to remove a disk, perhaps to use it for something else, you can remove it from the array by marking it as failed. Note that mdadm won't let you remove the disk from the array without marking it as failed – it is considered to be in use.

root@Ubuntu-1:~# mdadm --fail /dev/md0 /dev/sdc
mdadm: set /dev/sdc faulty in /dev/md0
root@Ubuntu-1:~# mdadm --remove /dev/md0 /dev/sdc
mdadm: hot removed /dev/sdc from /dev/md0

Note that the array is still considered to be degraded, and you'll still get warnings about it at boot time on a Ubuntu system.

Marking a disk as failed is persistent – rebooting, for example, won't clear the fault. To bring the disk back into the array, you'll need to add it explicitly using mdadm –-add.

Closing remarks on RAID

The use of RAID level 1, mirroring, is probably the simplest – both to set up and to recover from failures with. So long as the array is up the disks should, excepting in failure situations, be identical. This makes it easy to swap one disk for another. If you have more than two disks to create an array from, then questions about the RAID level start to arise. For additional protection you can create a mirrored pair with a hot spare. The spare will automatically come into service when one of the other disks in the array fails. Such an array is very robust – it will survive two disk failures in rapid succession. It does, however, mean allocating three disks to one filesystem.

Alternatively, you could create a RAID level-5 array, which uses a combination of data striping and redundancy to achieve a higher capacity than a RAID-1 mirror with the same disks, but at the cost of some loss of reliability. In general, three 20Gb disks in RAID-5 will give you a total capacity of 40Gb, rather than 20Gb for RAID-1 with a spare; but there is no guarantee of being able to recover from a failure of multiple disks.

Closing remarks

There's a lot to Linux storage management that this short article has not been able to cover. I haven't even touched on the various graphical tools that many Linux distributions now provide to automate some of these tasks. Although not strictly a Linux tools, it's worth becoming familiar with the gparted utility, which can do some fairly clever things with disk partitions that are very difficult to do from within a running Linux system.