Table of Contents
Linux Software Raid
Software RAID in Linux is realised using the dm-raid kernel module and can be
controlled by the mdadm
utility. Information and status of known raid
arrays can be gathered by reading /proc/mdstat
. Array details are
available using mdadm -D
:
# mdadm -D /dev/md1 /dev/md1: Version : 1.2 Creation Time : Fri Mar 16 01:13:24 2012 Raid Level : raid1 Array Size : 1048564 (1024.16 MiB 1073.73 MB) Used Dev Size : 1048564 (1024.16 MiB 1073.73 MB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Update Time : Fri Mar 16 01:13:36 2012 State : clean, resyncing Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Resync Status : 25% complete Name : base:1 (local to host base) UUID : c527d58e:bfd9cd43:ebb8a4f6:a812c75b Events : 4 Number Major Minor RaidDevice State 0 254 10 0 active sync /dev/dm-10 1 254 11 1 active sync /dev/dm-11
The following shows some more or less chaotic playing around with software raid arrays of various levels.
Creating a Level 1 Raid
Create the raid1 device /dev/md1
consisting of the two disks
/dev/lvg/raid1
and /dev/lvg/raid2
:
# mdadm --create -l 1 -n 2 /dev/md1 /dev/lvg/raid1 /dev/lvg/raid2
Check results:
# cat /proc/mdstat 0/0 n0-1@pts/13 Personalities : [raid6] [raid5] [raid4] [raid1] md1 : active raid1 dm-11[1] dm-10[0] 1048564 blocks super 1.2 [2/2] [UU]
Merging an existing Level 1 Array to Level 5 (the hard way)
The idea behind all this is having an existing raid1 array, and after adding a third disk of same size to the system the wish for converting to raid5. So after safely degrading the raid1 above, there are two disks available for initial setup of the new raid5 array.
Removing one of the Raid1 Devices
First, set the device faulty:
# mdadm /dev/md1 -f /dev/lvg/raid2 mdadm: set /dev/lvg/raid2 faulty in /dev/md1
Then finally remove it:
# mdadm /dev/md1 -r /dev/lvg/raid2 mdadm: hot removed /dev/lvg/raid2 from /dev/md1
Creating a (degraded) Level 5 Raid
The idea behind all this is having an existing raid1 array, and after adding a third disk of same size to the system the wish for converting to a raid5. So after safely degrading the raid1 above, there are two disks available for initial setup of the new raid5 array.
# mdadm --create -l 5 -n 3 /dev/md5 missing /dev/lvg/raid2 /dev/lvg/raid3 mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md5 started.
The word missing
is used as a special device string.
# cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] [raid1] md5 : active raid5 dm-12[2] dm-11[1] 2096128 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU] md1 : active raid1 dm-10[0] 1048564 blocks super 1.2 [2/1] [U_]
Completing the Transition from Raid1 to Raid5
Now one would create a filesystem on /dev/md5
, mount it and then copy over
all data from /dev/md1
. Afterwards, the old raid1 array can be completely
destructed and the single disk it consisted of added to the new array:
# mdadm --stop /dev/md1 mdadm: stopped /dev/md1 # mdadm /dev/md5 --add /dev/lvg/raid1 mdadm: added /dev/lvg/raid1 # cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] [raid1] md5 : active raid5 dm-10[3] dm-12[2] dm-11[1] 2096128 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU] [=>...................] recovery = 9.2% (97064/1048064) finish=1.1min speed=13866K/sec
Merging an existing Level 1 Array to Level 5 (the easy way)
Since dm-raid is actually able to change raid levels on the go, all the above can be reduced to just a single command. Given that resizing the filesystem is possible, which has to be done afterwards, of course:
# mdadm /dev/md1 --grow -n 3 -l 5 -a /dev/lvg/raid3 mdadm: level of /dev/md1 changed to raid5 mdadm: added /dev/lvg/raid3 mdadm: Need to backup 128K of critical section.. # cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] [raid1] md1 : active raid5 dm-12[2] dm-11[1] dm-10[0] 2097128 blocks super 1.2 level 5, 4k chunk, algorithm 2 [3/3] [UUU]
Ok, one limitation here is the device name. In order to change it, the array has to be brought offline (of course, since the different device name has to be mounted again anyway):
# mdadm --stop /dev/md1 mdadm: stopped /dev/md1 # mdadm --assemble /dev/md5 -U name /dev/lvg/raid1 /dev/lvg/raid2 /dev/lvg/raid3 mdadm: /dev/md5 has been started with 3 drives.
The result can be verified using mdadm -E
:
# mdadm -E /dev/lvg/raid3 /dev/lvg/raid3: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : c527d58e:bfd9cd43:ebb8a4f6:a812c75b Name : base:5 (local to host base) Creation Time : Fri Mar 16 01:13:24 2012 Raid Level : raid5 Raid Devices : 3 Avail Dev Size : 2097128 (1024.16 MiB 1073.73 MB) Array Size : 4194256 (2048.32 MiB 2147.46 MB) Data Offset : 24 sectors Super Offset : 8 sectors State : clean Device UUID : 9e065712:0eca28ca:c45ce117:020de68c Update Time : Fri Mar 16 01:29:36 2012 Checksum : 5371875b - correct Events : 55 Layout : left-symmetric Chunk Size : 4K Device Role : Active device 2 Array State : AAA ('A' == active, '.' == missing)
Note the Name
field which was base:1
before.
Various Adjustment Knobs
There is lots of interesting stuff in /sys/block/mdX/md/:
Sync Speed
When a raid array has to resync, it's speed may be controlled using the attributes sync_speed_min and sync_speed_max. The maximum synchronisation speed can be changed using a statement like:
# echo 1000000 > /sys/block/md1/md/sync_speed_max
The unit is kibibytes. In order to set the value for all raid devices in the system, one may use /proc/sys/dev/raid/speed_limit_{min,max} instead.
In sync_speed, the current speed is given in K/sec, 30s average.
Further Documentation
See the file Documentation/md.txt in your current kernel's sources.