====== Linux Software Raid ====== Software RAID in Linux is realised using the dm-raid kernel module and can be controlled by the ''mdadm'' utility. Information and status of known raid arrays can be gathered by reading ''/proc/mdstat''. Array details are available using ''mdadm -D'': # mdadm -D /dev/md1 /dev/md1: Version : 1.2 Creation Time : Fri Mar 16 01:13:24 2012 Raid Level : raid1 Array Size : 1048564 (1024.16 MiB 1073.73 MB) Used Dev Size : 1048564 (1024.16 MiB 1073.73 MB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Update Time : Fri Mar 16 01:13:36 2012 State : clean, resyncing Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Resync Status : 25% complete Name : base:1 (local to host base) UUID : c527d58e:bfd9cd43:ebb8a4f6:a812c75b Events : 4 Number Major Minor RaidDevice State 0 254 10 0 active sync /dev/dm-10 1 254 11 1 active sync /dev/dm-11 The following shows some more or less chaotic playing around with software raid arrays of various levels. ===== Creating a Level 1 Raid ===== Create the raid1 device ''/dev/md1'' consisting of the two disks ''/dev/lvg/raid1'' and ''/dev/lvg/raid2'': # mdadm --create -l 1 -n 2 /dev/md1 /dev/lvg/raid1 /dev/lvg/raid2 Check results: # cat /proc/mdstat 0/0 n0-1@pts/13 Personalities : [raid6] [raid5] [raid4] [raid1] md1 : active raid1 dm-11[1] dm-10[0] 1048564 blocks super 1.2 [2/2] [UU] ===== Merging an existing Level 1 Array to Level 5 (the hard way) ===== The idea behind all this is having an existing raid1 array, and after adding a third disk of same size to the system the wish for converting to raid5. So after safely degrading the raid1 above, there are two disks available for initial setup of the new raid5 array. ==== Removing one of the Raid1 Devices ==== First, set the device faulty: # mdadm /dev/md1 -f /dev/lvg/raid2 mdadm: set /dev/lvg/raid2 faulty in /dev/md1 Then finally remove it: # mdadm /dev/md1 -r /dev/lvg/raid2 mdadm: hot removed /dev/lvg/raid2 from /dev/md1 ==== Creating a (degraded) Level 5 Raid ==== The idea behind all this is having an existing raid1 array, and after adding a third disk of same size to the system the wish for converting to a raid5. So after safely degrading the raid1 above, there are two disks available for initial setup of the new raid5 array. # mdadm --create -l 5 -n 3 /dev/md5 missing /dev/lvg/raid2 /dev/lvg/raid3 mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md5 started. The word ''missing'' is used as a special device string. # cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] [raid1] md5 : active raid5 dm-12[2] dm-11[1] 2096128 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU] md1 : active raid1 dm-10[0] 1048564 blocks super 1.2 [2/1] [U_] ==== Completing the Transition from Raid1 to Raid5 ==== Now one would create a filesystem on ''/dev/md5'', mount it and then copy over all data from ''/dev/md1''. Afterwards, the old raid1 array can be completely destructed and the single disk it consisted of added to the new array: # mdadm --stop /dev/md1 mdadm: stopped /dev/md1 # mdadm /dev/md5 --add /dev/lvg/raid1 mdadm: added /dev/lvg/raid1 # cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] [raid1] md5 : active raid5 dm-10[3] dm-12[2] dm-11[1] 2096128 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU] [=>...................] recovery = 9.2% (97064/1048064) finish=1.1min speed=13866K/sec ===== Merging an existing Level 1 Array to Level 5 (the easy way) ===== Since dm-raid is actually able to change raid levels on the go, all the above can be reduced to just a single command. Given that resizing the filesystem is possible, which has to be done afterwards, of course: # mdadm /dev/md1 --grow -n 3 -l 5 -a /dev/lvg/raid3 mdadm: level of /dev/md1 changed to raid5 mdadm: added /dev/lvg/raid3 mdadm: Need to backup 128K of critical section.. # cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] [raid1] md1 : active raid5 dm-12[2] dm-11[1] dm-10[0] 2097128 blocks super 1.2 level 5, 4k chunk, algorithm 2 [3/3] [UUU] Ok, one limitation here is the device name. In order to change it, the array has to be brought offline (of course, since the different device name has to be mounted again anyway): # mdadm --stop /dev/md1 mdadm: stopped /dev/md1 # mdadm --assemble /dev/md5 -U name /dev/lvg/raid1 /dev/lvg/raid2 /dev/lvg/raid3 mdadm: /dev/md5 has been started with 3 drives. The result can be verified using ''mdadm -E'': # mdadm -E /dev/lvg/raid3 /dev/lvg/raid3: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : c527d58e:bfd9cd43:ebb8a4f6:a812c75b Name : base:5 (local to host base) Creation Time : Fri Mar 16 01:13:24 2012 Raid Level : raid5 Raid Devices : 3 Avail Dev Size : 2097128 (1024.16 MiB 1073.73 MB) Array Size : 4194256 (2048.32 MiB 2147.46 MB) Data Offset : 24 sectors Super Offset : 8 sectors State : clean Device UUID : 9e065712:0eca28ca:c45ce117:020de68c Update Time : Fri Mar 16 01:29:36 2012 Checksum : 5371875b - correct Events : 55 Layout : left-symmetric Chunk Size : 4K Device Role : Active device 2 Array State : AAA ('A' == active, '.' == missing) Note the ''Name'' field which was ''base:1'' before. ===== Various Adjustment Knobs ===== There is lots of interesting stuff in ///sys/block/mdX/md///: ==== Sync Speed ===== When a raid array has to resync, it's speed may be controlled using the attributes //sync_speed_min// and //sync_speed_max//. The maximum synchronisation speed can be changed using a statement like: # echo 1000000 > /sys/block/md1/md/sync_speed_max The unit is kibibytes. In order to set the value for all raid devices in the system, one may use ///proc/sys/dev/raid/speed_limit_{min,max}// instead. In //sync_speed//, the current speed is given in K/sec, 30s average. ===== Further Documentation ===== See the file //Documentation/md.txt// in your current kernel's sources.