User Tools

Site Tools


system:filesystem:raid:soft

Linux Software Raid

Software RAID in Linux is realised using the dm-raid kernel module and can be controlled by the mdadm utility. Information and status of known raid arrays can be gathered by reading /proc/mdstat. Array details are available using mdadm -D:

# mdadm -D /dev/md1
/dev/md1:
        Version : 1.2
  Creation Time : Fri Mar 16 01:13:24 2012
     Raid Level : raid1
     Array Size : 1048564 (1024.16 MiB 1073.73 MB)
  Used Dev Size : 1048564 (1024.16 MiB 1073.73 MB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

    Update Time : Fri Mar 16 01:13:36 2012
          State : clean, resyncing 
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

  Resync Status : 25% complete

           Name : base:1  (local to host base)
           UUID : c527d58e:bfd9cd43:ebb8a4f6:a812c75b
         Events : 4

    Number   Major   Minor   RaidDevice State
       0     254       10        0      active sync   /dev/dm-10
       1     254       11        1      active sync   /dev/dm-11

The following shows some more or less chaotic playing around with software raid arrays of various levels.

Creating a Level 1 Raid

Create the raid1 device /dev/md1 consisting of the two disks /dev/lvg/raid1 and /dev/lvg/raid2:

# mdadm --create -l 1 -n 2 /dev/md1 /dev/lvg/raid1 /dev/lvg/raid2

Check results:

# cat /proc/mdstat
0/0 n0-1@pts/13
Personalities : [raid6] [raid5] [raid4] [raid1]
md1 : active raid1 dm-11[1] dm-10[0]
      1048564 blocks super 1.2 [2/2] [UU]

Merging an existing Level 1 Array to Level 5 (the hard way)

The idea behind all this is having an existing raid1 array, and after adding a third disk of same size to the system the wish for converting to raid5. So after safely degrading the raid1 above, there are two disks available for initial setup of the new raid5 array.

Removing one of the Raid1 Devices

First, set the device faulty:

# mdadm /dev/md1 -f /dev/lvg/raid2
mdadm: set /dev/lvg/raid2 faulty in /dev/md1

Then finally remove it:

# mdadm /dev/md1 -r /dev/lvg/raid2
mdadm: hot removed /dev/lvg/raid2 from /dev/md1

Creating a (degraded) Level 5 Raid

The idea behind all this is having an existing raid1 array, and after adding a third disk of same size to the system the wish for converting to a raid5. So after safely degrading the raid1 above, there are two disks available for initial setup of the new raid5 array.

# mdadm --create -l 5 -n 3 /dev/md5 missing /dev/lvg/raid2 /dev/lvg/raid3
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md5 started.

The word missing is used as a special device string.

# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1]
md5 : active raid5 dm-12[2] dm-11[1]
      2096128 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU]

md1 : active raid1 dm-10[0]
      1048564 blocks super 1.2 [2/1] [U_]

Completing the Transition from Raid1 to Raid5

Now one would create a filesystem on /dev/md5, mount it and then copy over all data from /dev/md1. Afterwards, the old raid1 array can be completely destructed and the single disk it consisted of added to the new array:

# mdadm --stop /dev/md1
mdadm: stopped /dev/md1
# mdadm /dev/md5 --add /dev/lvg/raid1
mdadm: added /dev/lvg/raid1
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1] 
md5 : active raid5 dm-10[3] dm-12[2] dm-11[1]
      2096128 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU]
      [=>...................]  recovery =  9.2% (97064/1048064) finish=1.1min speed=13866K/sec

Merging an existing Level 1 Array to Level 5 (the easy way)

Since dm-raid is actually able to change raid levels on the go, all the above can be reduced to just a single command. Given that resizing the filesystem is possible, which has to be done afterwards, of course:

# mdadm /dev/md1 --grow -n 3 -l 5 -a /dev/lvg/raid3
mdadm: level of /dev/md1 changed to raid5
mdadm: added /dev/lvg/raid3
mdadm: Need to backup 128K of critical section..
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1] 
md1 : active raid5 dm-12[2] dm-11[1] dm-10[0]
      2097128 blocks super 1.2 level 5, 4k chunk, algorithm 2 [3/3] [UUU]

Ok, one limitation here is the device name. In order to change it, the array has to be brought offline (of course, since the different device name has to be mounted again anyway):

# mdadm --stop /dev/md1
mdadm: stopped /dev/md1
# mdadm --assemble /dev/md5 -U name /dev/lvg/raid1 /dev/lvg/raid2 /dev/lvg/raid3
mdadm: /dev/md5 has been started with 3 drives.

The result can be verified using mdadm -E:

# mdadm -E /dev/lvg/raid3
/dev/lvg/raid3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : c527d58e:bfd9cd43:ebb8a4f6:a812c75b
           Name : base:5  (local to host base)
  Creation Time : Fri Mar 16 01:13:24 2012
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 2097128 (1024.16 MiB 1073.73 MB)
     Array Size : 4194256 (2048.32 MiB 2147.46 MB)
    Data Offset : 24 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 9e065712:0eca28ca:c45ce117:020de68c

    Update Time : Fri Mar 16 01:29:36 2012
       Checksum : 5371875b - correct
         Events : 55

         Layout : left-symmetric
     Chunk Size : 4K

   Device Role : Active device 2
   Array State : AAA ('A' == active, '.' == missing)

Note the Name field which was base:1 before.

Various Adjustment Knobs

There is lots of interesting stuff in /sys/block/mdX/md/:

Sync Speed

When a raid array has to resync, it's speed may be controlled using the attributes sync_speed_min and sync_speed_max. The maximum synchronisation speed can be changed using a statement like:

# echo 1000000 > /sys/block/md1/md/sync_speed_max

The unit is kibibytes. In order to set the value for all raid devices in the system, one may use /proc/sys/dev/raid/speed_limit_{min,max} instead.

In sync_speed, the current speed is given in K/sec, 30s average.

Further Documentation

See the file Documentation/md.txt in your current kernel's sources.

system/filesystem/raid/soft.txt · Last modified: 2012/05/15 20:47 by 127.0.0.1