Intro to Nested-RAID: RAID-01 and RAID-10

On 2011年01月11日, in linux, tips, by netoearth

In the last article we reviewed the most basic RAID levels. In this article, we take things to the next level and discuss what is called “Nested RAID Levels”. These concepts can provide both performance and redundancy for a data-rich world. We then look at RAID-01 and RAID-10; two of the most common Nested RAID configurations.

In our last article, I introduced the standard RAID levels: RAID-0, RAID-1, RAID-2, RAID-3, RAID-4, RAID-5, and RAID-6. While it may have been a review for many, I think it’s always good to go back and brush up on fundamental concepts from time to time. These were “single” RAID levels that have a single number associated with them such as RAID-0, RAID-1, RAID-5, RAID-6, etc. Remember that the intent of RAID is to provide extra performance and/or extra data redundancy so these single RAID levels are defined based these principles.

We found that some of the RAID levels emphasize performance, and some emphasize data redundancy. There are also some that try to provide a balance of the two goals but perhaps emphasized data redundancy more than performance. After a while people recognized that these single RAID level configurations might be limiting what we could achieve with RAID. Perhaps we could use multiple RAID configurations to achieve a better balance of performance and data redundancy.

In this article I want to talk about a further development of RAID that combines various RAID levels to create a “Nested RAID”. Nested RAID is sometimes also called Hybrid RAID with the concept of combining the standard RAID levels to perhaps create new and better RAID configurations.

Fundamentals of Nested RAID

The goals in using Nested RAID are the same as for standard RAID levels – better performance and/or better data redundancy. But typically, Nested RAID configurations look for a balance of performance and data redundancy. They are achieved by combining two or more standard RAID levels to create a “hybrid” RAID configuration. Originally, Nested RAID levels are created to either improve the performance of a standard RAID level that emphasizes redundancy, such as RAID-1, or to improve the redundancy of a standard RAID level that emphasizes performance such as RAID-0.

If you want to get a little geeky about it, you can think of Nested RAID levels like cooking. You sprinkle a little RAID-1 here, a little RAID-5 there, and then add RAID-0 at the end for performance and you have your solution. Now that I’ve finally used a cooking metaphor in a Linux storage column let’s move on to more technical details.

Nested RAID levels are usually labeled with a series of numbers rather than a single number as in the standard RAID levels. The most common have two levels or two numbers. Generically they can be written as RAID X+Y or RAID XY (assuming only two levels). For example, we can write RAID-10 where X=RAID-1 and Y=RAID-0. In the numbering scheme, the first number on the left, which is “X” in the generic scheme, is the lowest level in the nest. So in the case of RAID-10, the Nested RAID starts with RAID-1 at the lowest level.

Then the numbering scheme moves to the right. Using the generic RAID-XY sequence, the next RAID level is “Y”, to add the higher RAID levels. In the example of RAID-10, the lowest level is RAID-1, and then RAID-0 is the highest level. So RAID-0 is layered on top of RAID-1 (performance on top of redundancy).

The order of the RAID levels in the naming sequence is significant. RAID X+Y is not the same as RAID Y+X (sorry – no symmetry). RAID Y+X starts with RAID Y at the lowest level and ends with RAID X on top of it. In the case of RAID 10 (RAID X+Y), X is RAID-1 and is the lowest level, and Y is RAID-0 and is the highest level. For RAID Y+X, the opposite is true. RAID-0 is “Y” and is the lowest level and RAID-1 is “X” and is the highest level.

One might think that RAID-10 (RAID 1+0) and RAID-01 (RAID 0+1) are really the same thing and to some degree they are. They actually have fairly similar performance, capacity, and storage efficiency, but the big difference is in how many disks have to be read from, and written to, during regeneration when a Nested RAID fails. This article will talk more about this in subsequent sections.

One last comment as well. You don’t have to stay with just two levels such as RAID-XY. You can create as many levels as the hardware, or software, will allow. So you can easily create RAID-XYZ if you like (and if the hardware or software will allow it).

With that said, let’s move on to some common Nested RAID Combinations, starting with RAID-01 or RAID 0+1.

RAID 0+1

In this Nested RAID configuration, you start my creating two identical groups of disks and applying RAID-0 across each group. Then you use RAID-1 to combine the two groups. Figure 1 below shows how this is laid out.

RAID_01.png
Figure 1: RAID-01 layout with Four Drives

In this example, four disks are used. Two groups of two disks each are created and each group is built with RAID-0 (for performance). Then the two groups are combined using RAID-1 (mirroring) for data redundancy.

The data flow in Figure 1 is fairly easy to follow. The data chunk “A” is sent to the RAID group at the highest level (RAID-1). The blocks making up the chunk “A” are mirrored by sending them to the two groups (RAID-1). Then the data is striped across the two disks in each group per the RAID-0. So in Figure 1 you can see where blocks A1, A2, A3, and so on reside on the disks. Be sure to notice how the block mirroring (RAID-1) works so that there are two copies of each block in the RAID configuration (i.e. disks 1 and 3 contain the same blocks).

Since it’s difficult to go into more depth with just four drives, let’s try an example with six drives as shown in Figure 2 below.

800px-Raid_01.svg.png

This particular example was taken from wikipedia. There are a total of six drives that are all the same size (120GB). This RAID-01 array has two sets of three drives each. Each set of three drives is built with RAID-0 (the lowest level) and then they are combined using RAID-1 (mirroring).

The capacity of a RAID-01 array are the fairly simple to compute. The capacity is computed as,

Capacity = (n/2) * min(disk sizes)

where n is the number of disks in the array and min(disk sizes) is the minimum common capacity across the drives (this indicates that you can use drives of different sizes). This equation also means that RAID-01 is not very capacity efficient since you lose half of the drives to gain better data redundancy (mirroring).

Examining the RAID layout for data redundancy and failures is very enlightening. In particular what happens when a drive is lost? Let’s assume drive 2 is lost (second drive from the left). Immediately the RAID-0 group that it belongs to fails. However, since we have a mirror of the RAID-0 group, we don’t lose any data (always a good thing). We can even tolerate losing disks 1 and 3 while disk 2 is down without losing any data because they are part of the first RAID-0 group. However, after losing disk 2, we cannot tolerate losing any disk in the second RAID-0 group (disk 4, disk 5, disk 6). So the redundancy of RAID-01 is one disk (worst case) and n/2 in the best case (but not likely to happen).

Table 1 below is a quick summary of RAID-01 with a few highlights.

Table 1 – RAID-01 Highlights

Raid Level Pros Cons Storage Efficiency Minimum Number of disks
RAID-01
  • Very good performance (read and write) with the exact level of performance depending upon how many drives in a RAID-0 stripe. RAID-1 results in very good read performance.
  • Reasonable data redundancy (can tolerate the loss of any one disk)
  • Number of disks lost with no loss of data access varies from 1 to n/2. However, the n/2 is unlikely to happen.
  • Very poor storage efficiency (50%)
  • With Nested RAID levels, more data redundancy is desired beyond the loss of one disk.
  • All disks have to be used in rebuild (more on that later in article).
50% assuming the drives are the same size 4 (Note: you have to use an even number of disks)

RAID 1+0 (RAID-10)

RAID-10 is one of the most popular Nested RAID configurations and builds the configuration in the opposite sequence of RAID-01. In Figure 3 below, a minimum configuration of four disks is shown.

RAID_10.png
Figure 3: RAID-10 layout with Four Drives

For RAID-10 the configuration begins with two groups of RAID-1 pairs (we’re assuming that RAID-1 is done with only two disks). Then RAID-0 is used to combine the RAID-1 groups. Compared to RAID-01 we are using a data redundant standard RAID level first (at the lowest level), and then using a performance standard RAID level at the highest level.

The data flow for RAID-10 is fairly simple to follow. The data in chunk “A” is first striped across the two RAID-1 pairs using RAID-0 (the highest RAID level). Then each RAID pair mirrors their particular block across the two disks (RAID-1). So mirrored data blocks are closer to one another in RAID-10 than in RAID-01.

To compare directly to RAID-01 let’s look at the same six-drive configuration as Figure 2. Figure 4 below shows the RAID-10 configuration using the same six drives.

800px-RAID_10_6Drives.svg.png
Figure 4: RAID-10 layout with Six Drives

In this case there are three pairs of RAID-1 disks that are combined with RAID-0. Comparing to RAID-01, RAID-10 first stripes the data using RAID-0 and then uses RAID-1 for data redundancy. RAID-01 does the opposite – data is mirrored first and then stripped.

The capacity of a RAID-10 array is the fairly simple to compute and is exactly the same as RAID-01. The capacity is computed as,

Capacity = (n/2) * min(disk sizes)

where n is the number of disks in the array and min(disk sizes) is the minimum common capacity across the drives (this indicates that you can use drives of different sizes). This equation also means that RAID-10 is not very capacity efficient since you lose half of the drives to gain better data redundancy (mirroring).

But what happens in RAID-10 if you lose a disk? For example, let’s again assume we lose disk 2 (second disk from the left). You can now tolerate the loss of another disk as long as it is not disk 1. You can actually lose one disk in each RAID-1 pair without losing data. However, if you lose both disks in a RAID-1 pair the RAID-0 configuration is lost. So, just like RAID-01, you can lose one disk with out losing access to the data. But you can lose up to n/2 disks without losing access to the data, IF they are correct disks (very unlikely to happen). So just like RAID-01, the redundancy of RAID-10 is one disk (worst case) and n/2 in the best case (but not likely to happen).

Table 2 below is a quick summary of RAID-10 with a few highlights.

Table 2 – RAID-10 Highlights

Raid Level Pros Cons Storage Efficiency Minimum Number of disks
RAID-10
  • Very good performance (read and write) with the exact level of performance depending upon how many pairs in a RAID-0 stripe. RAID-1 results in very good read performance.
  • Reasonable data redundancy (can tolerate the loss of one disk)
  • Number of disks lost with no loss of data access varies from 1 to n/2. However, the n/2 is unlikely to happen.
  • Only one disk is involved in rebuild (more on that later in article).
  • Very poor storage efficiency
  • With Nested RAID levels, more data redundancy is desired beyond the loss of one disk.
50% assuming the drives are the same size 4 (Note: you have to use an even number of disks)

Figure 2: RAID-01 layout with Six Drives

Comparison of RAID-01 and RAID-10

Nested RAID configurations are all about combining standard RAID levels to produce something better in terms of performance or data redundancy (access to data). How you mix the standard RAID levels is the key to producing what you want (just like cooking), but you have to examine the results from all angles to judge which one is better (if, in fact, one is better). Again – just like cooking (it may smell and look good but taste like dirt). So let’s examine RAID-01 and RAID-10 from different angles.

The capacity and storage efficiency are two common aspects to be compared. Both RAID-01 and RAID-10 only use half of the disks in terms of capacity. This is due to the RAID-1 (mirroring) in the RAID configuration. The result is that the capacity and storage efficiency for RAID-01 and RAID-10 are the same.

Another aspect that can be examined is the number of drives that can be lost without losing access to data. In this case, both RAID-01 and RAID-10 are the same. Losing one drive is the worst case for both configurations (i.e. fewest drives lost before losing access to data). But, both configurations can also lose up to n/2 drives before data access is lost, although the pattern of failed drives is unique to each configuration and unlikely to happen in real-life

It is difficult to compare performance for RAID-01 and RAID-10 since in many cases the performance is heavily dependent upon the performance of the RAID controller. Consequently, we can really only objectively say that the performance of the two configuration is about equal. But we can also say the performance of both is far better than the standard RAID levels except for RAID-0 (pure performance), which is one of the goals of Nested RAID configurations.

The last category in comparing RAID-01 and RAID-10 is RAID rebuild. This category covers the number of drives used in the RAID rebuild which affects the amount of time needed to rebuild the array. When a drive fails you want the RAID array to rebuild as quickly as possible because you may be able to tolerate the loss of another drive, but it is more likely that you cannot. So the faster you get the RAID array rebuilt so that it has no failed drives, the more reliable your data access will be.

In the case of RAID-01, you need to access all of the remaining drives in the array to rebuild the loss of a single drive. The reason is that you have a RAID-1 across two sets of drives that are RAID-0. The loss of a single drive means the entire RAID-0 group has failed. The only way to restore the lost data is to copy all of the blocks from the “healthy” part of the RAID-1 array to the “old” drives in the failed RAID-0 as well as the new drive in the failed RAID-0 group. So in the case of the configuration in Figure 2 assuming that drive 2 has failed, you have to read all of the blocks from disks 4, 5, and 6, and write those blocks to drives 1 and 3 which were there before, as well as the new disk 2. This means that every drive inthe RAID group is touched requiring more time for a rebuild. The result is an increased data access vulnerability while the rebuild is happening.

In the case of RAID-10, if a drive fails, the only it’s pair drive is accessed. Using the example in Figure 4, if drive 2, fails, then only drive 1 is accessed to restore the data on the new drive 2. Since you are only having to read the data on a single drive the rebuild time goes much faster.

In addition, since only one drive has to be touched we stay away from the magic number of blocks were we can tickle a URE (Unrecoverable Read Error). So in the case of RAID-01 we run a much greater risk of hitting a URE during the rebuild as compared to RAID-10.

Using Figures 2 and 4 we can do the math on how much data has to be read and how much as to be written during a rebuild.

RAID-01: Amount of data read = 360GB
RAID-10: Amount of data read = 120GB
RAID-01: Amount of data written = 360GB
RAID-10: Amount of data written = 120GB

So for this particular example, RAID-01 has to read and write three times as much data as RAID-10 (although it can read the data a bit faster because of the data striping). If you like, we can make this into simple formulas:

RAID-01: Amount of data read = (n/2) * single drive capacity
RAID-10: Amount of data read = single drive capacity
RAID-01: Amount of data written = (n/2) * single drive capacity
RAID-10: Amount of data written = single drive capacity

So as the number of drives increases and the capacity of a single drive increases, the longer it takes RAID-01 to rebuild. However,
the RAID-10 rebuild time is only affected by the capacity of a single disk not the number of them.

Table 3 below summarizes RAID-01 and RAID-10 from the above discussion.

Table 3 – RAID-01 and RAID-10 Comparison

Category RAID-01 RAID-10
Capacity (n/2) * min(disk sizes) (n/2) * min(disk sizes)
Storage Efficiency 50% 50%
Fewest number of failed drives without loss of data 1 1
Largest number of failed drives without loss of data n/2 n/2
Overall Performance Very good Very good
Amount of data read during rebuild (n/2) * capacity of single disk capacity of single disk
Amount of data written during rebuild (n/2) * capacity of single disk capacity of single disk
Minimum Number of Drives 4 4

Linux is Special

While we may be jumping the gun a little by talking about specific implementations of RAID, Linux has some special RAID modes that impact RAID-10 and provide unique configurations.

Remember that Linux has something called software RAID (the md driver which can be managed using mdadm). This is RAID as implemented in software in the Linux kernel using the CPU’s for RAID processing (as opposed to a dedicated RAID card). A few years ago, the “md” and mdadm maintainer, Neil Brown, created some rather unique RAID-10 options for Linux, that, in some cases, can use fewer than 4 drives (the “normal” minimum for RAID-10) and use an odd number of drives. There are two concepts used in RAID-10 for md: “near” and “far”.

In the “near” concepts each data chunk is repeated m times in a k-way stripe. However, it is not required that m evenly divide k as in the normal RAID-10. With two drives and assuming that m is two, the “special” near RAID-10 configuration just boils down to RAID-1.

Disk 1    Disk 2
------    ------
  A1        A1
  A2        A2
  A3        A3
  ..        ..

Notice that we can lose one drive and still retain access to the data just like RAID-1. But the Linux md drive thinks this is RAID-10 (allows for easier upgrading and expansion).

But, one of the cool things about RAID-10 using the md driver in Linux is that we can use an odd number of drives. In the case of three drives the data layout looks like the following.

Disk 1    Disk 2    Disk 3
------    ------    ------
  A1        A1        A2
  A2        A3        A3
  A4        A4        A5
  A5        A6        A6
  ..        ..        ..

Notice that you have 2 copies of each data chunk across the drives just like RAID-10 but the chunks are added in a round-robin fashion. As with RAID-10 we can also tolerate the loss of a single drive without losing access to the data. However, this configuration allows us to use an odd number of drives and get increased capacity. The capacity of this configuration is the following:

Capacity = 2/3 * capacity of single disk

For four disks, the data layout looks just like RAID-10.

Disk 1    Disk 2    Disk 3    Disk 4
------    ------    ------    ------
  A1        A1        A2        A2
  A3        A3        A4        A4
  A5        A5        A6        A6
  A7        A7        A8        A8
  ..        ..        ..        ..

As with RAID-10 we can tolerate the loss of one drive in general, and two drives if they don’t result in the loss of certain data chunks.

The phrase “near” comes from the fact that the mirrored data blocks are “near” to each other. According to Neil Brown, this means that you can get read and write performance that is similar to RAID-0 but with half the number of drives.

The “far” concept arranges the data more like RAID-01 than RAID-10. For a “far” data layout, you split the drives into f sections. All of the data chunks are repeated in each section but are offset by one device putting the mirrored data chunks “further” apart on the drives. Let’s assume that f is 2 meaning that a drive is broken into two parts. A two-drive “far” layout with f=2 would look like the following.

Disk 1    Disk 2
------    ------
  A1        A2
  A3        A4
  A5        A6
  ..        ..
  A2        A1
  A4        A3
  A6        A5
  ..        ..

Notice how the same data chunks reside on different drives so we can lose one drive and still have access to all of the data (just like RAID-1). However, the overall capacity with two drives is still the following.

Capacity = (n/2) * capacity of single disk

where n is the number of drives.

A three-drive configuration with f=2 looks like the following:

Disk 1    Disk 2    Disk 3
------    ------    ------
  A1        A2        A3
  A4        A5        A6
  A7        A8        A9
  ..        ..        ..
  A3        A1        A2
  A6        A4        A5
  A9        A7        A8
  ..        ..        ..

Again, we can lose up to one drive and retain access to all of the data, but we have increased the capacity of RAID array to the following:

Capacity = 2/3 * capacity of single disk

which is the same as in the “near” configuration.

Notice that for the “far” configuration the emphasis is on striping. This means that reads can be striped making them very fast, but writes are somewhat limited because of the head movement on the two drives (mirroring the data). According to this article the performance is,

  • Sequential read performance is excellent since it is RAID-0 striped
  • Random reads are maybe 10-20% better than conventional RAID-10 due to using the faster outer disk sectors (short stroking) and smaller average seek times
  • Sequential and Random writes are about the same performance as RAID-1 (i.e. not that good)

The conclusion is that “far” configurations are good for reading but not so good for writing. However, both “near” and “far” configurations allow you to use fewer than four disks and use an odd number of drives.

Just as Nested RAID allowed you to combine RAID levels, Linux md allows you to combine “near” and “far” RAID configurations. For example, in a m=2 and f=2 configuration with four drives (the minimum allowed) the data layout looks the following:

Disk 1    Disk 2    Disk 3    Disk 4
------    ------    ------    ------
  A1        A1        A2        A2
  A3        A3        A4        A4
  A5        A5        A6        A6
  A7        A7        A8        A8
  ..        ..        ..        ..
  A2        A2        A1        A1
  A4        A4        A3        A3
  A6        A6        A5        A5
  A8        A8        A7        A7
  ..        ..        ..        ..

Notice that there are four copies of each data chunk! Also notice that we can lose two drives without losing access to the data. However, in contrast to the normal RAID-10 we can lose any two drives and still retain access to all of the data. This is much better than RAID-10 where we can lose any single drive, but only specific combinations of two drives. The capacity of this particular four-drive “near” and “far” RAID configuration is the following.

Capacity = (n/2) * capacity of single disk

So we don’t gain any extra capacity relative to RAID-10 but we do gain the ability to lose any drives (a strong plus in my book).

Remember that with RAID-10 you have to have an even number of drives. The RAID-10 md drive allows you to use an odd number of drives. For the same “near” and “far” combination with m=2 and f=2, a five-drive configuration would look like the following:

Disk 1    Disk 2    Disk 3    Disk 4    Disk 5
------    ------    ------    ------    ------
  A1        A1        A2        A2        A3
  A3        A4        A4        A5        A5
  A6        A6        A7        A7        A8
  A8        A9        A9        A10       A10
  ..        ..        ..        ..        ..
  A2        A3        A1        A1        A2
  A5        A5        A3        A4        A4
  A7        A8        A6        A6        A7
  A10       A10       A8        A9        A9
  ..        ..        ..        ..        ..

Again, notice that there are four copies of the data on the arrays but we’ve increased the capacity of the RAID array because we’ve a fifth drive. So we have retained the ability to lose any two drives without losing data (a HUGE plus in my opinion), and we’ve gained capacity which is,

Capacity = 3/5 * capacity of single disk

Summary

Nested RAID was created to combine the performance and/or data redundancy features of the standard RAID levels to give us a wider range of RAID behavior. Moreover, it allows us to improve the data redundancy of standard RAID levels that are focused on performance or to improve the performance of standard RAID levels that are focused on data redundancy.

In this article, we reviewed the basics of Nested RAID and the typical naming convention for them. Don’t forget that the first number to the left is the lowest RAID configuration and then as you move to the right in the RAID name, more RAID levels are added. For example, RAID-01 starts with two groups of RAID-0 and then combines them with RAID-1 (mirroring).

How you combine the standard RAID levels can impact the overall behavior. This means that RAID-01 is not the same as RAID-10. While they are very similar, RAID-10 has better recovery behavior if you lose a drive and should have a faster rebuild time if a disk is lost. So the moral of the story is that RAID X+Y is not the same as RAID Y+X.

Finally we examined the Linux md RAID-10 driver and it’s unique capabilities. It has uses two concepts: “near” and “far” when deciding where to place data chunks on the drives. For both concepts, it allows you to use as few as two drives (normally RAID-10 requires at least four drives) and also allows you to use an odd number of drives (again, something that a normal RAID-10 doesn’t allow) which increases capacity. You can also combine the “near” and “far” concepts using a minimum of four drives, allowing you to lose any two drives in the array without losing data. In contrast, RAID-10 allows you to lose one drive and then certain other drives without losing data (an unlikely scenario).

In the next article in our RAID series, we’ll examine using RAID-5 and RAID-6 in Nested RAID to discover what kind of RAID combinations we can create and how they behave.

Comments are closed.