Linux: Rebuild RAID After Disk Failure

We begin by using list the partition tables present on the server to find the brand new hard disk extraupdate

[root@mail ~]# fdisk -lDisk /dev/hda: 500.1 GB, 500107862016 bytes
255 heads, sixty three sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/hda1 1 13 104391 fd Linux raid autodetect
/dev/hda2 14 6387 51199155 fd Linux raid autodetect
/dev/hda3 6388 6769 3068415 fd Linux raid autodetect
/dev/hda4 6770 60801 434012040 fd Linux raid autodetect
Disk /dev/hdc: 500.1 GB, 500107862016 bytes
255 heads, sixty three sectors/music, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/hdc doesn’t include a valid partition table
Disk /dev/md1: 52.Four GB, 52427816960 bytes
2 heads, 4 sectors/tune, 12799760 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk /dev/md1 does not include a legitimate partition table
Disk /dev/md3: 444.4 GB, 444428255232 bytes
2 heads, four sectors/track, 108502992 cylinders
Units = cylinders of eight * 512 = 4096 bytes
Disk /dev/md3 doesn’t include a legitimate partition desk
Disk /dev/md2: 3141 MB, 3141926912 bytes
2 heads, 4 sectors/music, 767072 cylinders
Units = cylinders of eight * 512 = 4096 bytes
Disk /dev/md2 doesn’t include a legitimate partition desk
Disk /dev/md0: 106 MB, 106823680 bytes
2 heads, four sectors/music, 26080 cylinders
Units = cylinders of eight * 512 = 4096 bytes
Disk /dev/md0 doesn’t incorporate a valid partition table
Disk /dev/sda: a thousand.2 GB, 1000204886016 bytes
255 heads, 63 sectors/music, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 1 121601 976760001 eighty three Linux
[root@mail ~]#
The above output indicates that there are two hard disks (hda and hdc). Partitions are seen on hda but hdc does not include a legitimate partition table. In truth this is the alternative for the defective disk.
To view the health of the Software RAID type cat /proc/mdstat

READ MORE :

 

[root@mail ~]# cat /proc/mdstat
Personalities: [raid1]
md0: energetic raid1 hda1[1]
104320 blocks [2/1] [_U]md2: lively raid1 hda3[1]
3068288 blocks [2/1] [_U]
md3: energetic raid1 hda4[1]
434011968 blocks [2/1] [_U]
md1: lively raid1 hda2[1]
51199040 blocks [2/1] [_U]
unused gadgets:
[root@mail ~]#

linux

Note that [2/1] suggests that there are 2 disks inside the array but only 1 is lively. The symbols [_U] imply that one of the two disks is missing from the array and must be reconfigured.
Since the new disk is /dev/hdc we need create the walls required for the RAID. Use fdisk to create the boot partition on the new disk.

[root@mail ~]# fdisk /dev/hdc
Device includes neither a legitimate DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in reminiscence best,
till making a decision to write them. After that, of direction, the previous
content may not be recoverable.The range of cylinders for this disk is set to 60801.
There is not anything wrong with that, however this is larger than 1024,
and will in certain setups motive problems with:
1) software that runs at boot time (e.G., vintage versions of LILO)
2) booting and partitioning software from different OSs
(e.G., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition desk four may be corrected with the aid of w(ceremony)
Command (m for assist): p
Disk /dev/hdc: 500.1 GB, 500107862016 bytes
255 heads, sixty three sectors/song, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
Command (m for assist): n
Command movement
e extended
p primary partition (1-four)
p
Partition range (1-four): 1
First cylinder (1-60801, default 1): 1
Last cylinder or +size or +sizeM or +sizeK (1-60801, default 60801): +100M
Command (m for help): p
Disk /dev/hdc: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/hdc1 1 13 104391 83 Linux
Command (m for assist): t
Selected partition 1
Hex code (kind L to list codes): l
zero Empty 1e Hidden W95 FAT1 80 Old Minix be Solaris boot
1 FAT12 24 NEC DOS 81 Minix / vintage Lin bf Solaris
2 XENIX root 39 Plan 9 eighty two Linux switch / So c1 DRDOS/sec (FAT-
3 XENIX usr 3c PartitionMagic eighty three Linux c4 DRDOS/sec (FAT-
four FAT16 <32M forty Venix 80286 84 OS/2 hidden C: c6 DRDOS/sec (FAT-
5 Extended forty one PPC PReP Boot eighty five Linux extended c7 Syrinx
6 FAT16 42 SFS 86 NTFS extent set da Non-FS data
7 HPFS/NTFS 4d QNX4.X 87 NTFS quantity set db CP/M / CTOS /.
Eight AIX 4e QNX4.X 2nd part 88 Linux plaintext de Dell Utility
9 AIX bootable 4f QNX4.X 3rd part 8e Linux LVM df BootIt
a OS/2 Boot Manag 50 OnTrack DM 93 Amoeba e1 DOS access
b W95 FAT32 fifty one OnTrack DM6 Aux ninety four Amoeba BBT e3 DOS R/O
c W95 FAT32 (LBA) fifty two CP/M 9f BSD/OS e4 SpeedStor
e W95 FAT16 (LBA) 53 OnTrack DM6 Aux a0 IBM Thinkpad hello eb BeOS fs
f W95 Ext’d (LBA) fifty four OnTrackDM6 a5 FreeBSD ee EFI GPT
10 OPUS fifty five EZ-Drive a6 OpenBSD ef EFI (FAT-12/sixteen/
11 Hidden FAT12 fifty six sfdisk /dev/hdc
Checking that no-one is using this disk right now…
OKDisk /dev/hdc: 60801 cylinders, 255 heads, 63 sectors/track
Old situation:
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
Device Boot Start End #cyls #blocks Id System
/dev/hdc1 0+ 12 13- 104391 fd Linux raid autodetect
/dev/hdc2 0 – 0 0 0 Empty
/dev/hdc3 0 – 0 0 0 Empty
/dev/hdc4 0 – 0 0 0 Empty
New situation:
Units = sectors of 512 bytes, counting from 0
Device Boot Start End #sectors Id System
/dev/hdc1 63 208844 208782 fd Linux raid autodetect
/dev/hdc2 208845 102607154 102398310 fd Linux raid autodetect
/dev/hdc3 102607155 108743984 6136830 fd Linux raid autodetect
/dev/hdc4 108743985 976768064 868024080 fd Linux raid autodetect
Warning: no primary partition is marked bootable (active)
This does not matter for LILO, but the DOS MBR will not boot this disk.
Successfully wrote the new partition table
Re-reading the partition table…
If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
to zero the first 512 bytes: dd if=/dev/zero of=/dev/foo7 bs=512 count=1
(See fdisk(8).)
[root@mail ~]#

Typing fdisk -l once again displays that the partitions on both disks are in fact identical

rebuild

[root@mail ~]# fdisk -lDisk /dev/hda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/hda1 1 13 104391 fd Linux raid autodetect
/dev/hda2 14 6387 51199155 fd Linux raid autodetect
/dev/hda3 6388 6769 3068415 fd Linux raid autodetect
/dev/hda4 6770 60801 434012040 fd Linux raid autodetect
Disk /dev/hdc: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/hdc1 1 13 104391 fd Linux raid autodetect
/dev/hdc2 14 6387 51199155 fd Linux raid autodetect
/dev/hdc3 6388 6769 3068415 fd Linux raid autodetect
/dev/hdc4 6770 60801 434012040 fd Linux raid autodetect
Disk /dev/md1: 52.4 GB, 52427816960 bytes
2 heads, 4 sectors/track, 12799760 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk /dev/md1 doesn’t contain a valid partition table
Disk /dev/md3: 444.4 GB, 444428255232 bytes
2 heads, 4 sectors/track, 108502992 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk /dev/md3 doesn’t contain a valid partition table
Disk /dev/md2: 3141 MB, 3141926912 bytes
2 heads, 4 sectors/track, 767072 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk /dev/md2 doesn’t contain a valid partition table
Disk /dev/md0: 106 MB, 106823680 bytes
2 heads, 4 sectors/track, 26080 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk /dev/md0 doesn’t contain a valid partition table
Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 1 121601 976760001 83 Linux
[root@mail ~]#

Now that the partitions are identical we can re-sync the information across the RAID configuration
Type mdadm -a /dev/md0 /dev/hdc1 to add /dev/hdc1 partition to the /dev/md0 RAID partition.

[root@mail ~]# mdadm -a /dev/md0 /dev/hdc1
mdadm: added /dev/hdc1
[root@mail ~]#
Type cat /proc/mdstat to view the re-syncing process

[root@mail ~]# cat /proc/mdstat
Personalities: [raid1]
md0: active raid1 hdc1[2] hda1[1]
104320 blocks [2/1] [_U]
[=>… ] restoration = 7.Eight% (8512/104320) end=0.7min speed=2128K/secmd2: lively raid1 hda3[1]
3068288 blocks [2/1] [_U]
md3: energetic raid1 hda4[1]
434011968 blocks [2/1] [_U]
md1: lively raid1 hda2[1]
51199040 blocks [2/1] [_U]
unused devices:
[root@mail ~]# cat /proc/mdstat
Personalities: [raid1]
md0: energetic raid1 hdc1[2] hda1[1]
104320 blocks [2/1] [_U]
[=======>… ] recuperation = 37.2% (39808/104320) finish=zero.5min velocity=1809K/sec
md2: active raid1 hda3[1]
3068288 blocks [2/1] [_U]
md3: energetic raid1 hda4[1]
434011968 blocks [2/1] [_U]
md1: lively raid1 hda2[1]
51199040 blocks [2/1] [_U]
unused devices:

The below suggests how the output ought to be when the re-syncing process has ended. Note the “[UU]”. This confirms that each disks are energetic.

[root@mail ~]# cat /proc/mdstat
Personalities: [raid1]
md0: active raid1 hdc1[0] hda1[1]
104320 blocks [2/2] [UU]md2: energetic raid1 hda3[1]
3068288 blocks [2/1] [_U]
md3: active raid1 hda4[1]
434011968 blocks [2/1] [_U]
md1: energetic raid1 hda2[1]
51199040 blocks [2/1] [_U]
unused devices:
[root@mail ~]#

The equal system needs to be completed on the opposite walls

[root@mail ~]# mdadm -a /dev/md1 /dev/hdc2
mdadm: delivered /dev/hdc2
[root@mail ~]# mdadm -a /dev/md2 /dev/hdc3
mdadm: brought /dev/hdc3
[root@mail ~]# mdadm -a /dev/md3 /dev/hdc4
mdadm: delivered /dev/hdc4
[root@mail ~]# cat /proc/mdstat
Personalities: [raid1]
md0: energetic raid1 hdc1[0] hda1[1]
104320 blocks [2/2] [UU]md2: active raid1 hdc3[2] hda3[1]
3068288 blocks [2/1] [_U]
resync=DELAYED
md3: active raid1 hdc4[2] hda4[1]
434011968 blocks [2/1] [_U]
resync=DELAYED
md1: energetic raid1 hdc2[2] hda2[1]
51199040 blocks [2/1] [_U]
[>… ] restoration = zero.0% (37376/51199040) end=501.5min speed=1698K/sec
unused devices:
[root@mail ~]#

Finally we want to put in GRUB on the second one disk’s MBR.
[root@mail ~]# grub
GNU GRUB version 0.97 (640K lower / 3072K higher reminiscence)[ Minimal BASH-like line editing is supported. For the first word, TAB
lists possible command completions. Anywhere else TAB lists the possible
completions of a device/filename.]
grub> find /grub/grub.Conf
(hd0,zero)
(hd1,0)
grub> root (hd1,0)
Filesystem type is ext2fs, partition kind 0xfd
grub> setup (hd1)
Checking if “/boot/grub/stage1” exists… No
Checking if “/grub/stage1” exists… Sure
Checking if “/grub/stage2” exists… Yes
Checking if “/grub/e2fs_stage1_5” exists… Sure
Running “embed /grub/e2fs_stage1_5 (hd1)”… 15 sectors are embedded.
Succeeded
Running “set up /grub/stage1 (hd1) (hd1)1+15 p (hd1,0)/grub/stage2 /grub/grub.Conf”… Succeeded
Done.
Grub> quit