Forums

Resolved
4 votes
Hi,
1. To install RAID can be a challenge.
2. To use >2TB disks with GPT partitions to boot your system can also present some difficulties if not done right.
3. To use UEFI booting instead of the old BIOS is absolutely rocket science. To the point that the Anaconda installer will crash when trying to write the boot loader config on the disk, at least on my ASUS motherboard based PC. (That is why I market this post as a Bug)

Now, combine all three above and you can look forward to hours behind the keyboard before you finally give up (or succeed). Well, two weeks ago I decided to try just that.

Today, I finally made it (I think...). Not sure what will happen at the next kernel update, but hopefully it will work. To give any other some hints on what to do if you would like to try this. Here you find some info:

Background: I have three 3 TB disks and an ordinary x86-64 PC, with UEFI enabled motherboard, that I run as "home server". The CPU load as well as the disk load is moderate at worst. I want RAID to have some fault tolerance, but also want maximum space for storage. I therefore decided to have RAID 5 for the root file system and RAID 1 for boot and swap. I prepared a USB-stick with the installation ISO using a Linux-system (See instructions on this site).

Note: When installing for UEFI, the installation media MUST be booted using UEFI (not by bios). In your "bios"-screen setup, your installation media should have a name similar to [UEFI]:my-usb-stick .

Note2: At least on my system, the installation will fail if I select to have the boot loader installed automatically. It may my ASUS motherboard that has some bugs in the firmware that causes this, I do not know. But I do have the lastest firmware installed.

Note3: When dealing the UEFI and GPT there are a number of things that changes from "how it always has been done". For instance, DO NOT use grub-install ! It will kill your setup if I understand correctly. Instead you should use grub2-mkconfig. Also, the files ends up in different directories, why following different advices on the Internet is not always a good idea...

Installation:
Insert the prepared usb-stick, start the PC and make sure it boot the usb using UEFI
Select your language and keyboard
Add "base" to programs to install

Press CTRL-ALT-F2 to open a command window
- For each drive do: gdisk /dev/sda . type "x" and then "z", Confirm = Y, Confirm = Y to erase MBR too.
- Enter gdisk /dev/sda again. Type n = new -> Partition = 1 -> Start = Enter -> End = +50M ->Type: ef00 (EFI boot partition) -> w = write (if you plan to have multiboot system, increase size)
- Repeat above two lines for each of /dev/sdb and /dev/sdc
- Format the EFI partition to FAT (do NOT use any other filesystem other than FAT16 or FAT32)
mkfs.vfat /dev/sda1
mkfs.vfat /dev/sdb1
mkfs.vfat /dev/sdc1
Now go back to the installation with Ctrl + Alt + F6

Select Installation target menu. Select all three disks. IMPORTANT: UN-SELECT boot loader to be installed on any disk (down to the left. We will do this manually instead)
Select "I will do the partitioning myself"
Press "Ready" and you will now see the partitioning menu.

Click on the "reload icon" down to the left. It will read the existing partitions from the disk again. You will be brought back to the menu. Choose "Ready" again.
Select the already defined EFI-partition on sda1. Enter text for the mounting point: /boot/efi
Select the already defined EFI-partition on sdb1. Enter text for the mounting point: /boot/efi2
Select the already defined EFI-partition on sdc1. Enter text for the mounting point: /boot/efi3

Select "+", select /boot, select 500M, select RAID 1, filesystem xfs (all three disks should be automatically selected)
Select "+", select /swap, select 4G (or any other suitable size for your system), select RAID 1, check that it will be on all three disks
Select "+", select /, select 100% (or empty field), select RAID 5, file system xfs
Press "Ready"

Activate your network connections
Start installation, set root password

IMPORTANT: When the installation is done, but BEFORE doing the reboot: Press CTRL-ALT-F2 again for the command window

Do this:
chroot /mnt/sysimage
nano /etc/fstab (check that there are mountpoints for sda1, sdb1 and sdc1 at /boot/efi, /boot/efi2 and /boot/efi3. Also add ,nofail on all three EFI-partitions to not stumble if one drive is down at boot) Save by pressing CTRL-o and exit by pressing CTRL-x

The /etc/fstab file should now look something like this

#
# /etc/fstab
# Created by anaconda on Fri Oct 30 18:08:18 2015
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=dcfd3f3c-5512-41ed-86ff-a98a8a1cc9f0 / xfs defaults 0 0
UUID=ad3c3481-2715-4743-9f22-d54fd050ecf1 /boot xfs defaults 0 0
UUID=0511-101A /boot/efi vfat umask=0077,shortname=winnt,nofail 0 0
UUID=071E-DAFB /boot/efi2 vfat umask=0077,shortname=winnt,nofail 0 0
UUID=0A12-82C5 /boot/efi3 vfat umask=0077,shortname=winnt,nofail 0 0
UUID=fff64f1a-553f-465c-ac40-640a955c5dbe swap swap defaults 0 0

check that sda1, sdb1 and sdc1 are mounted by entering df
Now it is time to fix so that the system will boot. dracut will need some kernel cmdline parameters (otherwise due to some stupid(?) new idea in UEFI/Centos7 the boot will not succeed since the raid is not automatically assembled...)

dracut --print-cmdline >> /etc/dracut.conf
nano /etc/dracut.conf
Encapsulate the whole last (very long...) line with qoutes and add kernel_cmdline+= so it looks like this:
kernel_cmdline+=" rd.md.uuid=afa91bad:eb4213d1:01870edc:bfe82fce more params... "
save and exit by pressing Ctrl-o Ctrl-x

dracut --force
grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg (I think you can ignore the warnings about "null" devices, but I do not know...)
cp -av /boot/efi/. /boot/efi2/ (this will copy the bootfiles to the second drive)
cp -av /boot/efi/. /boot/efi3/
efibootmgr --create --disk /dev/sdc --label "ClearOS 7 sdc" --load "\\EFI\\centos\\grubx64.efi" (creates a boot entry in your motherboard NVRAM for the third disc)
efibootmgr --create --disk /dev/sdb --label "ClearOS 7 sdb" --load "\\EFI\\centos\\grubx64.efi"
efibootmgr --create --disk /dev/sda --label "ClearOS 7 sda" --load "\\EFI\\centos\\grubx64.efi"

do: ctrl-alt-f6
Now you can go ahead and reboot

Done !

Your system shall now be possible to boot from any of the three drives using UEFI.


The RAID setup should now work, however IF it seems that the PC hangs during the reboot. WAIT for a couple of minutes before you do anything. Most likely you then end up in "Dracut emergency mode" due to that the raid was not "started":
If so, at the dracut prompt, type: mdadm --assemble --scan
Check that it now finds and starts all raid (md125, md126, md127)
Type: exit (to continue the booting, that now should work. However you need to find the configuration fault and then initate a new "dracut --force" and grub-mkconfig after that)


Hopefully you find this long post useful. Please note that I am everything but an expert in this field, and the above could be "NOT a good way" to configure your system. You have been warned.

/Fred
Saturday, October 31 2015, 10:19 PM
Like
1
Share this post:
Responses (1)
  • Accepted Answer

    chrille
    chrille
    Offline
    Sunday, January 03 2016, 12:32 AM - #Permalink
    Resolved
    0 votes
    Thanks for the information Fred,

    I had the same problem when installing 7.1 and after a couple of hours struggling I ended up with installing version 6.7.0 final since I, at that moment, needed the server online again. I will bookmark your post and when I have the time I will try your solution and get back to you. I also have Raid (1), 2 pieces of 3 TB disks and ASUS motherboard.

    Regards,

    :)

    /Chrille
    The reply is currently minimized Show
Your Reply