We have ClearOS 7 Community systems configured to boot with UEFI that are failing to boot after the latest GRUB update.
Error:
System BootOrder not found. Initializing defaults.
Creating boot entry "Boot0002" with label "ClearOS" for file "\EFI\clearos\shimx64.efi"
Failed to open \EFI\clearos\grubx64.efi -- Not Found
Failed to load image \EFI\clearos\grubx64.efi: Not Found
start_image() returned Not Found
StartImage failed: Not Found
I've determined that one of the latest GRUB updates is moving grubx64.efi from /boot/efi/EFI/clearos/ to /boot/efi/EFI/centos/ Of course the boot configuration is looking for the grubx64.efi to be in the clearos directory, not the (new with the update) centos one. Therefore boot fails.
I don't know precisely which of the following packages broke it, but the problem appeared after version 2.02-0.87.v7.6 of all the following were installed during a regular yum run:
grub2
grub2-common
grub2-efi-x64
grub2-pc
grub2-pc-modules
grub2-tools
grub2-tools-extra
grub2-tools-minimal
Error:
System BootOrder not found. Initializing defaults.
Creating boot entry "Boot0002" with label "ClearOS" for file "\EFI\clearos\shimx64.efi"
Failed to open \EFI\clearos\grubx64.efi -- Not Found
Failed to load image \EFI\clearos\grubx64.efi: Not Found
start_image() returned Not Found
StartImage failed: Not Found
I've determined that one of the latest GRUB updates is moving grubx64.efi from /boot/efi/EFI/clearos/ to /boot/efi/EFI/centos/ Of course the boot configuration is looking for the grubx64.efi to be in the clearos directory, not the (new with the update) centos one. Therefore boot fails.
I don't know precisely which of the following packages broke it, but the problem appeared after version 2.02-0.87.v7.6 of all the following were installed during a regular yum run:
grub2
grub2-common
grub2-efi-x64
grub2-pc
grub2-pc-modules
grub2-tools
grub2-tools-extra
grub2-tools-minimal
Share this post:
Accepted Answer
Nick Howitt wrote:
Are you booting off a rescue/installation disk?
The threading on this forum is truly, um...
Anyway...
Being a bit of a Linux clutz, I missed step 2a on my first attempt!
1. Boot to Recovery environment (installation CD/Flashdrive, under the Troubleshooting menu)
2. proceed with the mount sysimage option
2a ENTER
3. chroot /mnt/sysimage
4. cp /boot/efi/EFI/centos/grubx64.efi /boot/efi/EFI/clearos/grubx64.efi
5. grub2-mkconfig -o /boot/efi/EFI/clearos/grub.cfg
6. exit
7. exit (again) -- system will reboot; remove recovery drive.
Cheers
Responses (66)
-
Accepted Answer
If your system has installed this broken version, and you haven't rebooted, you can simply copy the file and you'll be fine to reboot:
cp /boot/efi/EFI/centos/grubx64.efi /boot/efi/EFI/clearos/grubx64.efi
However, if you've already rebooted and your system is in an error state, you may be be assisted with the following:
DISCLAIMER: Use at your own risk: this is a reconstruction of some steps from testing in our lab, posting in case it helps someone. YMMV.
1. Boot to Recovery environment (installation CD/Flashdrive, under the Troubleshooting menu)
2. proceed with the mount sysimage option
3. chroot /mnt/sysimage
4. cp /boot/efi/EFI/centos/grubx64.efi /boot/efi/EFI/clearos/grubx64.efi
5. grub2-mkconfig -o /boot/efi/EFI/clearos/grub.cfg
6. exit
7. exit (again) -- system will reboot; remove recovery drive. -
Accepted Answer
Thanks for the report and workaround. This was always my greatest fear and I know it happened a couple of years ago when it was last updated, but then it was patched. I've run it on a number of EFI systems,updating to it and rebooting, so I wonder why yours triggered it. All grub packages come from a single source so they are one big update.
The other thing I don't understand is why your grub2-mkconfig command works. I was not able to get it to work with the old or new version as it always just put its output to screen so I had to redirect the output to /boot/efi/EFI/clearos/grubx64.efi -
Accepted Answer
-
Accepted Answer
Hmm interesting. Well perhaps it's not as a widespread problem as I thought. When I was testing, I was doing a fresh install of ClearOS 7 (might have been a ISO of 7.7; not sure -- I'm not at the office right now), and then running yum update. We were encountering the problem going from version 2.02.2-0.76.v7 to 2.02-0.87.v7.6, FWIW. -
Accepted Answer
Nick Howitt wrote:
If anyone else gets this issue, please post a "me too" message to the thread. Looking at my primary UEFI system I don't see a problem, so I am not sure how to identify it.
Me too. It's a 2011 vintage I3 HP desktop. Edit: Happy to advise that Marvin's method got me up and running once more. ? Kudos. -
Accepted Answer
-
Accepted Answer
-
Accepted Answer
OK, there is now a fix available for testing:
(you don't need the --disablerepo switch, it just makes it a bit faster).rm -rf /var/cache/yum/
yum update grub2* --disablerepo=* --enablerepo=clearos-updates-testing
You are looking for a version number 2.02.2-0.87.v7.6.0.3 for all packages.
So far I have tested it on 3 non-EFI machines and it now creates the /boot/efi/EFI/clearos folder instead of /boot/efi/EFI/centos. I've rebooted all three. I've also installed it on my one UEFI machine, but I can't reboot it until Sunday morning. My other UEFI is a Proxmox server and it needs me to swap the disk in it to do a clean install of ClearOS then a grub2 update. I did it before for testing but the server is not in a good location and realistically it is not going to happen today.
I also have a UEFI machine in the US which I've updated and rebooted and it has come up correctly so it is looking good.
If anyone with a UEFI machine can also confirm updating and rebooting is OK, I'll release the package immediately. -
Accepted Answer
Excellent! Great work on this, Nick!
I can confirm your proposed fix does indeed work as advertised.
Steps I took:
1. Fresh installation of ClearOS 7.7 Community on the same hardware I was testing with yesterday.
2. Brought all packages up to date and rebooted
3. Installed grub update from testing repo per Nick's post
4. Verified that a /boot/efi/EFI/centos directory did Not appear after the grub update
5. Rebooted successfully.
All is well. Many thanks! -
Accepted Answer
Thanks for the post back. I'll release it now.
It was a horrible little one. I could see Dave had made a conditional change to a parameter > 2 years ago and it was not triggering. I made it absolute and tidied it up, moving it to near the top of the spec file with the rest of the parameters, not realising that there was a call to another program a bit later on in the spec file which overrode the setting. Moving it below the call to the other program fixed it. -
Accepted Answer
It happened to me too, but Marvins workaround doesn't work for me...
After step 3, I have nothing in /boot directory... so I can't copy the grubx64.efi
I have selected Troubleshooting
Then Rescue a ClearOS system
Then it asks me, to choose: 1 for Continue, 2 for Read-only mount, 3 for Skip to shell and 4 to Quit
I tried 1 and 2, but /boot is empty... When I try 3, then it says: chroot: failed to run command "/bin/bash": No such file or directory
Am I doing something wrong?
What can I do? Must I reinstall everything?? -
Accepted Answer
I agree with Dave Pitman. If you skip the Enter (like I did as well) you end up somewhere on you installation USB (/boot had a vmlinuz file and one other bit no folders) rather than on the ClearOS Disk. I'll mark Dave's post as accepted but kudos to Marvin as well for finding the issue and posting a solution so quickly.
There is a Rescue Mode HowTo but it looks like it needs to be updated with newer images and a small update to the instructions. -
Accepted Answer
-
Accepted Answer
Flash wrote:
Hmm, where should I hit ENTER? When I select Rescue a ClearOS system?
I hit ENTER like 20 times, and nothing happens, it still gives me the same 4 options...
Should I select 1, 2 or 3?
It's the default choice, 1.
Follow the amended instructions. Once you press enter, you get a different prompt and that's where the Marvin magic comes in.
Cheers -
Accepted Answer
After I select option 1, all I get is this:
==================================================================================================
==================================================================================================
==================================================================================================
==================================================================================================
And this goes on and on and on...
If I hit Alt+F2 then I get:
ClearOS 7 (Final)
Kernel 3.10.0-1160.25.1.el7.x86_64 on an x86_64
[anaconda root@localhost /]#
Is it here where I should hit the ENTER? Because it doesn't do anything...
If I hit Alt+F1 again, there are still just ====================================================================
And still no /boot/efi/EFI/centos or clearsos... nothing... after I hit chroot /mnt/sysimage -
Accepted Answer
Flash wrote:
After I select option 1, all I get is this:
==================================================================================================
==================================================================================================
==================================================================================================
==================================================================================================
And this goes on and on and on...
If I hit Alt+F2 then I get:
ClearOS 7 (Final)
Kernel 3.10.0-1160.25.1.el7.x86_64 on an x86_64
[anaconda root@localhost /]#
Is it here where I should hit the ENTER? Because it doesn't do anything...
If I hit Alt+F1 again, there are still just ====================================================================
And still no /boot/efi/EFI/centos or clearsos... nothing... after I hit chroot /mnt/sysimage
Sorry, beyond my skillset. Nick? -
Accepted Answer
I've updated the howto at https://documentation.clearos.com/content:en_us:kb_rescue_mode with all the screenshots updated. Try that. -
Accepted Answer
Thank you for the updated how to.
But after I hit 1 for Continue, at the picture in the how to there are only 2 rows of ==========================
I get gazillion rows of ======================== and it doesn't end...
I let it for 15 minutes and still only this ==========================
It doesn't get past this...
I don't see: Rescue mount
Your system has been mounted under /mnt/sysimage.
What's wrong? -
Accepted Answer
-
Accepted Answer
That does not look good and I don't know what to recommend.
If you do need to reinstall, if you have anything you need on the disk, use another distro's live cd to boot up, mount your ClearOS disk and pick off the files you want from ClearSOS to another disk before you reinstall. You may want a configuration backup from /var/clearos/configuration_backup. Flexshares are under /var/flexshare/shares. Is there anything else critical you have there which you want to recover?
If you hold on for a few hours, Marvin may come back online with some ideas. -
Accepted Answer
First, thanks Dave for updating the recovery instructions with the extra details. When I posted them, I was in a hurry to leave for an appointment, but wanted to get them out there quick. Glad it helped! I don't mind who gets the credit, Nick.
Flash, as to your issue, I'd agree with your speculation that it's not seeing your NVMe drive, so I'm assuming it's just retrying infinitely or something. The reason you're not able to see any EFI directories is that it's never getting to the point of mounting your installation disk: any shell you get at this point is just in the live recovery environment you're booted to.
Suggest that rather than having it "auto locate" your installation with option 1, you instead select option 3 for a shell in the recovery environment (again, this is Not a shell on your installation, just the live recovery OS you're running). Then run "lsblk" (without the quotes) to get a list of the storage devices the live environment can see.
If lsblk lists out your NVMe drive, great! You should be able to proceed to mount it manually and then proceed with chroot (using whatever directory you mount it as -- won't be sysimage if you use another name) and continuing with the recovery steps.
If lsblk does Not show your NVMe drive, you'll need to specify a driver file for it at the time of booting the recovery environment. See step 3 under Procedure 32.1 on this page in the RHEL Documentation for more information on this.
A side note about mounting the drive to recover files in the event boot recovery fails: ClearOS by default uses LVM for the main partition, which can complicate things a bit as the Ubuntu live CD, for example, doesn't include LVM support by default (IIRC). It's not an intractable situation, however, one just needs to go to a bit of extra work to add that package to the live CD image. Here's a blog post about adding LVM support to a Debian live CD that may be helpful if you decide to go this route.
I hope you're able to get the issue resolved! -
Accepted Answer
Thank you Marvin for the explanation, but I need a little more help.
The command "lsblk" lists my NVMe drive.
Which one should I mount? nvme0n1p1, nvme0n1p2, nvme0n1p3, nvme0n1p4 or nvme0n1p5?
p1 and p2 are boot partitions and p3 is / partition.
Is this the correct way: mount -t xfs /dev/nvme0n1p3 /mnt/sysimage ? Or must I mount p1 or p2?
Because if I do that, I still don't have nothing in /boot directory. -
Accepted Answer
-
Accepted Answer
-
Accepted Answer
If I mount p1 it says: wrong fs type, bad option, bad superblock on /dev/nvme0n1p1, missing codepage or helper program, or other error
If I mount p2, then it mounts. But then, if I do chroot /mnt/sysimage it says: failed to run command "/bin/bash": No such file or directory
If I go to /mnt/sysimage then there are files in there... -
Accepted Answer
Marvin Martin wrote:
Out of interest, won't /boot show as /mnt/sysimage/boot without chrooting? Agree that you'll have to chroot before recovering.
One redundant note just to be sure we're on the same page: after you successfully mount a partition in your recovery environment, you'll need to chroot /mnt/sysimage/ before you can go looking for boot/. -
Accepted Answer
Flash wrote:
p1 then is probably the msdos partition on a GPT labelled disk. Then p2 will be /boot.
If I mount p1 it says: wrong fs type, bad option, bad superblock on /dev/nvme0n1p1, missing codepage or helper program, or other error
If I mount p2, then it mounts. But then, if I do chroot /mnt/sysimage it says: failed to run command "/bin/bash": No such file or directory
If I go to /mnt/sysimage then there are files in there...
If it does not run bash, does it stay in sh? It should not matter as most commands are similar but I am worried.....
... because I wonder if you need to mount both; mount / first into /mnt/sysimage then perhaps chroot to it. Then mount p2 into /boot. Marvin may need to comment here. -
Accepted Answer
-
Accepted Answer
Out of interest, won't /boot show as /mnt/sysimage/boot without chrooting? Agree that you'll have to chroot before recovering.
Yes, you're correct. I thought about mentioning this, but since I'm thinking you'd need to have changed root anyway to reconfigure grub, I had left it out to keep the discussion simple.
Marvin may need to comment here.
Hmm. I'll try to help, but must say I'm not super-experienced in "non-standard" boot recovery situations!
Flash, when you're at the shell in your recovery environment (not chroot'd anywhere), run "lsblk -f" -- this should show the filesystem type for each partition, which will be helpful to specify the correct filesystem type when you mount a partition.
As far as referencing shell utilities not in a given partition when using chroot, this IBM article may be helpful. That said, Nick's comment about changing directory vs chroot is a good one to bring up here: if you just want to "see what's there", while at your recovery environment shell, you can simply "cd /mnt/sysimage/" and that should get you to the root of the mounted partition.
If this doesn't get you out of the bind, could you post the output of "ls -lh" of the root of the partitions after you mount them? (you'll either need to chroot or use the cd /mnt/sysimage/ method mentioned above). -
Accepted Answer
While you were posting I was writing this:
https://pario.no/2015/11/02/how-to-mount-lvm-partitions-from-rescue-mode/ and https://documentation.online.net/en/dedicated-server/rescue/mount-lvm-partition show how to mount an lvm main partition like / could be.
On my system:
(it didn't like the --all switch)[root@microserver ~]# lvm vgscan -v
Reading volume groups from cache.
Found volume group "main" using metadata type lvm2
[root@microserver ~]# lvm lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
root main -wi-ao---- 924.84g
swap main -wi-ao---- 2.00g
My /etc/fstab has:
So, presumably:/dev/mapper/main-root / ext4 defaults 1 1
will work but it is lightly inconsistent with the two examples I linked to which indicate:mount /dev/mapper/main-root /mnt/sysimage
should work, but, for me the two symlink together. Note there is no need to specify the filesystem type (xs, ext4 etc as it should be auto-detected)mount /dev/main/root /mnt/sysimage
If that works, you can chroot then issue the command "mount-a" which shuold mount all the other partitions. -
Accepted Answer
OK. I have now successfully mounted the p1 partition. It is vfat partition... don't now why.
I have now a /mnt/sysimage/EFI directory. In there there are BOOT centos and clearos.
Is this OK? Should I just copy grubx64.efi from centos to clearos?
If I write chroot /mnt/sysimage I still get the same answer: failed to run command "/bin/bash": No such file or directory -
Accepted Answer
-
Accepted Answer
So can I just copy the grubx64.efi or not?
Yes, IMO you can go for it.
I take Nick's last post to be all about how to mount an LVM partition along with the others so you could have all of them mounted in your recovery environment. I'm not sure that this is needed, but perhaps Nick has a particular reason for recommending this. Maybe for the grub reconfiguration? -
Accepted Answer
Flash,
I think that's some progress, though the "no such device" is a bit worrisome. Basically now the EFI system is starting up, but it's saying, hey, things aren't where I was expecting them. At this point, if we can trigger the grub2-mkconfig command on the right partition as per the recovery instructions, we might be getting places.
To do this:
Boot back into your recovery environment again as before, using the option 3 for the shell.
Mount the main system partition.
And chroot to that. (If this doesn't work, post the output of "lsblk -f" and "pwd")
Pick up with the recovery steps at #5 and tell us how it goes. If you get errors, please post the output.
EDIT: updated instructions. If I'm thinking correctly, you will need to be in the main system partition (not the boot one that I think you were in earlier). -
Accepted Answer
-
Accepted Answer
Please login to post a reply
You will need to be logged in to be able to post a reply. Login using the form on the right or register an account if you are new here.
Register Here »