Hello,
As you may be aware, it's long been on the BitFolk TODO list to
implement a better way (than pygrub) to boot your VPSes.
Unfortunately I have to explain some rather lengthy history so you
can know how things arrived at where they are now, and what the
options are for the future.
If you want to skip all that then please do at least skip down to
the heading "== Now ==" so you are aware of what I am proposing to
do. Without reading the history though it may not make sense.
== Ancient times ==
Kernels and initrds had to live outside of the guest in dom0. Users
couldn't update their own kernel but had to find some means to sync
their kernel and initrd to some place in the dom0 (the privileged
gust that controls everything else). People came up with various
hacks to allow users to do that, the disadvantage being that quite a
lot of that had to run in dom0.
== Still rather a long time ago ==
Someone invented pygrub, a Python implementation of something that
looked a but like GNU GRUB 0.x. It was capable of looking inside an
image file, block device or partitions on same, to try to find a
file called menu.lst.
If it found that file it would try to parse it like GRUB would,
display a menu emulating GRUB's menu, and pass the chosen kernel,
initrd and boot arguments to Xen.
Advantages:
- Users could install their own kernel packages, since these would
maintain a menu.lst file in their guest, and pygrub would then
pick up the changes when they next booted.
- Users could control their own kernel command line since this also
was in the menu.lst file.
Disadvantages:
- It still all runs in dom0. pygrub is opening an actual filesystem
that is supplied by the user, in the context of a user in dom0,
and that is not a particularly safe thing to be doing.
- It isn't actual GRUB, it's just trying to emulate it. That means
it doesn't behave quite the same and doesn't support every single
thing that is valid in a real GRUB menu.lst file.
- As a further consequence of the above, pygrub is relying on
userspace filesystem access libraries that aren't part of actual
GRUB. This sometimes lead to surprising discrepancies between what
GRUB was expected to support and what pygrub would support, such
as XFS filesystems, XZ-compressed initrds, etc.
- GRUB 0.x is now referred to as GRUB-legacy. GRUB moved on to 1.x,
it got a lot more complicated, and it now puts its configuration
in /boot/grub/grub.cfg. Many Linux distributions gave up on
GRUB-legacy and only support grub.cfg now, so users of such are
left to maintain their own menu.lst file.
On Debian and Ubuntu you can still install the package grub-legacy
and it will maintain menu.lst, but it's getting increasingly
persistent about wanting you to move to grub.cfg.
This is the boot method that BitFolk has almost always been using.
Up until a couple of months ago there was still one customer who for
various reasons had to have a kernel hard-coded in their dom0 config
file, but now there's none. Everyone's on pygrub.
== Recent-ish history ==
Around 2010 in light of the above disadvantages, a member of the Xen
project ported GRUB 0.x to boot as a paravirtual guest under Xen.
That's the same sort of thing that your VPSes are. So you get a VM
started, and the VM runs actual GRUB. Once GRUB knows what it wants
to boot, it chain loads to that.
This was known as "pvgrub", an awful name really since it differs
from pygrub in only the descender of one letter. Should have passed
that one by the marketing department first.
Advantages:
- Runs only inside a VM.
- Still allows user control.
Disadvantages:
- Part of the Xen code tree, not really packaged anywhere.
- Still only GRUB-legacy support.
Due to it still only supporting GRUB-legacy, which wasn't any
different from what pygrub supports, BitFolk did not pursue it.
We ran into a couple of sticky situations such as when XZ-compressed
kernels became popular in Debian and pygrub didn't support that, but
we worked around it¹.
Still it became more pressing to support a better boot system
because it would only be a matter of time before more distributions
cease to support menu.lst or do something else that pygrub doesn't
support.
== Just 3¼ years ago! ==
A GNU GRUB committer added PV-booting support to upstream GRUB 2:
https://lists.xen.org/archives/html/xen-devel/2013-11/msg01216.html
What this means is that using only the upstream GRUB 2 binaries you
can generate a GRUB-as-kernel image that can be booted as Xen PV
virtual machine, which can then do everything that GRUB 2 can
normally do. Namely look inside your block devices for GRUB configs
and boot your own kernels.
It took a couple of years for this to filter down to being supported
in Debian stable and to iron out some bugs, so, fast forward to…
== Now ==
My recent experimentations with GRUB indicate that it may now be a
good time to switch to this method of booting instead of pygrub.
Here's how I am thinking of doing it.
- BitFolk's 64-bit or 32-bit GRUB2 image is booted as your kernel,
in your VM.
- It checks for existence of:
- /boot/xen/pvboot-x86_64.elf
- /xen/pvboot-x86_64.elf
- /boot/xen/pvboot-i386.elf
- /xen/pvboot-i386.elf
on any of your block devices/partitions.
If any of those are found then a menu entry for chainloading to
your own bootloader at that path is created.
- It checks for existence of:
- /boot/grub/grub.cfg
- /grub/grub.cfg
on any of your block devices/partitions.
If either of those are found then a menu entry for reading your
GRUB 2 config from that path is created.
- It checks for existence of:
- /boot/grub/menu.lst
- /grub/menu.lst
on any of your block devices/partitions.
If either of those are found then a menu entry for reading a
GRUB-legacy config from that path is created.
- The available menu entries are displayed for 5 seconds. If no
selection is made then a boot attempt is made for each menu
entry, starting at the top and going in order.
- Selection of a menu entry (or allowing it to time out and be
selected by default) will result in some text being displayed for
2 seconds explaining what is going to happen, e.g.:
Loading your GRUB config at (xen/xvda,msdos1)/boot/grub/grub.cfg...
would indicate that a GRUB2 config was found on the first MSDOS
partition of the Xen disk called xvda.
The message and 2 second wait could be cancelled by pressing
Escape though obviously you'd have to be looking at the console to
do that.
As you can see, the order of boot method is:
1. Chainload to your own bootloader.
2. Use your GRUB2 config.
3. Use your GRUB-legacy config.
Although there is some limited interactivity in this proposal (you
get to select a menu item if you're looking at your console),
obviously servers do need to boot automatically, so assuming no
intervention then after 5 seconds it's going to pick the first
available from the above and then after 2 seconds more it will do
it.
Unfortunately the configuration for how GRUB will behave has to be
baked into the binary image itself so I can't think of an easy way
to allow you to select your own timeouts or remove them altogether,
not on a per-user basis anyway.
It does mean a 7 second delay to booting, or more if for some reason
one of the earlier steps fails. A step can fail if the image or
config is incorrect. BitFolk's GRUB will only be checking for the
existence of a file path, not that what is there is valid. So for
example if you have GRUB2 installed so there is a
/boot/grub/grub.cfg file present but then you also touch the file
/boot/xen/pvboot-x86_64.elf, it will try to chainload to that first.
If that fails then it will fall back to parsing /boot/grub/grub.cfg,
which will add another 2 second pause.
Perhaps it would be acceptable to reduce the initial 5 second
countdown to 2 seconds? Pressing any key aborts the countdown and
gives you as long as you like to choose a menu entry, as usual with
GRUB.
The chainloading option hasn't been mentioned yet. In practice I
don't expect many people will be interested in it, but it does
provide for running your own bootloader.
At the moment although you may have GRUB-legacy installed in order
to maintain a menu.lst file, your bootloader is never actually
executed, only its config file is parsed, by pygrub. I am proposing
to move to using GRUB2 to parse your grub.cfg, but still this is
BitFolk's GRUB2 binary that is being run, not yours.
If for some reason you did need to run your own bootloader, perhaps
because you need to boot off of something that is not supported by
the GRUB binary that BitFolk is using, then you could install your
own bootloader that supports the PV boot protocol.
As far as I am aware only Debian has distribution support for this.
As of Debian jessie, installing grub-xen in your guest installs a
GRUB-as-PV-kernel image in /boot/xen/pvboot-$ARCH.elf. Using this
option, the last thing that runs before your VPS's kernel would be
your VPS's bootloader.
As I say, I expect for most people that will be unneeded
complication, as you don't really care whose bootloader is being
run, just that it boots. But the option would be there should you
need it.
I am not intending for the switching to pvgrub from pygrub to be
optional or reversible. I have put some effort into trying to ensure
that pvgrub will boot from a guest's menu.lst and all customers
currently have a menu.lst, so I am hoping I do not need to build in
a setting for this. Naturally if we switch and then someone finds
out it doesn't work then we can switch them back to pygrub until the
problem is worked out.
At the moment I have tested the following configurations:
- 64-bit Debian jessie guest, single root filesystem on xvda1:
- Chainload to GRUB, parse /boot/grub/grub.cfg
- Parse /boot/grub/grub.cfg
- Parse /boot/grub/menu.lst
- i686 Debian jessie guest, single root filesystem on xvda1:
- Parse /boot/grub/grub.cfg
- Parse /boot/grub/menu.lst
- i686 Debian jessie guest, single root filesystem directly on xvda:
- Parse /boot/grub/menu.lst
If you have a planned reboot coming up, it would be useful to me if
you would let me do the reboot for you. I would take the opportunity
to switch you to pvgrub booting to check it works in your
configuration. If it didn't then I would just revert you back to
pygrub and leave you be.
So, that is what I am thinking of doing and how I am proposing it
will work. If you have any comments about this, especially if you
think it will not work for you, or you have suggestions about how it
could be done better, I really want to hear about it.
Thanks for reading this immense email!
Cheers,
Andy
¹ You may recall, we modified pygrub to detect an XZ-compressed
kernel and unpack it using the actual xz utility, until a newer
version of pygrub could be installed which supported those
kernels.
--
https://bitfolk.com/ -- No-nonsense VPS hosting
> I'd be interested to hear any (even two word) reviews of their sofas…
Provides seating.
— Andy Davidson
