PXE network boot - is this feasible? (EDIT: Yes it is, at least for Ubuntu)



  • At work we have to support a ton of operating systems, and managing OS disks has been a bit of a pain. Most of us have multiple hard drives, labelled with each OS or family of OS's, and we swap the disks out for whichever one we currently need. Those without enough disks have to deal with Acronis images instead, and constantly backup or restore images to switch operating system.

    I want to try to simplify this, and was reading up about PXE. Basically, it sounds like I could set up a single machine with lots of disk space as a PXE server, and we could use that to network boot our workstations into specific operating systems. When I have time I hope to try setting this up for testing.

    Here's what I hope to accomplish, is this feasible? I still know next to nothing about network booting.

    • Single server with tons of disk space to manage all the boot images
    • Each workstation gets its own slice of the PXE server, so OS images aren't shared between our systems since we all set them up differently
    • When PXE booting, it'll need some kind of customizable menu to select the operating system. Generally it'll include a list of various Linux distributions, sometimes multiples of the same distro but with different Linux kernels.
    • One disk per workstation that is shared among all the OS images, so all the common stuff we need regardless of OS is not duplicated a billion times.

    Finally, our workstations will retain their current disks and still boot locally for the most common 1 - 2 OS's (typically Windows).

    Any PXE gurus here who can comment on this scheme?



  • @mott555 said:

    Most of us have multiple hard drives, labelled with each OS or family of OS's, and we swap the disks out for whichever one we currently need

    You do know with some magic tricks (mostly workarounds for old OSes) you can have as many OSs in one disk as you can fit, right?

    For the network boot, my university uses some IBM tool branded Tivoli. It lets you choose between OSs on boot, checks the image on disk, rebuilds it from the server if it's corrupted (or if it feels like it), then launches, which is what I understand you need.

    I think this is the link, but I literally can't understand anything in the IBM website, it seems to be entirely composed of brand names and buzzwords.

    I'm fairly certain there's open source equivalents, but I know as much as you do about those. Worst case, you have to make your own script to load the images from server to disk.



  • @anonymous234 said:

    You do know with some magic tricks (mostly workarounds for old OSes) you can have as many OSs in one disk as you can fit, right?

    We've tried multibooting with GRUB and it works for a few OS's, but is pretty unwieldy in our situation. It doesn't label things very well (tends to not display the kernel version or whether the OS is 32-bit or 64-bit), and while the labels can be fixed they're almost always wiped out if any of the OS's goes through an update. Not to mention different distributions use different bootloaders and sometimes don't detect all the installed OS's, and different distributions even label the disks differently (sda1 in Ubuntu might be sda2 in CentOS) making it very tricky to set up a working bootloader. Bootloader management has been such a pain most of us switched to a single OS per disk, which just means we need a billion disks.

    @anonymous234 said:

    I'm fairly certain there's open source equivalents, but I know as much as you do about those. Worst case, you have to make your own script to load the images from server to disk.

    I found a guide for setting up a Debian-based PXE server which uses client MAC addresses to decide which images to serve, and also has a menu allowing various kernels/distros. It also mentioned using NFS for common files, so I think I've got a good starting guide to play around with.



  • The first thing you'll want to do is have PXE serve up iPXE firmware (http://www.ipxe.org/), since iPXE is in all ways better than plain PXE. In particular, it can fetch kernels and ramdisk images over HTTP instead of the painfully slow TFTP. iPXE also supports scripts that can do pretty much anything you need.

    I've set up an iPXE menu that can boot live versions of Fedora and Debian. I didn't configure any kind of persistent storage for it, but I know it's possible. You could also not have any persistent storage on them and do everything over NFS or Samba.



  • Ah yes, the default GRUB scripts do kind of suck. Consider these two options.

    1. Maybe you could manually write your own grub.cfg, it's actually a pretty simple file, just a few lines per boot option. Install it in its own partition and the linuxes probably won't mess with it. I don't know how well that would handle upgrades.

    2. Plop, or other dedicated boot managers.



  • Is there a reason that a massive hard drive and virtual machines wouldn't work?

    I'm guessing there's a reason, but doing this from PXE seems like over-engineering.



  • @loopback0 said:

    Is there a reason that a massive hard drive and virtual machines wouldn't work?

    Hardware. We build PCI and PCI Express cards that are not compatible with PCI Passthrough.



  • Ah, yeah, fair enough.



  • Wow, a perfect use-case for eSATA, I thought I'd never see one.



  • Oh, network boot. I remember being very surprised when I connected my laptop to the school's wired network for the first time and discovered that it was booting a completely different Windows 7 installation than the one on my laptop's harddisc.



  • @mott555 said:

    not compatible with PCI Passthrough.

    Fix that and you're all set! /s
    Why don't you support that?



  • lol...because exactly 0% of our customers need it. It would be nice to have internally, but really not worth the effort debugging why it doesn't work.



  • i'm curious: how do you actually dev PCI(e) boards? Burnable ROMs? breadbords? What languages (if any) do you use ?



  • I'm uncertain, I'm on the testing and software side of things and don't know much about the hardware itself. I know the hardware guys do some kind of FPGA work to build the core of our hardware, and I think we contract out board fabrication though it's our own design/layout.

    On my side of things, we talk to the board in C using memory-mapped I/O to specific registers which the board's firmware will use to decide what it needs to do.


  • Grade A Premium Asshole

    @mott555 said:

    At work we have to support a ton of operating systems, and managing OS disks has been a bit of a pain. Most of us have multiple hard drives, labelled with each OS or family of OS's, and we swap the disks out for whichever one we currently need. Those without enough disks have to deal with Acronis images instead, and constantly backup or restore images to switch operating system.

    Up until I read where you develop on PCI cards, this would have been a perfect application for virtualization like VMWare Workstation.

    Good luck. PXE has always seemed to me like one of those black arts that the old gray beards used. I have never used it myself, but this seems like a great application for it. On the other hand, would it be at all feasible to have a workstation to do your dev work, and then a lab with all the OS's you need installed on separate machines that everyone shares?



  • @Polygeekery said:

    On the other hand, would it be at all feasible to have a workstation to do your dev work, and then a lab with all the OS's you need installed on separate machines that everyone shares?

    We mostly do that already. I'm hoping to use PXE booting to help manage the lab computers more so than our main workstations.


  • Grade A Premium Asshole

    Approximately how many different OS configurations do you have to test?



  • Commonly probably 8 - 9, there are a few oddball real-time OS's we support too but I hardly ever deal with those.



  • @Rhywden said:

    Oh, network boot. I remember being very surprised when I connected my laptop to the school's wired network for the first time and discovered that it was booting a completely different Windows 7 installation than the one on my laptop's harddisc.

    How on earth is that possible? Can a network override your booting sequence? Or were your BIOS settings so weird?


  • Discourse touched me in a no-no place

    @marczellm said:

    How on earth is that possible? Can a network override your booting sequence?

    Having PXE a higher priority than the local disk I imagine. i.e. the opposite of this:


  • Winner of the 2016 Presidential Election

    @anonymous234 said:

    this is the link

    http://www-03.ibm.com/software/products/en/tivoprovmanaforimagsystxedit
    tivoprovmanaforimagsystxedit
    tivoprovmanaforimagsystxedit

    :wtf:?



  • Indeed. Which, for some reason, was the default setting on this laptop from a certain Korean company.



  • Ugh, our main Ethernet switch blocks the TFTP port for some reason. Everything works if I go around it, but that port will not go through the switch. :wtf: And I cannot get the TFTP server to work on any alternate port. Time to call the corporate helpdesk. I suspect I won't get anywhere.


  • Impossible Mission Players - A

    I would say feasible, so long as you have control over the local DHCP server (typically in the same box as the router). Once you have it giving the proper DHCP options, the rest is rather cut and dry, so long as you're not attempting to support (U)EFI devices.

    Once you got that lined up, you just need a TFTP server (I think it's UDP 53-ish?) and some higher transport method, NFS, HTTP, etc.
    And a small suite of tools to help you get things configured of course.


  • Fake News

    Can you move the test PC's to a different / isolated network? PXE does require a specially crafted DHCP server and TFTP combo. IT might sleep better if the DHCP server is away from the network, unsuspecting computer users might otherwise be unable to connect to a Windows domain.

    Of course, good luck if you were doing this already and the switch was supposedly configured by the IT overlords...



  • to be fair - neither rogue DHCP nor PXE servers are desireable, so blocking that is entirely sensible.


  • Winner of the 2016 Presidential Election

    You could abuse port forwarding to make the TFTP server work on a different port. It would be terrible and a WTF of its own but it would work...



  • I'm now 90% of the way there. My issues were several unrelated causes. Turns out our Cisco switch wasn't blocking TFTP, it was the VMware vswitch on the ESXi box hosting the TFTP server! After some research, I switched the TFTP server's emulated NIC from the VMXNET3 to the E1000 type and now that works just fine.

    Then, another engineer was testing a piece of LAN-connected hardware we're developing, and he was flooding the network. Not only did it make my TFTP server rather unreliable, but it also managed to break our new corporate-mandated LAN-connected electronic door locks :facepalm:.

    Between TFTP, the modified DHCP server, and an NFS server, I can now reliably boot the Ubuntu Live DVD. I'm trying to get a full-blown install moved to an NFS share next.

    Biggest issue at the moment is the system I'm using as a guinea pig has the slowest PXE ROM in the universe. It takes it 10 - 20 minutes to load the initial Syslinux boot selector which is presented via PXE/TFTP. I thought it was just my TFTP server being slow, until someone else showed me it was working on his system and booting within seconds. So I guess I'm going to look into this iPXE thing and see if I can replace the PXE ROM on this slow (but otherwise fast, it's a Haswell i5 processor) system.

    EDIT: Biggest issue #2 is my Ubuntu kernel with NFSRoot support kernel panics when loaded over NFS.



  • @mott555 said:

    LAN-connected electronic door locks

    Nice. Permanently?
    Also: fail safe or fail secure?

    @mott555 said:

    10 - 20 minutes

    That's just crazy. I'm surprised it actually works when it's that slow.



  • @swayde said:

    Nice. Permanently?Also: fail safe or fail secure?

    They came back when he returned from lunch and shut down his equipment. I think the doors refused to unlock from the outside but I didn't personally test them. I did hear some joking about how we now can lock ourselves out of the building and force a vacation day for everyone :smile:



  • Your network security is worrying . I'm honestly amazed they run doors/locks on the same equipment as general access.
    (and has not assured it'll work under worst case scenarios)
    And they're not blocking ports that flood the network, and they're not blocking rogue DHCP /pxe servers?
    Did you just set up a DHCP server on the general network, or did you get a new segment?



  • We already had a local DHCP server, all I had to do was set up the two options to support PXE. We're a small office with about 8 people in at any one time, though we're owned by a much larger corporation with IT requirements that don't make sense for a group as small as we are.



  • I think that you can flash iPXE firmware over the original PXE firmware in a NIC, but I wouldn't take the chance if you don't have to. You can configure the server to basically chain load the iPXE firmware instead.

    I definitely recommend taking the time to learn iPXE. It can do really fancy stuff with menus and conditionals, and you'll probably get an order of magnitude better speed when downloading the kernel and ramdisk over HTTP instead of TFTP.


  • Impossible Mission Players - A

    +1 this. Definitely get off of TFTP as soon as possible once you've loaded whatever pre-boot binaries you need.

    My network serves up the UNDIONLY image to clients first (which is pretty small), then use iPXE to ask an HTTP server what options are available to a given machine (according to MAC/UUID if applicable).

    Source available upon request. :stuck_out_tongue:



  • @mott555 said:

    Commonly probably 8 - 9, there are a few oddball real-time OS's we support too but I hardly ever deal with those.

    Do it the old fashioned way...

    Product page



  • Yeah honestly, I still think the eSATA port is the best solution here.



  • Well, got it 100% working with Ubuntu 14.04 on my main workstation. I'm not too worried about iPXE yet, even over TFTP the system boots pretty quickly. (well, except for the other problematic system where it's slow, which seems to have nothing to do with TFTP).

    Next I need to set this up with a couple different kernels and figure out the process for CentOS. Then once everyone's back in the office I can demo it and see if it'll make their support lives easier, if so I'll need to build up a machine with a few terabytes of disk space to act as a new file server for this. Then the question is do we want to spend the money to make it all SSD storage, or go with some large Western Digital spinners.

    Another nice thing I just realized is if we can get this NFS server on our backups somehow, every single network-installed-OS will be backed up. Not to mention we won't lose any installs through general use. If a weird client wants to run Ubuntu 14.04 with kernel 3.2 and needs support, that can be set up as an NFS install and retained indefinitely.



  • @blakeyrat said:

    Yeah honestly, I still think the eSATA port is the best solution here.

    Very few of our systems have eSATA ports, and I'm trying to find a way to not have a hundred trillion hard disks (that number is literally literal) floating around the office which is what we currently have.



  • @mott555 said:

    all SSD storage, or go with some large Western Digital spinners.

    Given that you're mostly serving huge image files, with a lot of linear read load, would SSD give such a huge benefit?


  • Impossible Mission Players - A

    SSDs are probably not going to give any benefit, as you're not going to be able to saturate even a SATA2 port from requests over Ethernet


  • Winner of the 2016 Presidential Election

    @mott555 said:

    literally literal

    You have 137789000000000 lbs of hard drives?

    Filed Under: @­BenLubar mode go!

    <!--inb4 woosh-, counterwoosh trolling-->


  • @mott555 said:

    I'll need to build up a machine with a few terabytes of disk space to act as a new file server for this

    You might find it easier to let Synology build it for you. Even their entry-level NAS boxes support NFS and TFTP/PXE over GigE.



  • Too late, I scrounged. We had enough unused equipment lying around to set up a decent 3 TB Ubuntu server without needing to purchase any hardware. So it's kind of in a pilot stage at the moment, if more people use it and it's considered useful we'll build up a proper rackmount server.

    I like the setup, it works well. For the first time ever, I have most of our supported Linux versions installed on my system, and setting them up via PXE/NFS is actually quicker than local installs since I don't have to deal with GRUB/LILO/whatever breaking the bootloader each time I add a new OS. Doesn't feel slow at all over Gigabit Ethernet but we'll see what happens if other users start doing this too.


Log in to reply
 

Looks like your connection to What the Daily WTF? was lost, please wait while we try to reconnect.