Recently I had an idea: what if I maintained a local server to host my Ubuntu install ISOs so I could load them through the PXE boot capability available on all of my computers? I would never need to create an Ubuntu install USB drive ever again — instead, I could just tell the BIOS or UEFI to boot through the network card, pick the ISO I want from a list, and boot right into it. Every x86/amd64 computer I’ve ever used (with the exception of Macs) has had an option in the BIOS/UEFI for booting through the Ethernet port, but it’s something I’ve never been able to try, and tutorials on it are usually old and/or incomplete. I finally decided to try to figure it out. I got it working after a lot of trial and error.

I found plenty of tutorials, but none of them had everything that I needed to know in order to get this setup working. Most of the tutorials that were Ubuntu-specific applied to really old releases (10.04 and below), and generally didn’t support both UEFI and BIOS simultaneously. I can verify that the process I figured out will work for both legacy BIOS and modern UEFI systems, and it will properly boot into an Ubuntu 16.04 desktop live ISO. Specifically, I have tested it with an Ubuntu MATE live ISO, but it should work with all other variants too.

dnsmasq

These instructions will use dnsmasq, which is a nice little DNS/DHCP/TFTP server program that you can install in Linux. If you are using something like DD-WRT, there is a good chance you already have dnsmasq! However, for these instructions it’s apparently important that you use the latest version as of this writing, which is 2.76. I didn’t test with earlier versions, but according to sources online, version 2.76 contains bug fixes for PXE booting with UEFI. So to start, you need to make sure you have dnsmasq 2.76. If your server is an Ubuntu 16.10 server, you can simply install dnsmasq with apt–it’s already the latest version. Otherwise, you may need to manually grab the .deb from 16.10, or compile it yourself. Keep in mind that desktop versions of Ubuntu run dnsmasq in the background for DNS, so you may have to set up something special if you want to run this server from a desktop install of Ubuntu without interfering with the existing dnsmasq. I’m not 100% sure–I did this on a server install so that wasn’t an issue.

If you already have a different DHCP server program, you can probably adapt these instructions to work with it, but you’re on your own to figure out the required syntax.

Proxy DHCP or not?

There is a decision you will have to make which will affect how you configure dnsmasq. Do you want to run dnsmasq as a normal DHCP server, or a proxy DHCP server? Let me explain the choices. As a forewarning, this may take a while to fully explain, but it’s all useful information.

When I say “normal DHCP server”, what I mean is that dnsmasq will replace your existing DHCP server. So you would set up dnsmasq to assign IP addresses to all of your computers and devices on your network, and turn off the DHCP server built into your home router. This new DHCP server provided by dnsmasq will recognize when your computer is trying to netboot, and will tell it to use TFTP to download a file from a server to boot. As you will see shortly, this is probably your best option. It’s understandable, though, if you don’t want to mess around with the DHCP server that you already have which is working just fine with no problems whatsoever. If you’re in that camp, there is another option available: a proxy DHCP server.

If you already have a DHCP server on your network that you can’t configure for PXE but you still want to use (e.g. the DHCP server in your home router), you can set up dnsmasq as a proxy DHCP server that works alongside your existing DHCP server. Your existing DHCP server will still be responsible for assigning IP addresses to all of your computers and devices, but the proxy DHCP server will provide the necessary information that PXE clients need for booting (the TFTP server IP address and file path). You will not have to change any configuration on your existing DHCP server; the proxy just provides additional information that all PXE-enabled BIOS/UEFI implementations are supposed to support. PXE was designed to support this use case, and every computer I’ve tested so far works with it just fine.

With both of these options available, the choice may seem obvious to you: use it as a proxy DHCP server! It’s a low-risk choice because you’re not messing with the configuration of anything else already on your network. In theory, this would be a great idea. In practice, though, there are problems with booting in proxy mode with UEFI. It’s not actually dnsmasq’s fault at all. It’s not the PXE protocol’s fault, and it’s not even the fault of your computer’s UEFI implementation. The fault lies in shim and grub. Let me explain further.

shim is a bootloader that serves a single basic purpose: to load grub. The reason it exists is because it’s possible to have it signed by Microsoft so that it can load on a UEFI computer that has Secure Boot enabled. So for Ubuntu, Canonical provides a version of shim signed by Microsoft that trusts binaries signed by Canonical. This signed shim can then load a version of grub that has been signed by Canonical, and grub will load a signed kernel, and so forth. The trick with shim is that it needs to know where to find grub. It attempts to auto-detect how it was loaded, and looks for a file called grubx64.efi next to itself. This works great when it was loaded from a hard drive or SSD. If it was loaded through netboot, it will try to load grubx64.efi from the same TFTP server and directory from which it was loaded, which it determines by looking at the original information provided by the DHCP server. Unfortunately, shim doesn’t currently detect this information properly when it is booted from a PXE proxy server. UEFI provides the information necessary to figure it out, but shim doesn’t currently look for it.

You could be like me and say, “who cares about Secure Boot?” Instead, just disable Secure Boot on your computers, and boot directly to grub instead of using shim. The problem is that grub has the exact same issue, and it needs to know how it was booted in order to figure out where to get grub.cfg. There is probably a way to work around this problem by embedding an intermediate grub.cfg into your grub binary (using grub-mkstandalone) that knows where to look for the real grub.cfg, but you won’t be using shim so you won’t support clients with Secure Boot enabled.

The easiest thing to do is just not use proxy mode for now if you want to support UEFI clients, especially UEFI clients with Secure Boot enabled. I will still explain how to set up proxy mode at the end of this post if you’re interested, but just know that if you do it, you will need to spend a bit more time figuring out how to get grub to grab its config file and any other supporting files over TFTP when netbooting a UEFI computer. I didn’t bother to figure this out, though, because…

There is actually a third option that may work OK for you if you were hoping you could use a proxy server: use dnsmasq as a normal DHCP server alongside your existing DHCP server, but tell it to assign IPs outside of the range that your existing DHCP server provides, and also tell it to ignore non-PXE clients. That way, it will allow your other server to handle all normal DHCP leases, but it will still be able to respond with the boot info when it sees a PXE client. Your other DHCP server will also respond to the PXE client’s boot request, but I believe the PXE client will ignore your other server because it doesn’t provide any PXE boot info. It works OK in my testing, anyway!

You may be thinking, “Doug, you’re crazy! Two DHCP servers on the same subnet?” I know it sounds weird, but it’s safe to have two DHCP servers on the same network as long as they don’t assign any overlapping IP addresses. Some people even recommend using such a setup intentionally for redundancy. For example, if your existing DHCP server assigns addresses from 192.168.1.100 to 192.168.1.199, you could safely tell your new DHCP server to assign addresses from 192.168.1.200 to 192.168.1.249 with no conflicts at all–normal DHCP clients will use whichever server responds first. It would definitely be wise to ensure your new DHCP server assigns the same default gateway and DNS servers that the existing server assigns, but if you’re configuring it to only respond to PXE clients, it’s probably not a huge deal because the PXE boot process probably isn’t going to need them on a simple home network.

Do initial setup

Regardless of what type of DHCP server you’re going to run, there is a bunch of common setup to do. Let’s get started!

TFTP directory

First of all, let’s create a TFTP directory on your server, and change it so it’s owned by you (replace “doug” with your username):

sudo mkdir /tftpboot
sudo chown doug:doug /tftpboot

Download and extract the live install ISO

Now, we need to download the install ISO. You can store it wherever you want, but the location you choose won’t end up mattering; we’re going to extract the contents of this ISO into a directory on your computer. Start by downloading the Ubuntu .iso that you want to make bootable. In my case, I downloaded Ubuntu MATE 16.04.2 LTS:

wget http://cdimage.ubuntu.com/ubuntu-mate/releases/16.04.2/release/ubuntu-mate-16.04.2-desktop-amd64.iso

After grabbing the ISO, you should mount it loopback so you can extract files from it:

mkdir /tmp/iso
sudo mount -oloop,ro ubuntu-mate-16.04.2-desktop-amd64.iso /tmp/iso

Now, we need to do several things. First of all, we need to put the kernel and initrd into your TFTP directory so TFTP clients will be able to access it. I like to make a directory in the TFTP server for each ISO I’m hosting:

mkdir /tftpboot/ubuntu-mate-16.04.2-desktop-amd64
cp /tmp/iso/casper/{vmlinuz.efi,initrd.lz} /tftpboot/ubuntu-mate-16.04.2-desktop-amd64/

Note that even though the kernel is named with a .efi extension, it will still boot a normal BIOS system just fine. The bootloader will know how to handle the file either way.

Now, you need to save a copy of all of the contents of the ISO somewhere locally that you can serve with NFS. I prefer to make a directory on the root of my drive for each ISO, but do what you want. Afterward, you can unmount the ISO.

sudo mkdir /ubuntu-mate-16.04.2-desktop-amd64
sudo cp -R /tmp/iso/* /tmp/iso/.disk /ubuntu-mate-16.04.2-desktop-amd64/
sudo umount /tmp/iso

Set up an NFS server

Set up your system as an NFS server so the PXE clients will be able to load the contents of the ISO that you extracted:

sudo apt-get install nfs-kernel-server

Add the following line to /etc/exports (I’m unsure which of the options chosen here are absolutely necessary, but they worked for me):

/ubuntu-mate-16.04.2-desktop-amd64           *(ro,sync,no_wdelay,insecure_locks,no_root_squash,insecure,no_subtree_check)

And finally, restart the NFS server so it recognizes your changes:

sudo service nfs-kernel-server restart

At this point, you are serving the contents of the ISO inside the directory /ubuntu-mate-16.04.2-desktop-amd64 with NFS, and you no longer need the .iso file. Note that some people might prefer to just keep the .iso file mounted loopback at all times instead of extracting the files from it. If you prefer to do it that way, you can probably set something up in /etc/fstab to automatically mount it, and then set up /etc/exports to share the mounted directory instead. Whatever you prefer!

Now, we need to set up some bootloaders that the PXE clients will load. We will use PXELINUX for BIOS clients and shim/grub for UEFI clients. Later on, we will set up dnsmasq to decide which bootloader to serve based on how the client identifies itself.

PXELINUX

For legacy BIOS clients, we are going to use PXELINUX. This is a very popular PXE bootloader based on SYSLINUX. It only supports BIOS clients, so this will only handle the BIOS half of things.

Start out by downloading SYSLINUX, which includes PXELINUX. Extract pxelinux.0 and a few libraries from it and place them into the TFTP server directory:

wget https://www.kernel.org/pub/linux/utils/boot/syslinux/syslinux-6.03.tar.xz
tar -xJf syslinux-6.03.tar.xz
cp syslinux-6.03/bios/core/pxelinux.0 /tftpboot/
cp syslinux-6.03/bios/com32/lib/libcom32.c32 /tftpboot/
cp syslinux-6.03/bios/com32/libutil/libutil.c32 /tftpboot/
cp syslinux-6.03/bios/com32/elflink/ldlinux/ldlinux.c32 /tftpboot/
cp syslinux-6.03/bios/com32/menu/vesamenu.c32 /tftpboot/

pxelinux.0 will be the file that is sent to BIOS PXE boot clients. After loading, it will be responsible for grabbing the other library files (ending in .c32) from the TFTP server and then it will display a list of menu choices, which we will define in a config file. Create the directory that will contain the config file:

mkdir /tftpboot/pxelinux.cfg

Now, create the file /tftpboot/pxelinux.cfg/default, which will contain the commands needed to boot into the live environment from the NFS share of the Ubuntu ISO contents. In the example below, your NFS server is 192.168.1.2:

DEFAULT vesamenu.c32
MENU TITLE Network boot

LABEL ubuntu-mate-16.04.2-desktop-amd64
  MENU LABEL ubuntu-mate-16.04.2-desktop-amd64
  KERNEL ubuntu-mate-16.04.2-desktop-amd64/vmlinuz.efi
  APPEND initrd=ubuntu-mate-16.04.2-desktop-amd64/initrd.lz root=/dev/nfs boot=casper netboot=nfs nfsroot=192.168.1.2:/ubuntu-mate-16.04.2-desktop-amd64 splash -- 

You can add as many “LABEL, MENU LABEL, KERNEL, APPEND” groups as you want to provide multiple selectable options.

shim and grub

For modern UEFI clients, we can’t use PXELINUX. Apparently, SYSLINUX also supports UEFI netbooting, so you could use it, but I read that the current version as of this writing (6.03) is very buggy with UEFI PXE booting. Instead, let’s use grub. Using grub will also allow you to use shim and support Secure Boot if you care about that.

Download Ubuntu’s version of shim-signed, and copy it into your TFTP directory. There are many ways to obtain it, such as downloading the dpkg file and extracting shim.efi.signed out of it; here is an easier way that works as of this writing:

wget https://launchpad.net/ubuntu/+archive/primary/+files/shim-signed_1.19~16.04.1.tar.xz
tar -xJf shim-signed_1.19~16.04.1.tar.xz
cp shim-signed-1.18~16.04.1/shim.efi.signed /tftpboot/shim.efi

Do the same with a signed version of grub, which Ubuntu’s signed shim will be able to load in a Secure Boot environment:

wget http://archive.ubuntu.com/ubuntu/dists/xenial/main/uefi/grub2-amd64/current/grubnetx64.efi.signed
mv grubnetx64.efi.signed /tftpboot/grubx64.efi

Create the directory that will contain grub’s config file:

mkdir /tftpboot/grub

And finally, create the file /tftpboot/grub/grub.cfg, which will contain the commands needed for booting into the Ubuntu live environment. Like before, the NFS server in this example is 192.168.1.2.

menuentry "ubuntu-mate-16.04.2-desktop-amd64" {
  linux ubuntu-mate-16.04.2-desktop-amd64/vmlinuz.efi root=/dev/nfs boot=casper netboot=nfs nfsroot=192.168.1.2:/ubuntu-mate-16.04.2-desktop-amd64 splash -- 
  initrd ubuntu-mate-16.04.2-desktop-amd64/initrd.lz
}

You can add as many “menuentry” items as you want to provide multiple selectable options.

Set up dnsmasq

We are now finished setting up the TFTP directory. All bootloaders and config files are in place. Now, we just need to set up the DHCP/TFTP server and tell it to boot BIOS clients using pxelinux.0, and UEFI clients using shim.efi. First, install dnsmasq 2.76 or greater using whatever method is most convenient for you. Based on which strategy you chose earlier, go to the subsection that matches the setup you want, and set up your dnsmasq.conf based on the given example.

Replace your existing DHCP server

Here is a (mostly) complete dnsmasq.conf file that sets up a full DHCP server. Note you will probably need to add additional options for the default gateway and DNS servers, but I’ve only supplied the parts of the config file necessary for PXE booting to work properly. Find a normal dnsmasq tutorial for info on the options to add for fully setting up a DHCP server. In the file below, it’s assumed that the server running dnsmasq is 192.168.1.2 and it is assigning IP addresses in the range of 192.168.1.50 to 192.168.1.99. Also, note that the configuration includes support for EFI32, which is a pretty uncommon architecture and will probably require a different build of grub and shim, if anyone has even made such a thing. You probably won’t have to worry about EFI32, so you can completely remove the lines referring to it if you want.

Another interesting tidbit is that this configuration doesn’t actually use PXE; it just uses the normal TFTP server and filename fields that are a part of DHCP. All PXE clients I’ve ever tested (both BIOS and UEFI) support this without any problems though. The reason I did it this way is because I discovered during testing that UEFI clients don’t seem to work properly when you actually use PXE with this setup. I never figured out why, but since it works to just set the DHCP boot server and filename, there’s no need for PXE.

# Don't function as a DNS server
port=0

# Log lots of extra information about DHCP transactions
log-dhcp

# Enable the built-in TFTP server
enable-tftp

# Set the root directory for files available via TFTP
tftp-root=/tftpboot

# Disable re-use of the DHCP servername and filename fields as extra
# option space. That's to avoid confusing some old or broken DHCP clients
dhcp-no-override

# Inspect the vendor class string and match the text to set the tag
dhcp-vendorclass=BIOS,PXEClient:Arch:00000
dhcp-vendorclass=UEFI32,PXEClient:Arch:00006
dhcp-vendorclass=UEFI,PXEClient:Arch:00007
dhcp-vendorclass=UEFI64,PXEClient:Arch:00009

# Set the boot filename based on the matching tag from the vendor class (above)
dhcp-boot=pxelinux.0,,192.168.1.2
dhcp-boot=net:UEFI32,shim.efi,,192.168.1.2
dhcp-boot=net:UEFI,shim.efi,,192.168.1.2
dhcp-boot=net:UEFI64,shim.efi,,192.168.1.2

# Give out IPs from 192.168.1.50 to 192.168.1.99
dhcp-range=192.168.1.50,192.168.1.99

If for some reason you’re directly booting into grub instead of shim, replace all instances of shim.efi with grubx64.efi instead.  This won’t work with Secure Boot enabled, so I’d definitely recommend using shim for maximum compatibility.

Set up a proxy DHCP server

This config file sets you up as a PXE proxy server. Like the above example, it is assumed that the dnsmasq server’s IP address is 192.168.1.2. I am also assuming that the other DHCP server is 192.168.1.1. Unlike the above example though, this example does actually use PXE, and it works properly with both BIOS and UEFI. Unfortunately, as I explained earlier in this post, shim and grub won’t play nicely because they won’t be able to automatically detect that they were loaded from a PXE proxy. Because of that problem, this configuration is mostly useless if you need to support UEFI. I’m just providing it for reference, especially to show the difference between setting up normal DHCP TFTP server/filename booting and actual PXE booting. If shim and grub are ever fixed to support PXE proxies properly in the future, this configuration will be much more useful.

The “Test PXE” and “Testing PXE” text can be set to whatever you want.

# Don't function as a DNS server
port=0

# Log lots of extra information about DHCP transactions
log-dhcp

# Enable the built-in TFTP server
enable-tftp

# Set the root directory for files available via TFTP
tftp-root=/tftpboot

# Disable re-use of the DHCP servername and filename fields as extra
# option space. That's to avoid confusing some old or broken DHCP clients
dhcp-no-override

# Set the boot file and server IP based on the architecture
pxe-service=x86PC, "Test PXE", pxelinux.0, 192.168.1.2
pxe-service=BC_EFI, "Test PXE", shim.efi, 192.168.1.2
pxe-service=X86-64_EFI, "Test PXE", shim.efi, 192.168.1.2

# PXE menu. The first part is the text displayed to the user. The second is
# the timeout, in seconds.
pxe-prompt="Testing PXE", 1

# We are proxying for the DHCP server 192.168.1.1
dhcp-range=192.168.1.1,proxy

Add a second DHCP server with a different IP range (and ignore non-PXE clients)

This method should allow you to preserve your existing DHCP server while still supporting PXE. Because of the problems with shim and grub mentioned above, I’d personally recommend using this method instead of the proxy method if you don’t want to mess with your existing DHCP server. Use the same configuration from the “Replace your existing DHCP server” section, but add the following lines to the config file, just after the dhcp-no-override line:

# Set a tag if it's a PXE client
dhcp-match=set:IsPXEClient,60,"PXEClient"
# Ignore non-PXE requests, to allow the existing DHCP server to handle them instead.
dhcp-ignore=tag:!IsPXEClient

And of course, update the dhcp-range setting to make sure it doesn’t overlap with your existing DHCP server’s IP address range.

As described earlier, the dnsmasq server in this configuration will ignore non-PXE clients, so your existing DHCP server will be fully in charge of assigning IP addresses to your computers and devices, with the exception of PXE boot attempts. Both your existing server and the dnsmasq server will respond to PXE requests, but your server should “win” because the existing DHCP server won’t respond to the PXE request with any of the required PXE information. I can’t guarantee this will work with every PXE client, but it seems to work fine with every computer I’ve tried to netboot.

Run dnsmasq and try it out

At this point, you have set up dnsmasq using one of the three strategies listed above. Now it’s time to try it out! Ensure that dnsmasq will start up and run. While testing, I like to leave the service stopped and instead run it manually with the -d flag so I can see all of the debug output:

sudo service dnsmasq stop
sudo dnsmasq -d

Now, tell a computer on your network to boot! Usually your computer has a boot menu that lets you pick various boot options. Sometimes the F12 key works, sometimes the Esc key works. Or, try making a VMware virtual machine and booting it without a hard drive. In many computers, you have to enable the PXE boot ROM or the “Networking Stack” in order to make it work. There’s almost always an option somewhere! Remember, this only works over Ethernet; if you have built-in Wi-Fi, it’s probably not going to work unless you have a really fancy BIOS that knows how to connect to a Wi-Fi network — I’ve never heard of such a thing.

If everything works OK, you should end up at a menu screen served by PXELINUX on a BIOS computer or grub on a UEFI computer. Pick the Ubuntu option you added to the config file earlier, and press enter to begin booting!

Sometimes, downloading the kernel and initrd can be slow. If you see a black screen and nothing’s happening, don’t panic. Open up Wireshark and see if anything’s happening, and be patient. In some cases, such as a VMware VM set up for EFI, grub takes forever to load the kernel and initrd over TFTP and sits with a black screen for a long time. dnsmasq doesn’t actually print out anything about a TFTP file transfer in progress; it only prints out a message after the file transfer succeeds. At some point, Ubuntu will load and attempt to mount the ISO contents over NFS.

Once I know that everything is working OK, then I kill the debug dnsmasq with ctrl-C and run it as a service instead (assuming I have it installed that way):

sudo service dnsmasq start

You might want to remove some of the debug options from the dnsmasq config file such as log-dhcp once you know everything is working OK, but keep in mind if you ever need to do any troubleshooting, it will be useful to turn it back on.

After installing, fix /etc/network/interfaces

When you install Ubuntu from an NFS mount, it does something weird: it sets up /etc/network/interfaces to manage your Ethernet card instead of allowing NetworkManager to handle it. It’s really easy to fix though. After booting into your new Ubuntu install, edit /etc/network/interfaces and remove the lines relating to your Ethernet card. The only two non-comment lines left when you’re finished should be:

auto lo
iface lo inet loopback

This is just a minor annoyance that is easy to solve. It’s probably possible to customize the Ubuntu install scripts to fix this automatically, but I spent way too much time figuring out how to netboot. I don’t feel like solving that tiny problem now. Post a comment on the blog if you figure out how to do it and I’ll add instructions to this post.

All done!

Not too bad, right? I hope this post can serve as a reference tool for sysadmins, developers, power users, etc. who want to netboot install modern Ubuntu systems. Maybe I didn’t search hard enough, but I couldn’t find any reference online with all of this information in one place. Let me know if you have any suggestions on ways I can improve these instructions, or if things change as time goes on! Maybe someday, shim and grub will support PXE proxy servers and I won’t have to tell people to steer clear of that option.

Special thanks

It wouldn’t be right if I didn’t give credit to the following sites that helped me figure out how to make this work and also helped me understand why things weren’t working properly:

Trackback

22 comments

  1. This post is really helpful to me, thanks for your sharing!

  2. Wow!. What an explanation.

  3. Thanks a lot for this tutorial. It really helped me a lot, I appreciate your effort. But I have got an issue at the end of it all. I’m trying this procedure in Virtualbox. When I boot a virtual client machine from the virtual server, it finds vmlinuz and initrd files and shows Ubuntu 16.04 text with the dots but then goes to BusyBox v1.22.1 screen saying: “Unable to find a live file system on the network” [110.626081] random:crng init done.

    If you can please suggest what may be causing it? I’m trying to PXEBoot Ubuntu 16.04.2 Xenial Xerus Desktop AMD64 version of the operating system.

  4. Hi abkap02,

    It sounds like your NFS server might not be working properly, or the path is wrong. I would verify that you can mount the NFS share from another computer, and that you have the NFS path and server correct in pxelinux.cfg/default and grub.cfg.

  5. Hi Doug,

    Thank you once again. I found the error, had not defined the correct nfsroot in pxelinux.cfg/default file.

    I was wondering if we can extend the same functionality to boot a client with a copy of installed operating system on the server instead of booting from a live image? That’ll be cool, isn’t it? The client will just need to connect to the server and it’ll load its operating system and will also store data the there. What do you think? Please do comment, thanks!

  6. Glad you got it working! Yes, I’ve actually done what you described with Ubuntu MATE 16.04 at work with an NFS root filesystem. The setup as far as the PXE server goes is the same, so my tutorial still shows you how to get everything set up for that. What I ended up doing (I think) was following the directions on this Ubuntu wiki page:

    https://help.ubuntu.com/community/DisklessUbuntuHowto

    I followed their recommendation of installing Ubuntu to a hard drive and then copying the install over to an NFS server. I’m pretty sure I did all of the config file customizations they recommended, and then used the kernel and initrd from the Ubuntu install as the kernel and initrd for pxelinux. I didn’t test with UEFI support and grub, but it should work. It might take a little bit of improvisation in case things have changed since that wiki page was written but for the most part I think it will all work fine. Then it becomes a question of whether or not you want to set up a swap file on the computer’s hard drive. The wiki describes some weird setup where you can put the swap file on the NFS server but if your computer has a hard drive it’s probably better to use the actual hard drive.

    I was originally thinking about using it to boot multiple computers but there are a lot of gotchas with that, some of which they mention at the bottom of the page. Stuff like multiple computers writing to the same log files, what happens if the same user is logged into two computers at the same time, /media probably shouldn’t be shared between multiple computers, etc. If you don’t have to worry about that stuff, this seems like it would be a pretty cool setup.

  7. Hi Doug,

    Thanks for helping me with that guide, I appreciate it. By now you must have got this that I’m fairly new to this Ubuntu ecosystem and am just trying to learn a few things about it. I did try everything you asked me to for setting up diskless booting and I think I’m quite close to actually getting it done but I’m stuck again and wanted to ask for your help.

    Giving you a brief description of what I did so far. I followed the same steps as in your tutorial to set up the server except downloading live ubuntu image. Then I switched to Diskless Ubuntu setup instructions. First I tried their 3rd way in which they asked to do a standalone installation of Ubuntu on a client machine (Ubuntu Mate in my case) and then mounting those files from client machine to nfs share folder on the server. Everything worked to the point of mounting OS files. But then it gave me some copy errors when I used these commands: cp -ax /. /mnt/. and cp -ax /dev/. /mnt/dev/. So I quit on it there and switched to their 1st method in which they ask to make use of debbootstrap by following the instructions on:

    https://help.ubuntu.com/community/Installation/OnNFSDrive link.

    I got a copy of Ubuntu installed on the server inside my /nfsshare directory. Then I installed linux kernel and initrd separately as: sudo chroot /nfsshare. sudo apt-cache search linux-image which gave me linux-image-4.4.0-21-generic as the stable version for x64 so I did, sudo apt-get install linux-image-4.4.0-21-generic. Then I copied vmlinuz-4.4.0-21-generic and initrd.img-4.4.0-21-generic to my tftpboot directory. Besides, as mentioned in the diskless booting tutorial, I edited the interfaces file, to manual and also configured fstab.

    Now when I try to boot a client machine, it gets an ip address via my dnsmasq server, finds vmlinuz and initrd but then gets stuck on:

    IP-Config: enp0s3 (my virtual machine’s nic’s name) hardware address mtu 1500 DHCP
    [ 4.043138] e1000: enp0s3 NIC Link is Up 1000 mbps Full Duplex, Flow Control RX
    IP-Config: no response after 2 secs – giving up
    IP-Config: enp0s3 hardware address mtu 1500 DHCP
    IP-Config: no response after 3 secs – giving up
    IP-Config: enp0s3 hardware address mtu 1500 DHCP
    IP-Config: no response after 4 secs – giving up
    .
    .
    .
    [ 58.326825 random: nonblocking pool is initialized
    [ 278.503506] Kernel panic – not syncing: Attempted to kill init! exit code=0x00000200
    [ 278.503506]
    , 278.505311] CPU: 0 PID: 1 Comm: init Not tainted 4.4.0-21-generic #37-Ubuntu
    .
    .
    .
    [ 278.511168] Call Trace:
    [ 278.511512] [] dump_stack+0x63/0x90
    [ 278.511512] [] panic+0xd3/0x215
    [ 278.511912] [] ? perf_event_exit_task+0xbe/0x350
    .
    .
    .
    [ 278.5124661] Kernel Offset: disabled
    [ 278.5150071] —[ end Kernel panic – not syncing: Attempt to kill init! exit code=0x00000200
    and then it sits here.

    I did google a bit and found something about adding nomodeset to the default configuration file in pxelinux.cfg. Did that too but to no good? I’m seeing no way out of this issue Doug. So asking for your help, hoping you’ll bring my sinking ship to shore once again. I’ll be highly obliged.

    Thanks!

  8. Hi abkap,

    It looks to me like the kernel is failing to get an IP address from your DHCP server for whatever reason, so it’s bailing before it gets a chance to try to do anything. Can you post the content of your pxelinux config file? The nomodeset thing is related to graphics, so I don’t think it makes a difference with the problem you’re seeing here. Have you tried rebooting your router and/or DHCP server, depending on how you have the PXE stuff set up?

  9. Hi Doug,

    Here’s my pxelinux.cfg/default file:

    DEFAULT vesamenu.c32
    MENU TITLE Network Boot

    LABEL ubuntu-mate
    MENU LABEL ubuntu-mate-diskless
    KERNEL ubuntu-mate/vmlinuz-4.4.0-21-generic
    APPEND initrd=ubuntu-mate/initrd.img-4.4.0-21-generic root=/dev/nfs nfsroot=192.168.1.75:/ubuntu-mate rw nomodeset ip=dhcp

    I am using VirtualBox virtualization tool for my server while running a proxy DHCP server (as per your tutorial) and have given a static ip address to the server itself and I’m trying to boot another virtual client machine via this server. This setup work’s flawless when I’m using ubuntu live image.

    And, yup I have restarted the server, dnsmasq, and nfs-kernel-server services running on it a couple of times to no avail.

    Thanks!

  10. Hi abkap,

    Hmm, very strange! Everything you’ve done for configuration seems correct to me. It pretty much matches what I did. Here is my exact APPEND line changed to use your filenames and IP addresses, in case there’s a difference. I did notice I added “,rw” at the end of the nfsroot, dunno if that makes a difference:

    APPEND root=/dev/nfs initrd=ubuntu-mate/initrd.img-4.4.0-21-generic nfsroot=192.168.1.75:/ubuntu-mate,rw ip=dhcp rw

    I can’t confirm 100% because I didn’t use the debootstrap strategy. Did you end up making the configuration changes to /etc/initramfs-tools/initramfs.conf as described in the wiki article I linked, and then regenerate initrd.img? I’m not 100% sure whether you are actually inside of the initramfs while this problem is occurring, but what I ended up doing was appending “BOOT=nfs” to the end of that file. Everything else was fine by default. After doing that, I had to regenerate initrd.img. That ended up being the initrd.img file that I used for booting and it worked fine with a VirtualBox VM.

  11. Hi Doug,

    I tried everything you said in your last comment but it didn’t work out so I’m falling back to the instructions on Diskless Ubuntu Installation tutorial.

    When I mount the client file system and try cp -ax /. /mnt/. and
    cp -ax /dev/. /mnt/dev/. commands it gives me the following error messages:

    root@client-VirtualBox:/# mount -t nfs -onolock 192.168.1.75:/nfsroot /mnt
    root@client-VirtualBox:/# cp -ax /. /mnt/.
    cp: cannot create directory ‘/mnt/././lost+found’: Read-only file system
    cp: cannot create symbolic link ‘/mnt/././initrd.img’: Read-only file system
    cp: cannot create symbolic link ‘/mnt/././vmlinuz’: Read-only file system
    cp: cannot create directory ‘/mnt/././bin’: Read-only file system
    cp: cannot create directory ‘/mnt/././run’: Read-only file system
    cp: cannot create directory ‘/mnt/././dev’: Read-only file system
    cp: cannot create directory ‘/mnt/././opt’: Read-only file system
    cp: cannot create directory ‘/mnt/././snap’: Read-only file system
    cp: cannot create directory ‘/mnt/././cdrom’: Read-only file system
    cp: cannot create directory ‘/mnt/././nfsroot’: Read-only file system
    cp: cannot create directory ‘/mnt/././home’: Read-only file system
    cp: cannot create directory ‘/mnt/././sbin’: Read-only file system
    cp: cannot create directory ‘/mnt/././etc’: Read-only file system
    cp: cannot create directory ‘/mnt/././mnt’: Read-only file system
    cp: cannot create directory ‘/mnt/././srv’: Read-only file system
    cp: cannot create directory ‘/mnt/././var’: Read-only file system
    cp: cannot create directory ‘/mnt/././boot’: Read-only file system
    cp: cannot create directory ‘/mnt/././lib64’: Read-only file system
    cp: cannot create directory ‘/mnt/././sys’: Read-only file system
    cp: cannot create directory ‘/mnt/././lib’: Read-only file system
    cp: cannot create directory ‘/mnt/././proc’: Read-only file system
    cp: cannot create directory ‘/mnt/././tmp’: Read-only file system
    cp: cannot create directory ‘/mnt/././media’: Read-only file system
    cp: cannot create directory ‘/mnt/././root’: Read-only file system
    cp: cannot create directory ‘/mnt/././usr’: Read-only file system
    cp: preserving times for ‘/mnt/./.’: Read-only file system
    root@client-VirtualBox:/# cp -ax /dev/. /mnt/dev/.
    cp: cannot create directory ‘/mnt/dev/.’: No such file or directory
    root@client-VirtualBox:/#

    192.168.1.75 is my server ip and I have installed Ubuntu 16.04.2 Xenial Xerius Desktop edition this time.

    Please help, thanks!

  12. Hi abkap,

    Are you able to create any files on the NFS share at all? Does “touch /mnt/blah” create a file “blah” on the NFS server, or does it also give you a read-only file system error? Is it possible that the NFS share has been set up without read/write privileges, or with strange permissions?

    I just remembered something: I had to set up the NFS share with a few specific options in order to make everything work correctly. I’m guessing that there is a difference in how you have your NFS share set up. Here is the relevant line from my server’s /etc/exports:

    /nfsroot *(rw,sync,no_subtree_check,insecure,no_root_squash)

    • rw makes it read/write.
    • sync makes sure things are saved to disk properly in case of power outages
    • no_subtree_check is complicated but it seems to be recommended to be done that way and is now the default
    • insecure seems to allow requests to come in from ports above 1024, dunno if it’s even necessary
    • no_root_squash makes root from a client actually be root in the server (rather than mapped to the nobody user), which is important for stuff like setuid files. The man page for exports states this option is mainly useful for diskless clients.

    Hopefully that will get you going! 🙂

  13. Hi Doug,

    Yeah I squared it out. The nfsroot folder had to be shared in read-write format instead of read-only. So /nfsroot *(rw,sync,no_subtree_check,insecure,no_root_squash) configuration in /etc/exports worked.

    But now cp -ax /. /mnt/. command hangs up and I’m trying to figure it out. Any suggestions?

  14. You’re almost there! 🙂

    You could try adding the -v option to the cp command so it prints out what is being copied, so you can see where it’s hanging. I don’t think it’s a great idea to do that copy while you’re booted into the system, so I’m sort of surprised that the wiki suggests it. Why don’t you boot from a live CD (or boot a live CD with PXE!) and do the cp command from there? Mount the hard drive and NFS share while you’re booted from the live CD and do the cp command from there.

    Alternatively if you really have to do it while you’re booted onto the hard drive, you could try doing a mount --bind of / to another directory, and then copying the contents of that other directory. Supposedly cp -x should do the same type of thing, but I think mount --bind is a cleaner way of accomplishing the same task. You might want to empty out your NFS share first in case it has a partial copy of your previous try.

    I hope that works for you!

  15. Hi Doug,

    Thanks for that last tip. I was not able to copy files from Ubuntu Desktop or Mate neither installed nor when I booted from a live cd or did a PXE boot but somehow it worked for Ubuntu Server. So I got all the files on my server and I’m planning to install a GUI later.

    Moving ahead, now I am getting some kinda Mount fail error. It just passes by very quickly when initial ram disk loads but I guess it says something like Failed to mount /dev/nfs and goes to emergency mode. I tried systemctl default command but it says Jobs for graphical.target cancelled.

    So I went through the Now What? section on Diskless Ubuntu wiki again where it says, “Reboot your client PC it should get its net config from the DHCP server, download the pxelinux image, download and boot the kernel, mount your root fs and continue to boot just as if it were using its internal HD”

    What does “mount your root fs” part mean here? Does it implies I have to manually mount the root file system from my server? Considering this, I did try
    mount 192.168.1.100:/nfsroot / command but it didn’t work.
    Then I tried mount -a and rebooted it said /var/lock is a symbolic directory so I commented out /var/lock in fstab and tried again but it again goes to the emergency mode and when I press Ctrl+D now it just sits there.

    What may be the issue now? Please do lemme know. I’ll appreciate it, thanks!

  16. Hi abkap,

    It’s not implying you have to manually mount the root filesystem — it’s saying that Ubuntu should automatically mount it over NFS. At this point, unfortunately I’m not sure what’s wrong. We know your NFS server is set up correctly. It also seems that your pxelinux.cfg/default file is correct based on your earlier comment. If it failed to mount /dev/nfs that seems like it’s having trouble mounting your NFS share. The only real idea I have left is to make sure that you regenerated your initramfs properly with the BOOT=nfs line added, and used that with pxelinux.cfg. If you did that correctly, then I’m not sure what else could be causing problems 🙁

  17. Hi Doug,

    Finally I got it working, yippee! Somehow initrd was not able to mount /var/run and /var/lock at boot so I commented them out in fstab. Besides I also added this bit of code:

    do
    mnt=`echo $rest | sed ‘s/^.* \(\/[^ ]*\) [A-Za-z].*$/\1/’`
    pidfile=/var/run/autofs/`echo $mnt | sed ‘s,/,_,g’`.pid

    + mkdir -p “$mnt”
    start-stop-daemon –start –pidfile $pidfile –quiet \
    –exec $DAEMON — –pid-file=$pidfile $rest

    to nfsroot/etc/init.d/autofs file. I don’t know if this or the other did the trick but its rolling now and I’m super happy. I got this code from Diskless Ubuntu wiki footnotes:

    https://bugs.launchpad.net/ubuntu/+source/autofs/+bug/226219

    You gonna say I’ve gone crazy about it all or what but wanna raise the bet now. I will be trying to boot a client from a proxy dns and nfs-kernel server setup on an Amazon EC2 instance by embedding this custom iPXE script:

    #!ipxe

    set server_ip
    set nfs_path /tftpboot/ubuntu
    kernel nfs://${server_ip}${nfs_path}/casper/vmlinuz.efi || read void
    initrd nfs://${server_ip}${nfs_path}/casper/initrd.lz || read void
    imgargs vmlinuz.efi initrd=initrd.lz root=/dev/nfs boot=casper netboot=nfs nfsroot=${server_ip}:${nfs_path} ip=dhcp splash quiet — || read void
    boot || read void

    into my ipxe.iso build which I’m planning to use to boot a client machine. I have set up a proxy dns and nfs kernel server on an EC2 instance for now.

    What you think, will it work? 🙂

  18. Glad you figured it out! I have no idea about the Amazon EC2 instance or using iPXE, but it definitely sounds interesting as long as you have enough bandwidth to pull it off over the internet!

  19. Hi Doug,

    Finally I did it. I’m able to boot an Ubuntu Desktop 16.04.2 live image from an AWS EC2 instance on a virtual client machine. Had to make a couple of additions to your server setup like installing an Apache2 server with nfs-kernel and dnsmasq to provide vmlinuz.efi and initrd.lz during PXE boot and a script to recognize dhcp server during boot but the rest of it all stayed the same.

    I’m using a 15Mbps connection at my home and it’s kinda laggy on the client side, taking almost 10 minutes to boot-up but I’m super-excited to get it done at the first place. At least we can say it works, which I was not much sure of ever-since I read about those Hyper-V issues which could have created some complications on the AWS side.

    Now I’ll be moving ahead to PXE boot an installed operating system like I did in the local server. Hope all goes well there too!

    Just wanted to keep you posted, as I’ll always be highly obliged for you got me going when I was dead stuck during this whole endeavor.

    Thanks again and take care!

  20. Hey abkap,

    Great job! I’m glad to hear you got it working. I can imagine it would be laggy because of the 15Mbps connection, but it’s still really cool to be able to boot a machine over the internet. Thanks for the update!

  21. where is locating the dnsmasq.conf file in 2.76 version.

  22. It should be in /etc.

Add your comment now