Setting up PXE netboot for x86 clients

Relevant docs

dhcptab (4)(although creating dhcptab by hand is not recommended, it's useful to understand the relationships)
Solaris DHCP Administration Guide
System Administration Guide: IP Services
Installation docs

Some things to realize before you start

You Must Control the DHCP server For the Subnet

You need control of the DHCP server that answers on the subnet in question. That might mean "private subnet" or it might mean "add configuration to the building DHCP server". But you can't have two DHCP servers answering the same PXE broadcast; that will just lead to trouble, as more than likely only one of them will have the right information to do a netboot. Also realize that I'm mostly talking about the stuff over and above "normal DHCP" setup here, so I'm glossing over "how you set up normal DHCP info", although I'll give a final full working example that will include a lot of important values that are not strictly used for PXE booting.

Macros Sent Automatically by the Solaris DHCP Server

The Solaris DHCP server sends a bunch of macros semi-automatically for you, in this order (which matters if more than one macro defines the same DHCP option) (see dhcptab(4)):

a macro named after the client-class (i.e. SUNW.i86pc for Solaris x86 PXE boot clients).
a macro named after the network number (and note that the DHCP server machine must have valid netmasks to identify the network appropriately, if the netmask can't be derived from the IP class)
the "default macro" or "server macro" (or "IP Address" macro as called in dhcptab(4)), noted in the network table for this client IP address
a macro named after the client-ID of the client (which, for Ethernet, is "01" followed by the MAC address, in uppercase hexadecimal..that is, for aa:bb:cc:dd:ee:ff, the client-ID is "01AABBCCDDEEFF".

(Note: with respect to the first of these automatic macros, the client-class: during a PXE boot, there are actually two different DHCP transactions, and they use different client-class identifiers. The first, done by the PXE BIOS, uses the client-class PXEClient:Arch:00000:UNDI:002001, and the second, done by the Solaris first network boot program nbp, uses SUNW.i86pc...so in some DHCP setups, you'll see both macros present. They are both client-class macros, and are used in different phases of the same PXE boot.)

Sun Vendor-Specific Options Must Be Added

There are a number of SUN-defined "vendor-specific" DHCP options, and the DHCP server must be taught about them. This is true even of the SUN DHCP server named in.dhcpd.

Quick overview of the PXE boot process

The PXE BIOS sends a DHCP request using client-class PXEClient:Arch:00000:UNDI:002001, which nets it the expected IP configuration parameters, and also the name of a TFTP boot server and boot pathname (for the nbp program).
nbp is downloaded and runs next; this is the first small Solaris network boot program, which announces itself as Solaris network boot.
nbp sends a new DHCP request using client-class SUNW.i86pc, so another response is sent from the DHCP server, containing things relevant to that client-class as well as all the things from the first request relevant to this client-ID, this network, this IP address, etc.
This response allows nbp to find and download inetboot, aka boot.bin, the familiar Solaris second-stage boot program.
inetboot uses the PXE BIOS for network driver services, and mounts the Solaris image using NFS, and proceeds as normal to run the Solaris kernel and installer.

Configuration steps

Now that we've said all that, let's talk about what you need to do to the DHCP server to set up for PXE booting.

If using a Solaris DHCP server (probably), create the vendor-specific options on the DHCP server. I cut'n'pasted the variable definitions from the Solaris DHCP Administration Guide, the first part of Example 4-1, and created the script add.netboot.options, included below, to do that. Note that the vendor-specific options are tied to a client-class identifier; they will only be supplied if the client-class in the DHCP request is set to one of the client-class IDs included in the macro definition. This allows different vendor spaces to exist, selected by client-class. (Note also that this means that the first PXE request, where client-class is PXEClient:Arch:00000:UNDI:002001, will *not* receive the SUN vendor-specific macros in its response.)
add the normal "network" and "addresses" entries using your favorite DHCP configuration method (graphical or cmdline utilities). The graphical manager dhcpmgr makes this significantly easier, as it pre-defines macros for the network and the clients when you use the address wizard to add addresses. Note that there's also a "default" macro for each served address, which I often name by the DHCP server, and put things relevant to the subnet in question there.
Now you have two choices. You can
1. use add_install_client with the -d switch. It will tell you something like this (with obvious substitutions for specific names and addresses):
```
To enable SUNW.i86pc in the DHCP server,
add an entry to the server with the following data:

   Install server      (SinstNM)  : kpt01
   Install server IP   (SinstIP4) : 129.153.72.231
   Install server path (SinstPTH) : /export/home/images/s9u2/04
   Root server name    (SrootNM)  : kpt01
   Root server IP      (SrootIP4) : 129.153.72.231
   Root server path    (SrootPTH) : /export/home/images/s9u2/04/Solaris_9/Tools/Boot

To enable PXE boot, create a macro definition called
PXEClient:Arch:00000:UNDI:002001 which has the following values:
   Boot file           (BootFile) : nbp.SUNW.i86pc
   Boot server IP      (BootSrvA) : 129.153.72.231
```
  and realize that the first message means "create a macro named SUNW.i86pc to hold these specific values of those options", and that the second means just "create the macro, don't bother tying it to any particular other macro or including it in some other macro" (which is important because you *can't* include a macro that has ':' in its name). So, create SUNW.i86pc and PXEClient:Arch:00000:UNDI:002001 as directed. (The PXEClient:Arch:00000:UNDI:002001 macro will also be sent automagically, because there are really two separate DHCP requests, one from the PXE BIOS and one from the nbp program; on the first request, the client-class is set to PXEClient:Arch:00000:UNDI:002001, and on the second request it's set to SUNW.i86pc. So both macros will be sent in different phases of the boot, because they're both client-ID macros as in d) above.)
2. if you want things to be different for each client, which we usually do inside Sun so we can install different releases from the same server, use add_install_client with both the -d and -e <macaddr> switches, which will advise you to create instead a macro named "01-followed-by-the-Ethernet-address-in-uppercase-hex" (that is, the default client-ID) that contains the entries shown. i.e.: for a machine with Ethernet address 00:07:e9:04:4a:bf, define a macro named 010007E9044ABF containing everything specific to that client. This will take the place of the SUNW.i86pc macro in the 2a), class-specific method. It will also recommend that you create a SbootFIL macro containing the name of the inetboot program link it sets up, which will be the client-ID. Note: this is a bug, and will be changed soon to the SbootURI form. See Outstanding Problems below.

Results

With all this done, the network macro (network number, say 10.0.0.0), the 'default' macro for the IP address, the SUNW.i86pc macro OR the client-ID macro, and the PXEClient:Arch:00000:UNDI:002001 macro will all be combined to create a composite DHCP response.

Real Working Example

Here's an example client-specific setup for two clients on the 10.11.12.0 network.


# pntadm -P 10.11.12.0

Client ID       Flags   Client IP       Server IP       Lease Expiration                Macro           Comment
01000347D56F52  00      10.11.12.104    129.153.72.231  08/09/2003                      kpt01           kpt02 LX50
0100010326A537  00      10.11.12.106    129.153.72.231  08/09/2003                      kpt01           3Com plugin card

This defines two IP address assignments, both dynamic. Each client will receive the 'kpt01' macro as the "default" or "server" macro from c) above.

# dhtadm -P

Name                    Type            Value

==================================================
01000347D56F52          Macro           :SinstNM="kpm01":SinstIP4=10.11.12.1:SinstPTH="/export/home/images/s9u2/04":SrootNM="kpm01":SrootIP4=10.11.12.1:SrootPTH="/export/home/images/s9u2/04/Solaris_9/Tools/Boot":SbootFIL="inetboot.stifix":BootFile="nbp.dmick":
0100010326A537          Macro           :SinstNM="kpm01":SinstIP4=10.11.12.1:SinstPTH="/export/home/images/s9u4/08a":SrootNM="kpm01":SrootIP4=10.11.12.1:SrootPTH="/export/home/images/s9u4/08a/Solaris_9/Tools/Boot":SbootFIL="inetboot.grizzly":BootFile="nbp.dmick":
10.11.12.0              Macro           :Subnet=255.255.255.0:RDiscvyF=1:Broadcst=10.11.12.255:
kpt01                   Macro           :Include=Locale:Timeserv=129.153.72.231:LeaseTim=86400:LeaseNeg:
PXEClient:Arch:00000:UNDI:002001        Macro           :BootFile="nbp.SUNW.i86pc":BootSrvA=10.11.12.1:
Locale                  Macro           :UTCoffst=-28800:
Sterm                   Symbol          Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,15,ASCII,1,0
SjumpsCF                Symbol          Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,14,ASCII,1,0
SsysidCF                Symbol          Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,13,ASCII,1,0
SinstPTH                Symbol          Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,12,ASCII,1,0
SinstNM                 Symbol          Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,11,ASCII,1,0
SinstIP4                Symbol          Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,10,IP,1,1
SbootRS                 Symbol          Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,9,NUMBER,2,1
Stz                     Symbol          Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,8,ASCII,1,0
SbootFIL                Symbol          Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,7,ASCII,1,0
SswapPTH                Symbol          Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,6,ASCII,1,0
SswapIP4                Symbol          Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,5,IP,1,0
SrootPTH                Symbol          Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,4,ASCII,1,0
SrootNM                 Symbol          Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,3,ASCII,1,0
SrootIP4                Symbol          Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,2,IP,1,1
SrootOpt                Symbol          Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,1,ASCII,1,0

Note that, in this example, there is no SUNW.i86pc client-class macro; everything is done with client-ID macros, so each client must be administered separately. Other environments may not need or want such close control, so might define a SUNW.i86pc macro to cover all clients, or might define both SUNW.i86pc macros *and* client-ID macros, so that some clients get special behavior, and some default.

The first client, client-ID 01000347D56F52, or "the one with MAC addr 00:03:47:d5:6f:52", will get a DHCP response including

PXEClient:Arch:00000:UNDI:002001, because that's the first client-class, so containing BootFile=nbp.SUNW.i86pc and BootSrvA=10.11.12.1. These are used for the very first TFTP request, to download the nbp program, the first stage of the Solaris boot. nbp prints a message Solaris network boot so you know it's working. (note that both of these values are relevant to that first request, and nothing else; no other DHCP use would need a BootFile or a BootSrvA.)
There is no SUNW.i86pc macro present, so there will be no client-class options for this client during the second DHCP request from nbp.
10.11.12.0, since that's the network number, so Subnet=255.255.255.0, RDiscvyF=1, and Broadcst=10.11.12.255. Note that all these values are germane to the 10.11.12.0 network, even if only doing "normal" DHCP.
kpt01 (because it's the 'default macro' defined in the network table), which gives the contents of the Locale macro (UTCoffst=-28800),Timeserv=129.153.72.231,LeaseTim=86400, and LeaseNeg. It could be argued that Locale and Timeserv belong in the network macro 10.11.12.0.
01000347D56F52, because it matches the client-ID. Further, because the client-class is set to SUNW.i86pc, the Sun vendor-specific options are available to this response, so it can include the S options. So this macro selects and includes SinstNM="kpm01", SinstIP4=10.11.12.1,SinstPTH="/export/home/images/s9u2/04",SrootNM="kpm01",SrootIP4="10.11.12.1",SrootPTH="/export/home/images/s9u2/04/ Solaris_9/Tools/Boot", SbootFIL="inetboot.stifix", BootFile="nbp.dmick". Note: this sets SbootFIL to something different than the default (the client-class string, or SUNW.i86pc), and also sets BootFile, which overrides the setting in PXEClient:Arch:00000:UNDI:002001, because the automatic macros are processed and added in order as above.

Notice, also, that the Sun vendor-specific option definitions show up in the dhtadm output.

Outstanding Problems

Bug 4809551 meant you couldn't specify a client-specific secondary boot (inetboot or so), so all clients had to share the same boot programs. This bug was fixed, but incorrectly, by changing nbp to use SbootFIL as the name of inetboot. This works, but confuses the "default load file name", so booting with kadb tries to load inetboot. The real fix should be to use SbootURI for the URI of the inetboot file (i.e. tftp:/<tftp-ip-addr>/inetboot) but that's not implemented yet. See bug 4905669.

It is currently impossible to select serial console when booting from PXE (the only way to do this previously was to change settings on the boot floppy, and since there is no boot floppy, some other solution is clearly required). The proposed solution is to allow the creation of client-specific bootenv.rc files on the install server, and is documented in bug 4769739, which will be included in the putback for the Claymore project, due into s10 in build s10_43.

Note also that different versions of add_install_client may do slightly-wrong things in advising you about macros and contents. Use this document as a sanity guide and adjust what add_install_client recommends.

add.netboot.options script

add.netboot.options script (including SbootURI):

#!/bin/ksh
# Load the Solaris vendor specific options. We'll start out supporting 
# the Ultra-1, Ultra-30, and i86 platforms. Changing -A to -M would replace
# the current values, rather than add them.
dhtadm -A -s SrootOpt -d 'Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,1,ASCII,1,0'
dhtadm -A -s SrootIP4 -d 'Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,2,IP,1,1'
dhtadm -A -s SrootNM -d 'Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,3,ASCII,1,0'
dhtadm -A -s SrootPTH -d 'Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,4,ASCII,1,0'
dhtadm -A -s SswapIP4 -d 'Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,5,IP,1,0'
dhtadm -A -s SswapPTH -d 'Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,6,ASCII,1,0'
dhtadm -A -s SbootFIL -d 'Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,7,ASCII,1,0'
dhtadm -A -s Stz -d 'Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,8,ASCII,1,0'
dhtadm -A -s SbootRS -d 'Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,9,NUMBER,2,1'
dhtadm -A -s SinstIP4 -d 'Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,10,IP,1,1'
dhtadm -A -s SinstNM -d 'Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,11,ASCII,1,0'
dhtadm -A -s SinstPTH -d 'Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,12,ASCII,1,0'
dhtadm -A -s SsysidCF -d 'Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,13,ASCII,1,0'
dhtadm -A -s SjumpsCF -d 'Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,14,ASCII,1,0'
dhtadm -A -s Sterm -d 'Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,15,ASCII,1,0'
dhtadm -A -s SbootURI -d 'Vendor=SUNW.Ultra-1 SUNW.Ultra-30 SUNW.i86pc,16,ASCII,1,0'