PXE Boot and CloneDeploy with tagged VLAN

  1. 4 weeks ago

    PXE Boot and CloneDeploy with tagged VLAN

    I was able to successfully save an image with CloneDeploy in a tagged VLAN environment.
    It needed three more or less difficult changes (it was harder to find out how than the doing).

    1. Enable PXE boot for the network interface card (Dualport Intel 10GBit 82599 X520)
    2. Enable VLAN for PXE boot
    3. Enable VLAN in CloneDeploy boot image and kernel (compile custom kernel + module and add script to initrd.xz)

    I thought I would write it down and share my experience.
    If you have additional improvements to this procedure let me know!
    !!! I AM STILL USING CloneDeploy v1.2.1 so be aware that the scripts / URL etc are different for v1.3 !!!

  2. 1 Network Interface Card (PXE)

    The first problem was to get pxe boot working at all for the Intel 10GBit nic as it did not show up in the bios nor at boot.
    I found out that you need to enable PXE manually with an Intel utility "BootUtil" that comes in different forms like for DOS for bootable media or EFI or windows exe.

    Download: https://downloadcenter.intel.com/download/19186/Intel-Ethernet-Connections-Boot-Utility-Preboot-images-and-EFI-Drivers

    This archive contains two important things:

    1. the BootUtil
    2. an up to date BootImage (Intel Boot Agent XE)

    With the Bootutil you can enable PXE boot and update the bootimage.
    Updating the BootImage was in my case important because I had v2.1.40 (10GbE) and VLAN is only supported since v2.2.05 (or since v1.3.95 for 1GbE and v1.0.00 for 40GbE).

    # List all NIC
    bootutil -E
    # Update BootImage (Intel(R) Boot Agent XE v2.4.16)
    # either use the -NIC=x with the number for your nic if you have different Intel cards or if you want to update all that are listed you can simply use -ALL
    #Bootutil -UP=PXE -NIC=x -FILE=BOOTIMG.FLB
    # Enable PXE Boot
    bootutil -NIC=x -FE
    #-SETUPENABLE or -ste	should activate the nic menu at boot that you can enter with ctrl + s but this seems only to be working with older bootutil and older Intel Boot Agent version ... unsure
    bootutil -SETUPENABLE -ALL

    Now it should be possible to see the NIC in BIOS and to use PXE but, but now comes the next problem:

  3. Edited 3 weeks ago by deployer

    2 How to set a VLAN ID for PXE Boot

    The BootUtil has some officially declared as "undocumented features" see details here: https://www.intel.com/content/dam/support/us/en/documents/network-and-i-o/ethernet-products/Boot_Agent_book_rev_1_9.pdf#G2.1053055

    #Program VLAN

    At this time if everything went OK it might be already working for you.
    I did an additional step before I found out about the undocumented features but I don't know if it is needed or recommended at all:
    I flashed a new eeprom v4.25 update from Dell but I am not sure if that is recommended or needed at all, so I would recommend to skip this step and only do it if it is not working. (see downn below for details)

    3 Create custom Kernel

    Compile Custom Kernel and enable 8021q

    ########### Compile Custom Kernel with VLAN Support
    # Prepare the system
    aptitude update
    aptitude upgrade
    # optional cleanup:
    #aptitude autoclean
    #apt autoremove --purge
    apt-get install build-essential subversion libqt4-dev bison flex gettext texinfo zlib1g-dev uuid-dev git libelf-dev
    # Download Linux Kernel Sources 
    mkdir kernel
    cd kernel
    cd ..
    mv kernel kerneldev
    cd kerneldev/
    wget http://docs.clonedeploy.org/files/64config.txt
    vi 64config.txt
    wget http://docs.clonedeploy.org/files/32config.txt
    wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.15.11.tar.xz
    tar xf linux-4.15.11.tar.xz
    cd linux-4.15.11/
    # Check current architecture (x86 or x86_64) 
    # copy your config 
    cp /root/kerneldev/64config.txt ./.config
    git clone https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
    cd kerneldev/linux-4.15.11/
    make xconfig
    ### Enable 8021q under "Networking Options"
    ### Make sure to double click it so that it will NOT be compiled as separate module!
    ### check the .config file that it really is: CONFIG_VLAN_8021Q=y
    ### if it is CONFIG_VLAN_8021Q=m it will be compiled as module
    # check cpu cores 
    # compile
    cd kerneldev/linux-4.15.11/
    make -j 4 bzImage
    # Copy image and give useful name
    cp arch/x86_64/boot/bzImage /home/xxx/bzImage_4.15.11x64
  4. Edited 3 weeks ago by deployer

    4 Make Custom BootImage and BootMenu

    1. Unpack initrd.xz
    2. Mount initrd
    3. Modify CloneDeploy Script: etc/init.d/S99cd
    4. Compress initrd to get initrd.xz

    You can find the Boot Images on a Windows CloneDeploy Server here:
    c:\Program Files (x86)\clonedeploy\tftpboot\images\initrd.xz

    Copy it to your Linux Server.

    mkdir image
    unxz initrd.xz
    sudo mount -t ext2 -o loop initrd image/
    sudo vi image/etc/init.d/S99cd

    These are the changes I made to the script:

    ############## Modify CloneDeploy Script in initrd.xz
    # This allows to specify an additional new kernel parameter cd_net_vlanid=1234
    # File: image/etc/init.d/S99cd
    # sudo vi image/etc/init.d/S99cd 
            #predefined static ip
            if [ -n "$cd_net_vlanid" ]; then
            echo "VLAN ID: ${cd_net_vlanid}"
            vconfig add ${net_if} ${cd_net_vlanid}
            echo "auto ${net_if}" >> /etc/network/interfaces
            echo "iface ${net_if}.${cd_net_vlanid} inet static" >> /etc/network/interfaces
              echo "address $cd_net_ip" >> /etc/network/interfaces
              echo "netmask $cd_net_netmask" >> /etc/network/interfaces
              echo "gateway $cd_net_gateway" >> /etc/network/interfaces
            echo "vlan-raw-device ${net_if}" >> /etc/network/interfaces
              echo "$cd_net_dns" >> /etc/resolv.conf
            ip link set ${net_if} up
              ifdown ${net_if}.${cd_net_vlanid}
              ifup ${net_if}.${cd_net_vlanid}
              sleep 5
            elif [ -n "$cd_net_ip" ]; then
              echo "iface $net_if inet static" >> /etc/network/interfaces
              echo "address $cd_net_ip" >> /etc/network/interfaces
              echo "netmask $cd_net_netmask" >> /etc/network/interfaces
              echo "gateway $cd_net_gateway" >> /etc/network/interfaces
              echo "$cd_net_dns" >> /etc/resolv.conf
              ifdown $net_if
              ifup $net_if
              sleep 5
    # finally unmount initrd and compress it
    sudo umount image/
    cat initrd | xz -9 --format=lzma --check=crc32 --compress --stdout > initrdvlan.xz
    # Copy the new initrdvlan.xz to your CloneDeploy Server as:
    # c:\Program Files (x86)\clonedeploy\tftpboot\images\initrdvlan.xz

    ###### Custom BootMenu
    (you need to modify the values for your environment)

    LABEL Client Console
    kernel kernels\4.15.11x64
    append initrd=images\initrdvlan.xz root=/dev/ram0 rw ramdisk_size=156000  web=http://10.x.x.x/clonedeploy/service/client.asmx/ USER_TOKEN= task=debug consoleblank=0 net_if=eth4 cd_net_vlanid=1234 cd_net_ip=10.x.x.x cd_net_netmask=255.255.255.x cd_net_gateway=10.x.x.1 
    MENU LABEL Client Console

    ##### Custom Image Profile
    (you need to modify the values for your environment)

    # Example URL: http://10.x.x.x/clonedeploy/views/images/profiles/pxe.aspx?imageid=51&profileid=54&cat=profiles
    Kernel: 4.15.11x64
    Boot Image: initrdvlan.xz
    Kernel Arguments: net_if=eth4 cd_net_vlanid=1234 cd_net_ip=10.x.x.x cd_net_netmask= cd_net_gateway=10.x.x.1 
  5. Additional Information

    # Dell PowerEdge NIC Card 82599 Firmware


    Dell PowerEdge NIC Card 82599 Firmware v4.25
    This driver is for Dell Dual-Port 10GE Daughter Mezzanine Adapter in PowerEdge C6100, C6105, C6145, C6220, C6220 II.
    Fixes & Enhancements
    Fixes: not applicable


    • MNG images now require 32 KB EEPROM.
    • PXE VLAN Configuration added.
    • Software Checksum implemented.
    • Core Clock Gate Disable is set

    Version 4.25, 4.25
    Release date
    28 May 2013
    Last Updated
    21 Aug 2014

    Dell PowerEdge Intel 82599 10Gb Mezzanine Card Firmware v4.25.1
    Dell Dual-Port 10GbE Daughter Mezzanine Adapter Firmware v4.25.1
    Fixes & Enhancements

    • EEPROM FW 4.25.1 disable VPD support to fix VPD error message.
    • EEPROM FW 4.25.1 update batch script to support DPN: X53DF (Subsystem ID: 004C) only.

    - The Inte 82599 10G Mezzanine card firmware can be updated correctly by fixed subsystem ID (004C) only and no impect for other Intel 82599 10G adapter card.
    Version 4.25.1, 4.25.1
    Release date
    28 Aug 2014
    Last Updated
    08 Oct 2014

    # PXE VLAN
    Source: Boot_Agent_book_rev_1_9.pdf

    8.0 VLAN / 802.1p Support

    Starting with GbE PXE version 1.3.95, 10 GbE PXE version 2.2.05 and 40 GbE PXE version 1.0.00, VLAN tagging and priority ta
    gging is supported on PCIe devices only.

    This feature is not disclosed in end user documentation. The EEPROM image must be
    configured for VLAN support before software utilities enable the setting of VLAN or
    To specify a priority value, the command line parameter -PXEVLANPRIORITY =
    <PRIORITYVALUE> must be used in conjunction
    with the -PXEVLAN parameter. If the -
    PXEVLANPRIORITY parameter is used without
    the -PXEVLAN parameter, it is ignored.
    This feature is not documented for end users but is available on supported devices.

    Values passed to the -PXEVLAN option must
    be a 12-bit number, in a range from 0 to
    4095 (FFF in hexadecimal), specified in de
    cimal or hexadecimal form. A value of 0
    disables VLANs and a value from 1 to 4095 enables VLANs and set the stated VLAN
    number. Any value exceeding this range is rejected by BootUtil.
    A hexadecimal form must be preceded by a 0x prefix. For example, PXEVLAN=0x24.
    Also, a decimal form must not be preceded
    by any prefix. For example, PXEVLAN=36.
    When -PXEVLANPRIORITY is entered, the option value is used as a VLAN priority value,
    in a range of 0 to 7, specified in decimal or hexadecimal form.
    A hexadecimal form must be preceded by the 0x prefix. For example,
     A decimal form must not be preceded by any prefix. For example,
     In all cases, the Bootutil command must also specify the NIC number. For example,
    BootUtil -NIC=5 -PXEVLAN=0x23 -PXEVLANPRIORITY=0x2.
    The EEPROM stores the VLAN configuration in a block. The starting address of the block
    is defined in word 0x3C for GbE PCIe devi
    ces and word 0x20 for 10 GbE PCIe devices.
    The total size of the block varies depending on how many ports the device supports.
    One, two or four ports can be supported.

  6. Edited 3 weeks ago by deployer

    UPDATE 2018-03-27:

    • commented some apt cleanup commands to have it optional
    • changed compression command to use stronger compression and crc32:
    cat initrd | xz -9 --format=lzma --check=crc32 --compress --stdout > initrdvlan.xz

    UPDATE 2018-03-21:
    Instead of using a module it is much better to handle it directly in the kernel.
    So it is not needed to compile, copy and load the module anymore.

  7. clonedeploy

    Mar 21 Administrator

    Thank you very much for this write up. I was going to suggest compiling into the kernel, but you beat me to it. I will try and get this moved over to the wiki. A question for you though. It looks like you are hard coding the vlan into the network card for the pxe boot. Why not just move the port into the correct vlan? Or is the imaging vlan seperate from the end result? Again, thanks, I know these write ups take a lot of time, and is much appreciated.

  8. clonedeploy

    Mar 21 Administrator

    I also meant to add, I will add your kernel options and S99 (doesn't exist any more in 1.3.x, is now /bin/lie_start) changes to future version.

  9. Thanks and yes it is a special case for me.
    The servers are connected to a vlan trunk port (Brocade VDX) because they serve as virtualization hosts (currently esxi).

    I could have used the internal 1Gbe that currently are not connetced at all but they were Broadcom and allow to specify a vlan id in the boot menu, but I have not tried it. I wanted to use what I have and a new Intel 10Gbe adapter should be able to support PXE and VLAN so that bugged me ;)
    I didn't expect that it would get this hard ...

    For the future it might be good to add vlan config options to the menu where you manually specify an IP if DHCP fails, too.


or Sign Up to reply!