PXE Boot and CloneDeploy with tagged VLAN



  • [h]PXE Boot and CloneDeploy with tagged VLAN[/h]

    I was able to successfully save an image with CloneDeploy in a tagged VLAN environment.
    It needed three more or less difficult changes (it was harder to find out how than the doing).

    1. Enable PXE boot for the network interface card (Dualport Intel 10GBit 82599 X520)
    2. Enable VLAN for PXE boot
    3. Enable VLAN in CloneDeploy boot image and kernel (compile custom kernel + module and add script to initrd.xz)

    I thought I would write it down and share my experience.
    [b]If you have additional improvements to this procedure let me know![/b]
    [i][b]!!! I AM STILL USING CloneDeploy v1.2.1 so be aware that the scripts / URL etc are different for v1.3 !!![/b][/i]



  • [h]1 Network Interface Card (PXE)[/h]

    The first problem was to get pxe boot working at all for the Intel 10GBit nic as it did not show up in the bios nor at boot.
    I found out that you need to enable PXE manually with an Intel utility "BootUtil" that comes in different forms like for DOS for bootable media or EFI or windows exe.

    Download: [url=https://downloadcenter.intel.com/download/19186/Intel-Ethernet-Connections-Boot-Utility-Preboot-images-and-EFI-Drivers]https://downloadcenter.intel.com/download/19186/Intel-Ethernet-Connections-Boot-Utility-Preboot-images-and-EFI-Drivers[/url]

    This archive contains two important things:

    1. the BootUtil
    2. an up to date BootImage (Intel Boot Agent XE)

    With the Bootutil you can enable PXE boot and update the bootimage.
    Updating the BootImage was in my case important because I had v2.1.40 (10GbE) and VLAN is only supported since [b]v2.2.05[/b] (or since v1.3.95 for 1GbE and v1.0.00 for 40GbE).

    [code]# List all NIC
    bootutil -E

    Update BootImage (Intel(R) Boot Agent XE v2.4.16)

    either use the -NIC=x with the number for your nic if you have different Intel cards or if you want to update all that are listed you can simply use -ALL

    #Bootutil -UP=PXE -NIC=x -FILE=BOOTIMG.FLB
    Bootutil -UP=PXE -ALL -FILE=BOOTIMG.FLB

    Enable PXE Boot

    bootutil -NIC=x -FE
    #-SETUPENABLE or -ste should activate the nic menu at boot that you can enter with ctrl + s but this seems only to be working with older bootutil and older Intel Boot Agent version ... unsure
    bootutil -SETUPENABLE -ALL[/code]

    Now it should be possible to see the NIC in BIOS and to use PXE but, but now comes the next problem:



  • [h]2 How to set a VLAN ID for PXE Boot[/h]

    The BootUtil has some officially declared as "undocumented features" see details here: https://www.intel.com/content/dam/support/us/en/documents/network-and-i-o/ethernet-products/Boot_Agent_book_rev_1_9.pdf#G2.1053055

    [code]#Program VLAN
    bootutil -NIC=x -PXEVLAN=YOUTVLANNUMBER -PXEVLANPRIORITY=0[/code]

    At this time if everything went OK it might be already working for you.
    I did an additional step before I found out about the undocumented features but I don't know if it is needed or recommended at all:
    I flashed a new eeprom v4.25 update from Dell but I am not sure if that is recommended or needed at all, so I would recommend to skip this step and only do it if it is not working. (see downn below for details)

    [h]3 Create custom Kernel[/h]
    Compile Custom Kernel and enable 8021q

    [code]########### Compile Custom Kernel with VLAN Support

    Prepare the system

    aptitude update
    aptitude upgrade

    optional cleanup:

    #aptitude autoclean
    #apt autoremove --purge

    apt-get install build-essential subversion libqt4-dev bison flex gettext texinfo zlib1g-dev uuid-dev git libelf-dev
    reboot

    Download Linux Kernel Sources

    mkdir kernel
    cd kernel
    cd ..
    mv kernel kerneldev
    cd kerneldev/
    wget http://docs.clonedeploy.org/files/64config.txt
    vi 64config.txt
    wget http://docs.clonedeploy.org/files/32config.txt
    wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.15.11.tar.xz
    tar xf linux-4.15.11.tar.xz
    cd linux-4.15.11/

    Check current architecture (x86 or x86_64)

    lscpu

    copy your config

    cp /root/kerneldev/64config.txt ./.config

    git clone https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
    cd kerneldev/linux-4.15.11/
    make xconfig

    Enable 8021q under "Networking Options"

    Make sure to double click it so that it will NOT be compiled as separate module!

    check the .config file that it really is: CONFIG_VLAN_8021Q=y

    if it is CONFIG_VLAN_8021Q=m it will be compiled as module

    check cpu cores

    nproc

    compile

    cd kerneldev/linux-4.15.11/
    make -j 4 bzImage

    Copy image and give useful name

    cp arch/x86_64/boot/bzImage /home/xxx/bzImage_4.15.11x64

    [/code]



  • [h]4 Make Custom BootImage and BootMenu[/h]

    1. Unpack initrd.xz
    2. Mount initrd
    3. Modify CloneDeploy Script: etc/init.d/S99cd
    4. Compress initrd to get initrd.xz

    You can find the Boot Images on a Windows CloneDeploy Server here:
    c:\Program Files (x86)\clonedeploy\tftpboot\images\initrd.xz

    Copy it to your Linux Server.

    [code]mkdir image
    unxz initrd.xz
    sudo mount -t ext2 -o loop initrd image/
    sudo vi image/etc/init.d/S99cd[/code]

    These are the changes I made to the script:

    [code]############## Modify CloneDeploy Script in initrd.xz

    This allows to specify an additional new kernel parameter cd_net_vlanid=1234

    File: image/etc/init.d/S99cd

    sudo vi image/etc/init.d/S99cd

        #predefined static ip
        if [ -n "$cd_net_vlanid" ]; then
        echo "VLAN ID: ${cd_net_vlanid}"
        vconfig add ${net_if} ${cd_net_vlanid}
        echo "auto ${net_if}" >> /etc/network/interfaces
        echo "iface ${net_if}.${cd_net_vlanid} inet static" >> /etc/network/interfaces
          echo "address $cd_net_ip" >> /etc/network/interfaces
          echo "netmask $cd_net_netmask" >> /etc/network/interfaces
          echo "gateway $cd_net_gateway" >> /etc/network/interfaces
        echo "vlan-raw-device ${net_if}" >> /etc/network/interfaces
          echo "$cd_net_dns" >> /etc/resolv.conf
        ip link set ${net_if} up
          ifdown ${net_if}.${cd_net_vlanid}
          ifup ${net_if}.${cd_net_vlanid}
          sleep 5
    
        elif [ -n "$cd_net_ip" ]; then
          echo "iface $net_if inet static" >> /etc/network/interfaces
          echo "address $cd_net_ip" >> /etc/network/interfaces
          echo "netmask $cd_net_netmask" >> /etc/network/interfaces
          echo "gateway $cd_net_gateway" >> /etc/network/interfaces
          echo "$cd_net_dns" >> /etc/resolv.conf
          ifdown $net_if
          ifup $net_if
          sleep 5
    

    finally unmount initrd and compress it

    sudo umount image/
    cat initrd | xz -9 --format=lzma --check=crc32 --compress --stdout > initrdvlan.xz

    Copy the new initrdvlan.xz to your CloneDeploy Server as:

    c:\Program Files (x86)\clonedeploy\tftpboot\images\initrdvlan.xz

    [/code]

    [b]###### Custom BootMenu[/b]
    (you need to modify the values for your environment)

    [code]LABEL Client Console
    kernel kernels\4.15.11x64
    append initrd=images\initrdvlan.xz root=/dev/ram0 rw ramdisk_size=156000 web=http://10.x.x.x/clonedeploy/service/client.asmx/ USER_TOKEN= task=debug consoleblank=0 net_if=eth4 cd_net_vlanid=1234 cd_net_ip=10.x.x.x cd_net_netmask=255.255.255.x cd_net_gateway=10.x.x.1
    MENU LABEL Client Console[/code]

    [b]##### Custom Image Profile[/b]
    (you need to modify the values for your environment)

    [code]# Example URL: http://10.x.x.x/clonedeploy/views/images/profiles/pxe.aspx?imageid=51&profileid=54&cat=profiles
    Kernel: 4.15.11x64
    Boot Image: initrdvlan.xz
    Kernel Arguments: net_if=eth4 cd_net_vlanid=1234 cd_net_ip=10.x.x.x cd_net_netmask=255.255.255.0 cd_net_gateway=10.x.x.1
    [/code]



  • [h]Additional Information[/h]

    WARNING: OPTIONAL AND MIGHT CAUSE TROUBLES

    Dell PowerEdge NIC Card 82599 Firmware

    v4.25.1
    [url=http://www.dell.com/support/home/en/en/debsdt1/Drivers/DriversDetails?driverId=G6Y8N]http://www.dell.com/support/home/en/en/debsdt1/Drivers/DriversDetails?driverId=G6Y8N[/url]
    v4.25
    [url=https://www.dell.com/support/home/en/en/debsdt1/Drivers/DriversDetails?driverId=N01KV]https://www.dell.com/support/home/en/en/debsdt1/Drivers/DriversDetails?driverId=N01KV[/url]

    [b]Dell PowerEdge NIC Card 82599 Firmware v4.25[/b]
    This driver is for Dell Dual-Port 10GE Daughter Mezzanine Adapter in PowerEdge C6100, C6105, C6145, C6220, C6220 II.
    Fixes & Enhancements
    Fixes: not applicable

    Enhancement:

    • MNG images now require 32 KB EEPROM.
    • PXE VLAN Configuration added.
    • Software Checksum implemented.
    • Core Clock Gate Disable is set
      Version
      Version 4.25, 4.25
      Category
      Network
      Release date
      28 May 2013
      Last Updated
      21 Aug 2014

    [b]Dell PowerEdge Intel 82599 10Gb Mezzanine Card Firmware v4.25.1[/b]
    Dell Dual-Port 10GbE Daughter Mezzanine Adapter Firmware v4.25.1
    Fixes & Enhancements
    Fixes:

    • EEPROM FW 4.25.1 disable VPD support to fix VPD error message.
    • EEPROM FW 4.25.1 update batch script to support DPN: X53DF (Subsystem ID: 004C) only.

    Enhancements:

    • The Inte 82599 10G Mezzanine card firmware can be updated correctly by fixed subsystem ID (004C) only and no impect for other Intel 82599 10G adapter card.
      Version
      Version 4.25.1, 4.25.1
      Category
      Network
      Release date
      28 Aug 2014
      Last Updated
      08 Oct 2014

    #########

    PXE VLAN

    #########
    Source: [url=https://www.intel.com/content/dam/support/us/en/documents/network-and-i-o/ethernet-products/Boot_Agent_book_rev_1_9.pdf#G2.1053055]Boot_Agent_book_rev_1_9.pdf[/url]

    [b]8.0 VLAN / 802.1p Support[/b]

    Starting with GbE PXE version 1.3.95, 10 GbE PXE version 2.2.05 and 40 GbE PXE version 1.0.00, VLAN tagging and priority ta
    gging is supported on PCIe devices only.

    This feature is not disclosed in end user documentation. The EEPROM image must be
    configured for VLAN support before software utilities enable the setting of VLAN or
    priority.
    To specify a priority value, the command line parameter -PXEVLANPRIORITY =
    <PRIORITYVALUE> must be used in conjunction
    with the -PXEVLAN parameter. If the -
    PXEVLANPRIORITY parameter is used without
    the -PXEVLAN parameter, it is ignored.
    This feature is not documented for end users but is available on supported devices.

    Values passed to the -PXEVLAN option must
    be a 12-bit number, in a range from 0 to
    4095 (FFF in hexadecimal), specified in de
    cimal or hexadecimal form. A value of 0
    disables VLANs and a value from 1 to 4095 enables VLANs and set the stated VLAN
    number. Any value exceeding this range is rejected by BootUtil.
    A hexadecimal form must be preceded by a 0x prefix. For example, PXEVLAN=0x24.
    Also, a decimal form must not be preceded
    by any prefix. For example, PXEVLAN=36.
    When -PXEVLANPRIORITY is entered, the option value is used as a VLAN priority value,
    in a range of 0 to 7, specified in decimal or hexadecimal form.
    A hexadecimal form must be preceded by the 0x prefix. For example,
    PXEVLANPRIORITY=0x5.
     A decimal form must not be preceded by any prefix. For example,
    PXEVLANPRIORITY=5.
     In all cases, the Bootutil command must also specify the NIC number. For example,
    BootUtil -NIC=5 -PXEVLAN=0x23 -PXEVLANPRIORITY=0x2.
    The EEPROM stores the VLAN configuration in a block. The starting address of the block
    is defined in word 0x3C for GbE PCIe devi
    ces and word 0x20 for 10 GbE PCIe devices.
    The total size of the block varies depending on how many ports the device supports.
    One, two or four ports can be supported.



  • UPDATE 2018-03-27:
    -commented some apt cleanup commands to have it optional
    -changed compression command to use stronger compression and crc32:
    [code]cat initrd | xz -9 --format=lzma --check=crc32 --compress --stdout > initrdvlan.xz[/code]

    UPDATE 2018-03-21:
    Instead of using a module it is much better to handle it directly in the kernel.
    So it is not needed to compile, copy and load the module anymore.



  • Thank you very much for this write up. I was going to suggest compiling into the kernel, but you beat me to it. I will try and get this moved over to the wiki. A question for you though. It looks like you are hard coding the vlan into the network card for the pxe boot. Why not just move the port into the correct vlan? Or is the imaging vlan seperate from the end result? Again, thanks, I know these write ups take a lot of time, and is much appreciated.



  • I also meant to add, I will add your kernel options and S99 (doesn't exist any more in 1.3.x, is now /bin/lie_start) changes to future version.



  • Thanks and yes it is a special case for me.
    The servers are connected to a vlan trunk port (Brocade VDX) because they serve as virtualization hosts (currently esxi).

    I could have used the internal 1Gbe that currently are not connetced at all but they were Broadcom and allow to specify a vlan id in the boot menu, but I have not tried it. I wanted to use what I have and a new Intel 10Gbe adapter should be able to support PXE and VLAN so that bugged me 😉
    I didn't expect that it would get this hard ...

    For the future it might be good to add vlan config options to the menu where you manually specify an IP if DHCP fails, too.