How to run OpenBCM SDK on Netberg Aurora switches

Broadcom Switch Chip

In this document, we will discuss integrating OpenBCM in Ubuntu 20.04 to enable the Broadcom switch chip basic function.

OpenBCM Broadcom Switch Software Development Kit contains the source code for Broadcom network switch APIs and drivers for programming Broadcom network switch silicon-based platforms.

High-level description

We will use Netberg’s Trident3-based switches as an example.

Aurora 621 has a Trident3-X5 (BCM56770) as the main data and 1G control planes.

Aurora 721 has a Trident3-X7 (BCM56870) as the main data and 1G control planes.

Broadcom Chip Connectivity

The diagram below shows how this system connects with the main chip BCM56770.

Figure 1. Broadcom Chip Connectivity

PCIe Connectivity

SDK application communicates with Broadcom chips through the PCIe interface. The figure below shows the PCIe connectivity. To know the bus/slot/function for each chip, NOS can get it via its root port. Below is a sample output from Ubuntu Linux.

root@ubuntu:~# lspci -s `setpci -s 00:0b.0 0x19.b`:00.0
04:00.0 Ethernet controller: Broadcom Limited Device b770 (rev 01)
Figure 2. PCIe Connectivity

Broadcom Software Package Integration


Install Ubuntu 20.04 and set it up following our guide:

Please follow both Post-installation steps and model-specific steps.

Broadcom Software Packages

SDK – Broadcom network switching Software Development Kit.

We will show an integration example using OpenBCM SDK 6.5.27.

Integration Flow


This package builds the basic application to enable the Broadcom switch. All the other packages are linked to this one.

The Makefile we use is sdk-6.5.27/make/x86-smp_generic_64-2_6.

Reference MAKE_LOCAL files are located here – sdk-6.5.27/make/local/esw

sdk-6.5.27/make/Make.local.template contains comments on all available makefile options.

For example, we pick sdk-6.5.27/make/local/esw/Make.pkg.56770 and put it as sdk-6.5.27/make/Make.local. This approach saves space, and the build procedure will use this one if it exists.

Step-by-Step Procedure

Below is a simple guideline to build basic applications with all required Broadcom software packages.

1. Put all software packages into the folder called build_root (you can, based on your development environment, adjust properly).

user@ubuntu:~/build_root$ ls -al
-rwxr--r-- 1 user user 248642263 Dec 7 07:07 sdk-all-6.5.27.tar.gz

2. Untar all of them.

user@ubuntu:~/build_root$ tar zxf sdk-all-6.5.27.tar.gz 
user@ubuntu:~/build_root$ ll
drwxr-xr-x 12 user user 4096 Dec 28 10:04 sdk-all-6.5.27/
-rwxr--r-- 1 user user 248642263 Dec 7 07:07 sdk-all-6.5.27.tar.gz

3. Make SDK to create bcm.user Option: the user can specify to compile with a particular kernel version by setting ‘KERNDIR’. Our example uses kernel 5.15.91.

export KERNDIR=/home/user/kernel/linux-headers-4.15.0-123-generic

make -C sdk-all-6.5.27/systems/linux/user/x86-smp_generic_64-2_6/

When the SDK build is completed, the basic application will be generated.

user@ubuntu:~/build_root$ ls -al sdk-all-6.5.27/build/linux/user/x86-smp_generic_64-2_6/ 
total 391544 
drwxrwxr-x 2 user user 4096 Dec 29 12:37 . 
drwxrwxr-x 3 user user 4096 Dec 28 10:04 .. 
-rwxrwxr-x 1 user user 101205848 Dec 29 12:37 bcm.user 
-rwxrwxr-x 1 user user 298537184 Dec 29 12:37 bcm.user.dbg 
-rw-rw-r-- 1 user user 125192 Dec 29 12:36 linux-kernel-bde.ko 
-rw-rw-r-- 1 user user 34624 Dec 29 03:52 linux-user-bde.ko 
-rwxrwxr-x 1 user user 1009376 Dec 29 12:37 netserve

Configurations and Initialization Flow

Netberg provides configuration files, a led binary file, sample LED code, and patch files for reference.

The tree below shows the package content. We will use these and SDK reference configuration files in the following sections to enable the switch’s basic function.

Configuration files, led binary file, sample codes, and led patch:

|-- config
|   |-- port_mode
|   |   |-- td3_pm_25g_100g.bcm
|   |   |-- td3_pm_25g_100g_break.bcm
|   |-- SI
|   |   |-- td3_si_qsfp_40g.bcm
|   |   |-- td3_si_qsfp_100g.bcm
|   |   |-- td3_si_sfpp_1g.bcm
|   |   |-- td3_si_sfpp_10g.bcm
|   |   |-- td3_si_sfpp_25g.bcm
|   |-- rc.soc
|   |-- td3_lanemap.bcm
|   |-- td3_sys.bcm
|-- led
|   |-- custom_led.bin
|-- patch
|   |-- sdk-6.5.27_ledproc.patch
|-- sample
|   |-- custom_led.c
|   |-- custom_led.h
|   |-- Make.local

Configurations Provided by SDK

Three types of files are required to enable the switch’s basic functionality.

  1. Binary files built from Broadcom software packages
  2. Configuration files to enable Broadcom software features
  3. Board-specific configuration files from Netberg according to hardware design.

The first step is to create a folder ‘$DEST’ to store relevant files and to copy the binary files into this folder.

mkdir -p DEST
cp -f sdk-6.5.27/build/linux/user/x86-smp_generic_64-2_6/bcm.user $DEST
cp -f sdk-6.5.27/build/linux/user/x86-smp_generic_64-2_6/linux-kernel-bde.ko $DEST
cp -f sdk-6.5.27/build/linux/user/x86-smp_generic_64-2_6/linux-user-bde.ko $DEST

The second step is to prepare the common configuration file.

cp -f sdk-all-6.5.27/rc/cmicfw/linkscan_led_fw.bin $DEST

The third step is to apply board specific configuration as described below.

Platform Specific Configurations

Configuration Files

Broadcom chips need configuration files to work properly with a board. Netberg provides several customized configuration files for all usage scenarios.

In this section, we will describe these files and their purpose.

The files are listed according to the build version it can use. It is important to check the revision and use the correct file to ensure the system behavior can be as expected.

File NamePurpose
config.bcmconfig.bcm is the entry point of configuration files for application initialization. We construct this file from the configs, td3_sys.bcm, td3_lanemap.bcm, and td3_[port mode].bcm.
td3_sys.bcmMAC system initial configuration.
td3_lanemap.bcmMAC lane configurations for port polarity, lane swap
td3_[port mode].bcm Port configuration.
td3_pm_25g_100g.bcm for 48 x 25G + 6 x 100G NIF interface
td3_pm_25g_100g_break.bcm for 48 x 25G + 4 x 100G + 8 x 25G port (100g breakout) NIF interface.

Note: The system only supports two 100g ports breakout due to logical number (total 64) limitation
custom_led.binLED binary file for NIF ports.
td3_si_qsfp_100g.bcmForce TX FIR settings on QSFP 100G mode
td3_si_qsfp_40g.bcmForce TX FIR settings on QSFP 40G mode
td3_si_sfpp_25g.bcmForce TX FIR settings on SFP28 25G mode
td3_si_sfpp_10g.bcmForce TX FIR settings on SFP28 10G mode
td3_si_sfpp_1g.bcmForce TX FIR settings on SFP28 1G mode

Here is an example on how to use these files:

cat NBA621-CFG/config/td3_sys.bcm > config.bcm
cat NBA621-CFG/config/td3_lanemap.bcm >> config.bcm

48 x 25G + 6 x 100G with Copper Cable Use Case

We recommend enabling link training when a front port connects with a copper cable. Link training provides a fast and convenient way to set the TX FIR automatically.

cat NBA621-CFG/config/port_mode/td3_pm_25g_100g.bcm >> config.bcm
cp NBA621-CFG/led/custom_led.bin custom_led.bin
cp NBA621-CFG/config/rc.soc rc.soc

Initialization flow

1. Insert kernel modules.

/bin/mknod /dev/linux-kernel-bde c 127 0 
/bin/mknod /dev/linux-user-bde c 126 0 
insmod linux-kernel-bde.ko debug=8 dmasize=32M 
insmod linux-user-bde.ko

2. Initialize the basic application.


Apply SI

48 x 25G + 6 x 100G mode: (25G logical port 1-28,33-52; 100G logical port 29-30,53-55,59)

BCM.0> port 1-28,33-52,29-30,53-55,59 en=0
BCM.0> phy control 1-28,33-52,29-30,53-55,59 lt=0
BCM.0> port 1-28,33-52 speed=25000 if=sr fd=1
BCM.0> port 29-30,53-55,59 speed=100000 if=sr4 fd=1
BCM.0> rcload NBA621-CFG/config/SI/td3_si_sfpp_25g.bcm
BCM.0> rcload NBA621-CFG/config/SI/td3_si_qsfp_100g.bcm
BCM.0> port 1-28,33-52,29-30,53-55,59 en=1

48 x 10G + 6 x 40G mode:

BCM.0> port 1-28,33-52,29-30,53-55,59 en=0
BCM.0> phy control 1-28,33-52,29-30,53-55,59 lt=0
BCM.0> port 1-28,33-52 speed=10000 if=sr fd=1
BCM.0> port 29-30,53-55,59 speed=40000 if=sr4 fd=1
BCM.0> phy control 1-28,33-52,29-30,53-55,59 lt=0
BCM.0> rcload td3_si_sfpp_10g.bcm
BCM.0> rcload td3_si_qsfp_40g.bcm
BCM.0> port 1-28,33-52,29-30,53-55,59 en=1

Management port

BCM56770 only supports one management port and must be assigned to logical port 64.

Besides, the management port of Aurora 621 only supports 1G speed due to the Atom C3508 limitation.

BCM.0> port 64 enable=0
BCM.0> port 64 speed=1000 fd=1
BCM.0> vlan remove 1 pbm=xe,ce,64
BCM.0> port 64 enable=1

LED reference code

Netberg provides an LED reference code to match common breakout and non-breakout port LED behavior.

The default LED behavior is as follows:

  • The port’s current speed equals the port’s max speed, then green.
  • If the port’s current speed is lower than the port’s max speed, then yellow. (include breakout speed)
  • Any tx/rx traffic in that port is active, then blinking.

Customers can refer to this section to modify the LED code according to requirements.


Broadcom provides examples in the SDK package for most of the use cases.

FlexPort logic

The Trident3 supports splitting and un-splitting QSFP28 ports without restarting the system. How to do it?

The system configuration files have it either as 4×10/25G or one 40/100G.

A developer can follow the example flow in SDK/src/examples/xgs/chip_name/flexport.

Figure 3. Where to find FlexPort examples?


Latest news