Broadcom Switch Chip

In this document, we will discuss integrating OpenBCM in Ubuntu 20.04 to enable the Broadcom switch chip basic function.

OpenBCM Broadcom Switch Software Development Kit contains the source code for Broadcom network switch APIs and drivers for programming Broadcom network switch silicon-based platforms.

High-level description

We will use Netberg’s Trident3-based switches as an example.

Aurora 621 has a Trident3-X5 (BCM56770) as the main data and 1G control planes.

Aurora 721 has a Trident3-X7 (BCM56870) as the main data and 1G control planes.

Broadcom Chip Connectivity

The diagram below shows how this system connects with the main chip BCM56770.

PCIe Connectivity

SDK application communicates with Broadcom chips through the PCIe interface. The figure below shows the PCIe connectivity. To know the bus/slot/function for each chip, NOS can get it via its root port. Below is a sample output from Ubuntu Linux.

root@ubuntu:~# lspci -s `setpci -s 00:0b.0 0x19.b`:00.0
04:00.0 Ethernet controller: Broadcom Limited Device b770 (rev 01)

Broadcom Software Package Integration

Pre-requisite

Install Ubuntu 20.04 and set it up following our guide: https://github.com/netbergtw/files/tree/master/debian-bsps

Please follow both Post-installation steps and model-specific steps.

Broadcom Software Packages

SDK – Broadcom network switching Software Development Kit.

We will show an integration example using OpenBCM SDK 6.5.27.

Integration Flow

SDK

This package builds the basic application to enable the Broadcom switch. All the other packages are linked to this one.

The Makefile we use is sdk-6.5.27/make/x86-smp_generic_64-2_6.

Reference MAKE_LOCAL files are located here – sdk-6.5.27/make/local/esw

sdk-6.5.27/make/Make.local.template contains comments on all available makefile options.

For example, we pick sdk-6.5.27/make/local/esw/Make.pkg.56770 and put it as sdk-6.5.27/make/Make.local. This approach saves space, and the build procedure will use this one if it exists.

Step-by-Step Procedure

Below is a simple guideline to build basic applications with all required Broadcom software packages.

1. Put all software packages into the folder called build_root (you can, based on your development environment, adjust properly).

user@ubuntu:~/build_root$ ls -al
-rwxr--r-- 1 user user 248642263 Dec 7 07:07 sdk-all-6.5.27.tar.gz

2. Untar all of them.

user@ubuntu:~/build_root$ tar zxf sdk-all-6.5.27.tar.gz 
user@ubuntu:~/build_root$ ll
drwxr-xr-x 12 user user 4096 Dec 28 10:04 sdk-all-6.5.27/
-rwxr--r-- 1 user user 248642263 Dec 7 07:07 sdk-all-6.5.27.tar.gz

3. Make SDK to create bcm.user Option: the user can specify to compile with a particular kernel version by setting ‘KERNDIR’. Our example uses kernel 5.15.91.

export KERNDIR=/home/user/kernel/linux-headers-4.15.0-123-generic

make -C sdk-all-6.5.27/systems/linux/user/x86-smp_generic_64-2_6/

When the SDK build is completed, the basic application will be generated.

user@ubuntu:~/build_root$ ls -al sdk-all-6.5.27/build/linux/user/x86-smp_generic_64-2_6/ 
total 391544 
drwxrwxr-x 2 user user 4096 Dec 29 12:37 . 
drwxrwxr-x 3 user user 4096 Dec 28 10:04 .. 
-rwxrwxr-x 1 user user 101205848 Dec 29 12:37 bcm.user 
-rwxrwxr-x 1 user user 298537184 Dec 29 12:37 bcm.user.dbg 
-rw-rw-r-- 1 user user 125192 Dec 29 12:36 linux-kernel-bde.ko 
-rw-rw-r-- 1 user user 34624 Dec 29 03:52 linux-user-bde.ko 
-rwxrwxr-x 1 user user 1009376 Dec 29 12:37 netserve

Configurations and Initialization Flow

Netberg provides configuration files, a led binary file, sample LED code, and patch files for reference.

The tree below shows the package content. We will use these and SDK reference configuration files in the following sections to enable the switch’s basic function.

Configuration files, led binary file, sample codes, and led patch:

NBA621-CFG
|-- config
|   |-- port_mode
|   |   |-- td3_pm_25g_100g.bcm
|   |   |-- td3_pm_25g_100g_break.bcm
|   |-- SI
|   |   |-- td3_si_qsfp_40g.bcm
|   |   |-- td3_si_qsfp_100g.bcm
|   |   |-- td3_si_sfpp_1g.bcm
|   |   |-- td3_si_sfpp_10g.bcm
|   |   |-- td3_si_sfpp_25g.bcm
|   |-- rc.soc
|   |-- td3_lanemap.bcm
|   |-- td3_sys.bcm
|-- led
|   |-- custom_led.bin
|-- patch
|   |-- sdk-6.5.27_ledproc.patch
|-- sample
|   |-- custom_led.c
|   |-- custom_led.h
|   |-- Make.local

Configurations Provided by SDK

Three types of files are required to enable the switch’s basic functionality.

1. Binary files built from Broadcom software packages
2. Configuration files to enable Broadcom software features
3. Board-specific configuration files from Netberg according to hardware design.

The first step is to create a folder ‘$DEST’ to store relevant files and to copy the binary files into this folder.

mkdir -p DEST
cp -f sdk-6.5.27/build/linux/user/x86-smp_generic_64-2_6/bcm.user $DEST
cp -f sdk-6.5.27/build/linux/user/x86-smp_generic_64-2_6/linux-kernel-bde.ko $DEST
cp -f sdk-6.5.27/build/linux/user/x86-smp_generic_64-2_6/linux-user-bde.ko $DEST

The second step is to prepare the common configuration file.

cp -f sdk-all-6.5.27/rc/cmicfw/linkscan_led_fw.bin $DEST

The third step is to apply board specific configuration as described below.

Platform Specific Configurations

Configuration Files

Broadcom chips need configuration files to work properly with a board. Netberg provides several customized configuration files for all usage scenarios.

In this section, we will describe these files and their purpose.

The files are listed according to the build version it can use. It is important to check the revision and use the correct file to ensure the system behavior can be as expected.

Here is an example on how to use these files:

cat NBA621-CFG/config/td3_sys.bcm > config.bcm
cat NBA621-CFG/config/td3_lanemap.bcm >> config.bcm

48 x 25G + 6 x 100G with Copper Cable Use Case

We recommend enabling link training when a front port connects with a copper cable. Link training provides a fast and convenient way to set the TX FIR automatically.

cat NBA621-CFG/config/port_mode/td3_pm_25g_100g.bcm >> config.bcm
cp NBA621-CFG/led/custom_led.bin custom_led.bin
cp NBA621-CFG/config/rc.soc rc.soc

Initialization flow

1. Insert kernel modules.

/bin/mknod /dev/linux-kernel-bde c 127 0 
/bin/mknod /dev/linux-user-bde c 126 0 
insmod linux-kernel-bde.ko debug=8 dmasize=32M 
insmod linux-user-bde.ko

2. Initialize the basic application.

#./bcm.user

Apply SI

48 x 25G + 6 x 100G mode: (25G logical port 1-28,33-52; 100G logical port 29-30,53-55,59)

BCM.0> port 1-28,33-52,29-30,53-55,59 en=0
BCM.0> phy control 1-28,33-52,29-30,53-55,59 lt=0
BCM.0> port 1-28,33-52 speed=25000 if=sr fd=1
BCM.0> port 29-30,53-55,59 speed=100000 if=sr4 fd=1
BCM.0> rcload NBA621-CFG/config/SI/td3_si_sfpp_25g.bcm
BCM.0> rcload NBA621-CFG/config/SI/td3_si_qsfp_100g.bcm
BCM.0> port 1-28,33-52,29-30,53-55,59 en=1

48 x 10G + 6 x 40G mode:

BCM.0> port 1-28,33-52,29-30,53-55,59 en=0
BCM.0> phy control 1-28,33-52,29-30,53-55,59 lt=0
BCM.0> port 1-28,33-52 speed=10000 if=sr fd=1
BCM.0> port 29-30,53-55,59 speed=40000 if=sr4 fd=1
BCM.0> phy control 1-28,33-52,29-30,53-55,59 lt=0
BCM.0> rcload td3_si_sfpp_10g.bcm
BCM.0> rcload td3_si_qsfp_40g.bcm
BCM.0> port 1-28,33-52,29-30,53-55,59 en=1

Management port

BCM56770 only supports one management port and must be assigned to logical port 64.

Besides, the management port of Aurora 621 only supports 1G speed due to the Atom C3508 limitation.

BCM.0> port 64 enable=0
BCM.0> port 64 speed=1000 fd=1
BCM.0> vlan remove 1 pbm=xe,ce,64
BCM.0> port 64 enable=1

LED reference code

Netberg provides an LED reference code to match common breakout and non-breakout port LED behavior.

The default LED behavior is as follows:

• The port’s current speed equals the port’s max speed, then green.
• If the port’s current speed is lower than the port’s max speed, then yellow. (include breakout speed)
• Any tx/rx traffic in that port is active, then blinking.

Customers can refer to this section to modify the LED code according to requirements.

Examples

Broadcom provides examples in the SDK package for most of the use cases.

FlexPort logic

The Trident3 supports splitting and un-splitting QSFP28 ports without restarting the system. How to do it?

The system configuration files have it either as 4×10/25G or one 40/100G.

A developer can follow the example flow in SDK/src/examples/xgs/chip_name/flexport.