ICOS and Linux shell management

DC switch NOS evolution

For many years, switches were black boxes with management through CLI/web or special API from vendors.

Cumulus Linux was the first product that aimed to change the industry and manage a switch like a Linux server with standard Linux management, monitoring, and provisioning tools. However, Cumulus Linux is aimed at the particular market with heavy L3 usage and offers a limited set of network protocols.

What’s going on with traditional CLI-oriented network OS?

They are going to be open too. Let’s take a look at ICOS, a feature-rich package from Broadcom that serves as a foundation for NOS on their ASICs.

Unlike servers, Ethernet switch does not allow simple Linux deployment with an ASIC drivers. Special integration efforts are required for a proper LED indication management, port mapping, power control, etc. Such integration could be implemented in many ways, let’s take a look at them.

Traditional one.

Common way was an embedded OS with boot right to the CLI interface, as it’s done on Aurora 220 (48 x 1G Base-T + 8 x 10G SFP+) box:

 FASTPATH Startup -- Main Menu
1 - Start FASTPATH Application
 2 - Display Utility Menu
 Select (1, 2):
 BOOT_PART = /dev/mtd8
 Copying /dev/mtd8 to RAM...done.
 Extracting FASTPATH .....done
 Loading FASTPATH ...../mnt/application
 done
 Loading hardware monitor
 Uncompressing apps.lzma
 SyncDB Running...
 DMA pool size: 8388608
 AXI unit 0: Dev 0xb344, Rev 0x01, Chip BCM56344_A0, Driver BCM56340_A0
 SOC unit 0 attached to PCI device BCM56344_A0
<10> Jun      3 10:24:51 0.0.0.0 General[fp_main_task]: bootos.c(199) 3 %%   Event(0xaaaaaaaa)   started!
(Unit 1)>
Applying Global configuration, please wait ...
Applying Interface configuration, please wait ...
User:admin
 Password:
 (Aurora 220) >

One can invoke a Linux shell by issuing a command linuxsh, but possibilities are limited. For example, lspci output is quite laconic:

# lspci
 #

Or ifconfig:

# ifconfig
 eth0 Link encap:Ethernet HWaddr 70:B3:D5:CC:F0:2C
 inet addr:192.168.0.62 Bcast:192.168.0.255 Mask:255.255.255.0
 inet6 addr: fe80::205:64ff:fe30:7222/64 Scope:Link
 UP BROADCAST RUNNING ALLMULTI MULTICAST MTU:1500 Metric:1
 RX packets:153 errors:0 dropped:0 overruns:0 frame:0
 TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:18963 (18.5 KiB) TX bytes:1760 (1.7 KiB)
 Interrupt:234 Base address:0x2000
    lo Link encap:Local Loopback
 inet addr:127.0.0.1 Mask:255.0.0.0
 inet6 addr: ::1/128 Scope:Host
 UP LOOPBACK RUNNING MTU:16436 Metric:1
 RX packets:124 errors:0 dropped:0 overruns:0 frame:0
 TX packets:124 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:0
 RX bytes:6723 (6.5 KiB) TX bytes:6723 (6.5 KiB)
 rt_vrf_0_drv Link encap:Ethernet HWaddr AA:23:48:A8:6F:89
 UP BROADCAST RUNNING MULTICAST MTU:12288 Metric:1
 RX packets:0 errors:0 dropped:0 overruns:0 frame:0
 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
    rt_vrf_0_stk Link encap:Ethernet HWaddr 32:71:C7:DC:11:A9
 UP BROADCAST RUNNING MULTICAST MTU:12288 Metric:1
 RX packets:0 errors:0 dropped:0 overruns:0 frame:0
 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

 

On embedded platforms, network administrators have limited ability to add and run their own software and are limited to using the supported OS tools for managing switch configuration and software.

Modern approach

x86 based switches are game changers.

Standard GRUB boot loader allows choosing among installed NOS (if detected) and ONIE environment.

GNU GRUB  version 2.02~beta2+e4a1fe391
+----------------------------------------------------------------------------+
 |*ICOS                                                                                            |
 | ONIE                                                                                             |
 |                                                                                                       |
 |                                                                                                       |
 |                                                                                                       |
 |                                                                                                       |
 |                                                                                                       |
 |                                                                                                       |
 |                                                                                                       |
 |                                                                                                       |
 |                                                                                                       |
 |                                                                                                       |
 +----------------------------------------------------------------------------+

After NOS chose:

Booting `ICOS'
 Loading ICOS ...
Loading ICOS initial ramdisk ...
Loading, please wait...
 Begin: Loading essential drivers ... done.
 Begin: Running /scripts/init-premount ... done.
 Begin: Mounting root file system ...
 Begin: Running /scripts/local-top ... done.
 Begin: Running /scripts/local-premount ... done.
 Begin: Running /scripts/local-bottom ... done.
 ........
 * Starting NTP server ntpd                                              [ OK ]
 * Stopping System V runlevel compatibility                              [ OK ]
Ubuntu 14.04 LTS nba720 ttyS1
nba720 login: admin
 Password:
 Last login: Mon Jun  6 07:33:30 UTC 2016 on ttyS1
 Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.16.0-29-generic x86_64)
 * Documentation:  https://help.ubuntu.com/
 admin@nba720:~$

 

A user can find himself right at the Linux shell (Ubuntu 14.04 LTS in our case). It’s a fully-functional Linux without any limitations and is binary compatible with the pre-built packages that are available in the online Ubuntu repositories. These packages can be installed on the x86 switch using the apt-get utility. Even C/C++ toolchain is available for native development!

ICOS runs in NOS-as-a-Service mode. The switching function is active immediately after boot.

root@nba720:~# lspci
 00:00.0 Host bridge: Intel Corporation Atom processor C2000 SoC Transaction Router (rev 02)
 00:01.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 1 (rev 02)
 00:02.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 2 (rev 02)
 00:03.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 3 (rev 02)
 00:04.0 PCI bridge: Intel Corporation Atom processor C2000 PCIe Root Port 4 (rev 02)
 00:0b.0 Co-processor: Intel Corporation Atom processor C2000 nCPM (rev 02)
 00:0e.0 Host bridge: Intel Corporation Atom processor C2000 RAS (rev 02)
 00:0f.0 IOMMU: Intel Corporation Atom processor C2000 RCEC (rev 02)
 00:13.0 System peripheral: Intel Corporation Atom processor C2000 SMBus 2.0 (rev 02)
 00:14.0 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
 00:14.1 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
 00:14.2 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
 00:14.3 Ethernet controller: Intel Corporation Ethernet Connection I354 (rev 03)
 00:16.0 USB controller: Intel Corporation Atom processor C2000 USB Enhanced Host Controller (rev 02)
 00:17.0 SATA controller: Intel Corporation Atom processor C2000 AHCI SATA2 Controller (rev 02)
 00:18.0 SATA controller: Intel Corporation Atom processor C2000 AHCI SATA3 Controller (rev 02)
 00:1f.0 ISA bridge: Intel Corporation Atom processor C2000 PCU (rev 02)
 00:1f.3 SMBus: Intel Corporation Atom processor C2000 PCU SMBus (rev 02)
 01:00.0 Ethernet controller: Broadcom Corporation Device b960 (rev 11)
 01:00.1 Ethernet controller: Broadcom Corporation Device b960 (rev 11)

 

ifconfig:

root@nba720:~# ifconfig
 bond1     Link encap:Ethernet  HWaddr 70:B3:D5:CC:F0:7e
 UP BROADCAST MASTER MULTICAST  MTU:1518  Metric:1
 RX packets:0 errors:0 dropped:0 overruns:0 frame:0
 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:0
 RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
 ...
 bond64    Link encap:Ethernet  HWaddr ca:0a:c6:54:d4:86
 UP BROADCAST MASTER MULTICAST  MTU:1518  Metric:1
 RX packets:0 errors:0 dropped:0 overruns:0 frame:0
 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:0
 RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
eth0      Link encap:Ethernet  HWaddr 70:B3:D5:CC:F0:7c
 inet addr:192.168.0.74  Bcast:192.168.0.255  Mask:255.255.255.0
 inet6 addr: fe80::205:64ff:fe2f:3c7c/64 Scope:Link
 UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
 RX packets:86 errors:0 dropped:0 overruns:0 frame:0
 TX packets:56 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:9635 (9.6 KB)  TX bytes:4922 (4.9 KB)
fpti1_0_2 Link encap:Ethernet  HWaddr 70:B3:D5:CC:F0:7e
 UP BROADCAST MULTICAST  MTU:1518  Metric:1
 RX packets:0 errors:0 dropped:0 overruns:0 frame:0
 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:500
 RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
 ...
 fpti1_0_36 Link encap:Ethernet  HWaddr 70:B3:D5:CC:F0:7e
 UP BROADCAST SLAVE MULTICAST  MTU:1518  Metric:1
 RX packets:0 errors:0 dropped:0 overruns:0 frame:0
 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:500
 RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
lo        Link encap:Local Loopback
 inet addr:127.0.0.1  Mask:255.0.0.0
 inet6 addr: ::1/128 Scope:Host
 UP LOOPBACK RUNNING  MTU:65536  Metric:1
 RX packets:240 errors:0 dropped:0 overruns:0 frame:0
 TX packets:240 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:0
 RX bytes:19104 (19.1 KB)  TX bytes:19104 (19.1 KB)
rfc5549if Link encap:Ethernet  HWaddr a6:2e:4d:c2:f8:5d
 inet addr:250.0.0.1  Bcast:250.0.0.3  Mask:255.255.255.252
 UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
 RX packets:0 errors:0 dropped:0 overruns:0 frame:0
 TX packets:1 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:500
 RX bytes:0 (0.0 B)  TX bytes:42 (42.0 B)
rt_vrf_0_drv Link encap:Ethernet  HWaddr 3e:f7:83:10:a8:a1
 UP BROADCAST RUNNING MULTICAST  MTU:9416  Metric:1
 RX packets:0 errors:0 dropped:0 overruns:0 frame:0
 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
rt_vrf_0_stk Link encap:Ethernet  HWaddr 06:8e:c0:35:1a:98
 UP BROADCAST RUNNING MULTICAST  MTU:9416  Metric:1
 RX packets:0 errors:0 dropped:0 overruns:0 frame:0
 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

The Linux device interfaces corresponding to physical ports are named fpti1_0_<n>. The <n> is the front panel port number. For example, the ICOS port 0/36 corresponds to the Linux interface fpti1_0_36. In our case, enumeration starts with 2 and ends with 36 because the first two ports are in break-out mode. bond<n> – LAG interface. rt1_0_<n> – routing interface.

What does it mean?

That means that you can manage the switch ports using standard Linux tools!

ethtool, ifconfig, route, ip now manage the switch like they did with servers before. You can install Quagga and manage routing features; a switch configuration data are kept in text format in the / mnt/fastpath in a startup-config file.

Cumulus Networks pioneered using standard Linux tools for the switch management until them traditional CLI and custom API approach was the only possible way to talk to the network. The industry does not stand still, and ICOS on x86 platform offers new capabilities.

 

Feature Overview

x86 Platform

Embedded Platform

Linux Distribution

Ubuntu 14.04 LTS with special packages.

Custom embedded Linux distribution.

Open Source Linux Packages

Binary-compatible packages can be downloaded from the Ubuntu mirror sites. Vendor provides a Debian package file for the customized kernel.

Not Supported.

Zero-Touch Provisioning

Loads a script specified via DHCP option

239. Executes commands in the script to install additional packages and provision the switch.

Able to automatically load a configuration file and upgrade the OS image.

ONIE Code Installation

ONIE is required for installing the initial image.

ONIE is optional.

Dual ICOS Images.

Not supported. The switch has one file system with ICOS and one file system with the ONIE image. If the ICOS file system is corrupted, the recovery must be done with ONIE.

Dual ICOS images are supported. Automatic switch-over between images is supported if one image is corrupted.

Code and Configuration Management with Puppet.

Puppet is preinstalled in the ONIE ICOS image. The user is responsible for configuring Puppet and for creating the Puppet manifests.

Supported on some embedded platforms. The embedded switches support loading Puppet and other Linux packages during the Zero-Touch Provisioning procedure. This feature should be used only in conjunction with ONIE.

File Management

Loading files, such as configuration, scripts, and code images is done using standard Linux utilities from the Linux shell or from tools such as Puppet. Any file transfer mode supported by Linux can be used to download files. The ICOS copy command is not supported.

Files are managed using the copy command from the ICOS command prompt. The copy command supports TFTP, FTP, SCP, and SFTP.

Configuration File Compression

Configuration files and scripts are stored uncompressed in the file system. This enables the user to view and edit the files.

To conserve NVRAM, the configuration and script files are compressed. The copy command must be used to offload the uncompressed version of the file to an external file server.

User Login Management

The recommended approach is to use native Linux user authentication methods, and then use commands (ICOS-console or ICOS-cli) to manage ICOS from the Linux shell. ICOS authentication methods are also available and can be optionally enabled.

User authentication is controlled by ICOS. ICOS supports statically configured user IDs and passwords, RADIUS, and TACACS+.

Domain Name Server (DNS) Client

The x86 platforms use the Linux DNS client. Administrators must configure the Linux DNS client outside of ICOS. ICOS for x86 does not include the internal DNS client, but instead uses the libresolv library to resolve the DNS names.

The DNS client is embedded in ICOS and is configured only using the ICOS CLI commands.

Time of Day

The x86 platforms rely on the Linux NTP to manage the time zone and the time  of day. The NTP is configured outside of ICOS. ICOS for x86 does not include the internal SNTP client.

ICOS uses an internal SNTP client and provides CLI commands for managing the SNTP client configuration and status.

Syslog

The x86 platforms send syslog events to the Linux System Logger. The recommended approach is to configure the Linux syslog client outside of ICOS. This allows all software applications running on the switch to generate the syslog events from the same client. The ICOS syslog client is also available and may be used instead of the Linux syslog client.

The ICOS supports only the internal syslog client. The Linux System Logger is not supported.

Ethernet Service Port

The x86 platforms provide read-only access within ICOS to view the service port IP address. The Ethernet service port IP address assignment is controlled by Linux.

ICOS provides CLI commands to enable DHCP or statically assign the IP address to the Ethernet service port.

Host name assignment

On the x86 platforms ICOS reads the host name from Linux and uses the name in the CLI prompt. The ICOS CLI command to set the host name is supported, but it only sets the name in Linux using sethostname() and does not affect the ICOS configuration file.

ICOS provides a CLI command to set the host name. The host name is saved in the ICOS configuration file.

Switch Database Management (SDM) Template – Used for configuring routing scaling factors

On x86 platforms, the SDM template can be set via an ICOS command and by writing the template number into the file /mnt/fastpath/sdm-templatetext.cfg. The file mechanism enables the administrator to avoid an extra ICOS restart by setting the template in the ZTP script before ICOS starts.

The SDM template is set via the ICOS CLI commands and the switch is rebooted for the new template to take effect.

Interface Management via Linux Interfaces

The x86 platforms support the ability to monitor and manage physical ports, LAGs, and Routing interfaces via Linux tools such as ethtool and iproute2 collection. The same operations are available through the ICOS user interface as well.

Not supported. ICOS physical ports, LAGs, and routing interfaces are managed through the ICOS user interface.

Packet transmit/ receive via Linux interfaces

The x86 platforms enable applications to send and receive packets on physical ports and routing interfaces via Linux interfaces. This feature enables tools like Wireshark or networks protocols like LLDPD to run on the switch.

Not Supported.

Route Table Management via Linux Commands

The x86 platforms enable third-party applications to add IPv4 and IPv6 routes using standard utilities, such as iproute2 or directly via the NETLINK socket. This feature enables third-party routing protocols, such as Quagga, to run on the switch. Not all routing features are supported via this mechanism. For example, the VRF feature is not supported.

Not Supported.

 

Now, with the new ICOS capabilities, two worlds have merged and available in a single OS.

See more info in our x86 platform guide.

 

NEWS

Latest news