MC-LAG
Table of Contents
MC-LAG (Link Aggregation, port-channel)
Example model & SONiC version:
MC-LAG (Link Aggregation, port-channel)
A multi-chassis link aggregation group (MLAG or MC-LAG) is a type of link aggregation group (LAG) with constituent ports that terminate on separate chassis, which provide redundancy in the event one of the chassis fails.
MC-LAG virtualizes two physical devices into one logical device.
Design considerations
- MCLAG domain consists of only two systems.
- Each system only joins one MC-LAG domain.
- Supports Known Unicast and BUM traffic.
- ARP reply and ND advertisement packet sync-up between MC-LAG peers
- FDB sync-up between MC-LAG peers
- Support pure L2 MC-LAG port and MC-LAG L3 routed port or MC-LAG L2 port joining L3 vlan interface
- VRRP, IGMP snooping, DHCP Relay, and STP features are not supported.
- Make sure isolation ACL is installed in the ASIC table before MCLAG-enabled portchannel is active
The MLAG control plane relies on the ICCPd service, which might not be active on a fresh installation to conserve resources.
These commands will enable the container (per a plethora of Internet articles):
admin@sonic:~$ sudo systemctl unmask iccpd
admin@sonic:~$ sudo systemctl start iccpd
admin@sonic:~$ sudo systemctl enable iccpdSONiC will occasionally remove the service back to its masked state, and it will end with a brief message in syslog:
INFO featured[8636]: Created symlink /etc/systemd/system/iccpd.service → /dev/null
INFO featured: Feature iccpd is stopped and disabledThe proper way to enable MC-LAG is by running a command:
admin@sonic:~$ sudo config feature state iccpd enabled
admin@sonic:~$ sudo config save -yAn alternative way is editing the config_db.json file, the “FEATURE” section.
"iccpd": {
"auto_restart": "enabled",
"delayed": "False",
"has_global_scope": "True",
"has_per_asic_scope": "False",
"high_mem_alert": "disabled",
"state": "disabled",
"support_syslog_rate_limit": "true"
},Example model & SONiC version:
- Aurora 830, Aurora 721, Aurora 621
- Netberg SONiC: sonic-broadcom-202311.n0
Syntax
config mclag
Usage: config mclag [OPTIONS] COMMAND [ARGS]...
Commands:
add Add MCLAG Domain
del Delete MCLAG Domain
keepalive-interval Configure MCLAG Keepalive timer value in secs
member
session-timeout Configure MCLAG Session timeout value in secs
unique-ip Configure Unique IP on MCLAG Vlan interfaceConfigure a Domain
config mclag { add | del } <domain_id> <source_ip_addr> <peer_ip_addr> <peer_ifname>Parameter Description
add: Add MCLAG domain.
del: Delete MCLAG domain.
domain-id: MCLAG domain ID.
source-ip-addr: MCLAG domain local IP.
peer-ip-addr: MCLAG domain peer IP.
peer-ifname: Interface of the backup link in the MCLAG domain in a Layer 2 scenario.Set a keepalive interval
config mclag keepalive-interval <domain-id> <time-in-secs>Parameter Description
domain-id: MCLAG domain ID.
time-in-secs: Time in seconds.Configure Members
config mclag member { add | del } <domain_id> <portchannel_names>Parameter Description
add: Add member MCLAG interfaces from MCLAG domain.
del: Delete member MCLAG interfaces from MCLAG domain.
domain-id: MCLAG domain ID.
portchannel-names: Downlink interface of the MCLAG domain.Configure the Session Timeout
config mclag session-timeout <domain_id> <time_in_secs>Parameter Description
domain-id: MCLAG domain ID.
time-in-secs: Time in seconds.Configure Unique IP on MCLAG Vlan interface
config mclag unique-ip { add | del } <interface_names>Parameter Description
add Add Unique IP on MCLAG Vlan interface
del Delete Unique IP from MCLAG Vlan interface
interface_names: MCLAG Vlan interface nameMC-LAG L2 configuration
This scenario runs a Layer 2 network design with VLANs and inter-VLAN routing.
- Create port-channels on all switches. Please refer to the LAG (Link-Aggregation) piece.
Both systems:
admin@nba621-1:~$ sudo config portchannel add PortChannel01
admin@nba621-1:~$ sudo config portchannel add PortChannel02
admin@nba621-1:~$ sudo config portchannel add PortChannel03
admin@nba621-1:~$ sudo config portchannel member add PortChannel01 Ethernet0
admin@nba621-1:~$ sudo config portchannel member add PortChannel02 Ethernet1
admin@nba621-1:~$ sudo config portchannel member add PortChannel03 Ethernet64
admin@nba621-1:~$ sudo config portchannel member add PortChannel03 Ethernet682. Create two VLANs: one for the MCLAG keepalive link and another for data forwarding. The MCLAG keepalive VLAN is Vlan 10, bound on PortChannel03.
Vlan 20 is for the Host traffic, bound to PortChannel01, PortChannel02, and PortChannel03. Please refer to the VLAN & Inter-VLAN Routing piece.
Both systems:
admin@nba621-1:~$ sudo config vlan add 10
admin@nba621-1:~$ sudo config vlan add 20
admin@nba621-1:~$ sudo config vlan member add 10 PortChannel03
admin@nba621-1:~$ sudo config vlan member add -u 20 PortChannel01
admin@nba621-1:~$ sudo config vlan member add 20 PortChannel02
admin@nba621-1:~$ sudo config vlan member add 20 PortChannel033. Create an MC-LAG domain and set a unique IP address on Vlan10.
nba621-1:
admin@nba621-1:~$ sudo config mclag add 1 192.168.7.1 192.168.7.2 PortChannel03
admin@nba621-1:~$ sudo config mclag unique-ip add Vlan10
admin@nba621-1:~$ sudo config mclag member add 1 PortChannel01
admin@nba621-1:~$ sudo config mclag member add 1 PortChannel02nba621-2:
admin@nba621-2:~$ sudo config mclag add 1 192.168.7.2 192.168.7.1 PortChannel03
admin@nba621-2:~$ sudo config mclag unique-ip add Vlan10
admin@nba621-2:~$ sudo config mclag member add 1 PortChannel01
admin@nba621-2:~$ sudo config mclag member add 1 PortChannel024. Add an IP address to Vlan10. Please refer to VLAN & Inter-VLAN Routing piece.
nba621-1:
admin@nba621-1:~$ sudo config interface ip add Vlan10 192.168.7.1/24 nba621-2:
admin@nba621-2:~$ sudo config interface ip add Vlan10 192.168.7.2/245. Check the MC-LAG status.
nba621-1:
admin@nba621-1:~$ mclagdctl dump state
The MCLAG's keepalive is: OK
MCLAG info sync is: completed
Domain id: 1
Local Ip: 192.168.7.1
Peer Ip: 192.168.7.2
Peer Link Interface: PortChannel03
Keepalive time: 1
sesssion Timeout : 15
Peer Link Mac: 10:be:99:00:b3:c5
Role: Active
MCLAG Interface: PortChannel02,PortChannel01
Loglevel: NOTICEnba621-2:
admin@nba621-2:~$ mclagdctl dump state
The MCLAG's keepalive is: OK
MCLAG info sync is: completed
Domain id: 1
Local Ip: 192.168.7.2
Peer Ip: 192.168.7.1
Peer Link Interface: PortChannel03
Keepalive time: 1
sesssion Timeout : 15
Peer Link Mac: 10:be:99:00:ab:55
Role: Standby
MCLAG Interface: PortChannel02,PortChannel01
Loglevel: NOTICE6. Check learned MACs:
admin@nba621-1:~$ show mac
No. Vlan MacAddress Port Type
----- ------ ----------------- ------------- -------
1 10 10:BE:99:00:AB:55 PortChannel03 Static
2 20 B4:96:91:B3:C7:CA PortChannel02 Static
3 20 B4:96:91:B3:C7:C8 PortChannel01 Dynamic
Total number of entries 3
admin@nba621-1:~$ mclagdctl dump mac -i 1
TYPE: S-STATIC, D-DYNAMIC; AGE: L-Local age, P-Peer age
No. TYPE MAC VID DEV ORIGIN-DEV AGE
1 S 10:be:99:00:ab:55 10 PortChannel03 PortChannel03 L
2 D b4:96:91:b3:c7:c8 20 PortChannel01 PortChannel01 P
3 D b4:96:91:b3:c7:ca 20 PortChannel02 PortChannel02 Ladmin@nba621-2:~$ show mac
No. Vlan MacAddress Port Type
----- ------ ----------------- ------------- -------
1 10 10:BE:99:00:B3:C5 PortChannel03 Static
2 20 B4:96:91:B3:C7:C8 PortChannel01 Static
3 20 B4:96:91:B3:C7:CA PortChannel02 Dynamic
Total number of entries 3
admin@nba621-2:~$ mclagdctl dump mac -i 1
TYPE: S-STATIC, D-DYNAMIC; AGE: L-Local age, P-Peer age
No. TYPE MAC VID DEV ORIGIN-DEV AGE
1 S 10:be:99:00:b3:c5 10 PortChannel03 PortChannel03 L
2 D b4:96:91:b3:c7:c8 20 PortChannel01 PortChannel01 L
3 D b4:96:91:b3:c7:ca 20 PortChannel02 PortChannel02 PDynamic – MAC entry learned locally. Static – MAC entry learned from a remote peer.
7. Verify the result:
user@server-1:~$ ping 192.168.10.2
PING 192.168.10.2 (192.168.10.2) 56(84) bytes of data.
64 bytes from 192.168.10.2: icmp_seq=1 ttl=64 time=0.198 ms
64 bytes from 192.168.10.2: icmp_seq=2 ttl=64 time=0.321 msPings persist even in a link failure event (or a cable removal).
MC-LAG L3 configuration
This scenario runs a Layer 3 network desing with IP peers and traffic routing.
1. Create port-channels on all switches. Please refer to the LAG (Link-Aggregation) piece.
Both systems:
admin@nba621-1:~$ sudo config portchannel add PortChannel01
admin@nba621-1:~$ sudo config portchannel add PortChannel02
admin@nba621-1:~$ sudo config portchannel add PortChannel03
admin@nba621-1:~$ sudo config portchannel member add PortChannel01 Ethernet0
admin@nba621-1:~$ sudo config portchannel member add PortChannel02 Ethernet1
admin@nba621-1:~$ sudo config portchannel member add PortChannel03 Ethernet64
admin@nba621-1:~$ sudo config portchannel member add PortChannel03 Ethernet682. Bind IP addresses to portchannel interfaces.
nba621-1:
admin@nba621-1:~$ sudo config interface ip add PortChannel01 192.168.9.1/24
admin@nba621-1:~$ sudo config interface ip add PortChannel02 192.168.10.1/24
admin@nba621-1:~$ sudo config interface ip add PortChannel03 192.168.7.1/24 nba621-2:
admin@nba621-2:~$ sudo config interface ip add PortChannel01 192.168.9.1/24
admin@nba621-2:~$ sudo config interface ip add PortChannel02 192.168.10.1/24
admin@nba621-2:~$ sudo config interface ip add PortChannel03 192.168.7.2/243. Create an MCLAG domain and bind members.
nba621-1:
admin@nba621-1:~$ sudo config mclag add 1 192.168.7.1 192.168.7.2 PortChannel03
admin@nba621-1:~$ sudo config mclag member add 1 PortChannel01
admin@nba621-1:~$ sudo config mclag member add 1 PortChannel02nba621-2:
admin@nba621-2:~$ sudo config mclag add 1 192.168.7.2 192.168.7.1
admin@nba621-2:~$ sudo config mclag member add 1 PortChannel01
admin@nba621-2:~$ sudo config mclag member add 1 PortChannel02 | The L3 case doesn’t need a peer link since all members are router interfaces. |
4. Check the MC-LAG status.
nba621-1:
admin@nba621-1:~$ mclagdctl dump state
The MCLAG's keepalive is: OK
MCLAG info sync is: completed
Domain id: 1
Local Ip: 192.168.7.1
Peer Ip: 192.168.7.2
Peer Link Interface: Unknown
Keepalive time: 1
sesssion Timeout : 15
Peer Link Mac: 00:00:00:00:00:00
Role: Active
MCLAG Interface: PortChannel02,PortChannel01
Loglevel: NOTICE
admin@nba621-1:~$ show interfaces portchannel
Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available,
S - selected, D - deselected, * - not synced
No. Team Dev Protocol Ports
----- ------------- ----------- ---------------------------
01 PortChannel01 LACP(A)(Up) Ethernet0(S)
02 PortChannel02 LACP(A)(Up) Ethernet1(S)
03 PortChannel03 LACP(A)(Up) Ethernet64(S) Ethernet68(S)nba621-2:
admin@nba621-2:~$ show interfaces portchannel
Flags: A - active, I - inactive, Up - up, Dw - Down, N/A - not available,
S - selected, D - deselected, * - not synced
No. Team Dev Protocol Ports
----- ------------- ----------- ---------------------------
01 PortChannel01 LACP(A)(Up) Ethernet0(S)
02 PortChannel02 LACP(A)(Up) Ethernet1(S)
03 PortChannel03 LACP(A)(Up) Ethernet68(S) Ethernet64(S)
admin@nba621-2:~$ mclagdctl dump state
The MCLAG's keepalive is: OK
MCLAG info sync is: completed
Domain id: 1
Local Ip: 192.168.7.2
Peer Ip: 192.168.7.1
Peer Link Interface: Unknown
Keepalive time: 1
sesssion Timeout : 15
Peer Link Mac: 00:00:00:00:00:00
Role: Standby
MCLAG Interface: PortChannel02,PortChannel01
Loglevel: NOTICE5. Check the results
user@server-1:~$ ping 192.168.10.2
PING 192.168.10.2 (192.168.10.2) 56(84) bytes of data.
64 bytes from 192.168.10.2: icmp_seq=1 ttl=63 time=0.095 ms
64 bytes from 192.168.10.2: icmp_seq=2 ttl=63 time=0.140 msPings persist even in a link failure event (or a cable removal).
Check the ARP cache:
admin@nba621-1:~$ mclagdctl dump arp -i 1
No. IP MAC DEV Flag
1 192.168.10.2 b4:96:91:b3:c7:ca PortChannel02 L
2 192.168.9.2 b4:96:91:b3:c7:c8 PortChannel01 Ladmin@nba621-2:~$ mclagdctl dump arp -i 1
No. IP MAC DEV Flag
1 192.168.10.2 b4:96:91:b3:c7:ca PortChannel02 R
2 192.168.9.2 b4:96:91:b3:c7:c8 PortChannel01 R