Home › Resources › Field Guides › ISL & Fabric Expansion
Field Guide · Storage OperationsAdding a Switch to a SAN Fabric: ISL & Trunking Runbook for Cisco MDS
Joining a new switch to a production fabric is one command sequence and one cable pair — and one badly-timed zone merge away from an isolated ISL. The switches negotiate everything themselves the moment the E-ports come up: principal switch, domain IDs, and a zone database merge whose rules are unforgiving of conflict. This runbook is the order that makes the negotiation boring: verify the fabric, prepare a clean new switch, bundle the ISLs into a port-channel, let the merge happen once, and prove it worked with device counts instead of optimism.
!!Command safety model
zone default-zone permit: a legacy shortcut that lets every unzoned device talk to every other — see section 05 for why it stays off. Everything else here is reversible configuration on ports that carry no traffic yet.
| Badge | Meaning |
|---|---|
| Read-only | Inspects state. Safe in production, any time. |
| Disruptive | Changes fabric or port state. Correct inside the change window; wrong outside it. |
| Destructive | Clears zone databases or overwrites configuration. Only in the troubleshooting paths, only deliberately. |
ENVReference fabric
Fabric B of the Boston SAN is outgrowing its single switch. Today, bos-mds-b (MDS 9148V, VSAN 20, domain 0x14) carries every B-side path between the ESXi hosts and the NetApp AFF pair. A new switch — bos-mds-b-new, freshly racked, factory-clean — joins the fabric over a two-link ISL bundled into port-channel 1: ports fc1/31 and fc1/32 on both switches — deliberately the same ports at both ends, which keeps the cabling documentation, the verification commands, and the on-call engineer’s muscle memory identical on either side of the link. Fabric A is untouched throughout — which is precisely why dual-fabric designs let you do this during business hours.
| Switch | Role | State | What is attached |
|---|---|---|---|
bos-mds-b | EXISTING — production | Live fabric · principal · domain 0x14 | Every host and storage port in Fabric B |
bos-mds-b-new | NEW — being added | Factory-clean · no zoneset · no domain yet | Nothing — devices migrate in a later change |
bos-mds-b; green = the NEW switch bos-mds-b-new; grey = run on both. Before typing anything, read your prompt — the hostname is in it — and match it to the tag. The destructive recovery step in section 07 exists precisely because someone once ran it on the wrong one.
bos-mds-b-new carries no devices and no zoneset, the fabric-wide negotiations — principal election, domain assignment, zone merge — have nothing to conflict with. Keeping the new switch empty until after the join succeeds is the single most important decision in this runbook.01What actually happens when fabrics join
The moment two E-ports see each other, three negotiations run without asking you. Principal switch election: every fabric has one principal that assigns domain IDs; when fabrics join, priorities and WWNs decide who keeps the job. Domain ID assignment: every switch needs a unique domain — a collision between two statically configured domains isolates the link, which is why section 02 checks before section 04 cables. Zone merge: each VSAN’s active zone databases are compared. The rules are strict: identical databases merge; one-side-empty adopts the other side’s (this is why the clean-switch strategy works); conflicting databases — same zone name, different members — isolate the VSAN on that ISL rather than guess.
02Pre-flight: verify the fabric, prepare the new switch
Compatibility and backups Read-only
Confirm both switches run compatible NX-OS releases (show version both sides; same major train is the comfortable answer — see the upgrade runbook if the new switch needs bringing forward first). Then back up the production switch — locally and off-box, named with hostname and date:
copy running-config startup-config
copy running-config tftp://10.10.20.25/bos-mds-b-20260724Who is principal, and are domains safe? Read-only
Know the fabric’s principal switch and every domain ID in play before the join. On a single-switch fabric the answer is trivial — that switch is principal — but the habit matters, and the domain list is what you check the new switch against.
show fcdomain vsan 20The local switch is the Principal Switch.
Local switch run time information:
State: Stable
Local switch WWN: 20:14:8c:94:1f:aa:bb:01
Running fabric name: 20:14:8c:94:1f:aa:bb:01
Running priority: 2
Current domain ID: 0x14(20)
Local switch configuration information:
State: Enabled
Auto-reconfiguration: Disabled
Contiguous-allocation: Disabled
Configured fabric name: 20:01:00:05:30:00:28:df
Configured priority: 128
Configured domain ID: 0x14(20) (preferred)Zoning mode must match Read-only
Basic and enhanced zoning do not mix across a fabric. Check the mode on the production switch; the new switch must run the same mode before the join (a clean switch defaults to basic — flip it to enhanced first if your fabric runs enhanced, as the reference fabric does).
show zone status vsan 20VSAN: 20 default-zone: deny distribute: full Interop: default
mode: enhanced merge-control: allow
session: none
hard-zoning: enabled broadcast: unsupported
smart-zoning: disabled
Default zone:
qos: none broadcast: unsupported ronly: unsupportedSet the mode on the new switch to match Disruptive
A clean switch defaults to basic. The reference fabric runs enhanced, so the new switch flips before the join — done now, while the switch is alone, the change is local, instant, and conflict-free:
configure terminal
zone mode enhanced vsan 20
exitconfigure terminal
no zone mode enhanced vsan 20
exitshow zone status vsan 20VSAN: 20 default-zone: deny distribute: full Interop: default
mode: enhanced merge-control: allowProve the new switch is clean Read-only
The check that prevents every merge horror story. On bos-mds-b-new, before any ISL configuration:
show zoneset active
show fcdomain domain-list
show vsanZoneset not present
--
VSAN 1:
Number of domains: 1
--
vsan 1 information
name:VSAN0001 state:activewrite erase and reload, or at minimum clear every zone database and VSAN it used to carry. A switch that remembers its old fabric is the classic source of the same-name-different-members conflict.
Create the VSAN on the new switch Disruptive
configure terminal
vsan database
vsan 20
exit
exit03Configure the ISL ports — trunking E-ports, port-channeled
Two physical ISLs, one logical link. Bundling them into a SAN port-channel is the recommended shape: aggregate bandwidth, and a single-link failure costs capacity instead of causing an FSPF topology change. Configure the same port-channel structure on both switches — mode active lets the channel protocol negotiate cleanly.
configure terminal
interface port-channel 1
switchport mode E
switchport trunk mode on
switchport trunk allowed vsan 20
switchport rate-mode dedicated
exit
interface fc1/31-32
switchport mode E
switchport trunk mode on
switchport trunk allowed vsan 20
channel-group 1 force
no shutdown
exit
exitconfigure terminal
interface port-channel 1
switchport mode E
switchport trunk mode on
switchport trunk allowed vsan 20
switchport rate-mode dedicated
exit
interface fc1/31-32
switchport mode E
switchport trunk mode on
switchport trunk allowed vsan 20
channel-group 1 force
no shutdown
exit
exitmode E makes the port an inter-switch link; trunk mode on upgrades it to TE — a trunking E-port able to carry multiple VSANs, each tagged; trunk allowed vsan 20 restricts the trunk to exactly the VSANs you intend (allowed lists are your blast-radius control — never leave a production trunk allowing all); channel-group 1 force binds the member port to the channel, adopting the channel’s settings. Preferring individual ISLs without a channel? Drop the port-channel blocks and channel-group lines — everything else is identical, and FSPF will load-balance the two parallel links per exchange.
vsan 20 interface fc1/31. For a trunking E-port this is unnecessary — the trunk-allowed list governs which VSANs the TE link carries; port VSAN membership decides where an N-port device logging in on that port would land. Set the allowed list, skip the membership edit, keep the mental model clean.
04Cable up and verify the link Disruptive
Connect both cables — fc1/31→fc1/31, fc1/32→fc1/32, like-for-like at both ends — and watch the negotiation land. Three reads, in order:
show interface fc1/31-32 brief-------------------------------------------------------------------------------
Interface Vsan Admin Admin Status SFP Oper Oper Port
Mode Trunk Mode Speed Channel
Mode (Gbps)
-------------------------------------------------------------------------------
fc1/31 1 E on trunking swl TE 32 port-channel1
fc1/32 1 E on trunking swl TE 32 port-channel1-------------------------------------------------------------------------------
Interface Vsan Admin Admin Status SFP Oper Oper Port
Mode Trunk Mode Speed Channel
Mode (Gbps)
-------------------------------------------------------------------------------
fc1/31 1 E on trunking swl TE 32 port-channel1
fc1/32 1 E on trunking swl TE 32 port-channel1The line to read on both switches: Status trunking, Oper Mode TE, full Oper Speed, and the Port Channel column binding each member to port-channel1. Anything else — notConnected (cabling/optic), errDisabled (config rejection), or a blank channel column (channel-group missing) — stops the runbook here.
show port-channel summary
show interface port-channel 1 trunk vsanInterface Total Ports Oper Ports First Oper Port
-------------------------------------------------------------------------------
port-channel 1 2 2 fc1/31
--
port-channel1 is trunking
Vsan 20 is up (None)show topology vsan 20
show fcdomain domain-list vsan 20FC Topology for VSAN 20 :
--------------------------------------------------------------------------------
Interface Peer Domain Peer Interface Peer IP Address(Switch Name)
--------------------------------------------------------------------------------
port-channel 1 0x15(21) port-channel 1 10.10.20.13(bos-mds-b-new)
--
Number of domains: 2
Domain ID WWN
--------- -----------------------
0x14(20) 20:14:8c:94:1f:aa:bb:01 [Local] [Principal]
0x15(21) 20:14:8c:94:1f:aa:cc:01What you just confirmed: both members trunking as TE at full speed inside the channel, VSAN 20 up on the trunk (not isolated — the word to fear is isolated, see section 07), the new switch visible in the topology, and two clean domains with the original principal still principal.
05The zone merge — verification and the default-zone rule
Because the new switch joined empty, the merge is an adoption: the fabric’s active zoneset propagates to bos-mds-b-new on its own. Your job is verification, not configuration:
show zone status vsan 20
show zoneset active vsan 20VSAN: 20 default-zone: deny distribute: full Interop: default
mode: enhanced merge-control: allow
--
zoneset name FabricB-zs vsan 20
zone name z_esx01_hba1_aff01a vsan 20
* fcid 0x140100 [pwwn 20:00:00:25:b5:0b:00:1f]
* fcid 0x140200 [pwwn 50:0a:09:82:8c:73:00:04]
zone name z_esx02_hba1_aff01b vsan 20
...default-zone on deny. Procedures in circulation add zone default-zone permit vsan 20 at this step “so devices can see each other.” Understand what that does: every device not in a zone gains access to every other unzoned device — the SAN equivalent of turning off the firewall because a rule was missing. If something cannot see its storage after a join, the fix is a correct zone, never an open default zone. The one legitimate use — brief, deliberate, documented — is legacy interop migration, and even then it ends with deny restored. In enhanced mode, remember changes are session-based: zone commit vsan 20 publishes; nothing needs committing here if the adoption did its job.
06Post-implementation checks
The device-count equality test Read-only
The name server is the fabric’s truth: every logged-in device, fabric-wide. Run it on both switches — the counts must match, because a merged fabric has one name server view. A mismatch means the merge is not as complete as it looks.
show fcns database vsan 20VSAN 20:
------------------------------------------------------------------
FCID TYPE PWWN VENDOR FC4-TYPE:FEATURE
------------------------------------------------------------------
0x140100 N 20:00:00:25:b5:0b:00:1f scsi-fcp:init
0x140200 N 50:0a:09:82:8c:73:00:04 NetApp scsi-fcp:target
...
Total number of entries = 14Link health and the log Read-only
show interface port-channel 1
show cfs status
show logging log | include "ISL|zone|fcdomain"Read the port-channel counters for errors (CRC, discards — a dirty optic shows up here first), confirm CFS distribution is enabled (the fabric’s config-sync transport), and scan the log for anything the join left behind. Quiet is correct.
Zone-file diff, the belt-and-braces way Read-only
For the change record: capture show zoneset active vsan 20 from both switches to files and diff them — identical output is the evidence the auditors and your future self want. On a Windows admin workstation, PowerShell does it in one line:
Compare-Object (Get-Content "C:\temp\bos-mds-b-zones.txt") (Get-Content "C:\temp\bos-mds-b-new-zones.txt")(no output = the two zone captures are identical, which is the pass condition)Save both switches Disruptive
copy running-config startup-config07Troubleshooting a failed merge
The symptom is almost always the same: the ISL comes up, but the VSAN on it reads isolated. Diagnose before touching anything:
show interface port-channel 1 trunk vsan
show zone status vsan 20
show port internal info interface fc1/31 | include isolationport-channel1 is trunking
Vsan 20 is isolated (isolation reason: zone merge failure)- Zone merge failure — the joining switch brought a conflicting database after all. The clean fix, when the new switch’s zoneset is expendable (it should be): clear its zone database for the VSAN —
clear zone database vsan 20Destructive, and it runs onbos-mds-b-newONLY: read the prompt, confirm it saysbos-mds-b-new, and only then type — onbos-mds-bthis same command erases the production zone database — then bounce the port-channel (shutdown/no shutdown) and let the adoption rerun. When both sides carry real zonesets, merge them deliberately instead: export, reconcile names and members, import — never by permitting the default zone. - Domain ID overlap — two statically-configured switches claimed the same domain. Reassign one (
fcdomain domain <n> preferred vsan 20) and restart the negotiation. Prevention was the section 02 domain-list check. - Zoning-mode mismatch — one side basic, one enhanced. Align the modes (enhanced wins in a modern fabric), then bounce the link.
- VSAN not allowed — the trunk-allowed lists disagree.
show interface port-channel 1 trunk vsanon both sides; make the allowed lists identical.
RBRunbook · condensed sequence
- Verify: NX-OS compatible both sides · production config backed up locally + TFTP ·
show fcdomain vsan 20(principal + domains) · zone mode matches · new switch provably clean (show zoneset active→ not present). - Prepare new switch: create VSAN 20 · set zoning mode to match the fabric.
- Configure both sides: port-channel 1 as trunking E, allowed VSAN 20 · member ports with
channel-group 1 force·no shutdown. - Cable both links · verify TE trunking, channel 2/2 up, VSAN 20 up not isolated · topology shows the peer · two unique domains.
- Verify the adoption: active zoneset present on the new switch ·
default-zone denyconfirmed · CFS enabled. - Prove it:
show fcns database vsan 20totals equal on both switches · zone capture diff empty · logs quiet. - Save both configs. Device migration onto the new switch is its own change, another day.
Fabric running out of ports — or out of people who remember how it was built?
WUC engineers plan and execute SAN expansions as change-controlled engagements: fabric assessment, domain and zoning review, ISL and port-channel design, join execution with device-count evidence, and the migration plan for what comes after.
Talk to engineering →FAQFrequently asked questions
Q01Is joining a new switch disruptive to the devices already on the fabric?
Not when the pre-checks pass: existing devices keep their sessions through a clean join — the fabric gains a domain and a bigger name server, nothing more. The disruptive scenarios are precisely the failure modes in section 07, which is why the boring pre-flight is most of this guide.
Q02Port-channel or two individual ISLs?
Port-channel, almost always: one logical link, aggregate bandwidth, and a member failure is invisible to FSPF. Individual ISLs still work — FSPF load-balances equal-cost paths per exchange — and remain the fallback when the platforms disagree about channel protocols. What you never want is one lonely ISL as a single point of fabric partition.
Q03E-port versus TE-port — which am I building?
You configure mode E; trunk mode on is what upgrades the negotiated result to TE (a trunking E-port carrying multiple tagged VSANs). On modern MDS-to-MDS links trunking negotiates on by default — setting it explicitly, with an explicit allowed list, just removes the surprises.
Q04Do I need to set a domain ID on the new switch?
For the join itself, no — the principal assigns one, and an empty switch accepts it. Many shops then configure the assigned ID as static afterwards, because some legacy devices bind FCIDs into their configuration and a future domain change would disturb them. Do whichever your fabric standard says; just never let two statics collide.
Q05The doc I inherited says to permit the default zone. Should I?
No — see the warning in section 05. It “fixes” visibility by disabling the SAN’s access control for every unzoned device. If a device cannot see its target after a join, a zone is missing or wrong; write the zone. The permit shortcut converts a zoning gap into a fabric-wide exposure that outlives the incident.
Q06When do the hosts and storage move to the new switch?
In a separate change, after the joined fabric has soaked. Device migration is port-by-port, multipath-aware work — one path at a time, verify at the host, then the partner path — and belongs in the zoning field guide’s territory. This runbook deliberately ends with a bigger fabric and zero moved devices.
Q07Does adding a switch need extra licensing?
The E/TE trunking and port-channels used here are base NX-OS features on current MDS platforms. Port activation licensing (on-demand ports) is the one to check — the new switch needs enough activated ports for the ISLs plus the devices that will eventually move to it.
Q08Same procedure for Fabric A?
Identical — and never in the same change window. Grow B, soak, then grow A: the same one-fabric-at-a-time discipline as every other fabric-wide change in this series.
RFReferences
- Cisco MDS 9000 NX-OS — Interfaces and Fabric configuration guides (per release)
- Cisco MDS 9000 NX-OS — Fabric Configuration Guide: zone merge behaviour and CFS
- Cisco MDS 9000 Series — product documentation
→From the same practice
- Cisco MDS Zoning: A Field Guide for NetApp AFF Dual-Fabric Setups
- Cisco MDS NX-OS Upgrade Runbook
- Cisco MDS NX-OS Command Reference
- Post-OEM network maintenance
Planning a fabric expansion on a SAN nobody fully documented?
Most fabrics we assess grew organically — domains nobody chose, zones nobody owns, and one ISL doing the work of four. WUC engineers deliver fabric expansions as evidence-driven changes: current-state assessment, ISL and port-channel design, the join executed against this runbook, and the device-migration plan that follows.
- Fabric topology and domain assessment before you grow
- ISL / port-channel design and change-window execution
- Zone hygiene review — including retiring any default-zone permits
- Post-join device migration planning
Prefer to talk it through first? Book a technical consultation → · View managed services
All hostnames, domains, WWNs, IPs and values in this guide are synthetic reference values. No customer environment is depicted.