The Cisco Nexus 1000V has two main components:
- Virtual supervisor module (VSM)
A VM running in vSphere ESXi server (a standalone ESXi server or a shared ESXi server hosting both VSM and VEM)
Provides CLI interface for managing Nexus 1000v switch
Controls multiple VEM as a single network device
- Virtual Ethernet module (VEM)
An addon module to be installed on ESXi hypervisor, which controls vem daemons.
A kind of vsphere distributed switch
Independent of VSM in terms of operation, if VSM fails, VEM continues continue to forward
traffic, even its parts of configuration can be managed by vemcmd.
Cisco N1000V specific traffic types:
- Management traffic: Traffic for the VSM management interface and for VMware vCenter Server falls into
this category. VMware vCenter Server requires access to the VMware ESX management interface to
monitor and configure the VMware ESX host. Management traffic usually has low bandwidth requirements,
but it should be treated as high-priority traffic
- Control traffic: Control traffic is generated by the Cisco Nexus 1000V Series and exchanged between the
primary and secondary VSMs as well as between the VSMs and VEMs. It requires little bandwidth (less
than 7 MB) but demands absolute priority.
- Packet traffic: Packet traffic transports selected packets to the VSM for processing. The bandwidth required
for the packet interface is extremely low, and its use is intermittent. If the Cisco Discovery Protocol and
IGMP features are turned off, there is no packet traffic at all.
- System vlan: system vlan enables VEM to forward traffic even when the communication to VSM is lost. The system VLAN is mandatory for above 3 types of traffic and VMware management interface, it is also recommended for other vmkernel traffic e.g. vMotion,ISCSI/NFS.
Cisco n1000V requirements:
vSphere ESX/ESXi 4 or higher (check the Compatibility guide in Cisco website for details)
vSphere ESX/ESXi must has enterprise plus license (n100v is a kind of distributed Switch)
vSphere vCenter (VSM needs to install plugin into vCenter)
ESX/ESXi host must have at least 2 NICs, if you plan to install both VSM and VEM in the same host.
N1000V VM must use thick disk type and network interface must use E1000
Cisco n1000v pros and cons
- Pros:
Because Cisco n1000v running Cisco NX-OS, it offers additional features over vSphere distributed switch.
Central configuration by NX-OS CLI, just feel like a physical switch. E.g all live VMs’s interface can be seen in “show run” , an access list can be applied in the interface.
True end to end QOS, apart from being allocated specific bandwidth by policy-map, traffic leaving Cisco n1000v is marked up with DSCP value, which the upstream Cisco switch understands.VMware NetIOC offer bandwidth allocation by “shares”, but it is local to VMware hypervisor only. (update: VMware,VMware 5 supports IEEE 802.1p CoS value, which makes end to end QOS possible)
True LACP port-channel, VMware doesn’t support LACP bonding without Cisco N1000v
- Cons:
Additional license cost
Unlike distributed switch, which is VMware built-in, VSM is third party VM. Even VSM supports HA and VSM failure doesn’t stop VEM from functioning. Without VSM, it is impossible to make configuration change.
Cisco n1000v deployment procedures
1) Download softwareDownload free trial of ESXi host and vCenter from VMware site, Download free trial of N1000V from Cisco website.(check the compatible guide to determine the exact versions)
In my test, I used
- VMware ESXi 5.0.0 build-474610
- VMware vCenter Server 5.0.0 build-623373
- Cisco N1000v version 4.2(1)SV1(5.1)
NOTE: N1000V installation procedure may vary with different versions, the following is for verson 4.2(1)SV1(5.1).
2) Install ESXi and vCenter
Create 3 port-groups for management, control and packet traffic of N1000V (You can create separate VLAN for each type, but sharing the same VLAN as management interface of ESXi host is sufficient , so all three port-groups use the same VLAN ID)
3) Install N1000V VSM
Unzip the downloaded Cisco N1000V, connect to vCenter, deploy the OVA file located in “Nexus1000v.4.2.1.SV1.5.1\VSM\Install”. Follow the wizard to deploy the OVA (NOTE: use the default setting, disk type must use thick provisioning, NIC must use E1000)
Start up N1000V VM, login to console with the credentials supplied earlier, run “setup” to start the wizard to configure n1000v. most options are self explanatory, the following are noted
Configure Advanced IP options (yes/no)? [n]: noEnable the ssh service? (yes/no) [y]: yesConfigure svs domain parameters? (yes/no) [y]: yesEnter SVS Control mode (L2 / L3) : L3#(VEM-VSM communication can operate in Layer 2 mode or Layer 3 mode, L3 is recommended.But N1000V is layer 2 device, ip routing is not supported)Configure Advanced IP options (yes/no)? [n]: noEnable the ssh service? (yes/no) [y]: yesConfigure svs domain parameters? (yes/no) [y]: yesEnter SVS Control mode (L2 / L3) : L3#(VEM-VSM communication can operate in Layer 2 mode or Layer 3 mode, L3 is recommended.But N1000V is layer 2 device, ip routing is not supported)
4) Establish connection between VSM and vCenter
#Launch the JAVA installerC:\Nexus1000v.4.2.1.SV1.5.1\VSM\Installer_App>java -jar Nexus1000V-install.jar VC#Following the wizard to establish connection between VSM and vCenter, the result is that a new distributed switch will be created in “networking “ view in vCenter.Verify the connection in N1000V CLI:nv1> show svs connections….config status: Enabledoperational status: Connectedsync status: Completeversion: VMware vCenter Server 5.0.0 build-623373vc-uuid: F1D0CEBA-C365-4F55-830D-A0B9BB6F8520#Don’t continue to the next step without a successful connection is seen.
5) Install N1000V VEM
C:\Nexus1000v.4.2.1.SV1.5.1\VSM\Installer_App>java -jar Nexus1000V-install.jar VEM#Launch the Java installer, which will connect to vCenter and VSM to push VEM module to ESX host.#You can also install the VEM module manually.C:\Nexus1000v.4.2.1.SV1.5.1\VEM\cross_cisco-vem-v140-4.2.1.1.5.1.0-2.0.1.vib#Transfer the file to ESXi 5 host and runESXi>esxcli software vib install -v /tmp/cross_cisco-vem-v140-4.2.1.1.5.1.0-2.0.1.vib
6) Establish connection between VEM and VSM
#Because VEM and VSM are on the host, you need to put VEM VSM into separate physical NICs in order to migrating from standard switch to N1000V switch.#Create port profile on N1000V#create an uplink profile to be linked to ESXi host pNIC# note the type is Ethernet and switchport is trunkport-profile type ethernet vm-uplinkvmware port-groupswitchport mode trunkswitchport trunk allowed vlan 3-10no shutdownsystem vlan 3,5-6state enabled#Create port-group for vmkernel management interface.#note the type is venthernet and “capability l3control” and system vlan are mandatoryport-profile type vethernet L3vmkernelcapability l3controlvmware port-groupswitchport mode accessswitchport access vlan 3no shutdownsystem vlan 3state enabled# the following 3 profiles for N1000V port-groups, system vlan is mandatoryport-profile type vethernet ds_ctlvmware port-groupswitchport mode accessswitchport access vlan 3no shutdownsystem vlan 3state enabledport-profile type vethernet ds_mgtvmware port-groupswitchport mode accessswitchport access vlan 3no shutdownsystem vlan 3state enabledport-profile type vethernet ds_pktvmware port-groupswitchport mode accessswitchport access vlan 3no shutdownsystem vlan 3state enabled#the following is for generic VM traffic, vCentre VM will be migrated to this in the first state.#system vlan is optionalport-profile type vethernet vmdatavmware port-groupswitchport mode accessswitchport access vlan 3no shutdownsystem vlan 3state enabled
After successfully execution of above commands, 5 objects will be created under the switch in “networking” view.
Right click the switch and select “add host”, in select physical adapter, select “vmnic1” in uplink group select “vm-uplink”
Don’t migrate vmk0 yet.
In migrate virtual machine networking, migrate N1000v from standard switch to its N1000V switch , migrate vCentre VM to “vmdata” group.
Make sure N1000V VM and vCenre VM were migrated successfully by checking that “show svs connections”
Next step is to migrate vmk0.
Click the N1000V switch, in configuration tab, select “manage hosts”, select migrate vmk0 to “L3vmkernel “ portgroup, now everything has been migrated from vmnic0 to vmnic1, vmnic0 is spare, you can create port-channel in N1000V and then migrate vmnic0 as well.
Only after a successful migration of vmk0, VEM to VSM connection is established and VEM module is seen in Switch
nv1> show moduleMod Ports Module-Type Model Status--- ----- -------------------------------- ------------------ ------------1 0 Virtual Supervisor Module Nexus1000V active *3 248 Virtual Ethernet Module NA ok
Other notes:
Unlike VMware standard switch, port-groups “grow” under a switch, which, in turn, is linked to pNIC, so there is a clear one to one mapping relationship between port-group and pNIC. In N1000V switch, the relationship between port-group and pNIC is in line with VLAN relationship.
Unlike VMware standard switch,port-group can be tagged for “vMotion” traffic in GUI directly, You have to this for N1000V in following steps
Switch to “hosts and clusters” view, click the host in management tab, click “vSphere Distributed Switch” view, click “manage virtual adapters” add a vmkernel adapter and in connection setting select “use this adapter for vMotion”.
#Sample QOS configuration bound for a port-grouppolicy-map type qos po-vmdataclass class-defaultpolice cir 1500 mbps bc 200 ms conform transmit violate set dscp dscp table pir-markdown-mapport-profile type vethernet vmdataservice-policy input po-vmdataservice-policy output po-vmdata#Control VEM in ESXi host directly when VSM is not available.#unblock a port by defining correct system vlanESXi > vemcmd show portLTL VSM Port Admin Link State PC-LTL SGID Vem Port Type49 DOWN UP BLK 0 testlinux.eth0ESXi > vemset system-vlan 3 ltl 49ESXi > vemcmd show portLTL VSM Port Admin Link State PC-LTL SGID Vem Port Type49 UP UP FWD 0 testlinux.eth0
Cisco Nexus 1000V Series Switches download and document links
http://www.cisco.com/en/US/products/ps9902/index.html
Posts
