Understand Cisco Nexus 1000v deployment and other components – Part 2

Welcome !

Flashback, have a cluster with two nodes and also i added addition standalone ESXi to view it clearly all environment.

Some documents and videos are old or a few new installations guides are available on cisco but on community very good articles are there i will also add it end of the this article

Okay, go nexus installer up should be on C:\Users\Administrator\Downloads\Nexus1000v.4.2.1.SV1.5.2b\Nexus1000v.4.2.1.SV1.5.2b\VSM\Installer_App folder which name is Nexus1000V-install.jar

I expect that you installed 64 bit java and java.exe is under C:\Program Files\Java\jre7\bin , then run command line tool and go inside related java folder and execute the command

“C:\Program Files\Java\jre7\bin>java -jar C:\Users\Administrator\Downloads\Nexus1000v.4.2.1.SV1.5.2b\Nexus1000v.4.2.1.SV1.5.2b\VSM\Installer_App\Nexus1000V-install.jar VSM”

Log in credentials for install connect vCenter to install VSM

Choose where to deploy

Your VSM ova file should be under related folders

C:\Users\Administrator\Downloads\Nexus1000v.4.2.1.SV1.5.2b\Nexus1000v.4.2.1.SV1.5.2b\VSM\Install\nexus-1000v.4.2.1.SV1.5.2b.ova

First i would like to install standalone and then i will implement second one, give the name of VM and choose the datastore to deploy

From articles i learned that for debugging purpose its better to use L3 mode and i told you i would like to put VSM on separate vlan and because of ip connectivity enough then if my VEMs can reach the VSM then everything is okay. Also on this configuration i did not choose the control port for L3 connectivity will check the requirement next

Give a name to your baby, admin pass like 12qwaszxcv!A because of password policy,ip address , mask, gw , domain id and choose the DC and set the native vlan id.

Good explanation about what is native vlan from IBM http://publib.boulder.ibm.com/infocenter/zvm/v5r4/index.jsp?topic=/com.ibm.zvm.v54.hcpa6/hcsc9b3181.htm

Domain id is a tag to understand or to be sure that right VSM and VEM are talking each other, if VEM and VSM domain id different then control packages or communications are dropped.

Review screen will be appear, no need a screenshot pls check and confirm and go installing ……

You can check it the installation also from vCenter

You can check the connectivity

if you need to migrate something you can go on , but i do not think it

Click finish and the see the summary of installation and close the window and lets switch to VEM installation
Check the VEMs are already installed or not ! Try ssh you ESXi node, if you an error okay VEM is not installed on ESXi nodes

VMware offers supported, powerful system administration tools.  Please
see http://www.vmware.com/go/sysadmintools for details.

The ESXi Shell can be disabled by an administrative user. See the
vSphere Security documentation for more information.
~ # vem status
-sh: vem: not found
~ #

Before past to the installation need to be something modify, flashback again , previous article i have a screenshot of ESXi nodes networking

Now i created extra VMK Port which name is nexusvmk we will use it later to move such VMK to Nexus distributed switch port group, i did it also on other node too ! if you want completely you can separate the pNIC for VSM to VEM communication on ESXi.

Lets continue the installation of the VEM

“C:\Program Files\Java\jre7\bin>java -jar C:\Users\Administrator\Downloads\Nexus1000v.4.2.1.SV1.5.2b\Nexus1000v.4.2.1.SV1.5.2b\VSM\Installer_App\Nexus1000V-install.jar VEM”

Type again vCenter credentials , no need to screenshot and then type the VSM credential

Choose the hosts want to install/deploy the VEM

Check the summary and click finish

I got an error about installation failed and provided log files find that error is SEVERE: Compatible vib not found on VSM and on community one of the question is unanswered https://communities.cisco.com/message/110675  okay don’t worry about it 😀

I have a update manager 😀  instead of ability to install the VEM vib via CLI i will show you the update manager, maybe its help to you for other kind of installations anybody want to install vib from cli

Go to vCenter home and click Update Manager

Select Patch Repository and click Import Patch

Click browse and choose the related VSM for related ESXi version, its under VEM folder

Patch upload process will be begun

Pls look the type its a exrenstion

Now need to create a baseline to apply it to ESXi host for install VEM

Pls set a name to baseline and type as a externsion

Pls filter the extensions like below

Add it

Next

and Next after this process over need to assign such baseline to all ESXi nodes via cluster or one by one

Go to one of host last tab and click attach to set the baseline

Select the baseline what we created , sorry for mismatch i was get the screenshot before as a baseline Install VEM but this is Install , sorry for bad screenshots 😦

Looks like its attached

And lest step pls remediate it

Select the patch

Select immediately

Noting to enter maintenance mode , click next

Some checkboxes are not selected pls fallow the screenshot

And thats it click finish and fallow the task manager of vCenter

Now connect to the one of the node to SSH and check the VEM status , if vem agent is running then okay it loaded

Back to the VSM for shortly , type the command below, VSM is active its on first module and right now no extra module was there

MYVSM# show module

Check any VEM are missing, of course for now nothing should be missed but such command very important for debug

MYVSM# show module vem missing
No missing module found.

Forgot to check VSM what did when its installed on vCenter , nothing to care about for now, we will create everything ourself

Now, we need to configure virtual distributed switch, physically every switch should  have a uplink 🙂 to communicate out side
Lets create uplink via VSM , i’m not a network guy but looks like very easy

First pls check the vlans are setted or not

MYVSM# show vlan VLAN

Name Status Ports

—- ——————————– ——— ——————————-

1 default active

770 VLAN0770 active

VLAN Type

—- —–

1 enet

770 enet

Remote SPAN VLANs

——————————————————————————-

Primary Secondary Type Ports

——- ——— ————— ——————————————-

Lets add the vLAN 773. In my config, remember nexusvmk on each ESXi nodes should communicate with VSM.

MYVSM# CONfigure T

Enter configuration commands, one per line. End with CNTL/Z.

MYVSM(config)# VLan 773

MYVSM(config-vlan)# name nexusvmkonescinodes

Configure Uplink 🙂  Flashback, i used system clan  773 because ESXi node VSM communcaiton vlan is it and have to specify as a system vlan for provide ability to communicate VEM with VSM when ESXi booted.

MYVSM# configure t
Enter configuration commands, one per line. End with CNTL/Z.

MYVSM(config)# port-profile type ethernet uplink

MYVSM(config-port-prof)# vmware port-group

MYVSM(config-port-prof)# switchport mode trunk MYVSM(config-port-prof)# switchport trunk allowed vlan all

MYVSM(config-port-prof)# switchport mode access

MYVSM(config-port-prof)# no shutdown

MYVSM(config-port-prof)# system vlan 773

MYVSM(config-port-prof)# state enabled

Check the vCenter Networking section, uplink is there 😀

Lets create a portgroup to connect nexusvmk interfaces there to communicate VEMs to VSM

MYVSM(config)# port-profile type vethernet l3control773

MYVSM(config-port-prof)# switchport mode access

MYVSM(config-port-prof)# switchport access vlan 773

MYVSM(config-port-prof)# vmware port-group

MYVSM(config-port-prof)# no shutdown

MYVSM(config-port-prof)# capability l3control

Warning: Port-profile ‘l3control773’ is configured with ‘capability l3control’. Also configure the corresponding access vlan as a system vlan in: * Port-profile ‘l3control773’. * Uplink port-profiles that are configured to carry the vlan MYVSM(config-port-prof)# 2013 Jan 17 08:06:00 MYVSM %MSP-1-CAP_L3_CONTROL_CONFIGURED: Profile is configured with capability l3control. Also configure the corresponding VLAN as system VLAN in this port-profile and uplink port-profiles that are configured to carry the VLAN to ensure no traffic loss.

MYVSM(config-port-prof)# system vlan 773

MYVSM(config-port-prof)# state enabled

Okay lets join the ESXi nodes to Cisco Nexus Distributed Switch

Choose the NICs which are will be uplink for ESXi node and which uplink profile we should add , here it is uplink port profile

Still i can not understand if everything is working L3 mode then why need to migrate VMK interface to created vmware port-group which name is l3control773

Overview and its okay.

You have to get such notification(s) on console of VSM

MYVSM(config-port-prof)# 2013 Jan 17 08:15:52 MYVSM %VEM_MGR-2-VEM_MGR_DETECTED: Host nn1 detected as module 3 2013 Jan 17 08:15:52 MYVSM %VEM_MGR-2-MOD_ONLINE: Module 3 is online

MYVSM# 2013 Jan 17 08:19:19 MYVSM %VEM_MGR-2-VEM_MGR_DETECTED: Host nn2 detected as module 4 2013 Jan 17 08:19:19 MYVSM %VEM_MGR-2-MOD_ONLINE: Module 4 is online

Check the modules

MYVSM# show module

Mod Ports Module-Type Model Status

— —– ——————————– —————— ————

1 0 Virtual Supervisor Module Nexus1000V active *

3 248 Virtual Ethernet Module NA ok

4 248 Virtual Ethernet Module NA ok

Mod Sw Hw

— —————— ————————————————

1 4.2(1)SV1(5.2b) 0.0

3 4.2(1)SV1(5.2b) VMware ESXi 5.1.0 Releasebuild-799733 (3.1)

4 4.2(1)SV1(5.2b) VMware ESXi 5.1.0 Releasebuild-799733 (3.1)

Mod MAC-Address(es) Serial-Num

— ————————————– ———-

1 00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8 NA

3 02-00-0c-00-03-00 to 02-00-0c-00-03-80 NA

4 02-00-0c-00-04-00 to 02-00-0c-00-04-80 NA

Mod Server-IP Server-UUID Server-Name

— ————— ———————————— ——————–

1 10.11.12.13 NA NA

3 10.110.1.16 00408113-d21d-b211-8000-00215ad4da30 10.110.1.16 4 10.110.1.21 00217719-d21d-b211-8000-00237d5e6996 nn2 * this terminal session

Go to Understand Cisco Nexus 1000v deployment and other components – Part 3

VM

Good Links

http://www.cisco.com/en/US/docs/switches/datacenter/nexus1000/sw/4_2_1_s_v_1_5_1/install_upgrade/vsm_vem/guide/n1000v_installupgrade_install.html

https://communities.cisco.com/docs/DOC-28631

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1020542

Troubleshooting

http://www.cisco.com/en/US/docs/switches/datacenter/nexus1000/sw/4_0_4_s_v_1_3/troubleshooting/configuration/guide/n1000v_trouble_5modules.html#wp1193265

Understand Cisco Nexus 1000v deployment and other components – Part 1

Pls read this part if you want to get short update about components and to see my test environment, if you do not need to such information pls switch part 2 

Cisco Nexus 1000v is NX-OS software which run on hypervisor, its support VN-Link server virtualisation tech.Cisco Nexus 1000v provide L2 switching,advanced networking and common management model.

Each ESXi node (also it can be Windows 2012 node) become a line card. Two components together make up Cisco Nexus switch.(VSM can be installed independent of the VEM,VEM itself provide package-forwarding capability)

VSM(Virtual Supervisor Module)

Provide CLI
Use Cisco NX-OS Software
Control all VEMs

VEM(Virtual Ethernet Module)

Alternatif for VMware Distributed Switch
Activate advanced network capability too Hypervisor
Provider dedicated switch port to VMs

Each ESXi node can have only one VEM installation, not more

Cisco Nexus 1000v use virtual chassis module, Slot of 1 and 2 always assigned to VSM, VEMs are between slot 3 to 66
Port Profile is virtual boundary between server and networking and Cisco Nexus 1000v provide Network Policy Management with port-profile

Switch# show port-profile name Basic-VM
port-profile Basic-VM
config attributes:
switchport mode access
switchport access vlan 53
no shutdown

VSM provide management plane much like Cisco 7000 series Supervisor module, single point of management and coordinating configurations and functions across VEMs. Different thing is VSM is software not a hardware thing. Installable via OVF and support HA installation.

VSM do not deal about data flow, all data move via VEM.

VSM Interface ;

SVS(Software Virtual Switch)

VSM is a virtual machine and require 3 vNICs which should be Intel 1000e driver.

vNIC1 (Management)

Interface adı mgmt0, generally not used for communication between VSM and VEM, its provider connectivity between vCenter and VSM. SVS L3 mode management could be communication interface between VSM and VEM. Generally this interface should be setter as a Network Adapter 2 on VSM VM config.

vNIC2 (Control)

Control interface used from inter-communication between VSM and VEM when SVS mode set L2 also this interface used for communication between VSMs when HA mode activated. When SVS mode set L3 still VSM HA used this interface for communication, very important and maybe you can set priority for this. Generally this interface should be setter as a Network Adapter 1 on VSM VM config.

vNIC3 (Packet)

its mostly used for internal communication of VEMs like CDP , IGMP protocols. The last vNIC is for packet interface and for L3 deployment all packet communication happen via L3 interface between VSM tp VEM.

Domain ID is kind of tag and used for intercommunication between VSM and VEM, if one of VEM get request from VSM with different ID then ignore it.

After integrate VSM with vCenter, to communicate between VSM and VEM , VSM send some data which named Opaque Data like below

Among other content, the vswitch data contains:

• Switch domain ID
• Switch name
• Control and packet VLAN IDs
• System port profiles

VEMs are like a line card but they are all independent switch from a forwarding perspective, its tightly integrated with ESXi and kernel component.
Max 64 VEM can be managed by one VSM pair.

Switch Port Interfaces ; vEth (Virtual Ethernet) ,ETH , Po (Port Channel)

vEth is new concept and describe switch port which connected VM vNIC or vmknic/vswif. Another point of view vEth is a port like physical switch which server connected it via wired 😀 . Its unique, no problem at vMotion time and statically bind when VM is created, its not unassigned when VM is suspended or down but when VM is deleted related beth wait next VM provisioning to assign VM.

Eth is link to pNIC of ESXi host.

Po is used for link aggregation.

Each VEM keeps its own mac table and do not know something about other VEM, just forward to up if related MAC is not in its table.
No spanning tree, its use loop-preventation strategy.

VEM to VSM Communication ;

L2 mode, VSM and VEMs should be on same L2 domain.

CLI Command

config t
svs-domain
svs mode L2

L3 mode, VSM and VEMs could be any ip network , they can communicate each other and can maintain with easy ping command

CLI Command

config t
svs-domain
svs mode L3 interface (mgmt0 or control0)

Layer 3 mode encapsulate packet and control frames via UDP, a new or existing l3control enabled port-group should be and vmkernel interfaces on ESXi node should be move on this port-group.

Node restart or such issues do not effect slot number of VEMs

VSM has a heartbeat with assigned VEMs. Every second VSM send a request every second and if it can not get response from VEMs, its remove the VEM from line card via VSM. VEM worked with last known state and continue forwarding. When connectivity come back and configuration changed then need to be VEM reprogrammed and this cause 1-15 second downtime on related VEM or ESXi node.

VSM to VEM communication is encrypted.

To deploy VSM;

• You can use ESXi managment vmkernel interface for  communication between VSM-VEM, then no need additional interface on node
• Or you can seperate ESXi and VSM to VEM communication interfaces
• End you can use the control interface for it but ESXi Managment VMKernel vlan and Control vlan should be on different vlans.

Port Profiles

Port profile needed to apply network policy and after its provisioned via VSM its appear on vCenter after creation now VMs can connect their vNICs to port group. Port profil are not static and can be changed any time when the system is running.

vEth Profiles

When newly created VM powered on, vEth created on Nexus 1000v switch and its appointed to port profile. Related vEth number is not deleted or used buy other VM, until the VM is deleted its vEth always will be exist.

Note for vmware users, a port profile and ethernet/uplink profiles icons are not changed for nexus 1000v on vCenter, same illustrations are already used.

System VLANs

Very special meanings it has. its like a read ahead or quick start like solution, try to up VSM and VEM communication quick as possible when ESXi starts. The control and packet VLANs must be defined as system VLANs. The service console interface and VMware vmkernel iSCSI or NFS interface should also be defined as system VLANs.

My Test Environment

A standalone vSphere 5.1 ESXi server which will host vCenter and Update Manager servers also VSM will run on it
For guests machines vSphere Standart Switch will be used
* A single VMKernel Port for ESXi management
* Two virtual switches , vSwitch0 and vSwitch1
* Two vlan ‘770’ and ‘773’
* Two port group ‘VM Network’ and ‘V770’

I planed to keep VSM on different network and VSM guest machine will be on vLAN 770, vCenter and Update manager on vLan 773

Two additional vSphere 5.1 ESXi servers which are already joined cluster under related datacenter we need, this nodes have ;

* Two VMKernel Port for ESXi management and iSCSI (For Now)
* Two virtual switches , vSwitch0 and vSwitch1
* Two vLan for Management of ESXi and for iSCSI trafik
* Additional one or more, in my configuration only one extra ethernet interface needed for add it Cisco Nexus 1000v switch for uplink needs of VM traffic, between VMs or internet traffic

Before installations pls get ready ;

Pls go to the link and search section write down 1000v and select NX-OS System Software link and download. You also need to install java on where VSM Installer will run 😀 download from this link and choose the relate version and supported OS. I saved all downloads and installed Java JRE on vCenter VM.

Understand Cisco Nexus 1000v deployment and other components – Part 2

A stupid problem – No translate group found on Cisco ASA Firewall

Today i faced stupid network access issue
I don’t know how but i started to get no translate group found between some of source and destinations. After leave the ASA for a long year, its really hard to adapt again but a cool feature which must be used at first time , its Packet Tracer

Second is NAT 😀  In my config why i need a NAT , back to the NAT screen and got it !

Pls check the Enable traffic through the firewall without address trasnlation, because if not ASA looks like try to find out some nat matches to allow access which is not needed in my config.

VM

Event ID 1000 / Login failure – retrying once… and vCenter Service is not starting

I’m adding this article after many search and adding here because to find out more quickly next time 🙂

 

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2015824

 

http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&popup=true&languageId=&externalID=2033620#rereg_vc_to_different_sso

 

Regards

VM