Part1 - Cisco MPLS L3VPN Lab

In this series of posts, we will look at various technologies used for Inter-AS design. To build these labs, we will use:

• OSPF
• IS-IS
• MPLS
• Layer3 VPN (VRFs)
• Multi-protocol BGP

A few years ago I passed the CCIE Service Provider certification. The lab side of this course is heavily focused on ISIS, BGP, OSPF and L2/L3 VPNs. These build together to form Inter-AS designs, with Option A, B and C being practiced for the lab.

InterAS Option C (which is knows as "MPLS VPN Inter-AS with ASBRs Exchanging IPv4 Routes and MPLS Labels" - Option C is much easier..) is a method to enable to AS's to join together, and allow IPv4/v6 routes to be distributed between the AS (via their route reflectors), and VPNv4/v6 routes to pass across the network to support end customer CE devices connected to the different AS's.

Assumed Knowledge

This series assumes some core understanding of the above listed technologies. I do provide config for each of the steps, but if you ever need to troubleshoot what has been loaded, you will need some basic understanding of how the tech works. If you have CCNP-SP/JNCIP-SP level knowledge, that should suit these posts.

To create this environment, I have used the following lab and images:

. Intel Core i7 with 32GB ram [10GB is needed for the lab to be built, so a 16GB system would be fine]
. 256GB SSD [20GB max is needed for the lab]

. VMware ESXi 6.7 [this can be built on any platform that EVE-NG can be installed on]

. EVE-NG community edition virtual environment (latest version - im using 2.0.3-95) with the following images:

. vIOS         - vios-adventerprisek9-m-15.5
. iosXR       - xrv-k9-6.0.0
. csr1000v   - csr1000v-3.16.00.S.155-3.S
. vSRX        - vsrx-12.1X47-D20.7-domestic

The Cisco router layout for the lab looks like this:

What are we trying to achieve ?

We need to get traffic between each separate customer (Cust1=YELLOW, Cust2=ORANGE) across the 2 networks, in a segmented VRF without customer routes appearing in either core network. The IPv4 core will have NONE of the customer routes within the RIBs of any of the core routers (this is handled via MP-BGPv4).
The aim is to create a scalable network in the core which can support lots more customers transiting traffic in the same way, via simple MPLS paths and Layer3 VPN tunnels.

When I build and design, normally Ill look to do it in layers. For the CCIE lab this is critical (as you are time limited and you need to build it Layer2 - Layer3 - IGP - BGP - special config, with verification of each step, otherwise troubleshooting faults is a nightmare), and it's great way to learn making sure each step is validated correctly.

The steps we will go through to build this design over the following posts are as follows:

Stage 1 - Build OSPF/ISIS Areas
Stage 2 - Configure BGP, Create ASBRs/PEs, use RR via R7/R8
Stage 3 - Enable MPLS
Stage 4 - Create InterAS OptC (multihop vpnv4 ebgp between RR lo)
Stage 5 - Connect CEs, CUST1=OSPF CUST2=BGP
Stage 6 - replace some routers with Juniper

Download the LAB from HERE
Download the base config from HERE

OK lets start the lab up. You can select all the devices to start them, and then let the routers fire up. This will take 5-10 minutes depending on your system performance, so give them all a chance to get up to the login prompts. The IOS XR units use cisco/cisco as the login, and the other devices should not need a login. Go through the routers and apply the base config to them all (making sure that interfaces are UP as well during the setup, and that you can ping the other end of each of the links you create). This sets up the IP addressing and port config, ready for Stage1.

Once you have Layer3 connectivity, lets begin and work on Stage1 - OSPF/ISIS. We need IGP connectivity, so that we can then move on to create the BGP peering and InterAS design.

AS21 - OSPF (R1,R2,R6,R7,R10)

 R1

router ospf 21
 address-family ipv4
 area 0
  interface Loopback0
  !
  interface GigabitEthernet0/0/0/0
  !
  interface GigabitEthernet0/0/0/1
  !
  interface GigabitEthernet0/0/0/2
  !
 !
!

R2

router ospf 21
 address-family ipv4
 area 0
  interface Loopback0
   passive enable
  !
  interface GigabitEthernet0/0/0/0
  !
  interface GigabitEthernet0/0/0/1
  !
  interface GigabitEthernet0/0/0/2
  !
 !
!

 R6

 router ospf 21
 network 6.6.6.6 0.0.0.0 area 0
 network 10.0.0.0 0.0.255.255 area 0

R7

 router ospf 21
 router-id 7.7.7.7
 network 7.7.7.7 0.0.0.0 area 0
 network 10.0.0.0 0.0.255.255 area 0

R10

 router ospf 21
 network 10.0.0.0 0.0.255.255 area 0
 network 10.10.10.10 0.0.0.0 area 0

Once completed, we should find we have OSPF adjacency to each connected peer, and a OSPF Database (and RIB) list with the interfaces and loopback addresses all listed for the AS:

Looks good - OSPF intra area routes are in the RIB (TYPE = O), and all router LSAs are present in the OSPF DB.

Lets create the ISIS peering for the other AS. Here we use metric-style wide to support the MPLS TLVs used in ISIS:

AS111 - ISIS (R3,R4,R5,R8,R9)

R3

router isis 1
 is-type level-2-only
 net 49.1111.0000.0000.0003.00
 address-family ipv4 unicast
  metric-style wide
 !
 interface Loopback0
  address-family ipv4 unicast
  !
 !
 interface GigabitEthernet0/0/0/0
  address-family ipv4 unicast
  !
 !
 interface GigabitEthernet0/0/0/1
  address-family ipv4 unicast
  !
 !
!

R4
router isis 1
 is-type level-2-only
 net 49.1111.0000.0000.0004.00
 address-family ipv4 unicast
  metric-style wide
 !
 interface Loopback0
  address-family ipv4 unicast
  !
 !
 interface GigabitEthernet0/0/0/1
  address-family ipv4 unicast
  !
 !
 interface GigabitEthernet0/0/0/2
  address-family ipv4 unicast
  !
 !
 interface GigabitEthernet0/0/0/3
  address-family ipv4 unicast
  !
 !
!

R5
router isis 1
 net 49.1111.0000.0000.0005.00
 is-type level-2-only
 metric-style wide
!
interface Loopback0
 ip router isis 1
!
interface GigabitEthernet2
 ip router isis 1
!
interface GigabitEthernet3
 ip router isis 1

R8
router isis
 net 49.1111.0000.0000.0008.00
 is-type level-2-only
 metric-style wide
!
interface Loopback0
 ip router isis
!
interface GigabitEthernet1
 ip router isis
!
interface GigabitEthernet2
 ip router isis

R9
router isis
 net 49.1111.0000.0000.0009.00
 is-type level-2-only
 metric-style wide
!
interface Loopback0
 ip router isis
!
interface GigabitEthernet1
 ip router isis
!
interface GigabitEthernet2
 ip router isis
!
interface GigabitEthernet3
 ip router isis

This should now show a complete ISIS network, with all routes and neighbours verified:

Great! Routes present (Type L2) and all loopbacks in the RIB. In the next post we will move on to Stage2 and get the BGP peering working. Check out the next post HERE

Part 2 - Cisco MPLS L3VPN Lab

In the previous post we looked at the IGP configuration. That's now verified and working, so lets move on the BGP and MPLS.

Stage 1 - Build OSPF/ISIS Areas - COMPLETE
Stage 2 - Configure BGP, Create ASBRs/PEs, use RR via R7/R8
Stage 3 - Enable MPLS
Stage 4 - Create InterAS OptC (multihop vpnv4 ebgp between RR lo)
Stage 5 - Connect CEs, CUST1=OSPF CUST2=BGP
Stage 6 - replace some routers with Juniper

Just as a reminder, here is our lab design:

Now for Stage2 - we need to create BGP peering within each AS (to the AS route reflector - R7 or R8) and also create a eBGP peering between AS's. We will also add in the MPLS label function to BGP, as we will need that for the next stage. A few points about the BGP peers:

1. We are using BGP IPv4 labeled-unicast for the peering, since MPLS labels need to be passed between BGP peers.
2. We will pass the loopback addresses between the eBGP peers via the network statement (rather than a redistribute statement)
3. IOS-XR blocks eBGP route passing by default, so we need the PASS route-policy in place to support the route flow
4. We use the next-hop-self on the ASBR so that routers inside the AS know how to get across to the other AS for their loopbacks and routes

AS21 (R1,R2,R6,R7,R10)

R1

router bgp 21
 bgp router-id 1.1.1.1
 address-family ipv4 unicast
  allocate-label all
 !
 neighbor 7.7.7.7
  remote-as 21
  update-source Loopback0
  address-family ipv4 labeled-unicast
   next-hop-self
  !

R2
route-policy PASS
  pass
end-policy
!
router bgp 21
 bgp router-id 2.2.2.2
 address-family ipv4 unicast
  network 1.1.1.1/32
  network 2.2.2.2/32
  network 6.6.6.6/32
  network 7.7.7.7/32
  network 10.10.10.10/32
  allocate-label all
 !
 neighbor 7.7.7.7
  remote-as 21
  update-source Loopback0
  address-family ipv4 labeled-unicast
   next-hop-self
  !
 !
 neighbor 10.0.29.2
  remote-as 111
  address-family ipv4 labeled-unicast
   route-policy PASS in
   route-policy PASS out
  !
 !
!

R6
router bgp 21
 bgp router-id 6.6.6.6
 bgp log-neighbor-changes
 neighbor 7.7.7.7 remote-as 21
 neighbor 7.7.7.7 update-source Loopback0
 neighbor 10.22.22.2 remote-as 2
 !
 address-family ipv4
  neighbor 7.7.7.7 activate
  neighbor 7.7.7.7 next-hop-self
  neighbor 7.7.7.7 send-label
  neighbor 10.22.22.2 activate
 exit-address-family
 !

R7
router bgp 21
 template peer-policy AS21
  route-reflector-client
  send-community both
  send-label
 exit-peer-policy
 !
 template peer-session AS21_SESS
  remote-as 21
  update-source Loopback0
 exit-peer-session
 !
 bgp router-id 7.7.7.7
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 1.1.1.1 inherit peer-session AS21_SESS
 neighbor 2.2.2.2 inherit peer-session AS21_SESS
 neighbor 6.6.6.6 inherit peer-session AS21_SESS
 neighbor 10.10.10.10 inherit peer-session AS21_SESS
 !
 address-family ipv4
  neighbor 1.1.1.1 activate
  neighbor 1.1.1.1 inherit peer-policy AS21
  neighbor 2.2.2.2 activate
  neighbor 2.2.2.2 inherit peer-policy AS21
  neighbor 6.6.6.6 activate
  neighbor 6.6.6.6 inherit peer-policy AS21
  neighbor 10.10.10.10 activate
  neighbor 10.10.10.10 inherit peer-policy AS21
 exit-address-family
 !

R10
router bgp 21
 bgp router-id 10.10.10.10
 bgp log-neighbor-changes
 neighbor 7.7.7.7 remote-as 21
 neighbor 7.7.7.7 update-source Loopback0
 neighbor 10.0.103.1 remote-as 111
 !
 address-family ipv4
  network 1.1.1.1 mask 255.255.255.255
  network 2.2.2.2 mask 255.255.255.255
  network 6.6.6.6 mask 255.255.255.255
  network 7.7.7.7 mask 255.255.255.255
  network 10.10.10.10 mask 255.255.255.255
  neighbor 7.7.7.7 activate
  neighbor 7.7.7.7 next-hop-self
  neighbor 7.7.7.7 send-label
  neighbor 10.0.103.1 activate
  neighbor 10.0.103.1 send-label
 exit-address-family

BGP peers should now be up, with the 5 local routes coming in from the 2 ASBRs. The RIB failure here is due to the OSPF route already being present in the RIB, so BGP can't add those routes in (as its less preferred).

OK on to the next AS:

AS111 (R3,R4,R5,R8,R9)
R3
route-policy PASS
  pass
end-policy
!
router bgp 111
 bgp router-id 3.3.3.3
 address-family ipv4 unicast
  network 3.3.3.3/32
  network 4.4.4.4/32
  network 5.5.5.5/32
  network 8.8.8.8/32
  network 9.9.9.9/32
  allocate-label all
 !
 neighbor 8.8.8.8
  remote-as 111
  update-source Loopback0
  address-family ipv4 labeled-unicast
   next-hop-self
  !
 !
 neighbor 10.0.103.2
  remote-as 21
  address-family ipv4 labeled-unicast
   route-policy PASS in
   route-policy PASS out
  !
 !
!

R4
router bgp 111
 bgp router-id 4.4.4.4
 address-family ipv4 unicast
  allocate-label all
 !
 address-family vpnv4 unicast
 !
 neighbor 8.8.8.8
  remote-as 111
  update-source Loopback0
  address-family ipv4 labeled-unicast
   next-hop-self
  !

R5
router bgp 111
 bgp router-id 5.5.5.5
 bgp log-neighbor-changes
 neighbor 8.8.8.8 remote-as 111
 neighbor 8.8.8.8 update-source Loopback0
 neighbor 10.22.222.2 remote-as 2
 !
 address-family ipv4
  neighbor 8.8.8.8 activate
  neighbor 8.8.8.8 next-hop-self
  neighbor 8.8.8.8 send-label
  neighbor 10.22.222.2 activate
 exit-address-family
 !

R8
router bgp 111
 template peer-policy AS111
  route-reflector-client
  send-community both
  send-label
 exit-peer-policy
 !
 template peer-session AS111_SESS
  remote-as 111
  update-source Loopback0
 exit-peer-session
 !
 bgp router-id 8.8.8.8
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 neighbor 3.3.3.3 inherit peer-session AS111_SESS
 neighbor 4.4.4.4 inherit peer-session AS111_SESS
 neighbor 5.5.5.5 inherit peer-session AS111_SESS
 neighbor 9.9.9.9 inherit peer-session AS111_SESS
 !
 address-family ipv4
  neighbor 3.3.3.3 activate
  neighbor 3.3.3.3 inherit peer-policy AS111
  neighbor 4.4.4.4 activate
  neighbor 4.4.4.4 inherit peer-policy AS111
  neighbor 5.5.5.5 activate
  neighbor 5.5.5.5 inherit peer-policy AS111
  neighbor 9.9.9.9 activate
  neighbor 9.9.9.9 inherit peer-policy AS111
 exit-address-family
 !

R9
router bgp 111
 bgp router-id 9.9.9.9
 bgp log-neighbor-changes
 neighbor 8.8.8.8 remote-as 111
 neighbor 8.8.8.8 update-source Loopback0
 neighbor 10.0.29.1 remote-as 21
 !
 address-family ipv4
  network 3.3.3.3 mask 255.255.255.255
  network 4.4.4.4 mask 255.255.255.255
  network 5.5.5.5 mask 255.255.255.255
  network 9.9.9.9 mask 255.255.255.255
  neighbor 8.8.8.8 activate
  neighbor 8.8.8.8 next-hop-self
  neighbor 8.8.8.8 send-label
  neighbor 10.0.29.1 activate
  neighbor 10.0.29.1 send-label
 exit-address-family

We should now have AS111 active, and all peers up, with routes from both AS111 AND loopbacks from AS21, due to the eBGP peering between AS.

Looking at the routing table, we now have BGP routes present (B) from the AS21. Now we need to move on to the next stage, and enable MPLS across the different networks, so that we can get end to end connectivity.

Stage3 - Enable MPLS

We need to turn on MPLS and LDP on each of the routers, and create a static route on the XR ASBR units (since they require a /32 host route between peers to create a correct label path). The easiest way to enable ldp on all the links is to use the auto-config under the IGP, which makes sure that for all IGP enabled interfaces, LDP is also enabled and labels will be created.

R1
router ospf 21
 mpls ldp auto-config

mpls ldp
 router-id 1.1.1.1
 address-family ipv4
 !
!

R2
router static
 address-family ipv4 unicast
  10.0.29.2/32 GigabitEthernet0/0/0/0
 !
!
router ospf 21
 mpls ldp auto-config
!
mpls ldp
 router-id 2.2.2.2
 address-family ipv4
 !
 interface GigabitEthernet0/0/0/0
 !
!

R3
router static
 address-family ipv4 unicast
  10.0.103.2/32 GigabitEthernet0/0/0/2
 !
!
router isis 1
 address-family ipv4 unicast
  mpls ldp auto-config
!
mpls ldp
 router-id 3.3.3.3
 address-family ipv4
 !
 interface GigabitEthernet0/0/0/2
  address-family ipv4
  !
 !
!

R4
router isis 1
 address-family ipv4 unicast
  mpls ldp auto-config
!
mpls ldp
 router-id 4.4.4.4
 address-family ipv4
 !
!

R5
router isis 1
 mpls ldp autoconfig
!
mpls ldp router-id Loopback0
!

R6
router ospf 21
 mpls ldp autoconfig
!

R7
router ospf 21
 mpls ldp autoconfig
!
mpls ldp router-id Loopback0
!

R8
router isis
 mpls ldp autoconfig
!

R9
router isis
 mpls ldp autoconfig
!
interface GigabitEthernet4
 mpls bgp forwarding
!
mpls ldp router-id Loopback0

R10
interface GigabitEthernet1
 mpls bgp forwarding
router ospf 21
 mpls ldp autoconfig
!
mpls ldp router-id Loopback0

So we should now have a full label switched path end to end between the loopbacks of the 2 AS - lets have a look !

Trace from router 5 lo0 to router 6 lo0:

We start with 2 labels in the stack, 24008 is maintained across AS111, then it is pop'ed at the ASBR, and label 27 then label 24005 takes the traffic across AS21 to the final destination.

If we have a look across the other diagonal from AS21 to AS111:

We get the same result, with 2 labels within AS21, then a single label in AS111.

Looking at the MPLS forwarding table on R1, we can also see that routes have either a label or a pop label on them, which is what we want to see. If we see No Label listed here, that normally points to an issue in the config (e.g. MPLS not enabled on one of the interfaces).

We now have end to end MPLS label path, so we are ready to move on to Stage4 and build the L3 VPN tunnels across the core. The next post is HERE

Part 3 - Cisco MPLS L3VPN Lab

In the previous post we looked at the BGP and MPLS configuration.

Stage 1 - Build OSPF/ISIS Areas - COMPLETE
Stage 2 - Configure BGP, Create ASBRs/PEs, use RR via R7/R8 - COMPLETE
Stage 3 - Enable MPLS - COMPLETE
Stage 4 - Create InterAS OptC (multihop vpnv4 ebgp between RR lo)
Stage 5 - Connect CEs, CUST1=OSPF CUST2=BGP
Stage 6 - replace some routers with Juniper

Lets move on to configure the final stages ! So far what we have is a joined network at the same level via MPLS+BGP. The issue is, for us to create the Layer3 VPN tunnels, each AS needs to know about the VPNV4 destinations in the other AS, so that MPLS paths can be built for the MPLS traffic from the CE devices. The secret of this ? Connect the route-reflectors within each AS, and share routes. A few notes on this design:


. the next hop needs to remain unchanged when it is sent between the route-reflectors, to make sure the route reflectors do not become part of the data path
. add the VPNV4 family to the do the route-reflectors first, to make the config update easier
. ebgp multihop is required for the route-reflectors to be able to reach each other across the AS paths
. the ASBRs do NOT need VPNV4 configured on them, since they are not taking part in the VPNV4 process (they are simply forwarding traffic based on MPLS label), BUT the ASBRs do need a static route to address a fault in the XRv code that doesn't allocate the route correctly

The following shows us the peerings that we will now create:

Stage 4 - Create InterAS OptC

We need to peer the route-reflectors, and turn on the VPNV4 address family for all BGP peers.

R7
router bgp 21
 neighbor 8.8.8.8 remote-as 111
 neighbor 8.8.8.8 ebgp-multihop 255
 neighbor 8.8.8.8 update-source Loopback0
 address-family vpnv4
  neighbor 1.1.1.1 activate
  neighbor 1.1.1.1 send-community extended
  neighbor 1.1.1.1 inherit peer-policy AS21
  neighbor 6.6.6.6 activate
  neighbor 6.6.6.6 send-community extended
  neighbor 6.6.6.6 inherit peer-policy AS21
  neighbor 8.8.8.8 activate
  neighbor 8.8.8.8 send-community both
  neighbor 8.8.8.8 next-hop-unchanged
 exit-address-family
!

R1
router bgp 21
 address-family vpnv4 unicast
 !
 neighbor 7.7.7.7
  address-family vpnv4 unicast
   next-hop-self
!

R2
router static
 address-family ipv4 unicast
  10.0.29.2/32 GigabitEthernet0/0/0/0
 !
!

R6
router bgp 21
!
 address-family vpnv4
  neighbor 7.7.7.7 activate
  neighbor 7.7.7.7 send-community extended
  neighbor 7.7.7.7 next-hop-self
 exit-address-family
 !


R8
router bgp 111
 neighbor 7.7.7.7 remote-as 21
 neighbor 7.7.7.7 ebgp-multihop 255
 neighbor 7.7.7.7 update-source Loopback0
 !
 !
 address-family vpnv4
  neighbor 4.4.4.4 activate
  neighbor 4.4.4.4 send-community extended
  neighbor 4.4.4.4 inherit peer-policy AS111
  neighbor 5.5.5.5 activate
  neighbor 5.5.5.5 send-community extended
  neighbor 5.5.5.5 inherit peer-policy AS111
  neighbor 7.7.7.7 activate
  neighbor 7.7.7.7 send-community both
  neighbor 7.7.7.7 next-hop-unchanged
 exit-address-family
!
!

R3
router static
 address-family ipv4 unicast
  10.0.103.2/32 GigabitEthernet0/0/0/2
 !
!

R4
router bgp 111
 address-family vpnv4 unicast
 !
 neighbor 8.8.8.8
  address-family vpnv4 unicast
   next-hop-self
  !
 !

R5
router bgp 111
 address-family vpnv4
  neighbor 8.8.8.8 activate
  neighbor 8.8.8.8 send-community extended
  neighbor 8.8.8.8 next-hop-self
 exit-address-family
 !

That's done ! Lets go have a look at the BGP output now on the route reflector:

R8# sh ip bgp all sum
For address family: IPv4 Unicast

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
3.3.3.3           4          111     102     123       58    0    0 01:27:10        8
4.4.4.4           4          111       9       32        58    0    0 00:05:11        0
5.5.5.5           4          111       6       20        58    0    0 00:01:32        0
7.7.7.7           4           21      10       10        43    0    0 00:00:26        5
9.9.9.9           4          111     115     136       58    0    0 01:27:21        6

For address family: VPNv4 Unicast

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
4.4.4.4           4          111       9      32        1    0    0 00:05:11        0
5.5.5.5           4          111       6      20        1    0    0 00:01:32        0
7.7.7.7           4           21      10      10        1    0    0 00:00:26        0

Looks good ! Now we have peering to the local AS PEs, as well as the other AS route reflector, via VPNv4.

Stage 5 - Connect CEs, CUST1=OSPF CUST2=BGP

Time to setup the VRFs on the PEs, and peer between the PE and CE devices, so that we can get traffic across the Inter-AS MPLS core between CEs.

We will setup the VRF and create the BGP/OSPF peering at the same time.

. for the OSPF peering, we will use process 1. 
. we need to redistribute from the OSPF/BGP peering into the VPNv4 BGP network, so we can transmit across the MPLS core.
. Route-Distinguisher and Route-Target need to be configured, which are specific to the CUST1/2 network peering (so that the PEs know which route-targets to import for that customer)

R1
vrf CUST1
 address-family ipv4 unicast
  import route-target
   100:100
  !
  export route-target
   100:100
  !
 !
!
 router ospf 1
 vrf CUST1
  router-id 1.1.1.1
  redistribute connected
  redistribute bgp 21
  address-family ipv4 unicast
  area 0
   interface GigabitEthernet0/0/0/3
   !
  !
 !
!
router bgp 21
 vrf CUST1
  rd 1:11
  address-family ipv4 unicast
   network 10.11.11.0/30
   redistribute ospf 1 match internal external nssa-external
!

R11
router ospf 1
 router-id 11.11.11.11
 network 0.0.0.0 255.255.255.255 area 0
!

R6
vrf definition CUST2
 rd 2:2
 route-target export 200:200
 route-target import 200:200
 !
 address-family ipv4
 exit-address-family
!
 interface GigabitEthernet3
 vrf forwarding CUST2
 ip address 10.22.22.1 255.255.255.252
!
 router bgp 21
 !
 address-family ipv4 vrf CUST2
  redistribute connected
  neighbor 10.22.22.2 remote-as 2
  neighbor 10.22.22.2 activate
  neighbor 10.22.22.2 send-community both
 exit-address-family

R12
router bgp 2
 bgp router-id 12.12.12.12
 bgp log-neighbor-changes
 neighbor 10.22.22.1 remote-as 21
 !
 address-family ipv4
  redistribute connected
  neighbor 10.22.22.1 activate
  neighbor 10.22.22.1 send-community both
  neighbor 10.22.22.1 allowas-in
 exit-address-family

R4

vrf CUST1
 address-family ipv4 unicast
  import route-target
   100:100
  !
  export route-target
   100:100
  !
 !
!
router ospf 1
 vrf CUST1
  router-id 4.4.4.4
  redistribute connected
  redistribute bgp 111
  address-family ipv4 unicast
  area 0
   interface GigabitEthernet0/0/0/0
   !
  !
 !
!
router bgp 111
 vrf CUST1
  rd 1:1
  address-family ipv4 unicast
   network 10.11.111.0/30
   redistribute ospf 1 match internal external nssa-external
  !
 !
!

R14
router ospf 1
 router-id 14.14.14.14
 network 0.0.0.0 255.255.255.255 area 0
!

R5
vrf definition CUST2
 rd 2:22
 route-target export 200:200
 route-target import 200:200
 !
 address-family ipv4
 exit-address-family
!
interface GigabitEthernet1
 vrf forwarding CUST2
 ip address 10.22.222.1 255.255.255.252
!
router bgp 111
 address-family ipv4 vrf CUST2
  redistribute connected
  neighbor 10.22.222.2 remote-as 2
  neighbor 10.22.222.2 activate
  neighbor 10.22.222.2 send-community both
 exit-address-family
!

R13
router bgp 2
 bgp router-id 13.13.13.13
 bgp log-neighbor-changes
 neighbor 10.22.222.1 remote-as 111
 !
 address-family ipv4
  redistribute connected
  neighbor 10.22.222.1 activate
  neighbor 10.22.222.1 send-community both
  neighbor 10.22.222.1 allowas-in
 exit-address-family
!

And we should now be configured ! Lets have a look at R13 and R14 routers, and see what the trace looks like across to their other CE's connected to the other AS:

R13# sh ip route

      10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
B        10.22.22.0/30 [20/0] via 10.22.222.1, 00:06:28
C        10.22.222.0/30 is directly connected, GigabitEthernet0/0
L        10.22.222.2/32 is directly connected, GigabitEthernet0/0
      12.0.0.0/32 is subnetted, 1 subnets
B        12.12.12.12 [20/0] via 10.22.222.1, 00:06:28
      13.0.0.0/32 is subnetted, 1 subnets
C        13.13.13.13 is directly connected, Loopback0

R13# traceroute 12.12.12.12 source lo0
Type escape sequence to abort.
Tracing the route to 13.13.13.13
VRF info: (vrf in name/id, vrf out name/id)
  1 10.22.222.1 5 msec 4 msec 5 msec
  2 10.0.35.1 [MPLS: Labels 24004/23/27 Exp 0] 18 msec 16 msec 17 msec
  3 10.0.39.2 [MPLS: Labels 23/27 Exp 0] 19 msec 44 msec 71 msec
  4 10.0.29.1 [MPLS: Labels 24002/27 Exp 0] 30 msec 16 msec 16 msec
  5 10.22.22.1 [AS 21] [MPLS: Label 27 Exp 0] 16 msec 16 msec 17 msec
  6 10.22.22.2 [AS 21] 18 msec 18 msec *

R14# sh ip route
      10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
O E2     10.11.11.0/30 [110/1] via 10.11.111.1, 00:07:58, GigabitEthernet0/0
C        10.11.111.0/30 is directly connected, GigabitEthernet0/0
L        10.11.111.2/32 is directly connected, GigabitEthernet0/0
        11.0.0.0/32 is subnetted, 1 subnets
O IA     11.11.11.11 [110/2] via 10.11.111.1, 00:07:58, GigabitEthernet0/0
        14.0.0.0/32 is subnetted, 1 subnets
C        14.14.14.14 is directly connected, Loopback0

R14# traceroute 11.11.11.11 so lo0
Type escape sequence to abort.
Tracing the route to 11.11.11.11
VRF info: (vrf in name/id, vrf out name/id)
  1 10.11.111.1 4 msec 3 msec 2 msec
  2 10.0.49.2 [MPLS: Labels 16/24005/24016 Exp 0] 19 msec 17 msec 18 msec
  3 10.0.39.1 [MPLS: Labels 24005/24016 Exp 0] 21 msec 22 msec 19 msec
  4 10.0.103.2 [MPLS: Labels 16/24016 Exp 0] 20 msec 21 msec 81 msec
  5 10.0.110.1 [MPLS: Label 24016 Exp 0] 20 msec 19 msec 21 msec
  6 10.11.11.2 21 msec *  22 msec

Exactly what we were looking for ! At the start of each trace, there are 3 labels. One gets stripped as they cross the core eBGP peering, and then one final label is left to PHP (penultimate hop - second to last) to the destination.

So thats done. We now have a working Cisco based Inter-AS option C design. Feel free to play with this design and config - its great for testing and also learning on OSPF/ISIS databases and peering designs, as well as VPNv4 route traffic and VRFs.

You can download the complete Cisco config HERE

In the final post, we will have a quick look at changing some of the routers from Cisco to Juniper. You can jump to that post HERE.

Part 4 - Cisco and Juniper MPLS L3VPN Lab

In the previous post we completed the Inter-AS design with Cisco.

Stage 1 - Build OSPF/ISIS Areas - COMPLETE
Stage 2 - Configure BGP, Create ASBRs/PEs, use RR via R7/R8 - COMPLETE
Stage 3 - Enable MPLS - COMPLETE
Stage 4 - Create InterAS OptC (multihop vpnv4 ebgp between RR lo) - COMPLETE
Stage 5 - Connect CEs, CUST1=OSPF CUST2=BGP - COMPLETE
Stage 6 - replace some routers with Juniper

In this final post, we will review the config required on the Juniper routers to take part in OSPF, ISIS and MPLS.

Our updated design looks like this, when we replace some of the Cisco routers with Juniper. To experiment, I have replaced R4, R7 and R10. This will give us a Juniper Route Reflector, ASBR and PE.

The core config required for Juniper is as follows (to remove the SRx security features that we don't need):

delete security
set security forwarding-options family inet6 mode packet-based
set security forwarding-options family mpls mode packet-based
set security forwarding-options family iso mode packet-based

Also some of the common commands that are needed (for the non Juniper amoung us) are:

set interfaces ge-0/0/0 unit 0 family inet address 10.0.210.2/30
set interfaces ge-0/0/1 unit 0 family inet address 10.0.110.2/30
set interfaces ge-0/0/2 unit 0 family inet address 10.0.103.2/30
set interfaces ge-0/0/3 unit 0 family inet address 10.0.107.2/30

set interfaces lo0 unit 0 family inet address 10.10.10.10/32

set system host R10

set system root-authentication encrypted-password "$1$Siw1mUfw$jKFGWROP5tRUrUe0idGXp."
[this sets it to Juniper1]

I won't go through all the steps to configure each router, since it is the same process as the Cisco units (just with the Juniper relevant command instead). The following are the configs for each of the devices:

R4
set interfaces ge-0/0/0 unit 0 family inet address 10.0.48.1/30
set interfaces ge-0/0/0 unit 0 family iso
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 10.11.111.1/30
set interfaces ge-0/0/2 unit 0 family inet address 10.0.49.1/30
set interfaces ge-0/0/2 unit 0 family iso
set interfaces ge-0/0/2 unit 0 family mpls
set interfaces ge-0/0/3 unit 0 family inet address 10.0.45.1/30
set interfaces ge-0/0/3 unit 0 family iso
set interfaces ge-0/0/3 unit 0 family mpls
set interfaces lo0 unit 0 family inet address 4.4.4.4/32
set interfaces lo0 unit 0 family iso address 49.1111.0000.0000.0004.00
set routing-options router-id 4.4.4.4
set routing-options autonomous-system 111
set protocols mpls interface lo0.0
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface ge-0/0/2.0
set protocols mpls interface ge-0/0/3.0
set protocols bgp family inet labeled-unicast resolve-vpn
set protocols bgp family inet-vpn unicast
set protocols bgp group as111 type internal
set protocols bgp group as111 local-address 4.4.4.4
set protocols bgp group as111 neighbor 8.8.8.8 family inet labeled-unicast
set protocols bgp group as111 neighbor 8.8.8.8 family inet-vpn unicast
set protocols bgp group as111 neighbor 8.8.8.8 export next-hop-self
set protocols isis level 2 wide-metrics-only
set protocols isis interface all ldp-synchronization
set protocols isis interface all level 1 disable
set protocols ldp interface all
set policy-options policy-statement bgp-to-ospf term 1 from protocol bgp
set policy-options policy-statement bgp-to-ospf term 1 then accept
set policy-options policy-statement bgp-to-ospf term 2 then reject
set policy-options policy-statement next-hop-self then next-hop self
set policy-options policy-statement vpnexport term 1 from protocol ospf
set policy-options policy-statement vpnexport term 1 then community add cust1_comm
set policy-options policy-statement vpnexport term 1 then accept
set policy-options policy-statement vpnexport term 2 then reject
set policy-options policy-statement vpnimport term 1 from protocol bgp
set policy-options policy-statement vpnimport term 1 from community cust1_comm
set policy-options policy-statement vpnimport term 1 then accept
set policy-options policy-statement vpnimport term 2 then reject
set policy-options community cust1_comm members target:100:100
set security forwarding-options family inet6 mode packet-based
set security forwarding-options family mpls mode packet-based
set security forwarding-options family iso mode packet-based
set routing-instances CUST1 instance-type vrf
set routing-instances CUST1 interface ge-0/0/1.0
set routing-instances CUST1 route-distinguisher 1:1
set routing-instances CUST1 vrf-import vpnimport
set routing-instances CUST1 vrf-export vpnexport
set routing-instances CUST1 protocols ospf export bgp-to-ospf
set routing-instances CUST1 protocols ospf area 0.0.0.0 interface ge-0/0/1.0

R7
set interfaces ge-0/0/0 unit 0 family inet address 10.0.107.1/30
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 10.0.17.2/30
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces lo0 unit 0 family inet address 7.7.7.7/32
set routing-options router-id 7.7.7.7
set routing-options autonomous-system 21
set routing-options resolution
set protocols mpls interface lo0.0
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface ge-0/0/1.0
set protocols bgp family inet labeled-unicast resolve-vpn
set protocols bgp group as21 type internal
set protocols bgp group as21 local-address 7.7.7.7
set protocols bgp group as21 family inet labeled-unicast
set protocols bgp group as21 family inet-vpn unicast
set protocols bgp group as21 cluster 7.7.7.7
set protocols bgp group as21 neighbor 2.2.2.2 family inet labeled-unicast
set protocols bgp group as21 neighbor 2.2.2.2 family inet-vpn unicast
set protocols bgp group as21 neighbor 6.6.6.6 family inet labeled-unicast
set protocols bgp group as21 neighbor 6.6.6.6 family inet-vpn unicast
set protocols bgp group as21 neighbor 1.1.1.1 family inet labeled-unicast
set protocols bgp group as21 neighbor 1.1.1.1 family inet-vpn unicast
set protocols bgp group as21 neighbor 10.10.10.10 family inet labeled-unicast
set protocols bgp group as21 neighbor 10.10.10.10 family inet-vpn unicast
set protocols bgp group as111 type external
set protocols bgp group as111 multihop no-nexthop-change
set protocols bgp group as111 local-address 7.7.7.7
set protocols bgp group as111 neighbor 8.8.8.8 multihop
set protocols bgp group as111 neighbor 8.8.8.8 family inet-vpn unicast
set protocols bgp group as111 neighbor 8.8.8.8 peer-as 111
set protocols ospf area 0.0.0.0 interface lo0.0 passive
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface all ldp-synchronization
set protocols ldp interface all
set security forwarding-options family inet6 mode packet-based
set security forwarding-options family mpls mode packet-based
set security forwarding-options family iso mode packet-based

R10
set interfaces ge-0/0/0 unit 0 family inet address 10.0.210.2/30
set interfaces ge-0/0/0 unit 0 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 10.0.110.2/30
set interfaces ge-0/0/1 unit 0 family mpls
set interfaces ge-0/0/2 unit 0 family inet address 10.0.103.2/30
set interfaces ge-0/0/2 unit 0 family mpls
set interfaces ge-0/0/3 unit 0 family inet address 10.0.107.2/30
set interfaces ge-0/0/3 unit 0 family mpls
set interfaces lo0 unit 0 family inet address 10.10.10.10/32
set routing-options router-id 10.10.10.10
set routing-options autonomous-system 21
set protocols mpls interface lo0.0
set protocols mpls interface ge-0/0/0.0
set protocols mpls interface ge-0/0/2.0
set protocols mpls interface ge-0/0/3.0
set protocols mpls interface ge-0/0/1.0
set protocols bgp family inet labeled-unicast
set protocols bgp group as21 type internal
set protocols bgp group as21 local-address 10.10.10.10
set protocols bgp group as21 neighbor 7.7.7.7 family inet labeled-unicast
set protocols bgp group as21 neighbor 7.7.7.7 export next-hop-self
set protocols bgp group ebgp type external
set protocols bgp group ebgp local-address 10.0.103.2
set protocols bgp group ebgp neighbor 10.0.103.1 family inet labeled-unicast
set protocols bgp group ebgp neighbor 10.0.103.1 export TO-AS111
set protocols bgp group ebgp neighbor 10.0.103.1 peer-as 111
set protocols ospf area 0.0.0.0 interface lo0.0 passive
set protocols ospf area 0.0.0.0 interface ge-0/0/0.0
set protocols ospf area 0.0.0.0 interface ge-0/0/1.0
set protocols ospf area 0.0.0.0 interface ge-0/0/3.0
set protocols ospf area 0.0.0.0 interface all ldp-synchronization
set protocols ldp interface all
set policy-options policy-statement TO-AS111 term 1 from route-filter 1.1.1.1/32 exact
set policy-options policy-statement TO-AS111 term 1 from route-filter 2.2.2.2/32 exact
set policy-options policy-statement TO-AS111 term 1 from route-filter 6.6.6.6/32 exact
set policy-options policy-statement TO-AS111 term 1 from route-filter 7.7.7.7/32 exact
set policy-options policy-statement TO-AS111 term 1 from route-filter 10.10.10.10/32 exact
set policy-options policy-statement TO-AS111 term 1 then accept
set policy-options policy-statement next-hop-self then next-hop self
set security forwarding-options family inet6 mode packet-based
set security forwarding-options family mpls mode packet-based
set security forwarding-options family iso mode packet-based

Once configured, you should find you have MPLS labels, routes and IGP/BGP peering up as per Cisco. A completed Inter-AS design, with Cisco and Juniper happily working together !

You can download the Juniper (only) config file HERE

Thanks for reading, and I wish you good learning in your journey towards CCIE (or any form of service provider knowledge). Feel free to leave a post or send me an email with any comments or questions !

CCIE Enterprise - LAB Equipment

Cisco have released the LAB equipment list for the updated CCIE Routing and Switching (to become CCIE Enterprise Infrastructure). You can't suggest they are not heading to SDN now !

https://learningnetwork.cisco.com/docs/DOC-36509

Virtual machines

• Cisco CSR 1000v Series Cloud Services Routers with Cisco IOS XE SD-WAN Release 16.12
• Cisco IOSv with Cisco IOS Software Release 15.8
• Cisco IOSv-L2 with Cisco IOS Software Release 15.2
• Cisco SD-WAN (vManage, vBond, vSmart, vEdge) Software Release 18.4

Physical Equipment

• Cisco Catalyst 9300 Series Switches Release 16.12
• Cisco DNA Center Release 1.4

Supporting virtual machines

• Cisco Identity Services Engine 2.6
• Microsoft Windows 10 Professional
• Ubuntu Desktop 18.04 LTS

This is going to be a very interesting test for those who take it as of late February 2020 - with a strong knowledge of Viptela and ISE (NAC) needed on top of previous CCIE RS knowledge.

The future of engineering - network engineers or software engineers ? Cisco SDx

With the constant push towards a software defined world, a question I get asked a lot is what do engineers need to focus on understanding and being skilled at in the current jobs and business market.

For a long time the network engineer was all that was required, and developers stayed within the boundaries of application development. But SDN (software defined networks) means the ability to code and program is now required, so network engineers are no longer needed right ? I would suggest the answer is no ! Let's see how Cisco has handled this recently, being one of the biggest network engineering development houses in the world (yes I'm thinking about my CCIE and its modern value !) with a review of the Cisco Live 2019 Melbourne session on Lessons Learnt from SDx.


At the start of 2017, Cisco decided to make a change. They had a stack of great network engineers, and were really starting to bring in large teams of software engineers as program-ability and SDN was really becoming key in the market.

They asked themselves - what do we need of the network engineers in the team ? This slide shows the different components that they come up with to define what a modern engineer needs to have skills across (its a big list!)

This is a good list of skills to have, and highlights the drive to the SDN world and away from hardware and CLI based networking.

Now Cisco needed to work out how to bring both software and network engineers together into the same SDx engineer. How?

• For the software engineers - give them network engineer training.
• For the network engineers - give them software training.

Easy right ? Sounds simple... Here's the process they then embarked upon:

And from all reports, this engineering change over the last 2 years is now working well across many parts of the Cisco business. All future engineers who join Cisco will be expected to join as SDx engineers, rather than software or hardware only roles.

One of the interesting requirements for the existing engineers training - was that any time they work on an environment with more than 3 routers - they are required to use automation and programming to configure them. Its a great training method ! Force people to have no other way to achieve the result, and it will drive them to learn whats needed.

So Cisco has chosen this path towards SDN, and I know of other large corporate's that are also heavily moving this way, with in reality it being the best way to future proof engineers and make sure that the exiting tech and new tech can be combined and designed/build/supported within the same teams.

So, anyone up for a python crash course ?

This topic was presented at Cisco Live! Melbourne 2019, as:
Transforming Infrastructure to SDx (ITMGEN-2132)

Cisco to change the certification progam

Recently Cisco has announced changes to their certification program - from the CCNA level upwards. Are these good changes ? I think so. Here are the highlights from the recent announcement:

1. CCNA and CCNP certification will NOT require you to have done any previous level exams to sit and complete them. This is a step in the right direction - if you are good enough to go straight to the CCNP study and learning level, you don't have to wade through the the more basic CCNA level knowledge that you may already have.
2. There is a new DevNet certification that has arrived (finally !). This is much more in line with future trends in the network engineering industry, with DevNet skills becoming a standard requirement of all engineers across teams. Cisco did a few really good sessions at Cisco Live Melbourne this year, where they detailed the 2 year Cisco journey for their internal engineers to move them to DevNet skilled engineers, rather than Network or Software engineers being a separate team member. DevNet can be done at the CCNA and CCNP levels at present.
3. Cisco has also introduced a specialist level certification. This is an interesting one and aligns with Juniper that has the JNCIA, JNCIS and JNCIP levels before the JNCIE. It looks like Cisco will use this to align with their partner certifications and for non standard tracks, as well as their normal certifications.
4. CCIE will now have a lifetime tenure for 20yr members who maintain for the whole time (im a while away form that one...).
5. Routing & Switching is now getting a re-brand to Enterprise Infrastructure, and Wireless is similarly being renamed to Enterprise Wireless. These are good changes, to reflect what the actual certification is geared towards.
6. They are also making the validity period 3 years for re-certification. This is great for CCIE's - the 2 year re-cert is well known to be a bit excessive as it stands now.

These changes are due to start in early 2020, so for any current programs that are being undertaken you can continue, as any certifications will be moved across to the new equivalent automatically after this time.
The full Cisco page and further information can be found HERE

Some humour to lighten up the day

Sometimes I just have to laugh...

VMware NSX Lab on ESXi - VCP6-NV Part2

In the previous post, we were able to get the 3x ESXi Nested hosts installed on the ESXi core server. Now we need to get the NSX Manager and Controllers setup.

There are a few steps needed here (well covered elsewhere):

1. Create Cluster
2. Move 3x ESXi VMs to cluster
3. Setup Distributed Switch
4. Make ESXi VMs talk to the dSwitch as their primary network path
5. Configure NTP on the host ESXi and all the ESXi VMs

Once this is completed, the system is ready for the deployment of the NSX Manager. This is an OVA file, so deploy it as a usual VM (to the ESXi core server, not one of the ESXi VMs !). You will have to go through and answer some questions passwords for CLI admin end privilege mode, DNS name and IP (this needs a static IP), DNS settings and NTP server list. Match this to your network and let the install complete.

Once the VM is running, you connect to the server via HTTPS. You will use the password you setup in the config above, and then this will get you in to the web GUI for the NSX Manager.

The 2 sections you need to change here (and won't need to change again) are Manage Appliance settings - for the NTP details, and Manage vCentre Registration - to connect the server to vCentre.

Once both are configured, you can exit the web interface, and go back to vCentre.

To access NSX manager, you need to go to the Networking and Security icon/menu option. Note this may take 1-2 mins to appear as the NSX manager initially talks to the vCentre to setup.

You need to prepare the Cluster for NSX and VXLan (done under the Host Preparation option under Installation and Upgrade)

The next item on the list is the NSX controller install. This is where the second trick came about. You can easily deploy the controller, but as per the way the LAB has been setup so far, it will sit in the Deploying phase until the system times it out and deletes the controller. So it never completes - and after some review I found that this was because any VM within one of the ESXi VMs (as all the controllers will be) does NOT is not able to get network access. I tested this by building a CentOS VM on one of the ESXi VMs, and I could not get any DHCP. Why ? 

The answer is HERE - Promiscuous Mode & Forged Transmits are required to be enabled on the dSwitch ! Makes complete sense when it is detailed like William Lam does, but of course you don't think about that when just creating the lab.

So make the required changes to the dSwitch, deploy the controller, and it will succeed. It took a while on my machine to be in an CONNECTED state (up to 10m) as I think it is looking for DNS resolution of the vSphere hosts names (which I don't have on my network), but in the end the controllers come up fine (note you can only build one at a time).

 

There is a recommended sequence for booting up the devices listed from VMware HERE. So it goes:

1. ESXi Host w/ vCenter Server
2. NSX Manager
3. ESXi VMs with the controllers (autostart should be on for the controllers)
4. Anything else

A strange issue that I had was that after the reboot of the ESXi server, the NSX manager would not show up in the security console of the Web Client (it said "No NSX Managers" available). This turned out to be a session issue, where the previous NSX session was still present. So to fix this - you have to close all the existing sessions and logout/login to the web client as per HERE. See image below.

Another issue that happened after a reboot was that once all was online and working, the NSX cluster would show as Not Ready (with no notes under it as to why the issue), and clicking Resolve would not change the status.
Some searching found THIS article (scroll to last comment), which suggested a reboot of the vCenter appliance. Completed that, and all green ! See original error screen that was encountered below. Some other guides can be found HERE as well on the issue of the NSX Cluster being Not Ready.

Now we are ready to LAB NSX SDN ! Looking forward to this part.

EDIT Feb 2019 - Having passed VCP6-NV, I have continued to work with this lab in different configurations. A few more notes came to be useful !
1. ESXi 6.5 GA is NOT supported by NSX 6.3 - you need to upgrade to 6.5 U1 or 6.7
2. For the distributed switch you need 2 NICs on each ESXi host (basically one for standard and one for distributed). Lots of trial and error here as I was playing with moving form the standard switch to the distributed version.

VMware NSX Lab on ESXi - VCP6-NV Part1

I'm presently working towards the VMware VCP6-NV certification, and as part of the learning process I wanted to setup the NSX on my home lab for some hands on experience. NSX is VMware's Software Defined Networking product (SDN), that will allow you to build virtual networks on demand across a physical underlying vSphere infrastructure. As shown above, it allows you to create a virtual, segmented environment on demand, without the need of adding or changing any hardware or physical networking to build your networks as required.

The purpose of these posts on NSX is to show how to set it all up on a single ESXi host, and some of the tricks you need to know to make it work that are not covered in the setup guides. The actual standard NSX setup steps that I followed are detailed in VMware NSX for vSphere Introduction and Installation and VMware NSX Cookbook (listed below), so to get a detailed instruction process, please refer to them.

These posts on NSX assume you have done the vSphere6 fundamentals course (or have a good working knowledge of vSphere). Most of the guides around are based on using VMware Workstation (which is fine depending on what hardware you have access to) which I wont try to repeat, but my lab is the following:

• Single ESXi 6.7 host
• 32GB Ram
• Intel Core i7
• 256SSD + 1TB Sata
• 2x Intel NIC

I originally built this server since I prefer to have VMs running on a separate machine to the one im working on. It got me through CCIE SP, so its gotta be able to handle NSX right ? Well as usual there are a few tricks to learn.

For reference, the following is the main guides/series I am using for study -

• VMware NSX Cookbook PDF (Bayu Wibowo, Tony Sangha)
• VCP6-NV Official Cert Guide 2V0-641 PDF (Elver Sena Sosa)
• VMware Certified Professional 6-Network Virtualization 2v0-641 Videos (Lynda - Bill Ferguson)
• VMware NSX for vSphere Introduction and Installation (Pluralsight- Jason Nash)
• VMWare VCP-NV NSX vBrownbag Podcast series Videos

After doing some of the videos and reading the docs, the design in my lab I want to build is the following (including version details) - 

• 3x ESXi hosts - v6.7 (to host the NSX controllers - 1 in each ESXi)
• 1x vSphere vCentre server - v6.7
• 1x NSX Manager - v6.4.3 (note I tested v6.4.4 but had install issues)
• 3x NSX Controllers - v6.4.3 (inherited from NSX manager)

Which should then allow me to build an NSX environment and test.

A note on the interface for vCenter - the HTML5 UI is not full featured yet, so some things (e.g. the NSX logical switches menu option - which is kind of key !)  may not work or be present, so use the FLEX client if you find you can't locate something you thought should be there (get to it via https://[vcentre_ip or DNS]/vsphere-client). The images I have in these posts are from the HTML5 interface.

I setup the vCentre server first, then spun up the ESXi hosts. Just make sure when you create ESXi hosts that you chose OS as OTHER and scroll down to the bottom of the list to select ESXi 6.5 (or whatever ESXi version you are using). You will get a warning that esxi is not supported, but that's fine as this is not production.

This is where the first tricks became apparent. Whilst I learnt this the long way AFTER installing esxi on to esxi and finding issues later, ill just point them out here to make the process easier. This action of installing esxi on top of esxi is called NESTED ESXi. Once you start doing some google searching on this, you will find some secrets (especially with v6.7).

1. Boot mode must be EFI, Not BIOS
2. The CPU virtual architecture needs to be enabled on the esxi VM, otherwise you can't create new VMs within the ESXi VM.

See below for images of where the options are in the vCentre server new VM setup.

Its also worth looking in to the requirements of the ESXi server hardware required to support the  NSX controllers. This can be accessed via the VMware website HERE, and the relevant content is in this table:

As you can see, each esxi server will need to be able to provide 4 vCPU, 28GB and 4GB ram to each controller to work. After some trial and error I found that I actually needed 10-12 GB of ram on each esxi to then be able to install 1 controller (along with 4vCPU and 40GB+ HDD), so it may be worth making that 12GB and 60GB-80GB depending on how much space you have. The controllers don't use much in the way of resources once installed:  approx. 2GB RAM and 1-2 vCPUs, but you have to get past the install part first ! 

Note that the underlying ESXi host will manage the RAM for you so even though you don't have 36GB (3x 12GB for the 3xESXi VMs), they will install and work fine.

Let the install for the Nested ESXi complete from there complete (and you need 3 hosts for the lab - so create a template and copy or build 2 more VMs the same). It is a good idea to set the management ports to static IPs in your LAN for easy access to the working hosts once they have completed setup (via the DCUI).

In the next post we'll look at the NSX install and the distributed switch requirements.