CCIE Enterprise - LAB Equipment

Cisco have released the LAB equipment list for the updated CCIE Routing and Switching (to become CCIE Enterprise Infrastructure). You can't suggest they are not heading to SDN now !

https://learningnetwork.cisco.com/docs/DOC-36509

Virtual machines

• Cisco CSR 1000v Series Cloud Services Routers with Cisco IOS XE SD-WAN Release 16.12
• Cisco IOSv with Cisco IOS Software Release 15.8
• Cisco IOSv-L2 with Cisco IOS Software Release 15.2
• Cisco SD-WAN (vManage, vBond, vSmart, vEdge) Software Release 18.4

Physical Equipment

• Cisco Catalyst 9300 Series Switches Release 16.12
• Cisco DNA Center Release 1.4

Supporting virtual machines

• Cisco Identity Services Engine 2.6
• Microsoft Windows 10 Professional
• Ubuntu Desktop 18.04 LTS

This is going to be a very interesting test for those who take it as of late February 2020 - with a strong knowledge of Viptela and ISE (NAC) needed on top of previous CCIE RS knowledge.

The future of engineering - network engineers or software engineers ? Cisco SDx

With the constant push towards a software defined world, a question I get asked a lot is what do engineers need to focus on understanding and being skilled at in the current jobs and business market.

For a long time the network engineer was all that was required, and developers stayed within the boundaries of application development. But SDN (software defined networks) means the ability to code and program is now required, so network engineers are no longer needed right ? I would suggest the answer is no ! Let's see how Cisco has handled this recently, being one of the biggest network engineering development houses in the world (yes I'm thinking about my CCIE and its modern value !) with a review of the Cisco Live 2019 Melbourne session on Lessons Learnt from SDx.


At the start of 2017, Cisco decided to make a change. They had a stack of great network engineers, and were really starting to bring in large teams of software engineers as program-ability and SDN was really becoming key in the market.

They asked themselves - what do we need of the network engineers in the team ? This slide shows the different components that they come up with to define what a modern engineer needs to have skills across (its a big list!)

This is a good list of skills to have, and highlights the drive to the SDN world and away from hardware and CLI based networking.

Now Cisco needed to work out how to bring both software and network engineers together into the same SDx engineer. How?

• For the software engineers - give them network engineer training.
• For the network engineers - give them software training.

Easy right ? Sounds simple... Here's the process they then embarked upon:

And from all reports, this engineering change over the last 2 years is now working well across many parts of the Cisco business. All future engineers who join Cisco will be expected to join as SDx engineers, rather than software or hardware only roles.

One of the interesting requirements for the existing engineers training - was that any time they work on an environment with more than 3 routers - they are required to use automation and programming to configure them. Its a great training method ! Force people to have no other way to achieve the result, and it will drive them to learn whats needed.

So Cisco has chosen this path towards SDN, and I know of other large corporate's that are also heavily moving this way, with in reality it being the best way to future proof engineers and make sure that the exiting tech and new tech can be combined and designed/build/supported within the same teams.

So, anyone up for a python crash course ?

This topic was presented at Cisco Live! Melbourne 2019, as:
Transforming Infrastructure to SDx (ITMGEN-2132)

Cisco to change the certification progam

Recently Cisco has announced changes to their certification program - from the CCNA level upwards. Are these good changes ? I think so. Here are the highlights from the recent announcement:

1. CCNA and CCNP certification will NOT require you to have done any previous level exams to sit and complete them. This is a step in the right direction - if you are good enough to go straight to the CCNP study and learning level, you don't have to wade through the the more basic CCNA level knowledge that you may already have.
2. There is a new DevNet certification that has arrived (finally !). This is much more in line with future trends in the network engineering industry, with DevNet skills becoming a standard requirement of all engineers across teams. Cisco did a few really good sessions at Cisco Live Melbourne this year, where they detailed the 2 year Cisco journey for their internal engineers to move them to DevNet skilled engineers, rather than Network or Software engineers being a separate team member. DevNet can be done at the CCNA and CCNP levels at present.
3. Cisco has also introduced a specialist level certification. This is an interesting one and aligns with Juniper that has the JNCIA, JNCIS and JNCIP levels before the JNCIE. It looks like Cisco will use this to align with their partner certifications and for non standard tracks, as well as their normal certifications.
4. CCIE will now have a lifetime tenure for 20yr members who maintain for the whole time (im a while away form that one...).
5. Routing & Switching is now getting a re-brand to Enterprise Infrastructure, and Wireless is similarly being renamed to Enterprise Wireless. These are good changes, to reflect what the actual certification is geared towards.
6. They are also making the validity period 3 years for re-certification. This is great for CCIE's - the 2 year re-cert is well known to be a bit excessive as it stands now.

These changes are due to start in early 2020, so for any current programs that are being undertaken you can continue, as any certifications will be moved across to the new equivalent automatically after this time.
The full Cisco page and further information can be found HERE

Some humour to lighten up the day

Sometimes I just have to laugh...

VMware NSX Lab on ESXi - VCP6-NV Part2

In the previous post, we were able to get the 3x ESXi Nested hosts installed on the ESXi core server. Now we need to get the NSX Manager and Controllers setup.

There are a few steps needed here (well covered elsewhere):

1. Create Cluster
2. Move 3x ESXi VMs to cluster
3. Setup Distributed Switch
4. Make ESXi VMs talk to the dSwitch as their primary network path
5. Configure NTP on the host ESXi and all the ESXi VMs

Once this is completed, the system is ready for the deployment of the NSX Manager. This is an OVA file, so deploy it as a usual VM (to the ESXi core server, not one of the ESXi VMs !). You will have to go through and answer some questions passwords for CLI admin end privilege mode, DNS name and IP (this needs a static IP), DNS settings and NTP server list. Match this to your network and let the install complete.

Once the VM is running, you connect to the server via HTTPS. You will use the password you setup in the config above, and then this will get you in to the web GUI for the NSX Manager.

The 2 sections you need to change here (and won't need to change again) are Manage Appliance settings - for the NTP details, and Manage vCentre Registration - to connect the server to vCentre.

Once both are configured, you can exit the web interface, and go back to vCentre.

To access NSX manager, you need to go to the Networking and Security icon/menu option. Note this may take 1-2 mins to appear as the NSX manager initially talks to the vCentre to setup.

You need to prepare the Cluster for NSX and VXLan (done under the Host Preparation option under Installation and Upgrade)

The next item on the list is the NSX controller install. This is where the second trick came about. You can easily deploy the controller, but as per the way the LAB has been setup so far, it will sit in the Deploying phase until the system times it out and deletes the controller. So it never completes - and after some review I found that this was because any VM within one of the ESXi VMs (as all the controllers will be) does NOT is not able to get network access. I tested this by building a CentOS VM on one of the ESXi VMs, and I could not get any DHCP. Why ? 

The answer is HERE - Promiscuous Mode & Forged Transmits are required to be enabled on the dSwitch ! Makes complete sense when it is detailed like William Lam does, but of course you don't think about that when just creating the lab.

So make the required changes to the dSwitch, deploy the controller, and it will succeed. It took a while on my machine to be in an CONNECTED state (up to 10m) as I think it is looking for DNS resolution of the vSphere hosts names (which I don't have on my network), but in the end the controllers come up fine (note you can only build one at a time).

 

There is a recommended sequence for booting up the devices listed from VMware HERE. So it goes:

1. ESXi Host w/ vCenter Server
2. NSX Manager
3. ESXi VMs with the controllers (autostart should be on for the controllers)
4. Anything else

A strange issue that I had was that after the reboot of the ESXi server, the NSX manager would not show up in the security console of the Web Client (it said "No NSX Managers" available). This turned out to be a session issue, where the previous NSX session was still present. So to fix this - you have to close all the existing sessions and logout/login to the web client as per HERE. See image below.

Another issue that happened after a reboot was that once all was online and working, the NSX cluster would show as Not Ready (with no notes under it as to why the issue), and clicking Resolve would not change the status.
Some searching found THIS article (scroll to last comment), which suggested a reboot of the vCenter appliance. Completed that, and all green ! See original error screen that was encountered below. Some other guides can be found HERE as well on the issue of the NSX Cluster being Not Ready.

Now we are ready to LAB NSX SDN ! Looking forward to this part.

EDIT Feb 2019 - Having passed VCP6-NV, I have continued to work with this lab in different configurations. A few more notes came to be useful !
1. ESXi 6.5 GA is NOT supported by NSX 6.3 - you need to upgrade to 6.5 U1 or 6.7
2. For the distributed switch you need 2 NICs on each ESXi host (basically one for standard and one for distributed). Lots of trial and error here as I was playing with moving form the standard switch to the distributed version.

VMware NSX Lab on ESXi - VCP6-NV Part1

I'm presently working towards the VMware VCP6-NV certification, and as part of the learning process I wanted to setup the NSX on my home lab for some hands on experience. NSX is VMware's Software Defined Networking product (SDN), that will allow you to build virtual networks on demand across a physical underlying vSphere infrastructure. As shown above, it allows you to create a virtual, segmented environment on demand, without the need of adding or changing any hardware or physical networking to build your networks as required.

The purpose of these posts on NSX is to show how to set it all up on a single ESXi host, and some of the tricks you need to know to make it work that are not covered in the setup guides. The actual standard NSX setup steps that I followed are detailed in VMware NSX for vSphere Introduction and Installation and VMware NSX Cookbook (listed below), so to get a detailed instruction process, please refer to them.

These posts on NSX assume you have done the vSphere6 fundamentals course (or have a good working knowledge of vSphere). Most of the guides around are based on using VMware Workstation (which is fine depending on what hardware you have access to) which I wont try to repeat, but my lab is the following:

• Single ESXi 6.7 host
• 32GB Ram
• Intel Core i7
• 256SSD + 1TB Sata
• 2x Intel NIC

I originally built this server since I prefer to have VMs running on a separate machine to the one im working on. It got me through CCIE SP, so its gotta be able to handle NSX right ? Well as usual there are a few tricks to learn.

For reference, the following is the main guides/series I am using for study -

• VMware NSX Cookbook PDF (Bayu Wibowo, Tony Sangha)
• VCP6-NV Official Cert Guide 2V0-641 PDF (Elver Sena Sosa)
• VMware Certified Professional 6-Network Virtualization 2v0-641 Videos (Lynda - Bill Ferguson)
• VMware NSX for vSphere Introduction and Installation (Pluralsight- Jason Nash)
• VMWare VCP-NV NSX vBrownbag Podcast series Videos

After doing some of the videos and reading the docs, the design in my lab I want to build is the following (including version details) - 

• 3x ESXi hosts - v6.7 (to host the NSX controllers - 1 in each ESXi)
• 1x vSphere vCentre server - v6.7
• 1x NSX Manager - v6.4.3 (note I tested v6.4.4 but had install issues)
• 3x NSX Controllers - v6.4.3 (inherited from NSX manager)

Which should then allow me to build an NSX environment and test.

A note on the interface for vCenter - the HTML5 UI is not full featured yet, so some things (e.g. the NSX logical switches menu option - which is kind of key !)  may not work or be present, so use the FLEX client if you find you can't locate something you thought should be there (get to it via https://[vcentre_ip or DNS]/vsphere-client). The images I have in these posts are from the HTML5 interface.

I setup the vCentre server first, then spun up the ESXi hosts. Just make sure when you create ESXi hosts that you chose OS as OTHER and scroll down to the bottom of the list to select ESXi 6.5 (or whatever ESXi version you are using). You will get a warning that esxi is not supported, but that's fine as this is not production.

This is where the first tricks became apparent. Whilst I learnt this the long way AFTER installing esxi on to esxi and finding issues later, ill just point them out here to make the process easier. This action of installing esxi on top of esxi is called NESTED ESXi. Once you start doing some google searching on this, you will find some secrets (especially with v6.7).

1. Boot mode must be EFI, Not BIOS
2. The CPU virtual architecture needs to be enabled on the esxi VM, otherwise you can't create new VMs within the ESXi VM.

See below for images of where the options are in the vCentre server new VM setup.

Its also worth looking in to the requirements of the ESXi server hardware required to support the  NSX controllers. This can be accessed via the VMware website HERE, and the relevant content is in this table:

As you can see, each esxi server will need to be able to provide 4 vCPU, 28GB and 4GB ram to each controller to work. After some trial and error I found that I actually needed 10-12 GB of ram on each esxi to then be able to install 1 controller (along with 4vCPU and 40GB+ HDD), so it may be worth making that 12GB and 60GB-80GB depending on how much space you have. The controllers don't use much in the way of resources once installed:  approx. 2GB RAM and 1-2 vCPUs, but you have to get past the install part first ! 

Note that the underlying ESXi host will manage the RAM for you so even though you don't have 36GB (3x 12GB for the 3xESXi VMs), they will install and work fine.

Let the install for the Nested ESXi complete from there complete (and you need 3 hosts for the lab - so create a template and copy or build 2 more VMs the same). It is a good idea to set the management ports to static IPs in your LAN for easy access to the working hosts once they have completed setup (via the DCUI).

In the next post we'll look at the NSX install and the distributed switch requirements.

BGP Route sharing between iBGP and eBGP

 

The  above scenario came up in some BGP review I was doing recently for a certification renewal, and the core rules of BGP need to be remembered to answer the question that was posed. Both AS's shown have full mesh iBPG peerings, and R4-R5 has dual eBGP peerings.

The question that was asked was: 

If R8 generates a route and puts it into BGP (via the network statement), will R4 advertise it to R1 AND will it be visible in the R1 routing table IF R1 receives it ?

So to answer this we need to go back to our core BGP concepts - what will BGP speakers do with routes they receive, and what is required for a BGP route to be installed in the routing table ?

To look at the first part of what will happen to the route from R8 - we look to the BGP route rules:

1. BGP speakers will NOT advertise iBPG learnt routes to iBGP peers

2. BGP speakers WILL advertise eBGP learnt routes to iBGP and eBGP peers

3. BGP speakers WILL advertise iBGP learnt routes to eBGP peers

From here we need to put this into the perspective of the scenario above.

The route starts on R8. From there is goes to R5 - so R5 now has a route that is has learnt from an iBGP peer. 

Based on Point 1 - this route will NOT be advertised back to any of its AS 65512 peer.

Based on Point 3 - this route WILL be advertised to its eBGP peer R4.

So R4 now has an eBGP route from R5 (originated in AS 65512).

What does R4 do with the route ? Based on point 2 - this route will be advertised to all of the iBGP peers of R5 since its an eBGP route. This confirms that R1 WILL get the route in its BGP routing table. So that's the first part of the question answered - YES R1 will get the route (at least to its BGP routing table).

The second part of the question is a bit harder to answer - as we don't have access to the BGP config on R4. Why does that matter ? Because if the peering between R4 and R1 is just a standard iBGP peering with no options added, it will not allow the route to be added to the R1 routing table. Let's review this!

The next-hop rules for BGP now apply to this question:

1. If a route comes in from an eBGP peer, its next-hop is changed to the IP of that eBGP peer

2. If a route comes in from an iBGP peer, its next-hop is unchanged

Here is a standard iBGP peering config from a Cisco IOS-XE router:

router bgp 111
 bgp router-id 9.9.9.9
 bgp log-neighbor-changes
 neighbor 8.8.8.8 remote-as 111
 neighbor 8.8.8.8 update-source Loopback0
 !
 address-family ipv4
  neighbor 8.8.8.8 activate
 exit-address-family

So R4 gets the route from R5 via eBGP and so changes the next-hop for the route to be R5 (as per point 1).

If the above is the config for both R4 and R1, then when R1 gets the route, the next hop of the route will point to R5 (as per point 2 - since the next-hop was not changed when the route was passed via iBGP from R4 - R1), which R1 will not know how to get to (unless its been advertised into AS65511 via another method). 

So if R1 doesn't have the route to the next-hop, as per BGP rules this BGP route cannot be installed in its routing table. 

What would need to be added on R4 to fix this ? A simple way is to change the next hop for the eBGP routes advertised by R4 into its AS, with the following additional command:

 address-family ipv4
  neighbor 8.8.8.8 activate

  neighbor 8.8.8.8 next-hop-self

  exit-address-family

 

This changes the next-hop that all of the iBGP peers in AS65511 see for eBGP routes to be R4's IP, which they of course all know, and this then allows the route to be added to their routing table.

So the answer to the second part of the question is - it depends! A much loved statement of IT consultants...