Monday, 9 March 2020

AWS to Meraki VPN Tunnel with Python Part3


The following screens can be used to check on the VPN status, now that the python app has completed all the steps and built both ends of the VPN Tunnel - AWS and Meraki ! 

In my case the EC2 instance has an IP of 172.31.30.146.

AWS CGW configured:

AWS VGW Configured:


AWS VPN Configured:

Meraki VPN Enabled:

  
Trace from Meraki LAN to EC2 instance - via VPN tunnel (1 hop): 


And as promised - here is the completed APP ready to run (NOTE you will need to update the variables at the top that have [REQUIRED] next to them.

Hope this blog series has been of use - and any questions leave in the box below or send me an email !



AWS to Meraki VPN Tunnel with Python Part2


In the previous post, we went through the initial setup steps for the AWS to Meraki VPN:

# Meraki info step
# -----------------------------------
# 1. Get MX public IP
#
# AWS VPN creation steps
# -----------------------------------
# 1. Create Customer Gateway
# 2. Create Virtual Private Gateway
# 3. Attach VGW to VPC
# 4. Create VPN Connection using CGW + VPG
# 5. View Downloaded config with public IP and password
# 6. Add LAN routes to VPN table
# 7. Propogate VPC subnet into VPG
#
# Meraki VPN Creation step
# ------------------------------------
# 1. read existing non Meraki VPN config
# 2. update config with AWS VPN peer detail

Now we will finish setting up both the AWS and then Meraki VPNs, and make sure they connect and provide private traffic flow from the Meraki LAN to the VPC EC2 instance.


# # 1. create a customer gateway (requires Meraki MX Public IP) #     response = client.create_customer_gateway(         BgpAsn = 65000,         PublicIp = meraki_public_ip,         Type'ipsec.1',         DeviceName = 'MerakiMX',         DryRun=False     )
    print ("\nCustomer Gateway result:")     print(json.dumps(response["CustomerGateway"], sort_keys=Trueindent=4))     print ("Customer GatewayID:", response["CustomerGateway"]["CustomerGatewayId"])     myCGW = response["CustomerGateway"]["CustomerGatewayId"]

This call uses the meraki IP we read from the previous post to create a CGW. BGP ASN is required - but not going to be used since this will be based on simple static routing.

# # 2. Create Virtual Private Gateway #     response = client.create_vpn_gateway(         Type='ipsec.1',     )
    print ("\nVirtual Private Gateway result:")     print(json.dumps(response["VpnGateway"], sort_keys=Trueindent=4))     print ("VGW ID:", response["VpnGateway"]["VpnGatewayId"])     myVGW = response["VpnGateway"]["VpnGatewayId"]

This call creates the VGW, which initially is standalone.


# # 3. Attach VGW to VPC #     response = client.attach_vpn_gateway(         VpcId = myVpc,         VpnGatewayId = myVGW     )          print ("\nAttachement result:")     print(json.dumps(response["VpcAttachment"], sort_keys=Trueindent=4))     print ("Status:", response["VpcAttachment"]["State"])     status = response["VpcAttachment"]["State"]     print ("Waiting for [attached] status (rechecking every 5s)")     while status != "attached":         time.sleep(5)         response = client.describe_vpn_gateways (            VpnGatewayIds=[                myVGW,            ],         ) #        print(json.dumps(response["VpnGateways"], sort_keys=True, indent=4))         for line in response["VpnGateways"]:             for line2 in line["VpcAttachments"]:                 status = line2["State"]                 print ("Status:",status)

Now we attach the VGW to our VPC, and then put a simple 5s times loop in to wait for the status of the attachement to complete (normally takes 15-20s)

# # 4. Create VPN Connection using CGW + VPG # 5. Display VPN config data (public IP/passphrase) #     response = client.create_vpn_connection(         CustomerGatewayId = myCGW,         Type = 'ipsec.1',         VpnGatewayId = myVGW,         Options = {             'StaticRoutesOnly'True         }     )
    myVPN = response["VpnConnection"]["VpnConnectionId"]     print ("VPN ID:", myVPN)     print ("\nVPN Configuration Data (showing VPN0)\n----------------------------------")
    doc = xmltodict.parse(response["VpnConnection"]["CustomerGatewayConfiguration"]) #    print(json.dumps(doc, sort_keys=True, indent=4))     for tunnels in doc["vpn_connection"]["ipsec_tunnel"]:         print ("VPN0 Passphrase:", tunnels["ike"]["pre_shared_key"])         myPassPhrase = tunnels["ike"]["pre_shared_key"]         print ("VPN0 Public IP:", tunnels["vpn_gateway"]["tunnel_outside_address"]["ip_address"])         aws_public_ip = tunnels["vpn_gateway"]["tunnel_outside_address"]["ip_address"]         break

This ties the CGW and VGW together, and creates the VPN connection. We then receive back a response which includes the VPN Gateway config, which is in XML format. Convert this to a dictionary variable with xmltodict to make it easier to read (and reference).


# # 6. Add remote LAN routes to VPN table #     client.create_vpn_connection_route(            DestinationCidrBlock = my_meraki_lan_subnet,            VpnConnectionId = myVPN         )
# # 7. Propogate VPN routes to route table #     client.enable_vgw_route_propagation(         RouteTableId = myRouteID,         GatewayId = myVGW     )

Then we add in our Meraki LAN subnet to the VPN connection created in #5, and connect our new VGW to the route table of the VPC so we can route traffic between the VPC and the VPN.

# # Wait for VPN to be ready to use #     status = response["VpnConnection"]["State"]     print ("\nStatus:", status)     print ("Waiting for [available] status (rechecking every 30s)")     while status != "available":         time.sleep(30)         response = client.describe_vpn_connections(             VpnConnectionIds=[                 myVPN,             ],         )         for line in response["VpnConnections"]:             status = line["State"]             print ("Status:",status)

To finish, we now wait for the VPN to be ready, so we can complete the Meraki VPN setup, with another simple 30s loop status check routine.

Now lets create the VPN tunnel on the Meraki !

# Return the third party VPN peers for an organization #     try:         non_meraki_peers = meraki_client.organizations.get_organization_third_party_vpn_peers(params["organization_id"])         #print (non_meraki_peers)     except APIException as e:         print(e) 
    print ("\n") # # Add the additional peer to the existing third party VPN peers for an organization #     try:         collect = {}         collect['organization_id'] = params["organization_id"]         update_organization_third_party_vpn_peers = UpdateOrganizationThirdPartyVPNPeersModel()         update_organization_third_party_vpn_peers.peers = []
        peer_num=0         # iterate through the existing peers and add to update variable         for line in non_meraki_peers:                update_organization_third_party_vpn_peers.peers.append(PeerModel())             update_organization_third_party_vpn_peers.peers[peer_num].name = line['name']             update_organization_third_party_vpn_peers.peers[peer_num].public_ip = line['publicIp']             if 'remoteId' in line:                 update_organization_third_party_vpn_peers.peers[peer_num].remote_id = line['remoteId']             update_organization_third_party_vpn_peers.peers[peer_num].private_subnets = line['privateSubnets']             update_organization_third_party_vpn_peers.peers[peer_num].ipsec_policies_preset = line['ipsecPoliciesPreset']             update_organization_third_party_vpn_peers.peers[peer_num].secret = line['secret']             update_organization_third_party_vpn_peers.peers[peer_num].network_tags = line['networkTags']             peer_num +=1
        # add new aws peer to variable             update_organization_third_party_vpn_peers.peers.append(PeerModel())         update_organization_third_party_vpn_peers.peers[peer_num].name = 'AWS Gateway'         update_organization_third_party_vpn_peers.peers[peer_num].public_ip = aws_public_ip         # note remote_id missing from meraki_sdk file - need to update
\python3\Lib\site-packages\meraki_sdk\models\peer_model.py
        update_organization_third_party_vpn_peers.peers[peer_num].remote_id = 'ec2@aws.com.au'         update_organization_third_party_vpn_peers.peers[peer_num].private_subnets = my_aws_lan_subnet         update_organization_third_party_vpn_peers.peers[peer_num].ipsec_policies_preset = 'aws'

        update_organization_third_party_vpn_peers.peers[peer_num].secret = myPassPhrase         update_organization_third_party_vpn_peers.peers[peer_num].network_tags = ['all']
        collect['update_organization_third_party_vpn_peers'] = update_organization_third_party_vpn_peers
        result = meraki_client.organizations.update_organization_third_party_vpn_peers(collect)         print("MX Non-Meraki VPN peers:\n", json.dumps(result, sort_keys=Trueindent=4))

This final routine reads in the existing non-Meraki VPN config (under Security->Site to Site VPN), and then adds in the new AWS VPN to the config, and writes it back to the dashboard. As per the GUI guide from the first page, Meraki provides an AWS VPN config, so that you don't need to deep dive on ipsec parameters and work out what should be configured to connect to AWS ipsec VPN  correctly (thus the ipsec policy = aws).

And we are complete ! AWS VPN is up, Meraki VPN is up, and we should be able to ping/ssh/rdp from the Meraki LAN to the VPN EC2 instance via the private IP.

In the final post, we will have a look at the status output and Ill provide a copy of the completed application to run.


AWS to Meraki VPN Tunnel with Python Part1



One if the great features of both AWS and Meraki is the support for third party IPSec VPNs. These VPN tunnels give you a secure way of communicating with your services within AWS or your devices and LAN behind the Meraki MX via private IP, but only needing internet access to create the tunnel. This AWS VPN can be used in a commercial environment, to provide private access to VPC instances and beyond from on premise or from the DC, and can also be a backup link behind a Direct Connect link for use in the case of failure of the Direct Connect service.

In my previous post, I showed how to create a Python application that managed an EC2 instance (to start/stop it) via AWS API calls. For this post series, I am going to build a VPN tunnel from my Meraki MX network to my AWS VPC via API calls to both platforms, so I can connect to the EC2 instance via private IP (in my VPC's case - to the 172.16.x.x range) rather than having to come in to a public IP.

The steps in this application are taken from the GUI visual setup detailed in this ritcsec post. Its assumed for our code that the VPC is setup, a subnet is created, you have built an EC2 instance (so we can ping it and ssh/rdp to it via the private IP across the tunnel) and have turned ON VPN participation for the LAN subnet(s) on the Meraki (Security-Site to Site-VPN Settings). 

You also need to configure Tags for the VPC and route table as per below, so that the application can find the resources when it searches the list.

The following code uses a few python modules, which need to be installed if not done already:

$ pip install meraki_sdk xmltodict boto3

The following sequence outlines the steps we will take to build this app, and successfully create a VPN tunnel from the Meraki MX to the AWS VPC. The finished app can be downloaded at the end of the last post, for those who want to skip ahead !

# Meraki info step
# -----------------------------------
# 1. Get MX public IP
#
# AWS VPN creation steps
# -----------------------------------
# 1. Create Customer Gateway
# 2. Create Virtual Private Gateway
# 3. Attach VGW to VPC
# 4. Create VPN Connection using CGW + VPG
# 5. View Downloaded config with public IP and password
# 6. Add LAN routes to VPN table
# 7. Propogate VPC subnet into VPG
#
# Meraki VPN Creation step
# ------------------------------------
# 1. read existing non Meraki VPN config
# 2. update config with AWS VPN peer detail

Firstly, we will import the required modules and setup the global variables to be used within the subroutines:

from meraki_sdk.meraki_sdk_client import MerakiSdkClient from meraki_sdk.exceptions.api_exception import APIException from meraki_sdk.models.update_organization_third_party_vpn_peers_model import UpdateOrganizationThirdPartyVPNPeersModel from meraki_sdk.models.peer_model import PeerModel
import boto3 import json import time import sys import xmltodict import re
meraki_public_ip = "1.1.1.1" aws_public_ip = "2.2.2.2" my_meraki_lan_subnet = "192.168.1.0/24" # Meraki LAN subnet - must be advertised
 to VPN on Meraki dashboard (Security->Site-to-Site VPN) [REQUIRED]
org_name = "My ORG NAME"                # Meraki ORG we want to work in [REQUIRED] network_name = "My Network"             # ORG network we want to configure [REQUIRED] x_cisco_meraki_api_key = 'XXX' #  [REQUIRED] vpcTag = "MyVPC"                        # Name tag for my VPC table [REQUIRED] myRegion = "us-east-1" myRouteTable = "MyLanRoute"             # Name tag for my AWS route table [REQUIRED] myPassPhrase = "password" my_aws_lan_subnet = ['172.31.16.0/20']  # AWS VPC subnet (VPC->subnets) [REQUIRED]
params = {}


The few items that are key here are:
my_meraki_lan_subnet: this is obtained from the Meraki dashboard, and shows the LAN subnet(s) you have in the Meraki network
org_name: this is the name of your Meraki ORG that has the MX in it
network_name: this is the name of the network in your Meraki ORG that has the MX in it
x_cisco_meraki_api_key: this is the API key of a Meraki dahsboard ADMIN
vpcTag and myRouteTable: these are the tags configured on the VPC and Route table in AWS, which are used to match by the search engine when it looks for them in the list
my_aws_lan_subnet: is the subnet used within the VPC for instances

These variables need to be pre-populated before we run the application. How to set them up can be found in the GUI walkthorugh listed above.

The Meraki API documentation can be found HERE.

Next we are ready to run the main application call, which will check on the org_name being valid, and then run the 3 subroutines:

# Meraki info step [meraki_read()]
# AWS VPN creation steps [aws_vpn()]
# Meraki VPN Creation step [meraki_vpn()]

 meraki_client = MerakiSdkClient(x_cisco_meraki_api_key) # # get Meraki organisation list #     try:         print ("\n--- Meraki to AWS VPN Application ---\n")         orgs = meraki_client.organizations.get_organizations()         for organisation in orgs:             print ("ORG ID and Name:", organisation["id"], organisation["name"])             if organisation["name"] == org_name:                 params["organization_id"] = organisation["id"]     except APIException as e:         print(e)         sys.exit(2)     if is_empty(params):         print (org_name, "NOT found in list")         exit()
    meraki_read()     aws_vpn()          meraki_vpn()

We now have the ORG ID, so can continue to the dashboard and get the network ID and device details (specifically the public IP of the MX unit - needed to setup the AWS VPN)
# # get network list in org #     params["network_id"] = ""     try:         print ("AP: networks.get_organization_networks. ORG:", org_name)         nets = meraki_client.networks.get_organization_networks(params)         for network in nets:             print(network["id"], network["name"])             if network["name"] == network_name:                 params["network_id"] = network["id"]     except APIException as e:         print(e)
    if is_empty(params["network_id"]):         print (network_name, "NOT found in list")         exit()
# # get device list in network #     try:         devices = meraki_client.devices.get_network_devices(params["network_id"])         for items in devices:             print (items["model"], items["lanIp"], items["firmware"], items["serial"])             if re.match ("^MX", items["model"]):                 network_mx = {'serial': items["serial"], 'model': items["model"]}     except APIException as e:         print(e)
    print ("\n") # # Get public IP #     try:         collect = {}         collect['network_id'] = params["network_id"]         collect['serial'] = network_mx["serial"]
        mx_uplink = meraki_client.devices.get_network_device_uplink(collect)         print (network_mx["model"], "Public IPs (WAN1<>WAN2):", mx_uplink[0]['publicIp'], "<>", mx_uplink[1]['publicIp'])     except APIException as e:         print(e)
    meraki_public_ip = mx_uplink[0]['publicIp']

This routine uses the Merkai API calls via the meraki_sdk to get the Public IP. The variables that are returned from the Meraki calls are normally lists or dictionaries, so we just need to iterate through them to get the relevant value we are looking for. You can do a print (variable) if you want to get a full look at all the returned values, or better still - use json.dumps to display it in an easy to ready format with:

# print(json.dumps(response, sort_keys=True, indent=4))

Now that we have the meraki client setup and the public IP obtained for the MX network, we can move on to the steps for the AWS VPN creation.

This code uses the boto3 module, which provides AWSCLI functions via the python application.

The AWS boto3 documentation can be found HERE.

First we need to seetup the AWS Client. Here we define a region, and also use the profile of 'james' which has the API authentication setup for AWS to my VPC. I do this so I can have multiple profiles, and call them in the application depending on which VPC I am working in (details on this can be found in the previous EC2 post).
   boto3.setup_default_session(profile_name='james')     ec2 = boto3.resource('ec2'region_name=myRegion)     client = boto3.client('ec2'region_name=myRegion)
    filters = [{'Name':'tag:Name''Values':[vpcTag]}]     vpcs = list(ec2.vpcs.filter(Filters=filters))     for vpc in vpcs:         myVpc = vpc.id         #response = client.describe_vpcs(VpcIds=[vpc.id,])         #print(json.dumps(response, sort_keys=True, indent=4))     print ("\nVPC ID:", myVpc)
    response = client.describe_route_tables()     for routes in response["RouteTables"]:         for myTags in routes["Tags"]:             if myTags["Key"] == "Name" and myTags["Value"] == myRouteTable:                 myRouteID = routes["RouteTableId"]         print ("Route Table ID: ", myRouteID)

Here we now have the boto3 client enabled, as well as the ID of the VPC and  the route table to be used with the VPN

In the next post, we will create the AWS VPN, and then create the Meraki VPN, and confirm they are setup and sending traffic via the private VPN !