AWS to Meraki VPN Tunnel with Python Part2

 

In the previous post, we went through the initial setup steps for the AWS to Meraki VPN:

# Meraki info step

# -----------------------------------
# 1. Get MX public IP
#
# AWS VPN creation steps
# -----------------------------------
# 1. Create Customer Gateway
# 2. Create Virtual Private Gateway
# 3. Attach VGW to VPC
# 4. Create VPN Connection using CGW + VPG
# 5. View Downloaded config with public IP and password
# 6. Add LAN routes to VPN table
# 7. Propogate VPC subnet into VPG
#
# Meraki VPN Creation step
# ------------------------------------
# 1. read existing non Meraki VPN config
# 2. update config with AWS VPN peer detail

Now we will finish setting up both the AWS and then Meraki VPNs, and make sure they connect and provide private traffic flow from the Meraki LAN to the VPC EC2 instance.

# # 1. create a customer gateway (requires Meraki MX Public IP) #     response = client.create_customer_gateway(         BgpAsn = 65000,         PublicIp = meraki_public_ip,         Type= 'ipsec.1',         DeviceName = 'MerakiMX',         DryRun=False     )
    print ("\nCustomer Gateway result:")     print(json.dumps(response["CustomerGateway"], sort_keys=True, indent=4))     print ("Customer GatewayID:", response["CustomerGateway"]["CustomerGatewayId"])     myCGW = response["CustomerGateway"]["CustomerGatewayId"]

This call uses the meraki IP we read from the previous post to create a CGW. BGP ASN is required - but not going to be used since this will be based on simple static routing.

# # 2. Create Virtual Private Gateway #     response = client.create_vpn_gateway(         Type='ipsec.1',     )
    print ("\nVirtual Private Gateway result:")     print(json.dumps(response["VpnGateway"], sort_keys=True, indent=4))     print ("VGW ID:", response["VpnGateway"]["VpnGatewayId"])     myVGW = response["VpnGateway"]["VpnGatewayId"]

This call creates the VGW, which initially is standalone.

# # 3. Attach VGW to VPC #     response = client.attach_vpn_gateway(         VpcId = myVpc,         VpnGatewayId = myVGW     )          print ("\nAttachement result:")     print(json.dumps(response["VpcAttachment"], sort_keys=True, indent=4))     print ("Status:", response["VpcAttachment"]["State"])     status = response["VpcAttachment"]["State"]     print ("Waiting for [attached] status (rechecking every 5s)")     while status != "attached":         time.sleep(5)         response = client.describe_vpn_gateways (            VpnGatewayIds=[                myVGW,            ],         ) #        print(json.dumps(response["VpnGateways"], sort_keys=True, indent=4))         for line in response["VpnGateways"]:             for line2 in line["VpcAttachments"]:                 status = line2["State"]                 print ("Status:",status)

Now we attach the VGW to our VPC, and then put a simple 5s times loop in to wait for the status of the attachement to complete (normally takes 15-20s)

# # 4. Create VPN Connection using CGW + VPG # 5. Display VPN config data (public IP/passphrase) #     response = client.create_vpn_connection(         CustomerGatewayId = myCGW,         Type = 'ipsec.1',         VpnGatewayId = myVGW,         Options = {             'StaticRoutesOnly': True         }     )
    myVPN = response["VpnConnection"]["VpnConnectionId"]     print ("VPN ID:", myVPN)     print ("\nVPN Configuration Data (showing VPN0)\n----------------------------------")
    doc = xmltodict.parse(response["VpnConnection"]["CustomerGatewayConfiguration"]) #    print(json.dumps(doc, sort_keys=True, indent=4))     for tunnels in doc["vpn_connection"]["ipsec_tunnel"]:         print ("VPN0 Passphrase:", tunnels["ike"]["pre_shared_key"])         myPassPhrase = tunnels["ike"]["pre_shared_key"]         print ("VPN0 Public IP:", tunnels["vpn_gateway"]["tunnel_outside_address"]["ip_address"])         aws_public_ip = tunnels["vpn_gateway"]["tunnel_outside_address"]["ip_address"]         break

This ties the CGW and VGW together, and creates the VPN connection. We then receive back a response which includes the VPN Gateway config, which is in XML format. Convert this to a dictionary variable with xmltodict to make it easier to read (and reference).

# # 6. Add remote LAN routes to VPN table #     client.create_vpn_connection_route(            DestinationCidrBlock = my_meraki_lan_subnet,            VpnConnectionId = myVPN         )
# # 7. Propogate VPN routes to route table #     client.enable_vgw_route_propagation(         RouteTableId = myRouteID,         GatewayId = myVGW     )

Then we add in our Meraki LAN subnet to the VPN connection created in #5, and connect our new VGW to the route table of the VPC so we can route traffic between the VPC and the VPN.

# # Wait for VPN to be ready to use #     status = response["VpnConnection"]["State"]     print ("\nStatus:", status)     print ("Waiting for [available] status (rechecking every 30s)")     while status != "available":         time.sleep(30)         response = client.describe_vpn_connections(             VpnConnectionIds=[                 myVPN,             ],         )         for line in response["VpnConnections"]:             status = line["State"]             print ("Status:",status)

To finish, we now wait for the VPN to be ready, so we can complete the Meraki VPN setup, with another simple 30s loop status check routine.

Now lets create the VPN tunnel on the Meraki !

# Return the third party VPN peers for an organization #     try:         non_meraki_peers = meraki_client.organizations.get_organization_third_party_vpn_peers(params["organization_id"])         #print (non_meraki_peers)     except APIException as e:         print(e) 
    print ("\n") # # Add the additional peer to the existing third party VPN peers for an organization #     try:         collect = {}         collect['organization_id'] = params["organization_id"]         update_organization_third_party_vpn_peers = UpdateOrganizationThirdPartyVPNPeersModel()         update_organization_third_party_vpn_peers.peers = []
        peer_num=0         # iterate through the existing peers and add to update variable         for line in non_meraki_peers:                update_organization_third_party_vpn_peers.peers.append(PeerModel())             update_organization_third_party_vpn_peers.peers[peer_num].name = line['name']             update_organization_third_party_vpn_peers.peers[peer_num].public_ip = line['publicIp']             if 'remoteId' in line:                 update_organization_third_party_vpn_peers.peers[peer_num].remote_id = line['remoteId']             update_organization_third_party_vpn_peers.peers[peer_num].private_subnets = line['privateSubnets']             update_organization_third_party_vpn_peers.peers[peer_num].ipsec_policies_preset = line['ipsecPoliciesPreset']             update_organization_third_party_vpn_peers.peers[peer_num].secret = line['secret']             update_organization_third_party_vpn_peers.peers[peer_num].network_tags = line['networkTags']             peer_num +=1
        # add new aws peer to variable             update_organization_third_party_vpn_peers.peers.append(PeerModel())         update_organization_third_party_vpn_peers.peers[peer_num].name = 'AWS Gateway'         update_organization_third_party_vpn_peers.peers[peer_num].public_ip = aws_public_ip         # note remote_id missing from meraki_sdk file - need to update
\python3\Lib\site-packages\meraki_sdk\models\peer_model.py         update_organization_third_party_vpn_peers.peers[peer_num].remote_id = 'ec2@aws.com.au'         update_organization_third_party_vpn_peers.peers[peer_num].private_subnets = my_aws_lan_subnet         update_organization_third_party_vpn_peers.peers[peer_num].ipsec_policies_preset = 'aws'

        update_organization_third_party_vpn_peers.peers[peer_num].secret = myPassPhrase         update_organization_third_party_vpn_peers.peers[peer_num].network_tags = ['all']
        collect['update_organization_third_party_vpn_peers'] = update_organization_third_party_vpn_peers
        result = meraki_client.organizations.update_organization_third_party_vpn_peers(collect)         print("MX Non-Meraki VPN peers:\n", json.dumps(result, sort_keys=True, indent=4))

This final routine reads in the existing non-Meraki VPN config (under Security->Site to Site VPN), and then adds in the new AWS VPN to the config, and writes it back to the dashboard. As per the GUI guide from the first page, Meraki provides an AWS VPN config, so that you don't need to deep dive on ipsec parameters and work out what should be configured to connect to AWS ipsec VPN  correctly (thus the ipsec policy = aws).

And we are complete ! AWS VPN is up, Meraki VPN is up, and we should be able to ping/ssh/rdp from the Meraki LAN to the VPN EC2 instance via the private IP.

In the final post, we will have a look at the status output and Ill provide a copy of the completed application to run.

AWS to Meraki VPN Tunnel with Python Part1

 

One if the great features of both AWS and Meraki is the support for third party IPSec VPNs. These VPN tunnels give you a secure way of communicating with your services within AWS or your devices and LAN behind the Meraki MX via private IP, but only needing internet access to create the tunnel. This AWS VPN can be used in a commercial environment, to provide private access to VPC instances and beyond from on premise or from the DC, and can also be a backup link behind a Direct Connect link for use in the case of failure of the Direct Connect service.

In my previous post, I showed how to create a Python application that managed an EC2 instance (to start/stop it) via AWS API calls. For this post series, I am going to build a VPN tunnel from my Meraki MX network to my AWS VPC via API calls to both platforms, so I can connect to the EC2 instance via private IP (in my VPC's case - to the 172.16.x.x range) rather than having to come in to a public IP.

The steps in this application are taken from the GUI visual setup detailed in this ritcsec post. Its assumed for our code that the VPC is setup, a subnet is created, you have built an EC2 instance (so we can ping it and ssh/rdp to it via the private IP across the tunnel) and have turned ON VPN participation for the LAN subnet(s) on the Meraki (Security-Site to Site-VPN Settings). 

You also need to configure Tags for the VPC and route table as per below, so that the application can find the resources when it searches the list.

The following code uses a few python modules, which need to be installed if not done already:

$ pip install meraki_sdk xmltodict boto3

The following sequence outlines the steps we will take to build this app, and successfully create a VPN tunnel from the Meraki MX to the AWS VPC. The finished app can be downloaded at the end of the last post, for those who want to skip ahead !

# Meraki info step

# -----------------------------------
# 1. Get MX public IP
#
# AWS VPN creation steps
# -----------------------------------
# 1. Create Customer Gateway
# 2. Create Virtual Private Gateway
# 3. Attach VGW to VPC
# 4. Create VPN Connection using CGW + VPG
# 5. View Downloaded config with public IP and password
# 6. Add LAN routes to VPN table
# 7. Propogate VPC subnet into VPG
#
# Meraki VPN Creation step
# ------------------------------------
# 1. read existing non Meraki VPN config
# 2. update config with AWS VPN peer detail

Firstly, we will import the required modules and setup the global variables to be used within the subroutines:

from meraki_sdk.meraki_sdk_client import MerakiSdkClient from meraki_sdk.exceptions.api_exception import APIException from meraki_sdk.models.update_organization_third_party_vpn_peers_model import UpdateOrganizationThirdPartyVPNPeersModel from meraki_sdk.models.peer_model import PeerModel
import boto3 import json import time import sys import xmltodict import re
meraki_public_ip = "1.1.1.1" aws_public_ip = "2.2.2.2" my_meraki_lan_subnet = "192.168.1.0/24" # Meraki LAN subnet - must be advertised
 to VPN on Meraki dashboard (Security->Site-to-Site VPN) [REQUIRED] org_name = "My ORG NAME"                # Meraki ORG we want to work in [REQUIRED] network_name = "My Network"             # ORG network we want to configure [REQUIRED] x_cisco_meraki_api_key = 'XXX' #  [REQUIRED] vpcTag = "MyVPC"                        # Name tag for my VPC table [REQUIRED] myRegion = "us-east-1" myRouteTable = "MyLanRoute"             # Name tag for my AWS route table [REQUIRED] myPassPhrase = "password" my_aws_lan_subnet = ['172.31.16.0/20']  # AWS VPC subnet (VPC->subnets) [REQUIRED]
params = {}

The few items that are key here are:
my_meraki_lan_subnet: this is obtained from the Meraki dashboard, and shows the LAN subnet(s) you have in the Meraki network
org_name: this is the name of your Meraki ORG that has the MX in it
network_name: this is the name of the network in your Meraki ORG that has the MX in it
x_cisco_meraki_api_key: this is the API key of a Meraki dahsboard ADMIN
vpcTag and myRouteTable: these are the tags configured on the VPC and Route table in AWS, which are used to match by the search engine when it looks for them in the list
my_aws_lan_subnet: is the subnet used within the VPC for instances

These variables need to be pre-populated before we run the application. How to set them up can be found in the GUI walkthorugh listed above.

The Meraki API documentation can be found HERE.

Next we are ready to run the main application call, which will check on the org_name being valid, and then run the 3 subroutines:

# Meraki info step [meraki_read()]

# AWS VPN creation steps [aws_vpn()]

# Meraki VPN Creation step [meraki_vpn()]

 meraki_client = MerakiSdkClient(x_cisco_meraki_api_key) # # get Meraki organisation list #     try:         print ("\n--- Meraki to AWS VPN Application ---\n")         orgs = meraki_client.organizations.get_organizations()         for organisation in orgs:             print ("ORG ID and Name:", organisation["id"], organisation["name"])             if organisation["name"] == org_name:                 params["organization_id"] = organisation["id"]     except APIException as e:         print(e)         sys.exit(2)     if is_empty(params):         print (org_name, "NOT found in list")         exit()
    meraki_read()     aws_vpn()          meraki_vpn()

We now have the ORG ID, so can continue to the dashboard and get the network ID and device details (specifically the public IP of the MX unit - needed to setup the AWS VPN)

# # get network list in org #     params["network_id"] = ""     try:         print ("AP: networks.get_organization_networks. ORG:", org_name)         nets = meraki_client.networks.get_organization_networks(params)         for network in nets:             print(network["id"], network["name"])             if network["name"] == network_name:                 params["network_id"] = network["id"]     except APIException as e:         print(e)
    if is_empty(params["network_id"]):         print (network_name, "NOT found in list")         exit()
# # get device list in network #     try:         devices = meraki_client.devices.get_network_devices(params["network_id"])         for items in devices:             print (items["model"], items["lanIp"], items["firmware"], items["serial"])             if re.match ("^MX", items["model"]):                 network_mx = {'serial': items["serial"], 'model': items["model"]}     except APIException as e:         print(e)
    print ("\n") # # Get public IP #     try:         collect = {}         collect['network_id'] = params["network_id"]         collect['serial'] = network_mx["serial"]

        mx_uplink = meraki_client.devices.get_network_device_uplink(collect)         print (network_mx["model"], "Public IPs (WAN1<>WAN2):", mx_uplink[0]['publicIp'], "<>", mx_uplink[1]['publicIp'])     except APIException as e:         print(e)
    meraki_public_ip = mx_uplink[0]['publicIp']

This routine uses the Merkai API calls via the meraki_sdk to get the Public IP. The variables that are returned from the Meraki calls are normally lists or dictionaries, so we just need to iterate through them to get the relevant value we are looking for. You can do a print (variable) if you want to get a full look at all the returned values, or better still - use json.dumps to display it in an easy to ready format with:

# print(json.dumps(response, sort_keys=True, indent=4))

Now that we have the meraki client setup and the public IP obtained for the MX network, we can move on to the steps for the AWS VPN creation.

This code uses the boto3 module, which provides AWSCLI functions via the python application.

The AWS boto3 documentation can be found HERE.

First we need to seetup the AWS Client. Here we define a region, and also use the profile of 'james' which has the API authentication setup for AWS to my VPC. I do this so I can have multiple profiles, and call them in the application depending on which VPC I am working in (details on this can be found in the previous EC2 post).

   boto3.setup_default_session(profile_name='james')     ec2 = boto3.resource('ec2', region_name=myRegion)     client = boto3.client('ec2', region_name=myRegion)
    filters = [{'Name':'tag:Name', 'Values':[vpcTag]}]     vpcs = list(ec2.vpcs.filter(Filters=filters))     for vpc in vpcs:         myVpc = vpc.id         #response = client.describe_vpcs(VpcIds=[vpc.id,])         #print(json.dumps(response, sort_keys=True, indent=4))     print ("\nVPC ID:", myVpc)
    response = client.describe_route_tables()     for routes in response["RouteTables"]:         for myTags in routes["Tags"]:             if myTags["Key"] == "Name" and myTags["Value"] == myRouteTable:                 myRouteID = routes["RouteTableId"]         print ("Route Table ID: ", myRouteID)

Here we now have the boto3 client enabled, as well as the ID of the VPC and  the route table to be used with the VPN

In the next post, we will create the AWS VPN, and then create the Meraki VPN, and confirm they are setup and sending traffic via the private VPN !

AWS EC2 API access with Python

AWS has always had a great GUI interface, but the API side of calls is the programmatic way to work with AWS. AWS prides itself on having a rich set of API's, allowing most (if not all) operations to be created via API rather than having to head to the GUI to achieve the results.

I run a Windows 2K8 VM on AWS, which I use as a jumpbox to get to public services and clients public IPs (RDP/SSH/Teamviewer etc.). I had to set this up due to the very restrictive nature of the firewall I work behind and the corporate PC I work on blocking these services outbound - something not uncommon for corporate networks.

The instance I have setup for use is a standard Windows 2008 R2 AMI with 2 CPU, 4GB and a public IP (not an EIP - so not fixed as I don't want to be paying for the EIP when I don't have the instance on).

The following Python APP was built to support this via a script call rather than me having to log in to the GUI and manage the EC2 VM - I only start up the VM when I need to use it, so don't want to leave it running 24/7 and just costing money for no use (thus the beauty of AWS !).

To use the following script, you will need Python installed (in my case it's 3.6) and Boto3 (the Python SDK for the AWS CLI API access). Boto3 works with the AWS CLI, which is a command line way to run API calls.

The following guides are useful to setup these items ready to build the APP:

Python Installation:
https://wiki.python.org/moin/BeginnersGuide

PIP installation (used to install Python modules):
https://github.com/BurntSushi/nfldb/wiki/Python-&-pip-Windows-installation

BOTO3 install (simple with PIP):
https://pypi.org/project/boto3/

AWS CLI Install:
https://aws.amazon.com/cli/

From a user security point of view, you need to have an IAM user with an API key (programmatic) configured, who has the ability to manage EC2 instances. The API keys get stored via the AWS CLI program to C:\Users\[user]\.aws\ (for Windows), in 2 files called config and credentials.
Run '$ aws configure'  to set this up.

Once that is all built, you can create a new file called AWS-EC2.py (or a name of your choice - with the .py extension), and then use IDLE (right click on the file and chose Edit with IDLE) to put the code in. IDLE is a basic Python text editor that comes with the Python installer, with good recognition of code and structures for Python.

The APP has a few sections - whats imported (modules to be used), our variables, and the subroutines (def:). Firstly we need to import the required modules into Python:

# --------------------------
import boto3
import time
import subprocess
# --------------------------


This supports the AWS module, our retry timer for reading the public IP, and the command line process to open the remote desktop app in windows.

Next our variables:

# --------------------------
myTag1 = "MyInstance"
myTag2 = "JJ-Jumpbox"
myRegion = "us-east-1"
cmdCommand = "C:\Windows\System32\mstsc.exe"
# --------------------------

The myTag1/2 is a Tag I have on created my instance to uniquely identify it, the region is where it is hosted, and the command is the Windows Remote Desktop APP on my PC.

Now we have the first 2 routines - the start and stop of an EC2 instance calls:

# --------------------------
def start_ec2(ec2_instance_id, client):
    response = client.start_instances(
        InstanceIds=[
            ec2_instance_id,
        ],
        DryRun=False # change to true to test only
    )
    return response

def stop_ec2(ec2_instance_id, client):
    response = client.stop_instances(
        InstanceIds=[
            ec2_instance_id,
        ],
        DryRun=False
    )
    return response
# --------------------------

These routines use the boto3 API calls to start_instances and stop_instances, to manage the state of the EC2 instance.

Now for the main function, that runs through to read the current config, and uses the above routines to make changes to the instance.

# --------------------------
def main():
    boto3.setup_default_session(profile_name='default')
    ec2client = boto3.client('ec2', region_name=myRegion)

    response = ec2client.describe_instances()
    for reservation in response["Reservations"]:
        for instance in reservation["Instances"]:
            #print(instance) - use this if you want to see ALL the instance fields
            print(instance["InstanceId"], instance["InstanceType"], instance["State"], instance["Placement"], instance["PrivateIpAddress"], instance["KeyName"], instance["Platform"])
            for mytags in instance["Tags"]:
                if mytags["Key"] == myTag1 and mytags["Value"] == myTag2:
                    myInstanceId = instance["InstanceId"]
                    myStatus = instance["State"]
# --------------------------

The first part of the main function creates a boto3 client with access to our AWS environment (as defined with keys in the AWS CLI). Note the profile_name option is present just so I can use different profiles when i need to change between customers for other uses of the code.

describe_instances() is called, to list all instances that are present in this regions EC2 instance list. We then go through that list to find the one with the JJ-Jumpbox TAG reference, and find what the current state of the instance is.
# 'pending'|'running'|'shutting-down'|'terminated'|'stopping'|'stopped'

# --------------------------
    if myStatus['Name'] == 'stopped':
        print ("Instance", myTag1, myTag2)
        yesno = input ("Do you want to START the instance ? {y or n}")
        if yesno == "y":
            result = start_ec2(myInstanceId, ec2client)
            print ("\n")
            print (result)
            print ("\n")

            publicIP = {}
            publicIP["ip"] = 0        
            while publicIP["ip"] == 0:
                print ("Waiting for public IP (rechecking every 10s)")
                time.sleep(10)
                response = ec2client.describe_instances(
                    InstanceIds=[
                        myInstanceId,
                    ],
                DryRun=False
                )
               # print (response) - use this if you want to see ALL the response fields
                for reservation in response["Reservations"]:
                    for instance in reservation["Instances"]:
                        myState = instance["State"]
                        print (myState)
                        if  myState["Name"] == "running":
                            publicIP["ip"] = instance["PublicIpAddress"]

            print ("\n")               
            print ("Public IP is:", publicIP["ip"])
            process = subprocess.Popen(cmdCommand.split(), stdout=subprocess.PIPE) 
# --------------------------

If the state is stopped - we are given the choice to run it. The instance is then started, and a 10s while loop is started, checking each time to see if the instance state is changed to 'running', so we can find out the public IP of the instance to return to the user (to enter in to the RDP box). Once the IP is obtained, we exit the while loop, show the IP and open the mstsc.exe app which is Windows Remote Desktop.

# --------------------------           
    elif myStatus['Name'] == 'running':
        print ("Instance", myTag1, myTag2)
        yesno = input ("Do you want to STOP the instance ? {y or n}")
        if yesno == "y":
            result = stop_ec2(myInstanceId, ec2client)
            print ("\n")
            print (result)
            print ("\n")
    else:
        print ("\n")
        print ("Status is:", myStatus['Name'])
                  
if __name__ == "__main__":
    main()
# --------------------------

The other options in this main routine are to stop the instance - which is displayed if it is found to be already running when the APP is run. The last option is to show its status (ie not running or stopped) - so you can see what it is doing (starting up, shutting down or terminated).

That's it ! You should now have a working Python app that can talk to your AWS environment via IAM API key, and start/stop/review your EC2 instances. To make this work for your own EC2 units, just add a TAG to the instance, and change the myKey variable at the start of the script.

Where to from here ? The boto3 guide has a list of all the functions you can call via the API (there;s a lot !), so have a read through and see what else is in the list of things you would like to do with your EC2 (and other) AWS service.

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html

Its also possible to do this via AWS lambda... that's for anther day of learning !

AWS Cloudformation LAB Part3

In the previous posts, we succeeded in getting the Linux web server running via a template in CloudFormation. Now we want to add in support for SSH access from the internet to the Linux Instance. Luckily AWS has designed their product around the ability to just update whats changed, rather than having to tear it down and rebuild it all again with the updated configuration.

To do this, go to the Update option in the CloudFormation stack page.

From here we want to select Edit template in designer, and then View in Designer, so we can access the existing template that is currently active.

This will take us back in to the designer. Click on the PublicSecurity resource, and make sure you have YAML mode on and that you are in the Component tab for the SecurityGroup resource.

Add in the following addition text to the SecurityGroupIngress:
        # --- This opens the SSH port on the SecurityGroup from any source inbound
        - IpProtocol: tcp
          FromPort: '22'
          ToPort: '22'
          CidrIp: 0.0.0.0/0

Once added, you can then click on the Cloud with Arrow icon top left, to then publish this update.

Chose Next at each of the following screens to keep all the other existing setting the same, then this will take you back to the Stack building page. As you can see from the progress, only the SecurityGroup is updated, and the rest of the design is left as is (so its fast).

Now you will be able to SSH to your web server ! Note you will need to login using your .pem file that you created when you setup the keypair for this LAB. A basic guide from AWS can be found HERE.

The updated LAB YAML template can be downloaded from HERE

So that's it - you are now well on your way with CloudFormation ! I hope you enjoyed the posts, and take them as just the start to continue your AWS development and usage.

The AWS CloudFormation guide can be accessed HERE, and has some really good details on design, usage, and sample snippets that you can add in to your own templates.

AWS Cloudformation LAB Part2

In the previous post, we used designer to create the LAB we wanted in a simple graphical way. Now we will go in to the template parts and add in the parameters we need to join it all together.

In the designer, click on each resource below, then in the template section at the bottom of the screen click the Components TAB (making sure you are in YAML mode as per the button on the right. JSON will work - but the scripts I am using are in YAML so you need to enter it as YAML first then you can convert it to JSON). Delete the initial {} next to the Properties: string if it is present.

Resource:
     VPC
Add to Properties:
      EnableDnsSupport: 'true'
      EnableDnsHostnames: 'true'
      # --- this configures the private subnet block for the whole VPC
      CidrBlock: 10.0.0.0/16
      Tags:
        -
          Key: 'Name'
          Value: 'CloudFormation WebServer'

Result sample:

Resource:
    Subnet
Add to Properties:
      # --- this configures the specific subnet within the VPC we will be working with
      CidrBlock: 10.0.0.0/24
      Tags:
        -
          Key: 'Name'
          Value: 'CloudFormation WebServer'

Resource:
     Route
Add to Properties:
     # --- this configures a default route outbound to the IGW
     DestinationCidrBlock: 0.0.0.0/0

Resource:
      ACLIN
Add to Properties:
      # --- this configures no traffic blocking for inbound traffic via the ACL
      CidrBlock: 0.0.0.0/0
      Egress: 'false'
      PortRange:
        From: '0'
        To: '65535'
      Protocol: -1
      RuleAction: Allow
      RuleNumber: 100

Resource:
      ACLOUT
Add to Properties:
    # --- this configures no ACL blocking for outbound traffic
    CidrBlock: 0.0.0.0/0
      Egress: 'true'
      PortRange:
        From: '0'
        To: '65535'
      Protocol: '-1'
      RuleAction: Allow
      RuleNumber: 100

Resource:
      PublicSecurity
Add to Properties:
      # --- this permits port 80 inbound to the web server from any source
      GroupDescription: Web Ingress Group
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: '80'
          ToPort: '80'
          CidrIp: 0.0.0.0/0
      Tags:
        -
          Key: 'Name'
          Value: 'CloudFormation WebServer'

Resource:
      IGW
Add to Properties:
      Tags:
        -
          Key: 'Name'
          Value: 'CloudFormation WebServer'

Resource:
      wwwInstance
Delete all the existing text below Properties, and replace with:
      # --- !Ref is a variable, and this one references the AMI type we chose at the start
      InstanceType: !Ref 'InstanceType'
      ImageId: !Ref LatestAmiId
      # --- this variable holds the Key Pair for the Linux server access (which you need to create BEFORE the CloudFormation template is run)
      KeyName: !Ref KeyName
      NetworkInterfaces:
        - GroupSet:
            - Ref: PublicSecurity
          SubnetId: !Ref Subnet
          AssociatePublicIpAddress: 'false'
          DeviceIndex: '0'
          DeleteOnTermination: 'true'
      Tags:
        -
          Key: 'Name'
          Value: 'CloudFormation WebServer'
      # --- this is the script that will be run on the wwwInstance once it is spun up, to enable http and a string to show on the screen for success !This is the part I consider the secret of this script, as it runs the required setup on the linux server so its ready for use once completed
      UserData:
        'Fn::Base64': !Sub >
          #!/bin/bash -xe

          sudo yum update -y 

          sudo yum install httpd -y 

          sudo systemctl start httpd 

          sudo systemctl enable httpd

          echo "<html><body><p><h1 align=center>Congrats - you have a working
          web server !!</h1></body></html>" > /var/www/html/index.html

Resource:
      Overall Template (click outside the VPC to get to this section)
Add to Parameters:
  # --- each of these variable are choices we will get at the start that go into the template
  LatestAmiId:
    Type: 'AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>'
    Default: /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2
  InstanceType:
    Description: 'Enter t1.micro, t2.micro, or t3.micro. Default is t3.micro.'
    Type: String
    Default: t3.micro
    AllowedValues:
      - t1.micro
      - t2.micro
      - t3.micro
  KeyName:
    Description: Name of an EC2 KeyPair to enable SSH access to the instance.
    Type: 'AWS::EC2::KeyPair::KeyName'
    ConstraintDescription: must be the name of an existing EC2 KeyPair.

Add to Outputs (to the right of Parameters and Mappings):
  # --- these options will display once the CloudFormation template has completed running
  CloudInfo:
    Value: 'Here are some sample outputs from CloudFormation:'
  FQDN:
    Description: Public DNS for web server
    Value: !GetAtt wwwInstance.PublicDnsName
  InstanceID:
    Description: The Instance ID
    Value: !Ref wwwInstance


Once that's all entered, you can chose the Tick at the top left to 'Validate template', then click Create Stack next to the tick to launch ! This will take you through a few screens to Create the stack - chose next to each, select a name for the Stack (e.g. CloudForm-LAB), an instance type, and the keypair name (this needs to have been created prior via EC2 - KeyPairs - for OpenSSH), and then Next, Next and Create stack.

This will take you to the creation screen, where you can click the Circle Arrow to watch the setup of all resources. If all is well, after about 2 mins it will say CREATE_COMPLETE on the left.

If you get a ROLLBACK_IN_PROGRESS and CREATE_FAIL, just scroll down to see what the issue was that failed the template. Then you can DELETE the Stack, and once that's cleared go back in and edit the template issue and run again.

Once it has completed, click on the Outputs link to get the public DNS for the web server, so you can test if its working.

If all is correct, simply copy the FQDN address to your web browser, and you should see (note it may take a minute from when the lab finished until the httpd is running):

Congrats - you have a working web server !!

Perfect ! You have just automated a full AWS VPC environment that is templated and can be rebuilt any time.

Click HERE to download the full template used in this LAB

Now we want to be able to SSH to this instance, but we forgot to enable port 22 as part of the secuirty rule set. In the next post, we will load an updated template to fix that and allow us to SSH in to the linux server for further changes.

AWS Cloudformation LAB Part1

  

AWS Cloud is so much more than just a cloud storage provider, something I learnt well during my Masters when for my final (capstone) subject in 2019 I wrote a paper on AWS and its emerging applications. I am presently working through study for the AWS Certified Solution Architect Professional certification, and one of many features that is covered is Cloud Formation.

Cloud Formation is an AWS tool that allows you to script a complete environment to be setup (and as we will see - then change it) via a template design, rather than manually setting up all the components. Great for something that you need to roll out in a repeated (or scalable) fashion, and then have the ability easily to remove all the components once done with the project, so you don't have to go in to each area and delete each resource (and possible leave some resource(s) in place costing you wasted money!).

In our LAB here we will go through and initially build a Linux instance with a web server within a VPC, with internet access via a public IP. A similar walk through from AWS on this design can be found here:

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/working-with-templates-cfn-designer-walkthrough-createbasicwebserver.html

In our case we will make some slightly different options and methods, as well as use the GUI Designer to make it a bit more graphical and easy to understand what is connecting to where for the resources. Once the lab is build, then we can go to the template editor and add our options/parameters in to the YAML code and build it.

The design we will create looks like this:

So lets get started !

The components we need to make up our lab are:

1. A VPC to encompass the instance, rules and routes
2. A EC2 linux instance to run the web server
3. In and Out ACL rules
4. An Inbound Security Group
5. An EIP
6. An Internet Gateway

It may be worth building this in a region NOT where you currently have resources - just to make it easier to identify which resources are part of the LAB, and being able to trouble shoot or delete resources easily if anything goes wrong with the automatic method.

You will need to log in to your AWS account, and go to CloudFormation (just search it on the main home page). Once in to the CloudFormation page, click on Create stack, then Create template in Designer. You should now be in the drag-n-drop designer, where we can place our resources and build our connections.

From here you can build templates in virtually all AWS services - for us on the left go to EC2. All the resources we will use for this LAB are located under this heading.

As per the above design picture, you now need to add in a VPC. Within the VPC add a SecurityGroup, a Route Table (then a Route inside the RouteTable), a Subnet (then an instance inside the subnet) and a NetworkACL (with 2 NetworkACLEntries inside it).
Outside the VPC add an EIP and an Internet Gateway.

Once that's setup, you should have something looking like this:

You can download the template up to this stage from HERE

To make things easier, from here go through and rename a few of the resources, just so they make more sense when you read the template.

Right click on the following resources, click on the Eye picture, then in the box at the bottom of the screen chose YAML on the right and Components at the bottom tab, and change the name below the 'Resources' line:

• InternetGateway: IGW (just type over the EC2IG2UXKH string in the example above)
• EIP: EIP
• ROUTE: Route
• Left ACL entry: ACLIN
• Right ACL entry: ACLOUT
• SecurityGroup: PublicSecurity
• Instance: wwwInstance
• Subnet: Subnet

Now when we look at the diagram (and later the template) the references will be a bit clearer. You can do ALL the resources if you want - but at a minimum I have done the above.

Last thing for this section is to link the resources together. For this you will use the purple and pink dots around each resource. If you hover your mouse over each one around a specific resource, you will find it relates to different functions. Link the following by putting your mouse over the relevant dot around the resource, click and drag it to the related resource, as per the following:

• EIP - to wwwInstance
• wwwInstance - 'SecurityGroup' to PublicSecurity
• NetworkACL - 'SubnetNetworkAclAssociation' to Subnet
• RouteTable - 'SubnetRouteTableAssociation' to Subnet
• Route - 'InternetGateway' to IGW
• IGW - 'VPCGatewayAttachement' to VPC
• Route - 'Depends on' link 'VPCGatewayAttachement to VPC'

We have now completed the designer part of the setup, with resources and links in place, ready for the parameters to be configured. Your image should look something like the one at the top of this post. Note I will publish the complete template at the end of the series.

You can download the updated template from HERE

In Part2 we will look at setting up the required options in the template to then allow us to build the LAB !