Wednesday, 22 April 2020
Change of BLOG SITE
I am now moving away from Blogger to another hosted service:
http://blog.connect-it.com.au
Please update any links you have and head there for all future comments !
Monday, 9 March 2020
AWS to Meraki VPN Tunnel with Python Part3
The following screens can be used to check on the VPN status, now that the python app has completed all the steps and built both ends of the VPN Tunnel - AWS and Meraki !
In my case the EC2 instance has an IP of 172.31.30.146.
AWS CGW configured:
AWS VPN Configured:
Meraki VPN Enabled:
Trace from Meraki LAN to EC2 instance - via VPN tunnel (1 hop):
And as promised - here is the completed APP ready to run (NOTE you will need to update the variables at the top that have [REQUIRED] next to them.
Hope this blog series has been of use - and any questions leave in the box below or send me an email !
AWS to Meraki VPN Tunnel with Python Part2
In the previous post, we went through the initial setup steps for the AWS to Meraki VPN:
# Meraki info step
# -----------------------------------# 1. Get MX public IP
#
# AWS VPN creation steps
# -----------------------------------
# 1. Create Customer Gateway
# 2. Create Virtual Private Gateway
# 3. Attach VGW to VPC
# 4. Create VPN Connection using CGW + VPG
# 5. View Downloaded config with public IP and password
# 6. Add LAN routes to VPN table
# 7. Propogate VPC subnet into VPG
#
# Meraki VPN Creation step
# ------------------------------------
# 1. read existing non Meraki VPN config
# 2. update config with AWS VPN peer detail
Now we will finish setting up both the AWS and then Meraki VPNs, and make sure they connect and provide private traffic flow from the Meraki LAN to the VPC EC2 instance.
#
# 1. create a customer gateway (requires Meraki MX Public IP)
#
response = client.create_customer_gateway(
BgpAsn = 65000,
PublicIp = meraki_public_ip,
Type= 'ipsec.1',
DeviceName = 'MerakiMX',
DryRun=False
)
print ("\nCustomer Gateway result:") print(json.dumps(response["CustomerGateway"], sort_keys=True, indent=4)) print ("Customer GatewayID:", response["CustomerGateway"]["CustomerGatewayId"]) myCGW = response["CustomerGateway"]["CustomerGatewayId"]
print ("\nCustomer Gateway result:") print(json.dumps(response["CustomerGateway"], sort_keys=True, indent=4)) print ("Customer GatewayID:", response["CustomerGateway"]["CustomerGatewayId"]) myCGW = response["CustomerGateway"]["CustomerGatewayId"]
This call uses the meraki IP we read from the previous post to create a CGW. BGP ASN is required - but not going to be used since this will be based on simple static routing.
#
# 2. Create Virtual Private Gateway
#
response = client.create_vpn_gateway(
Type='ipsec.1',
)
print ("\nVirtual Private Gateway result:") print(json.dumps(response["VpnGateway"], sort_keys=True, indent=4)) print ("VGW ID:", response["VpnGateway"]["VpnGatewayId"]) myVGW = response["VpnGateway"]["VpnGatewayId"]
print ("\nVirtual Private Gateway result:") print(json.dumps(response["VpnGateway"], sort_keys=True, indent=4)) print ("VGW ID:", response["VpnGateway"]["VpnGatewayId"]) myVGW = response["VpnGateway"]["VpnGatewayId"]
This call creates the VGW, which initially is standalone.
#
# 3. Attach VGW to VPC
#
response = client.attach_vpn_gateway(
VpcId = myVpc,
VpnGatewayId = myVGW
)
print ("\nAttachement result:")
print(json.dumps(response["VpcAttachment"], sort_keys=True, indent=4))
print ("Status:", response["VpcAttachment"]["State"])
status = response["VpcAttachment"]["State"]
print ("Waiting for [attached] status (rechecking every 5s)")
while status != "attached":
time.sleep(5)
response = client.describe_vpn_gateways (
VpnGatewayIds=[
myVGW,
],
)
# print(json.dumps(response["VpnGateways"], sort_keys=True, indent=4))
for line in response["VpnGateways"]:
for line2 in line["VpcAttachments"]:
status = line2["State"]
print ("Status:",status)
Now we attach the VGW to our VPC, and then put a simple 5s times loop in to wait for the status of the attachement to complete (normally takes 15-20s)
#
# 4. Create VPN Connection using CGW + VPG
# 5. Display VPN config data (public IP/passphrase)
#
response = client.create_vpn_connection(
CustomerGatewayId = myCGW,
Type = 'ipsec.1',
VpnGatewayId = myVGW,
Options = {
'StaticRoutesOnly': True
}
)
myVPN = response["VpnConnection"]["VpnConnectionId"] print ("VPN ID:", myVPN) print ("\nVPN Configuration Data (showing VPN0)\n----------------------------------")
doc = xmltodict.parse(response["VpnConnection"]["CustomerGatewayConfiguration"]) # print(json.dumps(doc, sort_keys=True, indent=4)) for tunnels in doc["vpn_connection"]["ipsec_tunnel"]: print ("VPN0 Passphrase:", tunnels["ike"]["pre_shared_key"]) myPassPhrase = tunnels["ike"]["pre_shared_key"] print ("VPN0 Public IP:", tunnels["vpn_gateway"]["tunnel_outside_address"]["ip_address"]) aws_public_ip = tunnels["vpn_gateway"]["tunnel_outside_address"]["ip_address"] break
myVPN = response["VpnConnection"]["VpnConnectionId"] print ("VPN ID:", myVPN) print ("\nVPN Configuration Data (showing VPN0)\n----------------------------------")
doc = xmltodict.parse(response["VpnConnection"]["CustomerGatewayConfiguration"]) # print(json.dumps(doc, sort_keys=True, indent=4)) for tunnels in doc["vpn_connection"]["ipsec_tunnel"]: print ("VPN0 Passphrase:", tunnels["ike"]["pre_shared_key"]) myPassPhrase = tunnels["ike"]["pre_shared_key"] print ("VPN0 Public IP:", tunnels["vpn_gateway"]["tunnel_outside_address"]["ip_address"]) aws_public_ip = tunnels["vpn_gateway"]["tunnel_outside_address"]["ip_address"] break
This ties the CGW and VGW together, and creates the VPN connection. We then receive back a response which includes the VPN Gateway config, which is in XML format. Convert this to a dictionary variable with xmltodict to make it easier to read (and reference).
#
# 6. Add remote LAN routes to VPN table
#
client.create_vpn_connection_route(
DestinationCidrBlock = my_meraki_lan_subnet,
VpnConnectionId = myVPN
)
# # 7. Propogate VPN routes to route table # client.enable_vgw_route_propagation( RouteTableId = myRouteID, GatewayId = myVGW )
# # 7. Propogate VPN routes to route table # client.enable_vgw_route_propagation( RouteTableId = myRouteID, GatewayId = myVGW )
Then we add in our Meraki LAN subnet to the VPN connection created in #5, and connect our new VGW to the route table of the VPC so we can route traffic between the VPC and the VPN.
#
# Wait for VPN to be ready to use
#
status = response["VpnConnection"]["State"]
print ("\nStatus:", status)
print ("Waiting for [available] status (rechecking every 30s)")
while status != "available":
time.sleep(30)
response = client.describe_vpn_connections(
VpnConnectionIds=[
myVPN,
],
)
for line in response["VpnConnections"]:
status = line["State"]
print ("Status:",status)
To finish, we now wait for the VPN to be ready, so we can complete the Meraki VPN setup, with another simple 30s loop status check routine.
Now lets create the VPN tunnel on the Meraki !
# Return the third party VPN peers for an organization
#
try:
non_meraki_peers = meraki_client.organizations.get_organization_third_party_vpn_peers(params["organization_id"])
#print (non_meraki_peers)
except APIException as e:
print(e)
print ("\n") # # Add the additional peer to the existing third party VPN peers for an organization # try: collect = {} collect['organization_id'] = params["organization_id"] update_organization_third_party_vpn_peers = UpdateOrganizationThirdPartyVPNPeersModel() update_organization_third_party_vpn_peers.peers = []
peer_num=0 # iterate through the existing peers and add to update variable for line in non_meraki_peers: update_organization_third_party_vpn_peers.peers.append(PeerModel()) update_organization_third_party_vpn_peers.peers[peer_num].name = line['name'] update_organization_third_party_vpn_peers.peers[peer_num].public_ip = line['publicIp'] if 'remoteId' in line: update_organization_third_party_vpn_peers.peers[peer_num].remote_id = line['remoteId'] update_organization_third_party_vpn_peers.peers[peer_num].private_subnets = line['privateSubnets'] update_organization_third_party_vpn_peers.peers[peer_num].ipsec_policies_preset = line['ipsecPoliciesPreset'] update_organization_third_party_vpn_peers.peers[peer_num].secret = line['secret'] update_organization_third_party_vpn_peers.peers[peer_num].network_tags = line['networkTags'] peer_num +=1
# add new aws peer to variable update_organization_third_party_vpn_peers.peers.append(PeerModel()) update_organization_third_party_vpn_peers.peers[peer_num].name = 'AWS Gateway' update_organization_third_party_vpn_peers.peers[peer_num].public_ip = aws_public_ip # note remote_id missing from meraki_sdk file - need to update
\python3\Lib\site-packages\meraki_sdk\models\peer_model.py update_organization_third_party_vpn_peers.peers[peer_num].remote_id = 'ec2@aws.com.au' update_organization_third_party_vpn_peers.peers[peer_num].private_subnets = my_aws_lan_subnet update_organization_third_party_vpn_peers.peers[peer_num].ipsec_policies_preset = 'aws'
print ("\n") # # Add the additional peer to the existing third party VPN peers for an organization # try: collect = {} collect['organization_id'] = params["organization_id"] update_organization_third_party_vpn_peers = UpdateOrganizationThirdPartyVPNPeersModel() update_organization_third_party_vpn_peers.peers = []
peer_num=0 # iterate through the existing peers and add to update variable for line in non_meraki_peers: update_organization_third_party_vpn_peers.peers.append(PeerModel()) update_organization_third_party_vpn_peers.peers[peer_num].name = line['name'] update_organization_third_party_vpn_peers.peers[peer_num].public_ip = line['publicIp'] if 'remoteId' in line: update_organization_third_party_vpn_peers.peers[peer_num].remote_id = line['remoteId'] update_organization_third_party_vpn_peers.peers[peer_num].private_subnets = line['privateSubnets'] update_organization_third_party_vpn_peers.peers[peer_num].ipsec_policies_preset = line['ipsecPoliciesPreset'] update_organization_third_party_vpn_peers.peers[peer_num].secret = line['secret'] update_organization_third_party_vpn_peers.peers[peer_num].network_tags = line['networkTags'] peer_num +=1
# add new aws peer to variable update_organization_third_party_vpn_peers.peers.append(PeerModel()) update_organization_third_party_vpn_peers.peers[peer_num].name = 'AWS Gateway' update_organization_third_party_vpn_peers.peers[peer_num].public_ip = aws_public_ip # note remote_id missing from meraki_sdk file - need to update
\python3\Lib\site-packages\meraki_sdk\models\peer_model.py update_organization_third_party_vpn_peers.peers[peer_num].remote_id = 'ec2@aws.com.au' update_organization_third_party_vpn_peers.peers[peer_num].private_subnets = my_aws_lan_subnet update_organization_third_party_vpn_peers.peers[peer_num].ipsec_policies_preset = 'aws'
update_organization_third_party_vpn_peers.peers[peer_num].secret = myPassPhrase
update_organization_third_party_vpn_peers.peers[peer_num].network_tags = ['all']
collect['update_organization_third_party_vpn_peers'] = update_organization_third_party_vpn_peers
result = meraki_client.organizations.update_organization_third_party_vpn_peers(collect) print("MX Non-Meraki VPN peers:\n", json.dumps(result, sort_keys=True, indent=4))
collect['update_organization_third_party_vpn_peers'] = update_organization_third_party_vpn_peers
result = meraki_client.organizations.update_organization_third_party_vpn_peers(collect) print("MX Non-Meraki VPN peers:\n", json.dumps(result, sort_keys=True, indent=4))
This final routine reads in the existing non-Meraki VPN config (under Security->Site to Site VPN), and then adds in the new AWS VPN to the config, and writes it back to the dashboard. As per the GUI guide from the first page, Meraki provides an AWS VPN config, so that you don't need to deep dive on ipsec parameters and work out what should be configured to connect to AWS ipsec VPN correctly (thus the ipsec policy = aws).
And we are complete ! AWS VPN is up, Meraki VPN is up, and we should be able to ping/ssh/rdp from the Meraki LAN to the VPN EC2 instance via the private IP.
In the final post, we will have a look at the status output and Ill provide a copy of the completed application to run.
AWS to Meraki VPN Tunnel with Python Part1
One if the great features of both AWS and Meraki is the support for third party IPSec VPNs. These VPN tunnels give you a secure way of communicating with your services within AWS or your devices and LAN behind the Meraki MX via private IP, but only needing internet access to create the tunnel. This AWS VPN can be used in a commercial environment, to provide private access to VPC instances and beyond from on premise or from the DC, and can also be a backup link behind a Direct Connect link for use in the case of failure of the Direct Connect service.
In my previous post, I showed how to create a Python application that managed an EC2 instance (to start/stop it) via AWS API calls. For this post series, I am going to build a VPN tunnel from my Meraki MX network to my AWS VPC via API calls to both platforms, so I can connect to the EC2 instance via private IP (in my VPC's case - to the 172.16.x.x range) rather than having to come in to a public IP.
The steps in this application are taken from the GUI visual setup detailed in this ritcsec post. Its assumed for our code that the VPC is setup, a subnet is created, you have built an EC2 instance (so we can ping it and ssh/rdp to it via the private IP across the tunnel) and have turned ON VPN participation for the LAN subnet(s) on the Meraki (Security-Site to Site-VPN Settings).
You also need to configure Tags for the VPC and route table as per below, so that the application can find the resources when it searches the list.
The following code uses a few python modules, which need to be installed if not done already:
$ pip install meraki_sdk xmltodict boto3
The following sequence outlines the steps we will take to build this app, and successfully create a VPN tunnel from the Meraki MX to the AWS VPC. The finished app can be downloaded at the end of the last post, for those who want to skip ahead !
# Meraki info step
# -----------------------------------# 1. Get MX public IP
#
# AWS VPN creation steps
# -----------------------------------
# 1. Create Customer Gateway
# 2. Create Virtual Private Gateway
# 3. Attach VGW to VPC
# 4. Create VPN Connection using CGW + VPG
# 5. View Downloaded config with public IP and password
# 6. Add LAN routes to VPN table
# 7. Propogate VPC subnet into VPG
#
# Meraki VPN Creation step
# ------------------------------------
# 1. read existing non Meraki VPN config
# 2. update config with AWS VPN peer detail
Firstly, we will import the required modules and setup the global variables to be used within the subroutines:
from meraki_sdk.meraki_sdk_client import MerakiSdkClient
from meraki_sdk.exceptions.api_exception import APIException
from meraki_sdk.models.update_organization_third_party_vpn_peers_model import UpdateOrganizationThirdPartyVPNPeersModel
from meraki_sdk.models.peer_model import PeerModel
import boto3 import json import time import sys import xmltodict import re
meraki_public_ip = "1.1.1.1" aws_public_ip = "2.2.2.2" my_meraki_lan_subnet = "192.168.1.0/24" # Meraki LAN subnet - must be advertised
to VPN on Meraki dashboard (Security->Site-to-Site VPN) [REQUIRED] org_name = "My ORG NAME" # Meraki ORG we want to work in [REQUIRED] network_name = "My Network" # ORG network we want to configure [REQUIRED] x_cisco_meraki_api_key = 'XXX' # [REQUIRED] vpcTag = "MyVPC" # Name tag for my VPC table [REQUIRED] myRegion = "us-east-1" myRouteTable = "MyLanRoute" # Name tag for my AWS route table [REQUIRED] myPassPhrase = "password" my_aws_lan_subnet = ['172.31.16.0/20'] # AWS VPC subnet (VPC->subnets) [REQUIRED]
params = {}
import boto3 import json import time import sys import xmltodict import re
meraki_public_ip = "1.1.1.1" aws_public_ip = "2.2.2.2" my_meraki_lan_subnet = "192.168.1.0/24" # Meraki LAN subnet - must be advertised
to VPN on Meraki dashboard (Security->Site-to-Site VPN) [REQUIRED] org_name = "My ORG NAME" # Meraki ORG we want to work in [REQUIRED] network_name = "My Network" # ORG network we want to configure [REQUIRED] x_cisco_meraki_api_key = 'XXX' # [REQUIRED] vpcTag = "MyVPC" # Name tag for my VPC table [REQUIRED] myRegion = "us-east-1" myRouteTable = "MyLanRoute" # Name tag for my AWS route table [REQUIRED] myPassPhrase = "password" my_aws_lan_subnet = ['172.31.16.0/20'] # AWS VPC subnet (VPC->subnets) [REQUIRED]
params = {}
The few items that are key here are:
my_meraki_lan_subnet: this is obtained from the Meraki dashboard, and shows the LAN subnet(s) you have in the Meraki network
org_name: this is the name of your Meraki ORG that has the MX in it
network_name: this is the name of the network in your Meraki ORG that has the MX in it
x_cisco_meraki_api_key: this is the API key of a Meraki dahsboard ADMIN
vpcTag and myRouteTable: these are the tags configured on the VPC and Route table in AWS, which are used to match by the search engine when it looks for them in the list
my_aws_lan_subnet: is the subnet used within the VPC for instances
These variables need to be pre-populated before we run the application. How to set them up can be found in the GUI walkthorugh listed above.
The Meraki API documentation can be found HERE.
Next we are ready to run the main application call, which will check on the org_name being valid, and then run the 3 subroutines:
# Meraki info step [meraki_read()]
# AWS VPN creation steps [aws_vpn()]
# Meraki VPN Creation step [meraki_vpn()]
meraki_client = MerakiSdkClient(x_cisco_meraki_api_key)
#
# get Meraki organisation list
#
try:
print ("\n--- Meraki to AWS VPN Application ---\n")
orgs = meraki_client.organizations.get_organizations()
for organisation in orgs:
print ("ORG ID and Name:", organisation["id"], organisation["name"])
if organisation["name"] == org_name:
params["organization_id"] = organisation["id"]
except APIException as e:
print(e)
sys.exit(2)
if is_empty(params):
print (org_name, "NOT found in list")
exit()
meraki_read() aws_vpn() meraki_vpn()
meraki_read() aws_vpn() meraki_vpn()
We now have the ORG ID, so can continue to the dashboard and get the network ID and device details (specifically the public IP of the MX unit - needed to setup the AWS VPN)
#
# get network list in org
#
params["network_id"] = ""
try:
print ("AP: networks.get_organization_networks. ORG:", org_name)
nets = meraki_client.networks.get_organization_networks(params)
for network in nets:
print(network["id"], network["name"])
if network["name"] == network_name:
params["network_id"] = network["id"]
except APIException as e:
print(e)
if is_empty(params["network_id"]): print (network_name, "NOT found in list") exit()
# # get device list in network # try: devices = meraki_client.devices.get_network_devices(params["network_id"]) for items in devices: print (items["model"], items["lanIp"], items["firmware"], items["serial"]) if re.match ("^MX", items["model"]): network_mx = {'serial': items["serial"], 'model': items["model"]} except APIException as e: print(e)
print ("\n") # # Get public IP # try: collect = {} collect['network_id'] = params["network_id"] collect['serial'] = network_mx["serial"]
if is_empty(params["network_id"]): print (network_name, "NOT found in list") exit()
# # get device list in network # try: devices = meraki_client.devices.get_network_devices(params["network_id"]) for items in devices: print (items["model"], items["lanIp"], items["firmware"], items["serial"]) if re.match ("^MX", items["model"]): network_mx = {'serial': items["serial"], 'model': items["model"]} except APIException as e: print(e)
print ("\n") # # Get public IP # try: collect = {} collect['network_id'] = params["network_id"] collect['serial'] = network_mx["serial"]
mx_uplink = meraki_client.devices.get_network_device_uplink(collect)
print (network_mx["model"], "Public IPs (WAN1<>WAN2):", mx_uplink[0]['publicIp'], "<>", mx_uplink[1]['publicIp'])
except APIException as e:
print(e)
meraki_public_ip = mx_uplink[0]['publicIp']
meraki_public_ip = mx_uplink[0]['publicIp']
This routine uses the Merkai API calls via the meraki_sdk to get the Public IP. The variables that are returned from the Meraki calls are normally lists or dictionaries, so we just need to iterate through them to get the relevant value we are looking for. You can do a print (variable) if you want to get a full look at all the returned values, or better still - use json.dumps to display it in an easy to ready format with:
# print(json.dumps(response, sort_keys=True, indent=4))
Now that we have the meraki client setup and the public IP obtained for the MX network, we can move on to the steps for the AWS VPN creation.
This code uses the boto3 module, which provides AWSCLI functions via the python application.
The AWS boto3 documentation can be found HERE.
First we need to seetup the AWS Client. Here we define a region, and also use the profile of 'james' which has the API authentication setup for AWS to my VPC. I do this so I can have multiple profiles, and call them in the application depending on which VPC I am working in (details on this can be found in the previous EC2 post).
boto3.setup_default_session(profile_name='james')
ec2 = boto3.resource('ec2', region_name=myRegion)
client = boto3.client('ec2', region_name=myRegion)
filters = [{'Name':'tag:Name', 'Values':[vpcTag]}] vpcs = list(ec2.vpcs.filter(Filters=filters)) for vpc in vpcs: myVpc = vpc.id #response = client.describe_vpcs(VpcIds=[vpc.id,]) #print(json.dumps(response, sort_keys=True, indent=4)) print ("\nVPC ID:", myVpc)
response = client.describe_route_tables() for routes in response["RouteTables"]: for myTags in routes["Tags"]: if myTags["Key"] == "Name" and myTags["Value"] == myRouteTable: myRouteID = routes["RouteTableId"] print ("Route Table ID: ", myRouteID)
filters = [{'Name':'tag:Name', 'Values':[vpcTag]}] vpcs = list(ec2.vpcs.filter(Filters=filters)) for vpc in vpcs: myVpc = vpc.id #response = client.describe_vpcs(VpcIds=[vpc.id,]) #print(json.dumps(response, sort_keys=True, indent=4)) print ("\nVPC ID:", myVpc)
response = client.describe_route_tables() for routes in response["RouteTables"]: for myTags in routes["Tags"]: if myTags["Key"] == "Name" and myTags["Value"] == myRouteTable: myRouteID = routes["RouteTableId"] print ("Route Table ID: ", myRouteID)
Here we now have the boto3 client enabled, as well as the ID of the VPC and the route table to be used with the VPN
In the next post, we will create the AWS VPN, and then create the Meraki VPN, and confirm they are setup and sending traffic via the private VPN !
Sunday, 2 February 2020
AWS EC2 API access with Python
AWS has always had a great GUI interface, but the API side of calls is the programmatic way to work with AWS. AWS prides itself on having a rich set of API's, allowing most (if not all) operations to be created via API rather than having to head to the GUI to achieve the results.
I run a Windows 2K8 VM on AWS, which I use as a jumpbox to get to public services and clients public IPs (RDP/SSH/Teamviewer etc.). I had to set this up due to the very restrictive nature of the firewall I work behind and the corporate PC I work on blocking these services outbound - something not uncommon for corporate networks.
The instance I have setup for use is a standard Windows 2008 R2 AMI with 2 CPU, 4GB and a public IP (not an EIP - so not fixed as I don't want to be paying for the EIP when I don't have the instance on).
The following Python APP was built to support this via a script call rather than me having to log in to the GUI and manage the EC2 VM - I only start up the VM when I need to use it, so don't want to leave it running 24/7 and just costing money for no use (thus the beauty of AWS !).
To use the following script, you will need Python installed (in my case it's 3.6) and Boto3 (the Python SDK for the AWS CLI API access). Boto3 works with the AWS CLI, which is a command line way to run API calls.
The following guides are useful to setup these items ready to build the APP:
Python Installation:
https://wiki.python.org/moin/BeginnersGuide
PIP installation (used to install Python modules):
https://github.com/BurntSushi/nfldb/wiki/Python-&-pip-Windows-installation
BOTO3 install (simple with PIP):
https://pypi.org/project/boto3/
AWS CLI Install:
https://aws.amazon.com/cli/
From a user security point of view, you need to have an IAM user with an API key (programmatic) configured, who has the ability to manage EC2 instances. The API keys get stored via the AWS CLI program to C:\Users\[user]\.aws\ (for Windows), in 2 files called config and credentials.
Run '$ aws configure' to set this up.
Once that is all built, you can create a new file called AWS-EC2.py (or a name of your choice - with the .py extension), and then use IDLE (right click on the file and chose Edit with IDLE) to put the code in. IDLE is a basic Python text editor that comes with the Python installer, with good recognition of code and structures for Python.
The APP has a few sections - whats imported (modules to be used), our variables, and the subroutines (def:). Firstly we need to import the required modules into Python:
# --------------------------
import boto3
import time
import subprocess
# --------------------------
This supports the AWS module, our retry timer for reading the public IP, and the command line process to open the remote desktop app in windows.
Next our variables:
# --------------------------
myTag1 = "MyInstance"
myTag2 = "JJ-Jumpbox"
myRegion = "us-east-1"
cmdCommand = "C:\Windows\System32\mstsc.exe"
# --------------------------
The myTag1/2 is a Tag I have on created my instance to uniquely identify it, the region is where it is hosted, and the command is the Windows Remote Desktop APP on my PC.
Now we have the first 2 routines - the start and stop of an EC2 instance calls:
# --------------------------
def start_ec2(ec2_instance_id, client):
response = client.start_instances(
InstanceIds=[
ec2_instance_id,
],
DryRun=False # change to true to test only
)
return response
def stop_ec2(ec2_instance_id, client):
response = client.stop_instances(
InstanceIds=[
ec2_instance_id,
],
DryRun=False
)
return response
# --------------------------
These routines use the boto3 API calls to start_instances and stop_instances, to manage the state of the EC2 instance.
Now for the main function, that runs through to read the current config, and uses the above routines to make changes to the instance.
# --------------------------
def main():
boto3.setup_default_session(profile_name='default')
ec2client = boto3.client('ec2', region_name=myRegion)
response = ec2client.describe_instances()
for reservation in response["Reservations"]:
for instance in reservation["Instances"]:
#print(instance) - use this if you want to see ALL the instance fields
print(instance["InstanceId"], instance["InstanceType"], instance["State"], instance["Placement"], instance["PrivateIpAddress"], instance["KeyName"], instance["Platform"])
for mytags in instance["Tags"]:
if mytags["Key"] == myTag1 and mytags["Value"] == myTag2:
myInstanceId = instance["InstanceId"]
myStatus = instance["State"]
# --------------------------
The first part of the main function creates a boto3 client with access to our AWS environment (as defined with keys in the AWS CLI). Note the profile_name option is present just so I can use different profiles when i need to change between customers for other uses of the code.
describe_instances() is called, to list all instances that are present in this regions EC2 instance list. We then go through that list to find the one with the JJ-Jumpbox TAG reference, and find what the current state of the instance is.
# 'pending'|'running'|'shutting-down'|'terminated'|'stopping'|'stopped'
# --------------------------
if myStatus['Name'] == 'stopped':
print ("Instance", myTag1, myTag2)
yesno = input ("Do you want to START the instance ? {y or n}")
if yesno == "y":
result = start_ec2(myInstanceId, ec2client)
print ("\n")
print (result)
print ("\n")
publicIP = {}
publicIP["ip"] = 0
while publicIP["ip"] == 0:
print ("Waiting for public IP (rechecking every 10s)")
time.sleep(10)
response = ec2client.describe_instances(
InstanceIds=[
myInstanceId,
],
DryRun=False
)
# print (response) - use this if you want to see ALL the response fields
for reservation in response["Reservations"]:
for instance in reservation["Instances"]:
myState = instance["State"]
print (myState)
if myState["Name"] == "running":
publicIP["ip"] = instance["PublicIpAddress"]
print ("\n")
print ("Public IP is:", publicIP["ip"])
process = subprocess.Popen(cmdCommand.split(), stdout=subprocess.PIPE)
# --------------------------
If the state is stopped - we are given the choice to run it. The instance is then started, and a 10s while loop is started, checking each time to see if the instance state is changed to 'running', so we can find out the public IP of the instance to return to the user (to enter in to the RDP box). Once the IP is obtained, we exit the while loop, show the IP and open the mstsc.exe app which is Windows Remote Desktop.
# --------------------------
elif myStatus['Name'] == 'running':
print ("Instance", myTag1, myTag2)
yesno = input ("Do you want to STOP the instance ? {y or n}")
if yesno == "y":
result = stop_ec2(myInstanceId, ec2client)
print ("\n")
print (result)
print ("\n")
else:
print ("\n")
print ("Status is:", myStatus['Name'])
if __name__ == "__main__":
main()
# --------------------------
The other options in this main routine are to stop the instance - which is displayed if it is found to be already running when the APP is run. The last option is to show its status (ie not running or stopped) - so you can see what it is doing (starting up, shutting down or terminated).
That's it ! You should now have a working Python app that can talk to your AWS environment via IAM API key, and start/stop/review your EC2 instances. To make this work for your own EC2 units, just add a TAG to the instance, and change the myKey variable at the start of the script.
Where to from here ? The boto3 guide has a list of all the functions you can call via the API (there;s a lot !), so have a read through and see what else is in the list of things you would like to do with your EC2 (and other) AWS service.
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html
Its also possible to do this via AWS lambda... that's for anther day of learning !
Saturday, 4 January 2020
AWS Cloudformation LAB Part3
In the previous posts, we succeeded in getting the Linux web server running via a template in CloudFormation. Now we want to add in support for SSH access from the internet to the Linux Instance. Luckily AWS has designed their product around the ability to just update whats changed, rather than having to tear it down and rebuild it all again with the updated configuration.
To do this, go to the Update option in the CloudFormation stack page.
From here we want to select Edit template in designer, and then View in Designer, so we can access the existing template that is currently active.
This will take us back in to the designer. Click on the PublicSecurity resource, and make sure you have YAML mode on and that you are in the Component tab for the SecurityGroup resource.
Add in the following addition text to the SecurityGroupIngress:
# --- This opens the SSH port on the SecurityGroup from any source inbound
- IpProtocol: tcp
FromPort: '22'
ToPort: '22'
CidrIp: 0.0.0.0/0
Once added, you can then click on the Cloud with Arrow icon top left, to then publish this update.
Chose Next at each of the following screens to keep all the other existing setting the same, then this will take you back to the Stack building page. As you can see from the progress, only the SecurityGroup is updated, and the rest of the design is left as is (so its fast).
Now you will be able to SSH to your web server ! Note you will need to login using your .pem file that you created when you setup the keypair for this LAB. A basic guide from AWS can be found HERE.
The updated LAB YAML template can be downloaded from HERE
So that's it - you are now well on your way with CloudFormation ! I hope you enjoyed the posts, and take them as just the start to continue your AWS development and usage.
The AWS CloudFormation guide can be accessed HERE, and has some really good details on design, usage, and sample snippets that you can add in to your own templates.
Subscribe to:
Posts (Atom)