09/03/2017: List Deleteable Security Groups
I inherited an AWS VPC that grew organically. This meant about 150 security
groups with no clear way to know which ones are important. Until I learned about
the describe_network_interfaces function, creating a report about security
group usage seemed like too much work.
Even with knowledge of that function, this bit of python was more work than I thought. Therefore, I’m sharing it.
import boto3
security_groups = find_deleteable_security_groups(boto3.client('config'), boto3.client('cloudformation'))
for security_group in security_groups:
print(security_group)
def find_deleteable_security_groups(ec2Client, cfmClient):
# Will hold security group information.
security_groups = {}
# Will hold network interface information.
network_interfaces = {}
# Will hold backward references of security group. For example, securiy group A
# might have no assets but still be referenced by security group B.
backward_references = {}
for security_group in ec2Client.describe_security_groups()['SecurityGroups']:
sg_id = security_group['GroupId']
network_interfaces[sg_id] = ec2Client.describe_network_interfaces(Filters=[{'Name': 'group-id', 'Values': [ sg_id ] }])['NetworkInterfaces']
security_group['referenced_groups'] = {}
for permission in security_group['IpPermissions']:
userid_group_pairs = permission['UserIdGroupPairs']
for group in userid_group_pairs:
group_id = group['GroupId']
security_group['referenced_groups'][group_id] = 1
if group_id not in backward_references:
backward_references[group_id] = {}
backward_references[group_id][sg_id] = 1
security_groups[sg_id] = security_group
results = {}
for sg_id in sorted(security_groups):
network_interface = network_interfaces[sg_id]
interface_count = len(network_interface)
if security_groups[sg_id]['GroupName'] == 'default':
results[sg_id] = { 'Count' : interface_count, 'Message' : 'Default' }
else:
printed = False
sg = security_groups[sg_id]
referenced_group_count = len(sg['referenced_groups'])
backward_reference_count = 0
if sg_id in backward_references:
backward_reference_count = len(backward_references[sg_id])
if interface_count > 0:
results[sg_id] = { 'Count' : interface_count }
printed = True
else:
response = ec2Client.describe_tags(Filters=[ { 'Name': 'resource-id', 'Values': [ sg_id ] }, { 'Name': 'key', 'Values': [ 'aws:cloudformation:stack-id' ] } ])
if 'Tags' in response:
if len(response['Tags']) > 0:
components = response['Tags'][0]['Value'].split('/')
stack_name = components[1]
stack_id = cfmClient.Stack(stack_name).stack_id
results[sg_id] = { 'Count' : interface_count, 'StackName' : stack_name, 'StackId' : stack_id }
printed = True
if printed == False and referenced_group_count > 0:
results[sg_id] = { 'Count' : interface_count, 'References' : sg['referenced_groups'] }
printed = True
if printed == False and backward_reference_count > 0:
results[sg_id] = { 'Count' : interface_count, 'ReferencedBy' : backward_references[sg_id] }
printed = True
if printed == False:
results[sg_id] = { 'Count' : interface_count, 'Message' : 'Deleteable' }
return results
04/14/2017: Unable to Connect with AWS Managed Elasticseach - SOLVED
TR;DR - You don’t need to specify a port number when connecting to the AWS managed Elasticsearch service.
That was an easy issue to resolve. While you’re here, let me show you the CloudFormation template that I am using to create a three-node AWS-managed Elasticsearch cluster.
AWSTemplateFormatVersion: 2010-09-09
Description: Elasticsearch Stack
Parameters:
pAwsAccountId:
Type: String
Default: 929717587211
Description: AWS Account Id
pSourceIP:
Type: String
Default: 34.201.116.224
Description: IP of the server accessing Elasticsearch
Resources:
rElasticsearch:
Type: AWS::Elasticsearch::Domain
Properties:
AccessPolicies:
Version: 2012-10-17
Statement:
-
Effect: Allow
Principal:
AWS: "*"
Action: "es:*"
Condition:
IpAddress:
aws:SourceIp:
- !Ref pSourceIP
Resource: !Join [ "", [ "arn:aws:es:", !Ref "AWS::Region", ":", !Ref "AWS::AccountId", ":domain/", !Ref "AWS::StackName", "/*" ] ]
AdvancedOptions:
rest.action.multi.allow_explicit_index: true
indices.fielddata.cache.size: ""
DomainName: !Ref "AWS::StackName"
EBSOptions:
EBSEnabled: true
Iops: 0
VolumeSize: 20
VolumeType: gp2
ElasticsearchClusterConfig:
InstanceCount: 3
InstanceType: t2.medium.elasticsearch
ElasticsearchVersion: 5.1
Tags:
- Key: Name
Value: !Join [ ":", [ !Ref "AWS::StackName", elasticsearch ] ]
Outputs:
oElasticsearchId:
Description: the id of elasticsearch
Value: !Ref rElasticsearch
Export:
Name: !Join [ ":", [ !Ref "AWS::StackName", elasticsearch-id ] ]
oElasticsearchArn:
Description: the arn of elasticsearch
Value: !GetAtt rElasticsearch.DomainArn
Export:
Name: !Join [ ":", [ !Ref "AWS::StackName", elasticsearch-arn ] ]
oElasticsearchEndpoint:
Description: the endpoint of elasticsearch
Value: !GetAtt rElasticsearch.DomainEndpoint
Export:
Name: !Join [ ":", [ !Ref "AWS::StackName", elasticsearch-endpoint ] ]
As soon as the cluster is in the active state, I can use curl (from the source ip server) to access it.
Get the Endpoint from the Elasticseach console.
export ES_ENDPOINT=https://search-app-05-elasticsearch-stack-foy7lzlceajw4walf5yape6iru.us-east-1.es.amazonaws.com
curl -XGET $ES_ENDPOINT/_cat/indices?v