How to Easily Create an AWS EC2 Inventory with AWS CLI

Posted by: Rastko Vasiljevic April 13, 2020

During this current COVID-19 pandemic, it would seem a lot of people are doing some sort of an inventory review. Whether it’s for cost cutdown purposes or just to make the documentation of current projects and do an infrastructure inventory overview, it’s good to know how to obtain an EC2 server inventory on AWS using AWS CLI. When it comes down to cost optimization, AWS offers us quite a few tools that can be used to analyze EC2 resources and gain visibility into the cost associated with these resources. 

Tools like AWS Billing Dashboard, CostExplorer or Cost and Usage Reports can give us valuable insights into this. However, sometimes you just want a quick and simple report that will give you additional, more technical information about your resources. Perhaps in CSV output so that it can be analyzed in a Spreadsheet software.

When we’re talking about EC2 instances, we might want to see how many are running in each region, how many instances there are with a certain Tag name and value combination. For example, how many instances there are for Production, Staging or Development environments. We can also examine certain instance types and check if some of them, combined with a certain environment, are running longer than expected. Or perhaps we just want to see the total amount of Volumes attached to an instance and what the total volume size for each instance is.

In order to do all this, we’ll need some raw data. Afterwards, this data can be analyzed in any Spreadsheet software of your choosing. But first, we’ll need to collect all the information that we need. 

For this, we’ll be using AWS CLI. Installing AWS CLI won’t be covered in this blog post, but make sure that you’ve followed the instructions to install it properly. Also, make sure you configure AWS CLI and/or use a Named Profile. 

The first thing we want to do is to get all the information about all the instances. This is quite simple by using the AWS CLI:

$ aws –profile superadmins –region eu-central-1 ec2 describe-instances

The options –profile and –region are optional. If no profile name is defined, AWS CLI will use the profile that is defined as default. The same goes for the region option. If it’s not provided, the default region (the one that was set when configuring the AWS CLI) will be used. 

The output of this command can be very long, so here’s an example output of one EC2 instance:

{
    "Reservations": [
        {
            "Groups": [],
            "Instances": [
                {
                    "AmiLaunchIndex": 0,
                    "ImageId": "ami-0ec1ba09723e5bfac",
                    "InstanceId": "i-0bd2e3bf4dd08c87e",
                    "InstanceType": "t2.micro",
                    "KeyName": "superadmins",
                    "LaunchTime": "2020-04-05T17:45:50.000Z",
                    "Monitoring": {
                        "State": "disabled"
                    },
                    "Placement": {
                        "AvailabilityZone": "eu-central-1b",
                        "GroupName": "",
                        "Tenancy": "default"
                    },
                    "PrivateDnsName": "ip-172-31-35-135.eu-central-1.compute.internal",
                    "PrivateIpAddress": "172.31.35.135",
                    "ProductCodes": [],
                    "PublicDnsName": "ec2-35-159-46-16.eu-central-1.compute.amazonaws.com",
                    "PublicIpAddress": "35.159.46.16",
                    "State": {
                        "Code": 16,
                        "Name": "running"
                    },
                    "StateTransitionReason": "",
                    "SubnetId": "subnet-a19fa2dc",
                    "VpcId": "vpc-927e60f9",
                    "Architecture": "x86_64",
                    "BlockDeviceMappings": [
                        {
                            "DeviceName": "/dev/xvda",
                            "Ebs": {
                                "AttachTime": "2020-04-05T17:45:51.000Z",
                                "DeleteOnTermination": true,
                                "Status": "attached",
                                "VolumeId": "vol-04bc0ba0fffcff280"
                            }
                        },
                        {
                            "DeviceName": "/dev/sdb",
                            "Ebs": {
                                "AttachTime": "2020-04-05T17:45:51.000Z",
                                "DeleteOnTermination": true,
                                "Status": "attached",
                                "VolumeId": "vol-02d10aece7a0b2dbe"
                            }
                        }
                    ],
                    "ClientToken": "",
                    "EbsOptimized": false,
                    "EnaSupport": true,
                    "Hypervisor": "xen",
                    "NetworkInterfaces": [
                        {
                            "Association": {
                                "IpOwnerId": "amazon",
                                "PublicDnsName": "ec2-35-159-46-16.eu-central-1.compute.amazonaws.com",
                                "PublicIp": "35.159.46.16"
                            },
                            "Attachment": {
                                "AttachTime": "2020-04-05T17:45:50.000Z",
                                "AttachmentId": "eni-attach-096e56caf186b77a4",
                                "DeleteOnTermination": true,
                                "DeviceIndex": 0,
                                "Status": "attached"
                            },
                            "Description": "",
                            "Groups": [
                                {
                                    "GroupName": "launch-wizard-2",
                                    "GroupId": "sg-09c749abf91fd44d6"
                                },
                                {
                                    "GroupName": "launch-wizard-1",
                                    "GroupId": "sg-08587d6adccfec289"
                                }
                            ],
                            "Ipv6Addresses": [],
                            "MacAddress": "06:6b:17:77:80:5c",
                            "NetworkInterfaceId": "eni-02ec5c7c86b0e9f88",
                            "OwnerId": "195799635119",
                            "PrivateDnsName": "ip-172-31-35-135.eu-central-1.compute.internal",
                            "PrivateIpAddress": "172.31.35.135",
                            "PrivateIpAddresses": [
                                {
                                    "Association": {
                                        "IpOwnerId": "amazon",
                                        "PublicDnsName": "ec2-35-159-46-16.eu-central-1.compute.amazonaws.com",
                                        "PublicIp": "35.159.46.16"
                                    },
                                    "Primary": true,
                                    "PrivateDnsName": "ip-172-31-35-135.eu-central-1.compute.internal",
                                    "PrivateIpAddress": "172.31.35.135"
                                }
                            ],
                            "SourceDestCheck": true,
                            "Status": "in-use",
                            "SubnetId": "subnet-a19fa2dc",
                            "VpcId": "vpc-927e60f9",
                            "InterfaceType": "interface"
                        }
                    ],
                    "RootDeviceName": "/dev/xvda",
                    "RootDeviceType": "ebs",
                    "SecurityGroups": [
                        {
                            "GroupName": "launch-wizard-2",
                            "GroupId": "sg-09c749abf91fd44d6"
                        },
                        {
                            "GroupName": "launch-wizard-1",
                            "GroupId": "sg-08587d6adccfec289"
                        }
                    ],
                    "SourceDestCheck": true,
                    "Tags": [
                        {
                            "Key": "Environment",
                            "Value": "Development"
                        },
                        {
                            "Key": "Name",
                            "Value": "Instance Two"
                        }
                    ],
                    "VirtualizationType": "hvm",
                    "CpuOptions": {
                        "CoreCount": 1,
                        "ThreadsPerCore": 1
                    },
                    "CapacityReservationSpecification": {
                        "CapacityReservationPreference": "open"
                    },
                    "HibernationOptions": {
                        "Configured": false
                    },
                    "MetadataOptions": {
                        "State": "applied",
                        "HttpTokens": "optional",
                        "HttpPutResponseHopLimit": 1,
                        "HttpEndpoint": "enabled"
                    }
                }
            ],
            "OwnerId": "195799635119",
            "ReservationId": "r-01c1e015b1772735d"
        }
    ]
}

The default output is in JSON and when we have a lot of instances, the output for all the instances will be a long JSON output that isn’t practical for us. In order to make this more readable, we can modify the output by using the –output text option of the AWS CLI. This gives us a plain text output:

RESERVATIONS    195799635119 r-01c1e015b1772735d
INSTANCES       0 x86_64         False True xen ami-0ec1ba09723e5bfac   i-0bd2e3bf4dd08c87e t2.micro superadmins     2020-04-05T17:45:50.000Z ip-172-31-35-135.eu-central-1.compute.internal  172.31.35.135 ec2-35-159-46-16.eu-central-1.compute.amazonaws.com 35.159.46.16    /dev/xvda ebs True
        subnet-a19fa2dc hvm     vpc-927e60f9
BLOCKDEVICEMAPPINGS     /dev/xvda
EBS     2020-04-05T17:45:51.000Z        True attached vol-04bc0ba0fffcff280
BLOCKDEVICEMAPPINGS     /dev/sdb
EBS     2020-04-05T17:45:51.000Z        True attached vol-02d10aece7a0b2dbe
CAPACITYRESERVATIONSPECIFICATION        open
CPUOPTIONS      1 1
HIBERNATIONOPTIONS      False
METADATAOPTIONS enabled 1       optional applied
MONITORING      disabled
NETWORKINTERFACES               interface 06:6b:17:77:80:5c       eni-02ec5c7c86b0e9f88 195799635119 ip-172-31-35-135.eu-central-1.compute.internal       172.31.35.135 True in-use subnet-a19fa2dc vpc-927e60f9
ASSOCIATION     amazon ec2-35-159-46-16.eu-central-1.compute.amazonaws.com     35.159.46.16
ATTACHMENT      2020-04-05T17:45:50.000Z        eni-attach-096e56caf186b77a4 True    0 attached
GROUPS  sg-09c749abf91fd44d6    launch-wizard-2
GROUPS  sg-08587d6adccfec289    launch-wizard-1
PRIVATEIPADDRESSES      True ip-172-31-35-135.eu-central-1.compute.internal  172.31.35.135
ASSOCIATION     amazon ec2-35-159-46-16.eu-central-1.compute.amazonaws.com     35.159.46.16
PLACEMENT       eu-central-1b         default
SECURITYGROUPS  sg-09c749abf91fd44d6    launch-wizard-2
SECURITYGROUPS  sg-08587d6adccfec289    launch-wizard-1
STATE   16 running
TAGS    Environment     Development
TAGS    Name Instance Two

Both outputs give us the same information but in different formats. As you can see, there’s quite a lot of information that goes into describing one instance. Normally, we would like to know just a few of them. In order to do so, we can use the –query option that most AWS CLI commands offer. The query option is quite powerful and lets you control the output of the AWS CLI command and filter it.

Let’s say that we want to get the information like: InstanceID, InstanceType and the value of the Name tag. We can do this by filtering the output using –query :

$ aws ec2 describe-instances --instance-ids i-0bd2e3bf4dd08c87e --output text --query 'Reservations[].Instances[].[InstanceId, InstanceType, [Tags[?Key==Name].Value] [0][0] ]'

As you can see, the output is just one line of text with tab as a delimiter:

i-0bd2e3bf4dd08c87e     t2.micro        Instance Two

In this example we’re extracting Tag values so that they’re included in our output. It’s usually the Tags that describe our instances and they can provide us with information like which environment the instance belongs to or what the name of the instance is, so it’s very important to use Tags correctly. From here, it’s easy to modify the tab delimiter into a comma and in a few commands, we have a CSV file that’s ready to be imported and analyzed. We can also  modify the query filter and include much more additional information:

$ aws ec2 describe-instances \
--output text \
--query 'Reservations[].Instances[].[InstanceId, InstanceType, ImageId,
KeyName, State.Name, LaunchTime, Placement.AvailabilityZone, Placement.Tenancy,
PrivateIpAddress, PrivateDnsName, PublicDnsName, PublicIpAddress, SubnetId, VpcId,
[Tags[?Key==Name].Value] [0][0], [Tags[?Key==Environment].Value] [0][0] ]'

In addition to all the information that the command ec2 describe-instances has to offer, it doesn’t give us insight into the details of the EBS Volumes attached. To be more specific, we don’t know the size of the volumes attached. We can use the command ec2 describe-volumes that will provide us with this kind of information:

$ aws ec2 describe-volumes

A truncated output would be:

{
"Volumes": [
{
"Attachments": [
{
"AttachTime": "2020-04-05T16:10:15.000Z",
"Device": "/dev/xvda",
"InstanceId": "i-03ef573bf2c103e1a",
"State": "attached",
"VolumeId": "vol-0a453064ea50ef708",
"DeleteOnTermination": true
}
],
"AvailabilityZone": "eu-central-1b",
"CreateTime": "2020-04-05T16:10:15.412Z",
"Encrypted": false,
"Size": 8,
"SnapshotId": "snap-06f71b9398ff7f371",
"State": "in-use",
"VolumeId": "vol-0a453064ea50ef708",
"Iops": 100,
"VolumeType": "gp2"
}

Same as before, with the ec2 describe-instances command – we can modify the output of this command and filter out the data.

$ aws ec2 describe-volumes \
--output text \
--filters "Name=status,Values=in-use" \
--query 'Volumes[*].[Attachments[0].InstanceId,VolumeId,Attachments[0].State,
AvailabilityZone,Size,State,Iops,VolumeType]'

The output of this command is also tab delimited:

i-03ef573bf2c103e1a vol-0a453064ea50ef708 attached eu-central-1b 8 in-use 100 gp2
i-0bd2e3bf4dd08c87e vol-02d10aece7a0b2dbe attached eu-central-1b 15 in-use 100 gp2
i-0bd2e3bf4dd08c87e vol-04bc0ba0fffcff280 attached eu-central-1b 8 in-use 100 gp2

You may have noticed that in this command we’ve used the –filter option to additionally modify the output so that it only shows Volumes that are currently in use. At this point, we have outputs that can easily be converted to a CSV format and they both have the same identifier – the “InstanceID”, by which we can join the two CSV files with a tool like csvjoin. By doing so, we’ve created an EC2 server inventory file that can be used to analyze all EC2 instances. Now, you might be asking yourself: how can I automate this whole process and go through all the regions instead of doing one region at a time? So far, all of this can be scripted easily and automated, but the key thing is to get all the regions and loop through them. To get all the regions, we can also use the AWS CLI with the command describe-regions :

$ ec2 describe-regions --output text --query 'Regions[].[RegionName]'

Hopefully, this blog post helped you understand the powerful options that AWS CLI has to offer, as well as how to use some of the commands that can help you build your EC2 instance inventory file.

Don’t forget

you can always reach to us for an AWS consultation.

Start

Keep reading

A developer’s handbook for security best practices

20.11.2020

One thing that you notice once you embark on the arcane ways of offensive InfoSec (a.k.a penetration testing/red team engagement) is that your success is the direct consequence of someone else’s error.

read more

The Benefits & Implementation of the DevSecOps (SecDevOps) Approach to SDLC

10.11.2020

The DevSecOps (SecDevOps) approach to software development is based on incorporating security in each and every stage of the cycle.

read more

SaaS: Cloud-native versus On-prem

05.11.2020

Over the last decade, we witness that so many start-up businesses are SaaS-based solutions, mainly because the payment model moves from license-based to the subscription-based.

read more