During this current COVID-19 pandemic, it would seem a lot of people are doing some sort of an inventory review. Whether it’s for cost cutdown purposes or just to make the documentation of current projects and do an infrastructure inventory overview, it’s good to know how to obtain an EC2 server inventory on AWS using AWS CLI. When it comes down to cost optimization, AWS offers us quite a few tools that can be used to analyze EC2 resources and gain visibility into the cost associated with these resources.
Tools like AWS Billing Dashboard, CostExplorer or Cost and Usage Reports can give us valuable insights into this. However, sometimes you just want a quick and simple report that will give you additional, more technical information about your resources. Perhaps in CSV output so that it can be analyzed in a Spreadsheet software.
When we’re talking about EC2 instances, we might want to see how many are running in each region, how many instances there are with a certain Tag name and value combination. For example, how many instances there are for Production, Staging or Development environments. We can also examine certain instance types and check if some of them, combined with a certain environment, are running longer than expected. Or perhaps we just want to see the total amount of Volumes attached to an instance and what the total volume size for each instance is.
In order to do all this, we’ll need some raw data. Afterwards, this data can be analyzed in any Spreadsheet software of your choosing. But first, we’ll need to collect all the information that we need.
For this, we’ll be using AWS CLI. Installing AWS CLI won’t be covered in this blog post, but make sure that you’ve followed the instructions to install it properly. Also, make sure you configure AWS CLI and/or use a Named Profile.
The first thing we want to do is to get all the information about all the instances. This is quite simple by using the AWS CLI:
$ aws –profile superadmins –region eu-central-1 ec2 describe-instances
The options –profile and –region are optional. If no profile name is defined, AWS CLI will use the profile that is defined as default. The same goes for the region option. If it’s not provided, the default region (the one that was set when configuring the AWS CLI) will be used.
The output of this command can be very long, so here’s an example output of one EC2 instance:
{
"Reservations": [
{
"Groups": [],
"Instances": [
{
"AmiLaunchIndex": 0,
"ImageId": "ami-0ec1ba09723e5bfac",
"InstanceId": "i-0bd2e3bf4dd08c87e",
"InstanceType": "t2.micro",
"KeyName": "superadmins",
"LaunchTime": "2020-04-05T17:45:50.000Z",
"Monitoring": {
"State": "disabled"
},
"Placement": {
"AvailabilityZone": "eu-central-1b",
"GroupName": "",
"Tenancy": "default"
},
"PrivateDnsName": "ip-172-31-35-135.eu-central-1.compute.internal",
"PrivateIpAddress": "172.31.35.135",
"ProductCodes": [],
"PublicDnsName": "ec2-35-159-46-16.eu-central-1.compute.amazonaws.com",
"PublicIpAddress": "35.159.46.16",
"State": {
"Code": 16,
"Name": "running"
},
"StateTransitionReason": "",
"SubnetId": "subnet-a19fa2dc",
"VpcId": "vpc-927e60f9",
"Architecture": "x86_64",
"BlockDeviceMappings": [
{
"DeviceName": "/dev/xvda",
"Ebs": {
"AttachTime": "2020-04-05T17:45:51.000Z",
"DeleteOnTermination": true,
"Status": "attached",
"VolumeId": "vol-04bc0ba0fffcff280"
}
},
{
"DeviceName": "/dev/sdb",
"Ebs": {
"AttachTime": "2020-04-05T17:45:51.000Z",
"DeleteOnTermination": true,
"Status": "attached",
"VolumeId": "vol-02d10aece7a0b2dbe"
}
}
],
"ClientToken": "",
"EbsOptimized": false,
"EnaSupport": true,
"Hypervisor": "xen",
"NetworkInterfaces": [
{
"Association": {
"IpOwnerId": "amazon",
"PublicDnsName": "ec2-35-159-46-16.eu-central-1.compute.amazonaws.com",
"PublicIp": "35.159.46.16"
},
"Attachment": {
"AttachTime": "2020-04-05T17:45:50.000Z",
"AttachmentId": "eni-attach-096e56caf186b77a4",
"DeleteOnTermination": true,
"DeviceIndex": 0,
"Status": "attached"
},
"Description": "",
"Groups": [
{
"GroupName": "launch-wizard-2",
"GroupId": "sg-09c749abf91fd44d6"
},
{
"GroupName": "launch-wizard-1",
"GroupId": "sg-08587d6adccfec289"
}
],
"Ipv6Addresses": [],
"MacAddress": "06:6b:17:77:80:5c",
"NetworkInterfaceId": "eni-02ec5c7c86b0e9f88",
"OwnerId": "195799635119",
"PrivateDnsName": "ip-172-31-35-135.eu-central-1.compute.internal",
"PrivateIpAddress": "172.31.35.135",
"PrivateIpAddresses": [
{
"Association": {
"IpOwnerId": "amazon",
"PublicDnsName": "ec2-35-159-46-16.eu-central-1.compute.amazonaws.com",
"PublicIp": "35.159.46.16"
},
"Primary": true,
"PrivateDnsName": "ip-172-31-35-135.eu-central-1.compute.internal",
"PrivateIpAddress": "172.31.35.135"
}
],
"SourceDestCheck": true,
"Status": "in-use",
"SubnetId": "subnet-a19fa2dc",
"VpcId": "vpc-927e60f9",
"InterfaceType": "interface"
}
],
"RootDeviceName": "/dev/xvda",
"RootDeviceType": "ebs",
"SecurityGroups": [
{
"GroupName": "launch-wizard-2",
"GroupId": "sg-09c749abf91fd44d6"
},
{
"GroupName": "launch-wizard-1",
"GroupId": "sg-08587d6adccfec289"
}
],
"SourceDestCheck": true,
"Tags": [
{
"Key": "Environment",
"Value": "Development"
},
{
"Key": "Name",
"Value": "Instance Two"
}
],
"VirtualizationType": "hvm",
"CpuOptions": {
"CoreCount": 1,
"ThreadsPerCore": 1
},
"CapacityReservationSpecification": {
"CapacityReservationPreference": "open"
},
"HibernationOptions": {
"Configured": false
},
"MetadataOptions": {
"State": "applied",
"HttpTokens": "optional",
"HttpPutResponseHopLimit": 1,
"HttpEndpoint": "enabled"
}
}
],
"OwnerId": "195799635119",
"ReservationId": "r-01c1e015b1772735d"
}
]
}
The default output is in JSON and when we have a lot of instances, the output for all the instances will be a long JSON output that isn’t practical for us. In order to make this more readable, we can modify the output by using the –output text option of the AWS CLI. This gives us a plain text output:
RESERVATIONS 195799635119 r-01c1e015b1772735d
INSTANCES 0 x86_64 False True xen ami-0ec1ba09723e5bfac i-0bd2e3bf4dd08c87e t2.micro superadmins 2020-04-05T17:45:50.000Z ip-172-31-35-135.eu-central-1.compute.internal 172.31.35.135 ec2-35-159-46-16.eu-central-1.compute.amazonaws.com 35.159.46.16 /dev/xvda ebs True
subnet-a19fa2dc hvm vpc-927e60f9
BLOCKDEVICEMAPPINGS /dev/xvda
EBS 2020-04-05T17:45:51.000Z True attached vol-04bc0ba0fffcff280
BLOCKDEVICEMAPPINGS /dev/sdb
EBS 2020-04-05T17:45:51.000Z True attached vol-02d10aece7a0b2dbe
CAPACITYRESERVATIONSPECIFICATION open
CPUOPTIONS 1 1
HIBERNATIONOPTIONS False
METADATAOPTIONS enabled 1 optional applied
MONITORING disabled
NETWORKINTERFACES interface 06:6b:17:77:80:5c eni-02ec5c7c86b0e9f88 195799635119 ip-172-31-35-135.eu-central-1.compute.internal 172.31.35.135 True in-use subnet-a19fa2dc vpc-927e60f9
ASSOCIATION amazon ec2-35-159-46-16.eu-central-1.compute.amazonaws.com 35.159.46.16
ATTACHMENT 2020-04-05T17:45:50.000Z eni-attach-096e56caf186b77a4 True 0 attached
GROUPS sg-09c749abf91fd44d6 launch-wizard-2
GROUPS sg-08587d6adccfec289 launch-wizard-1
PRIVATEIPADDRESSES True ip-172-31-35-135.eu-central-1.compute.internal 172.31.35.135
ASSOCIATION amazon ec2-35-159-46-16.eu-central-1.compute.amazonaws.com 35.159.46.16
PLACEMENT eu-central-1b default
SECURITYGROUPS sg-09c749abf91fd44d6 launch-wizard-2
SECURITYGROUPS sg-08587d6adccfec289 launch-wizard-1
STATE 16 running
TAGS Environment Development
TAGS Name Instance Two
Both outputs give us the same information but in different formats. As you can see, there’s quite a lot of information that goes into describing one instance. Normally, we would like to know just a few of them. In order to do so, we can use the –query option that most AWS CLI commands offer. The query option is quite powerful and lets you control the output of the AWS CLI command and filter it.
Let’s say that we want to get the information like: InstanceID, InstanceType and the value of the Name tag. We can do this by filtering the output using –query :
$ aws ec2 describe-instances --instance-ids i-0bd2e3bf4dd08c87e --output text --query 'Reservations[].Instances[].[InstanceId, InstanceType, [Tags[?Key==Name].Value] [0][0] ]'
As you can see, the output is just one line of text with tab as a delimiter:
i-0bd2e3bf4dd08c87e t2.micro Instance Two
In this example we’re extracting Tag values so that they’re included in our output. It’s usually the Tags that describe our instances and they can provide us with information like which environment the instance belongs to or what the name of the instance is, so it’s very important to use Tags correctly. From here, it’s easy to modify the tab delimiter into a comma and in a few commands, we have a CSV file that’s ready to be imported and analyzed. We can also modify the query filter and include much more additional information:
$ aws ec2 describe-instances \
--output text \
--query 'Reservations[].Instances[].[InstanceId, InstanceType, ImageId,
KeyName, State.Name, LaunchTime, Placement.AvailabilityZone, Placement.Tenancy,
PrivateIpAddress, PrivateDnsName, PublicDnsName, PublicIpAddress, SubnetId, VpcId,
[Tags[?Key==Name].Value] [0][0], [Tags[?Key==Environment].Value] [0][0] ]'
In addition to all the information that the command ec2 describe-instances has to offer, it doesn’t give us insight into the details of the EBS Volumes attached. To be more specific, we don’t know the size of the volumes attached. We can use the command ec2 describe-volumes that will provide us with this kind of information:
$ aws ec2 describe-volumes
A truncated output would be:
{
"Volumes": [
{
"Attachments": [
{
"AttachTime": "2020-04-05T16:10:15.000Z",
"Device": "/dev/xvda",
"InstanceId": "i-03ef573bf2c103e1a",
"State": "attached",
"VolumeId": "vol-0a453064ea50ef708",
"DeleteOnTermination": true
}
],
"AvailabilityZone": "eu-central-1b",
"CreateTime": "2020-04-05T16:10:15.412Z",
"Encrypted": false,
"Size": 8,
"SnapshotId": "snap-06f71b9398ff7f371",
"State": "in-use",
"VolumeId": "vol-0a453064ea50ef708",
"Iops": 100,
"VolumeType": "gp2"
}
Same as before, with the ec2 describe-instances command – we can modify the output of this command and filter out the data.
$ aws ec2 describe-volumes \
--output text \
--filters "Name=status,Values=in-use" \
--query 'Volumes[*].[Attachments[0].InstanceId,VolumeId,Attachments[0].State,
AvailabilityZone,Size,State,Iops,VolumeType]'
The output of this command is also tab delimited:
i-03ef573bf2c103e1a vol-0a453064ea50ef708 attached eu-central-1b 8 in-use 100 gp2
i-0bd2e3bf4dd08c87e vol-02d10aece7a0b2dbe attached eu-central-1b 15 in-use 100 gp2
i-0bd2e3bf4dd08c87e vol-04bc0ba0fffcff280 attached eu-central-1b 8 in-use 100 gp2
You may have noticed that in this command we’ve used the –filter option to additionally modify the output so that it only shows Volumes that are currently in use. At this point, we have outputs that can easily be converted to a CSV format and they both have the same identifier – the “InstanceID”, by which we can join the two CSV files with a tool like csvjoin. By doing so, we’ve created an EC2 server inventory file that can be used to analyze all EC2 instances. Now, you might be asking yourself: how can I automate this whole process and go through all the regions instead of doing one region at a time? So far, all of this can be scripted easily and automated, but the key thing is to get all the regions and loop through them. To get all the regions, we can also use the AWS CLI with the command describe-regions :
$ ec2 describe-regions --output text --query 'Regions[].[RegionName]'
Hopefully, this blog post helped you understand the powerful options that AWS CLI has to offer, as well as how to use some of the commands that can help you build your EC2 instance inventory file.
