How To: Reinforcing the bastion host

Posted by: Boris Vezmar May 11, 2021

Category: Topic: AWS

So, you have your crew scattered around the globe and those that have static IP are few and far between. You solved the server access by introducing the bastion host, but new issues arise all the time. You were able to plug one leak only to discover another; bastion host which must be widely accessible. VPN springs to mind, but they all use multiple VPNs already. Not very convenient. There must be a better way!

Amazon introduced the EC2 instance connect feature back in 2019, and most of us immediately noticed it in our Dashboards. It was a tad confusing at first as only Amazon Linux 2 instances, sort of natively supported it. But over time, Ubuntu was included too and now we have something to work with. With the Centos project being put to sleep, Ubuntu and Amazon Linux are most likely candidates if you are in the AWS circles (and we are!).

So, what is special here? We can connect to our instances from the dashboard and that’s very convenient, but is there more to it than meets the eye? Well, yes. There is.

Reinforcing the bastion host is a 3 step process

The same mechanisms that let you connect from the Dashboard will let you connect from your PC. Once you dive a bit deeper, you will discover that instance metadata is heavily used, in conjunction with AWS CLI. If it can be used in the Dash, it can on your terminal too.

The basic logic is to request a temporary SSH key pair be pushed to your instance metadata, fetch it, then use it to log in. Key pair will be generated and used and will perish in 60 seconds. Sounds good?

The mechanism involves having a user, or users, with the right permissions to do that, and having EC2 instance connect installed on your server instances as well as your PC from which you connect. This guide assumes you have AWS CLI already installed with AWS profile configured to use your IAM keys.

The whole endeavor is a 3-step process:

Step 1: Setting the IAM

Let’s configure your user to be able to connect. If you have multiple AWS accounts which you manage, you might benefit from using the same user on all of them. It will save you some valuable time.

Your IAM will need AWS managed policy called “EC2InstanceConnect” which will now include EC2:DescribeInstance policy to accompany it. It used to be two policies allowing you to use SSH and CLI separately, but kind people of Amazon merged those recently.

In our example, we opted for creating a new policy in which we will define where keys can be pushed to.

    "Version": "2012-10-17",
    "Statement": [
            "Sid": "0912853760",
            "Effect": "Allow",
            "Action": "ec2:DescribeInstances",
            "Resource": "*"
            "Sid": "1278354910",
            "Effect": "Allow",
            "Action": "ec2-instance-connect:SendSSHPublicKey",
            "Resource": "arn:aws:ec2:us-east-1:678756478094:instance/i-0bcd21134f6bf6fe0"

You might be tempted to give your user “AdministratorAccess” but it’s a double-edged sword. That user will then be able to push the keys to any user of any instance, creating room for mischief. It can be handled though, by using explicit deny policies, but we will leave that to you to decide. Let’s play safe for now.

Keep reading

How To Easily Create An AWS EC2 Inventory With AWS CLI

Learn how

Above policy will let IAM access instance metadata, and then push the temporary key pair, but only to this exact resource. In our case, bastion used for this example was located in the region us-east-1, belongs to account 678756478094 and its ID is i-0bcd21134f6bf6fe0

This will ensure our users will be able to connect to bastion only.

Step 2: Installing EC2 on the servers

Once you have your IAM all configured it’s time to sort the servers. Amazon Linux 2 is being shipped with the EC2 instance connect bundled, but there’s been a recent bug with “eic_run_authorized_keys” not being able to run properly, so make sure you do a “yum update” before you start. Latest kernel (at the time of this writing, Linux 4.14.214-160.339.amzn2.x86_64 works well) is also highly desirable.

If for any reason the package is not there, please run:

$ sudo yum install ec2-instance-connect

On your ubuntu 16+ servers you will need to install EC2 instance connect by running:

$ sudo apt-get install ec2-instance-connect

Once that is done, a quick ls of “/opt/aws/bin/“ for Amazon Linux and “/usr/share/ec2-instance-connect/“ for Ubuntu will show you four new files:


These will make it possible for your instance to lookup temporary ssh key pair.

Maybe the most important change will be two lines written in your sshd.conf

AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys %u %f
AuthorizedKeysCommandUser ec2-instance-connect

These will be ran when you try and connect. Keep in mind that if these were used before for any reason, install will detect them and will not change them in any way, making this setup non-functioning. Make sure if you have these entries anywhere, that you can safely remove them beforehand.

Step 3: AWS CLI on local PC

Let’s move to our PC.

You can get EC2 instance connect CLI package in many different ways, but we will cover two that we prefer.

Fetch the latest package and install it using pip:

$ aws s3api get-object --bucket ec2-instance-connect --key cli/ec2instanceconnectcli-latest.tar.gz ec2instanceconnectcli-latest.tar.gz
$ sudo pip install ec2instanceconnectcli-latest.tar.gz

The other way would be to use pip repository:

$ pip install ec2instanceconnectcli

While pip repos are famous for being up-to-date, you might be better off with fetching the latest and greatest from the source itself. Your choice, we won’t judge.

Now that the complicated part is done, all we need is to connect.

$ mssh -r us-east-1 -u bastion-profile-aws ec2-user@ i-0bcd21134f6bf6fe0

# mssh -r us-east-1 -u bastion-profile-aws ec2-user@ i-0bcd21134f6bf6fe0
       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
[ec2-user@ip-10-100-1-1 ~]$ _

Voila. Your users/clients can now use regular SSH that you have already configured and jump to any server you allow them to. Limiting your server to be able to accept SSH traffic only from the bastion instance IP will greatly improve your security.

Pros of this configuration are:

We breathe AWS

SuperAdmins AWS architects know how to make your cloud work. Drop us a line - the consultation is free.


Keep reading

How to Migrate Your Application from Monolith to Microservices


According to a 2020 O’Reilly report on microservices adoption, 77% of businesses have already adopted microservices, while 92% of them state they are experiencing success after migrating monolith to microservices. The trend to en-masse start using microservices application architecture comes as no surprise considering the multiple benefits that the microservices architecture provides, including: Improved resilience […]

read more

What is Cloud Bursting and Does Your Business Need it?


Back in 2014, NASA launched a satellite called OCO-2 (Orbiting Carbon Observatory 2) to gain more insight into the Earth’s carbon uptake. Two years later, they had petabytes of gathered data that needed to be processed which would have taken over 3 months and cost about $200,000 if they had used on-premise data centers for […]

read more

How Managed Cloud & Open Source Solutions Fuel Business Growth


With the modern business landscape becoming more and more saturated and competitive in recent years, companies worldwide face an ongoing challenge of emerging through the noise and initiating growth. The resources are often suboptimal, which can make the process of developing a cost-effective and sustainable business strategy an insurmountable problem for many companies.  Luckily, with […]

read more