How To: Reinforcing the bastion host

So, you have your crew scattered around the globe and those that have static IP are few and far between. You solved the server access by introducing the bastion host, but new issues arise all the time. You were able to plug one leak only to discover another; bastion host which must be widely accessible. VPN springs to mind, but they all use multiple VPNs already. Not very convenient. There must be a better way!

Amazon introduced the EC2 instance connect feature back in 2019, and most of us immediately noticed it in our Dashboards. It was a tad confusing at first as only Amazon Linux 2 instances, sort of natively supported it. But over time, Ubuntu was included too and now we have something to work with. With the Centos project being put to sleep, Ubuntu and Amazon Linux are most likely candidates if you are in the AWS circles (and we are!).

So, what is special here? We can connect to our instances from the dashboard and that’s very convenient, but is there more to it than meets the eye? Well, yes. There is.

Reinforcing the bastion host is a 3 step process

The same mechanisms that let you connect from the Dashboard will let you connect from your PC. Once you dive a bit deeper, you will discover that instance metadata is heavily used, in conjunction with AWS CLI. If it can be used in the Dash, it can on your terminal too.

The basic logic is to request a temporary SSH key pair be pushed to your instance metadata, fetch it, then use it to log in. Key pair will be generated and used and will perish in 60 seconds. Sounds good?

The mechanism involves having a user, or users, with the right permissions to do that, and having EC2 instance connect installed on your server instances as well as your PC from which you connect. This guide assumes you have AWS CLI already installed with AWS profile configured to use your IAM keys.

The whole endeavor is a 3-step process:

• Make sure your IAM can push keys and connect.
• Install EC2 instance connect package on your servers.
• Install AWS CLI on the local PC.

Step 1: Setting the IAM

Let’s configure your user to be able to connect. If you have multiple AWS accounts which you manage, you might benefit from using the same user on all of them. It will save you some valuable time.

Your IAM will need AWS managed policy called “EC2InstanceConnect” which will now include EC2:DescribeInstance policy to accompany it. It used to be two policies allowing you to use SSH and CLI separately, but kind people of Amazon merged those recently.

In our example, we opted for creating a new policy in which we will define where keys can be pushed to.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "0912853760",
            "Effect": "Allow",
            "Action": "ec2:DescribeInstances",
            "Resource": "*"
        },
        {
            "Sid": "1278354910",
            "Effect": "Allow",
            "Action": "ec2-instance-connect:SendSSHPublicKey",
            "Resource": "arn:aws:ec2:us-east-1:678756478094:instance/i-0bcd21134f6bf6fe0"
        }
    ]
}

You might be tempted to give your user “AdministratorAccess” but it’s a double-edged sword. That user will then be able to push the keys to any user of any instance, creating room for mischief. It can be handled though, by using explicit deny policies, but we will leave that to you to decide. Let’s play safe for now.

Above policy will let IAM access instance metadata, and then push the temporary key pair, but only to this exact resource. In our case, bastion used for this example was located in the region us-east-1, belongs to account 678756478094 and its ID is i-0bcd21134f6bf6fe0

This will ensure our users will be able to connect to bastion only.

Step 2: Installing EC2 on the servers

Once you have your IAM all configured it’s time to sort the servers. Amazon Linux 2 is being shipped with the EC2 instance connect bundled, but there’s been a recent bug with “eic_run_authorized_keys” not being able to run properly, so make sure you do a “yum update” before you start. Latest kernel (at the time of this writing, Linux 4.14.214-160.339.amzn2.x86_64 works well) is also highly desirable.

If for any reason the package is not there, please run:

$ sudo yum install ec2-instance-connect

On your ubuntu 16+ servers you will need to install EC2 instance connect by running:

$ sudo apt-get install ec2-instance-connect

Once that is done, a quick ls of “/opt/aws/bin/“ for Amazon Linux and “/usr/share/ec2-instance-connect/“ for Ubuntu will show you four new files:

eic_curl_authorized_keys
eic_harvest_hostkeys
eic_parse_authorized_keys
eic_run_authorized_keys

These will make it possible for your instance to lookup temporary ssh key pair.

Maybe the most important change will be two lines written in your sshd.conf

AuthorizedKeysCommand /opt/aws/bin/eic_run_authorized_keys %u %f
AuthorizedKeysCommandUser ec2-instance-connect

These will be ran when you try and connect. Keep in mind that if these were used before for any reason, install will detect them and will not change them in any way, making this setup non-functioning. Make sure if you have these entries anywhere, that you can safely remove them beforehand.

Step 3: AWS CLI on local PC

Let’s move to our PC.

You can get EC2 instance connect CLI package in many different ways, but we will cover two that we prefer.

Fetch the latest package and install it using pip:

$ aws s3api get-object --bucket ec2-instance-connect --key cli/ec2instanceconnectcli-latest.tar.gz ec2instanceconnectcli-latest.tar.gz
$ sudo pip install ec2instanceconnectcli-latest.tar.gz

The other way would be to use pip repository:

$ pip install ec2instanceconnectcli

While pip repos are famous for being up-to-date, you might be better off with fetching the latest and greatest from the source itself. Your choice, we won’t judge.

Now that the complicated part is done, all we need is to connect.

$ mssh -r us-east-1 -u bastion-profile-aws ec2-user@ i-0bcd21134f6bf6fe0

# mssh -r us-east-1 -u bastion-profile-aws ec2-user@ i-0bcd21134f6bf6fe0
       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/
[ec2-user@ip-10-100-1-1 ~]$ _

Voila. Your users/clients can now use regular SSH that you have already configured and jump to any server you allow them to. Limiting your server to be able to accept SSH traffic only from the bastion instance IP will greatly improve your security.

Pros of this configuration are:

• In case of the credentials leak, you only need to disable the affected IAM keys and the gateway to your servers is permanently closed
• Your users need to do IAM authentication only for Bastion host making it simple to use
• You can regularly rotate access keys for your users, with them spending minimal amount of time to update their AWS profile locally
• IAM policy allows many customizations including the MFA enforcement