How find to an EC2 instance that isn't publicly accessible

Question

I'm on AWS and have two EC2 instances. I have an application server (app-server) that's world-accessible and I'd like it to talk to secret-server over the internal network. So, my app-server is basically running:

r = requests.get('http://10.1.2.3/stuff') 

However, secret-server's private IP changes whenever I shut it down/upgrade it/whatever. How is app-server supposed to find it again? secret-server.us-east-1.elasticbeanstalk.com resolves to the public IP, which is blocked via security groups. The private DNS address is, unhelpfully, ip-10.1.2.3.ec2.internal, which of course changes whenever the IP address changes.

Basically, I want to be able to configure a static private IP address or attach a DNS name whenever the EC2 instance changes that resolves to the private IP.

I'm using Elastic Beanstalk, if that matters.

Options I see:

• Restrict the subnet to one IP address, but that seems suboptimal.
• Add a second step to the deploy that sets the IP address to something static that hopefully AWS won't use for anyone else. This seems fragile and easy to forget.
• As suggested in the comment below, I tried adding a route53 route with secret-server's Elastic Beanstalk environment as the alias, but then the domain resolves to the public IP.

Answer 1

That's what security groups are for so you don't have to worry about IP addressing. In the App-Server security allow the traffic to the Secret Server's Security group only on the port the App server needs to talk to the Secret server.

Now if if we are talking in App, you can provision an internal Load balancer and place the secret server behind it that way the you can reference the load balancer's DNS name and not have to worry about the secret server's IP. Or you do what other have suggested and create a Private IP on a NIC and re-associate that one with new instance, that doesn't scale as well as the internal ELB.

https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-create-internal-load-balancer.html

Answer 2

Figured it out. Basically the answer is that you set up an internal load balancer, but it's a bit more complicated because Elastic Beanstalk requires you to configure your network in a specific way.

Here are the steps:

• Create a public subnet to be your DMZ.
• Create a NAT gateway that lives in your public subnet.
• Create a routing table for your public subnet that routes all intra-VPC traffic to the VPC's CIDR block and 0.0.0.0/0 to your internet gateway.
• Create a routing tables for the private subnets (where your EB applications will run) that routes intra-VPC traffic to the VPC's CIDR block and 0.0.0.0/0 to the nat address.
• Wait ~5 minutes for your changes to propagate.
• Go to Configuration -> Scaling and select that you want to enable load balancing.
• When it give you options to modify: don't change anything about the ELB and EC2 instances' subnets! I think it's a bug in the UI, but if you click any of the checkboxes, it will only allow you to have an ELB or an EC2 instance in that availability zone's subnet, but then the page will error out because you need one of each. Finally, select the option to have an "Internal" load balancer.
• Select save and wait for your config to update.

At that point, secret-server.us-east-1.elasticbeanstalk.com will resolve to a private IP.

If you enable the private load balancer but don't set up the NAT gateway, routes, and subnets, secret-server.us-east-1.elasticbeanstalk.com will resolve to a private IP. Unfortunately, your service will also transition to Severely Degraded and have no logs available (unless you pre-configured your network in ways that seem unlikely). This is because Elastic Beanstalk, when it starts up an EC2 instance, downloads some setup scripts from S3. However, your load-balanced Elastic Beanstalk EC2 instances will not be able to reach the internet, even if you haven't restricted outbound communication in security groups, because they won't have public IPs.

The solution is to set up a NAT gateways that all of your private subnets route to and a "DMZ" subnet that hosts the gateway and can actually access the internet. Amazon's docs have an example of this: VPC with Public and Private Subnets (NAT):

The instances in the public subnet can send outbound traffic directly to the Internet, whereas the instances in the private subnet can't. Instead, the instances in the private subnet can access the Internet by using a network address translation (NAT) gateway that resides in the public subnet.

I really feel like Elastic Beanstalk could maybe do some of this for you or at least document it better, but whatever. At least it's working.