DEV Community

Hugo Prudente
Hugo Prudente

Posted on

Using AWS Network ACLs With NAT Gateway

#aws #natgateway #nacl #networking

It’s quite common the mistakes made when using the AWS Network ACLs for adding that extra layer of security in your VPC.

Given the fact that Network ACLs are stateless, meaning that the Inbound (Ingress) should have a matching rule for Outbound (Egress).

With this post, you will learn how to identify such particularities using an AWS ElasticBeanstalk environment as an example, due to its sensitivity regarding network access requirements.

Common Issues

  • Fail to access any HTTPS/TLS endpoint resulting in timeout
  • Fail to sync NTP servers

Difference between Security Group and Network ACL (NACL)

The main difference between the Security Group and the Network ACL (NACL) is the
the context where they are applied and the type of rules they provided.

  • A Security Group:
    Stateful: Therefore you don't need a rule that allows response traffic for inbound requests.
    Local: Therefore it applies only to the instance or service to which the security group is attached to.

  • A Network ACL:
    Stateless: Therefore this rule is required to allow response traffic for inbound requests on the outbound rules.
    Global: Therefore it applies to all services that are placed on the subnet that is attached to.

AWS Network NACL & Security Group

Scenario 1 - (Web Tier) Public Subnet with Network ACL without AWS NAT Gateway.

Using the diagram below as an example to configure the Network ACL's for an ElasticBeanstalk environment.

Diagram:

+------------------+ | | | +----------+ | | | INSTANCE | | | +----------+ | | | SG-001 | | | +----------+ | +------------------+ | Subnet Pub | +------------------+ + +--------------+ |---| ACL-001 | v +--------------+ +------------------+ | IGW | +------------------+
Enter fullscreen mode Exit fullscreen mode

As presented on the common issues, the simplified flow for UDP and HTTPS TCPDump packet below displays the return of 2 (two) actions from the EC2 belonging to the ElasticBeanstalk test environment.

UDP 172.31.0.31:123 > 0.amazon.pool.ntp.org UDP 0.amazon.pool.ntp.org > 172.31.0.31:2000 TCP 172.31.0.31:443 > aws.amazon.com TCP aws.amazon.com > 172.31.0.31:3000
Enter fullscreen mode Exit fullscreen mode

As we notice from the packet analysis the returning port is ephemeral and as we already know that Network ACL are stateless we need to account for that during the rules creation. Due to the Security Group being stateful, the above behavior will work properly.

Solution

The solution, in this case, is a mix between the Security Group (SG-001) and the Network ACLs (ACL-001) where we can see:

Security Group presents inbound traffic for TCP/HTTP, TCP/HTTPS for standard ports 80 and 443 from everywhere and outbound traffic from the instance to everywhere.

Security Group Inbound Rule (SG-001) 

Type |Protocol |Port Range |Source --------|-----------|-----------|------ HTTP |TCP |80 |0.0.0.0/0 HTTPS |TCP |443 |0.0.0.0/0
Enter fullscreen mode Exit fullscreen mode

Security Group Outbound Rule (SG-001) 

Type |Protocol |Port Range |Source --------|-----------|-----------|------ All |Traffic |ALL |ALL |0.0.0.0/0
Enter fullscreen mode Exit fullscreen mode

Network ACL Inbound (ACL-001) 

Rule # Type Protocol Port Range Destination Allow / Deny 102 HTTP (80) TCP (6) 80 0.0.0.0/0 ALLOW 103 HTTPS (443) TCP (6) 443 0.0.0.0/0 ALLOW 201 Custom TCP Rule TCP (6) 1024-65535 0.0.0.0/0 ALLOW 202 Custom UDP Rule UDP (17) 123 0.0.0.0/0 ALLOW * ALL Traffic ALL ALL 0.0.0.0/0 DENY
Enter fullscreen mode Exit fullscreen mode

Network ACL Outbond (ACL-001) 

Rule # Type Protocol Port Range Destination Allow / Deny 101 HTTP (80) TCP (6) 80 0.0.0.0/0 ALLOW 202 HTTPS (443) TCP (6) 443 0.0.0.0/0 ALLOW 102 Custom TCP Rule TCP (6) 1024-65535 0.0.0.0/0 ALLOW 103 Custom UDP Rule) UDP (17) 1024-65535 0.0.0.0/0 ALLOW * ALL Traffic ALL ALL 0.0.0.0/0 DENY
Enter fullscreen mode Exit fullscreen mode

As we know that ElasticBeanstalk does more than serve traffic but also has agents that connect to AWS API's we notice two different sets of configurations in the Network ACL's.

  • Rules 102 and 103 allow inbound traffic for ports 80 and 443 following by its outbound pair, the 102 and 103 that are responsible to allow ephemeral port out to answer the requests.
  • Rules 202 and 203 are using in the reverse order, an agent will post from the EC2 to AWS API, we analyze it inverting the tables. In this case, outbound requires 80 and 443 for the request, and the response will return in an ephemeral port on the inbound rules.

Scenario 2 - (Worker Tier) Private Subnet with Network ACL and AWS NAT Gateway.

Again using the diagram below as an example to configure the Network ACL's for an ElasticBeanstalk environment.

Diagram:

+------------------+ | | | +----------+ | | | INSTANCE | | | +----------+ | | | SG-002 | | | +----------+ | +------------------+ | Subnet Private | +------------------+ + | +--------------+ |---| ACL-002 | v +--------------+ +------------------+ | | | +----------+ | | | NAT GW | | | | SG-002 | | | +----------+ | +------------------+ | Subnet Pub | +------------------+ + +--------------+ |---| ACL-003 | v +--------------+ +------------------+ | IGW | +------------------+
Enter fullscreen mode Exit fullscreen mode

Same as before as presented on the common issues, the simplified flow for UDP and HTTPS TCPDump packet below displays the return of 2 (two) actions from the EC2 belonging to the ElasticBeanstalk test environment.

UDP 172.31.0.31:123 > 172.31.0.34:2000 UDP 172.31.0.34:2000 > 0.amazon.pool.ntp.org UDP 0.amazon.pool.ntp.org > 172.31.0.34:2000 UDP 172.31.0.34:2000 > 172.31.0.31:3000 TCP 172.31.0.31:443 > 172.31.0.33:2000 TCP 172.31.0.33:2000 > aws.amazon.com TCP aws.amazon.com > 172.31.0.33:3000 TCP 172.31.0.33:3000 > 172.31.0.31:3000
Enter fullscreen mode Exit fullscreen mode

We can see that this is a little bit different than before, now 4 (four) pairs of packets with a extra jump returning different ports for each pair.

It happens due to a particular behavior of the AWS NAT Gateway. AWS Nat Gateway encapsulates packets to achieve higher performance, causing the change on the packet header during its life cycle.

Solution

The solution, in this case, is a also a mix between the Security Group (SG-002) and the Network ACLs but this case ACL-002 and ACL-003 where we can see:

Security Group Inbound Rule (SG-002) 

Type Protocol Port Range Source HTTP TCP 80 0.0.0.0/0 Custom UDP UDP 123 0.0.0.0/0 Custom UDP UDP 1024-65535 0.0.0.0/0 Security Group Outbond Rule (SG-002) Type Protocol Port Range Source All Traffic ALL ALL 0.0.0.0/0
Enter fullscreen mode Exit fullscreen mode

Network ACL Inbound - Private Subnet (ACL-002) 

Rule # Type Protocol Port Range Destination Allow / Deny 102 HTTP (80) TCP (6) 80 0.0.0.0/0 ALLOW 103 HTTPS (443) TCP (6) 443 0.0.0.0/0 ALLOW 202 Custom TCP Rule TCP (6) 1024-65535 0.0.0.0/0 ALLOW 203 Custom UDP Rule UDP (17) 123 0.0.0.0/0 ALLOW * ALL Traffic ALL ALL 0.0.0.0/0 DENY
Enter fullscreen mode Exit fullscreen mode

Network ACL Outbond - Private Subnet (ACL-002) 

Rule # Type Protocol Port Range Destination Allow / Deny 202 HTTP (80) TCP (6) 80 0.0.0.0/0 ALLOW 203 HTTPS (443) TCP (6) 443 0.0.0.0/0 ALLOW 102 Custom TCP Rule TCP (6) 1024-65535 0.0.0.0/0 ALLOW 103 Custom UDP Rule) UDP (17) 1024-65535 0.0.0.0/0 ALLOW * ALL Traffic ALL ALL 0.0.0.0/0 DENY
Enter fullscreen mode Exit fullscreen mode

Network ACL Inbound - Public Subnet (ACL-003) 

Rule # Type Protocol Port Range Destination Allow / Deny 102 HTTP (80) TCP (6) 80 0.0.0.0/0 ALLOW 103 HTTPS (443) TCP (6) 443 0.0.0.0/0 ALLOW 202 Custom TCP Rule TCP (6) 1024-65535 0.0.0.0/0 ALLOW 203 Custom UDP Rule UDP (17) 123 0.0.0.0/0 ALLOW * ALL Traffic ALL ALL 0.0.0.0/0 DENY
Enter fullscreen mode Exit fullscreen mode

Network ACL Outbond - Public Subnet (ACL-003) 

Rule # Type Protocol Port Range Destination Allow / Deny 202 HTTP (80) TCP (6) 80 0.0.0.0/0 ALLOW 203 HTTPS (443) TCP (6) 443 0.0.0.0/0 ALLOW 102 Custom TCP Rule TCP (6) 1024-65535 0.0.0.0/0 ALLOW 103 Custom UDP Rule) UDP (17) 1024-65535 0.0.0.0/0 ALLOW * ALL Traffic ALL ALL 0.0.0.0/0 DENY
Enter fullscreen mode Exit fullscreen mode

Again we have the rule set for the Network ACL where.

  • ACL-002 Rules 102 and 103 allow inbound traffic for ports 80 and 443 following by its outbound pair, the 102 and 103 that are responsible to allow ephemeral port out to answer the requests.
  • ACL-002 Rules 202 and 203 are using in the reverse order, an agent will post from the EC2 to AWS API, we analyze it inverting the tables. In this case, outbound requires 80 and 443 for the request, and the response will return in an ephemeral port on the inbound rules.
  • We would expect only outbound traffic for the NAT Gateway, but due to the encapsulation, NAT Gateway is opening a new TCP connection with the backend using the new port requiring the same set as before.

These problems only happen if you tight the security on the Network ACL used by the AWS NAT Gateway, as the AWS default NACL that allows all the traffic inbound and outbound.

Originally posted on: hugoprudente.github.io/en/posts/

References:

Top comments (1)

Collapse
 
cgill27 profile image
Craig Gill •

For time sync, Amazon Linux 2 uses the built-in instance endpoint to sync time with, so no need to have traffic go external for time sync:
docs.aws.amazon.com/AWSEC2/latest/...