Join Log in
Score:0
avatar Server

On AWS how can the ENI of my squid proxy become a blackhole in my route table if the EC2 instance still exists?

zw flag
Taylor

enter image description here

Been googling like crazy and can't find an answer. We have three AZs/subnets since we're in Ohio. But this diagram is close enough to explain the issue.

We've set up squid proxies to filter outbound traffic from one of our services.

  1. For each AZ, app servers are in a private subnet.
  2. Then there's a proxy in each public subnet for that AZ.
  3. The route table for the private subnet points 0.0.0.0 to the ENI of the proxy in the corresponding public subnet.

Over time, outbound traffic from each subnet died. It took us a bit to figure out what was going wrong so as each subnet died, we removed the instances in that subnet from the ALB for the service and motored on with a hobbled service while we researched. Yesterday the third subnet died and we decided to "route around" the proxies directly to the NAT gateway for each subnet. When we got to the route table, we noticed the ENI of each proxy was listed as a blackhole.

We've inspected

  1. Proxy instance logs
  2. ENI allocation times, and
  3. Cloudtrail logs

...looking for any indication as to why the ENIs had become invalid breaking our default routes. Nothing useful at all.

  1. The instances have been up for over three weeks
  2. The ENI allocation time stamp matches up with the instance creation time
  3. The boot logs don't show any reboots
  4. Cloudtrail doesn't show any modifications to the ENIs / instances.

We're stumped. How can our route table "suddenly" contain a route to an ENI that doesn't exist?

66
0 + 0
cn flag
shearn89
If the proxy is in the same subnet in which the route for 0/0 routes to the proxy, isn't that a circular route? Wouldn't you want the proxy in one subnet and the service in another?
0
Reply
zw flag
Taylor
@shearn89 thanks for responding. I need to update my description. App servers are in a private subnet, no internet access. Proxies are in a public subnet with a NAT gateway. Private subnet route has 0.0.0.0 > ENI of proxy in corresponding public subnet (same AZ). Public subnet route, 0.0.0.0 > nat gateway. The rub here is that with no explanation, the private subnet RTB suddenly showed 0.0.0.0 route as a blackhole.
0
Reply
Tim avatar
gp flag
Tim
@Taylor if you want to update your question please edit it, rather than commenting. You should probably draw a diagram to help people understand your setup / scenario and ideally break your wall of text into paragraphs for readability.
0
Reply
Tim avatar
gp flag
Tim
You could consider using AWS Network Firewall instead of squid proxies. It's probably a lot more expensive, but it's also likely much more reliable. https://aws.amazon.com/blogs/networking-and-content-delivery/deployment-models-for-aws-network-firewall/ I wonder if your ENIs turned into black hole routes because the instance was rejecting traffic, or something like that
0
Reply
cn flag
shearn89
The public subnet can't have the NAT GW in it and also have a default route pointing to that GW, as where does the NAT GW talk to? Public subnet needs a default route to an Internet GW, and then you could seperately route the proxies to the NAT GW via a specific table.
0
Reply
zw flag
Taylor
@Tim per you suggestions, slightly less of a wall of text and a borrowed diagram that is close enough to illustrate the point.
0
Reply
Tim avatar
gp flag
Tim
I suggest that you get AWS Support for a month and talk to them. With your permission AWS support can look directly into your account to help you solve this somewhat unusual problem. I suspect the answer will be in the CloudTrail logs, but finding anything in CloudTrail can be difficult.
1
Reply
ธงชาติไทย
Elon Musk
I sit in a Tesla and translated this thread with Ai:

mangohost

Post an answer

Most people don’t grasp that asking a lot of questions unlocks learning and improves interpersonal bonding. In Alison’s studies, for example, though people could accurately recall how many questions had been asked in their conversations, they didn’t intuit the link between questions and liking. Across four studies, in which participants were engaged in conversations themselves or read transcripts of others’ conversations, people tended not to realize that question asking would influence—or had influenced—the level of amity between the conversationalists.