1

I'm in a bit of a pickle here. I have a specific application (one that I cannot modify) which I need to automatically deploy on 3 AWS EC2 instances using CloudFormation, and I need all 3 of them to be aware of each other as they boot up, or at least before any traffic hits them.

In short, what needs to happen is that as the machines spin up, they need to each run a local command (a windows shell script) that includes the hostnames or IPs of all 3 machines. Imagine something like:

c:\>node startReplication.js server1.aws.com,server2.aws.com,server3.aws.com

What makes this tougher is that I cannot use static IPs or names, I need this to be completely dynamic for each stack. I also cannot use any non-AWS-native tools such as Chef or Terraform or any other 3rd party, for reasons too irrelevant to specify here. Everything must be done via native AWS services.

I've tried doing something like this:

"UserData" : { "Fn::Base64" : { "Fn::Join" : [ ",", [ { "Fn::GetAtt" : [ "server1", "PublicDnsName" ] }, { "Fn::GetAtt" : [ "server2", "PublicDnsName" ] }, { "Fn::GetAtt" : [ "server3", "PublicDnsName" ] } ] } }

To only pass in the dns names, haven't even reached the point of figuring out how to actually run a script/command - but already this fails due to a circular reference.

What I understand is that CloudFormation looks at these references as dependencies - so to reference "server1" I need it to be created, but if all 3 machines need all 3 references, I've hit a brick wall.

I'm not experienced enough with AWS (actually, not at all) to figure out an alternative path for this, but I had some theoretical ideas which you might be able to confirm, or suggest one of your own:

  1. Have each machine register itself to some external place - such as an S3 bucket, AWS Config, SQS and so on.. And when CF is done, run a lambda that uses that data to somehow initiate a shell script on all 3 machines.
  2. Maybe CF has a way of running a shell script when all instances are created (using a Wait?) and their DNS names are available?
  3. Developing some kind of agent service to run on all machines, have one that will get references to the other two using UserData and then have it send that information to the other two to trigger said script
  4. Maybe running cf-init from a node process launched with userdata script to get this information and pass it on to subsequent scripts that need to run (not sure if that's how cf-init works and if it happens after all 3 machines have dns names allocated)

I want to avoid an overly complicated or convoluted solution as I've got both limited knowledge and limited time (isn't that the recipe for success nowadays?)

I hope I've made the issue clear enough. Thanks in advance!

Improve this question
1
  • Not sure if it helps, but Server 1 doesn't need Server 1's info, and the others don't need their own. A running instance can learn its own information from the Instance Metadata Service. Commented Jul 9, 2017 at 19:28

1 Answer 1

Reset to default
0

Use the CloudFormation stack name as a subdomain for the 3 server domain names. That way you know what the DNS names will be and can inject it into the config file.

"DNSRecordInstance1": { "Type": "AWS::Route53::RecordSet", "Properties": { "HostedZoneName": { "Ref": "HostedZone" }, "Name": { "Fn::Join": [ ".", [ "server1", { "Ref": "AWS::StackName" }, ".", { "Ref": "HostedZone" } ] ] }, "Type": "A", "TTL": "900", "ResourceRecords": [ { "Fn::GetAtt": [ "Instance1", "PrivateIp" ] } ] } }

This will create a DNS record server1.<stack-name>.<hosted-zone>, for example server1.test-stack.example.com. You do it for all 3 instances.

Then in each Instance's Metadata you can create the config file straight away because you know what the DNS names will be and don't need to know what the IPs will be - DNS will handle that.

"Instance1": { "Type": "AWS::EC2::Instance", "Properties": { [...] "UserData": { "Fn::Base64": { "Fn::Join": ["", [ "<script>\n", "cfn-init.exe -v -s ", { "Ref" : "AWS::StackId" }, " -r Instance1", " --region ", { "Ref" : "AWS::Region" }, "\n", "</script>" ] ] } } }, "Metadata": { "AWS::CloudFormation::Init": { "config": { "files": { "c:\\servers.conf": { "content": { "Fn::Join": ["", [ "server1.", { "Ref": "AWS::StackName" }, ".", { "Ref": "HostedZone" }, "\n", "server2.", { "Ref": "AWS::StackName" }, ".", { "Ref": "HostedZone" }, "\n", "server3.", { "Ref": "AWS::StackName" }, ".", { "Ref": "HostedZone" }, "\n" ]]} } } } [...]

This will create C:\servers.conf with a list of 3 DNS names of your servers. Again, do it on each of your instances and you're done :)

The above template snippets should also help you with running the Cloud Init scripts. Make sure you keep the Instances references correct, i.e. cfn-init.exe -r Instance1 in Instance1's UserData. Update that to cfn-init.exe -r Instance2 for Instance2, etc. The label behind -r must be the resource name where it is defined.

Hope that helps!

Improve this answer
3
  • Thanks for the extremely detailed answer! I now finally see how cfn-init is actually used. I'll give it a try. Commented Jul 10, 2017 at 9:14
  • Fantastic, works like a charm! Thanks! I've made a few changes to your template code as there was a missing comma in one of the FN::Joins and the final URLs generated for the servers.conf file should use StackName and not StackId. My only concern with this is Route53's pricing - if I use these DNS names for replication, I might be getting a lot of traffic on them. If that happens, I'll just resolve them to IPs and use those for my configuration. Commented Jul 10, 2017 at 12:28
  • @motig88 I'm glad I could help and sorry for the typos. Regarding Route53 pricing don't worry - the TTL (Time to live) was set to 900 seconds (= 15 minutes) in my example, the servers should only re-resolve the names every 15 minues. That won't cost you anything. You can make the TTL even higher - 3600 (1 hour). Also as long as they keep the TCP connection up they probably won't need to re-resolve at all. Only when re-establishing the connection after it drops. Route53 never costs me anything relevant :) Commented Jul 11, 2017 at 0:40

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.