« Amazon EC2 - What You May Not Have Known »

Amazon’s Elastic Compute Cloud (EC2) has the goal of providing flexible computing capacity in the form of a service. This service provides the user with the ability to quickly scale to the demands of an application by booting or shutting down servers in a matter of minutes. Since all these machines run in a virtual environment, you only need to pay for the resources you use. More detailed information can be found on Amazon’s EC2 home page - http://aws.amazon.com/ec2/  Much of the documentation provided by Amazon was straightforward and easy to follow so for a full walk-through see http://docs.amazonwebservices.com/AWSEC2/2007-08-29/GettingStartedGuide/.  We will assume that the reader is familiar the basics of EC2.

This article focuses on the problematic aspects of EC2 - issues that can lead to serious problems or technicalities that if ignored, can lead to frustrating hours wasted on troubleshooting and debugging.  We've learned that the single most important thing you can do for your EC2 environment is to give it a dynamic DNS solution it can use to overcome the DHCP nature of virtual machines.  Now what can you do for yourself, you ask?  Take a look at the gotchas we encountered and save yourself from dealing with the same problems.

DHCP, Dynamic IPs and DynDNS.com

One side-effect of these virtual servers is that each time one boots up, DHCP assigns them a new IP address. In all of our experience we've never been assigned the same IP after deploying a new instance of a machine.  This is highly undesirable since our web application running on Amazon EC2 would become unreachable by it's dns name and old external IP if we ever had to re-deploy a server instance after some failure.  It became evident that if a server were to go down, we'd be dealing with a significant amount of down-time.  In order to ensure that we wouldn't have to deal with a time-consuming process of modifying the configuration of servers and waiting for updates to our domain provider's DNS to propagate (up to 96 hours), we implemented a dynamic DNS solution.

Our application had the additional complexity of requiring inter-server communication. But with a new internal IP address on every re-deployment, the new locations of the server would be unknown to the others. Once again we decided that we needed these servers to have stable aliases in order to avoid reconfiguring each machine whenever we had to re-deploy a server.

In order to ensure a quick recovery after one of our servers went down, we needed to be able to update DNS entries and have the changes take effect immediately. Amazon suggests using dynamic DNS solutions such as DynDNS and ZoneEdit.  We decided to go with DynDNS because it appeared to offer better support and documentation for the service itself, as well as better instructions on how to set up  recommended update clients such as ddclient.  The ddclient tool is responsible for monitoring a machine’s IP address and updating DNS entries when a change is detected. Here is what we did to implement the dynamic DNS service for EC2:

1. Go to https://www.dyndns.com/services/ and sign up for a free ‘Dynamic DNS’ account or a paid ‘Custom DNS’ account if you want to stick with an existing domain.
2. Create place holder records for entries that you expect to be updated dynamically by ddclient (you can start with a bogus value like 10.10.10.10 to make it obvious when it changes).
3. Under your preferences you can also pre-activate your solution to speed things up if you plan on delegating the name service over to DynDNS soon.
4. Go to https://www.dyndns.com/support/clients/unix.html to download ddclient and follow the instructions in the Knowledge Base article to get the client installed.
5. Paste the following and update your login, password and 'custom' server list in your ddclient.conf file:
   

   use=cmd, cmd='curl http://169.254.169.254/2007-08-29//meta-data/public-ipv4'
   login=xxxxx
   password=xxxxx
   protocol=dyndns2
   server=members.dyndns.org
   wildcard=YES
   custom=yes, your.server1.com, your.server2.com
           

6. Note that the client requires the perl-IO-Socket-SSL module to be
   installed so using yum, the following command should do the trick -
   "yum install perl-IO-Socket-SSL.noarch"
7. You can also choose to make sure the ddclient daemon is running when
   the machine boots up by using chkconfig with the service or start  it
   with "service ddclient start"

Note that the url in the configuration above is Amazon's recommended way to obtain a server's external IP address and http://169.254.169.254/2007-08-29//meta-data/local-ipv4 will give you an machine's internal IP.  That’s it. After getting the client set up you can log into your DynDNS account to see that your records are being updated and now you can access your servers using those DNS names without worrying about unexpected changes to the IP address of your servers.

Here are some things to consider when using ddclient.  ddclient maintains a cache by default in “/var/cache/ddclient/” which can prevent any updates to DynDNS if a record is updated outside of the client - remember to delete the cache in this situation.  If you want to keep both a machine's external and internal dns names up to date, you would need to run multiple instances of the ddclient daemon. Note however, that you must modify both the startup script (provided by them) to handle multiple instances as well as the .pid values to be unique in each of the .conf files. This may be too much work so a simple alternative is to have separate cron-jobs that call ddclient with the ‘–force’ flag along with the location of each ddclient.conf file with the ‘–file’ parameter.

Other ‘Gotchas’ and Issues We Ran Into

This section covers unexpected issues that we ran into our first time around working with EC2. These issues are centered around 3 areas - choices in machine images, packaging your images and disk usage on the virtual servers.  It’s helpful to be aware of these issues because you may otherwise end up wasting time troubleshooting the same problems we did.

Choices in Machine Images

Remember that you don't have to use Amazon’s base Fedora Core 4 images to build your machines. Before spending too much time configuring and customizing an AMI, find one that suits your needs from the start so you won't have to redo any work later on down the road. Check out the list of public AMIs in Amazon’s resource center for something that is more suitable for your needs: http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=101 We started with the standard Fedora Core 4 build but eventually moved to a more up-to-date enterprise CentOS build provided by rightscale.com for better maintainability and security.

Packaging Your Image

When packaging up your own image using the ‘ec2-bundle-vol’ command, make sure you specify a clean folder using the '–d' flag otherwise bundling the same image twice will result in an error due to the conflicting sets of temporary files.

Also, use the '–p' flag to specify a prefix/name for your image otherwise when you upload the AMI and look at your list of images with the "ec2-describe-images –o self" command it will be very hard to differentiate between all the images that you’ve created.  For example, we used something like "ec2-bundle-vol -k pk-XXXXXXXXXXXXXX.pem -u 123456789 -c cert-XXXXXXXXXXXXX.pem -d /mnt/cleantempfolder -p web-server-v1".

Machine Disk Usage

When working with your image note that the main drive/partition (where the system files are) has a very limited capacity (10 GB in our case). So when dealing with large files/directories use ‘/mnt’ as it has over 100 GB.  We've experienced all sorts of failures after accidentally maxing out the main partition.  Remember that if you are running an application that generates log files or temporary/residual files on disk, you will need to make sure you don't cause failures by filling up the main partition with large files and directories.

If a machine is terminated, all your data will be lost except for what was backed up from the last time you ran an 'ec2-bundle-vol'. Be mindful of where you put your files because when bundling your machine image, many directories are excluded by default so it's easy to lose data.   Check the sample output of the ‘ec2-bundle-vol’ command to see which directories don't get backed up - http://docs.amazonwebservices.com/AWSEC2/2007-08-29/GettingStartedGuide/creating-an-image.html

Good Luck!

With a dynamic DNS solution, Amazon EC2 servers face significantly less down-time if something goes wrong and are much easier to maintain in the long run. Amazon has provided a set of very useful tools that make it simple to build, upload and deploy customized machine images. Overall, I have to say that Amazon’s EC2 platform has been relatively easy to work with and is worth considering as a hosting solution.

Comments

[Davis]

I tried EC2 and ended up at Slicehost due to some of the problems you pointed out above.

[Kevin Day]

Thanks for the post! I'm wrangling with EC2 now, and this helps out a lot.

[wids]

For the inter-server communication mechanism we used a register mechanism whereby they shared the same ec3 credentials and wrote a file in a particular directory with their ip address, then the controller would ls the directory every minute to see if any new m/c's had come up and assign work accordingly. wfm :-)

[Oliver Chan]

I'm glad that some of you have found this useful. It would have been nice if Amazon had provided an integrated dynamic DNS service as part of the package. Luckily, implementing the dynDNS solution is a one time thing as long as you remember to bundle it up into your image. :)

[Eamon Leonard]

Hi Oliver,

I just wanted to say thanks for putting this together. I'm looking into EC2 right now, and it's great to have identified the above at this stage than further down the line.

You've definitely saved me time.

Cheers.

[Ishtiaq Ahmed]

Hi guyz,

its nice to see all the info over to the single place. My question is that after I creted my AMi and do some configuration changes in Apache.. How to update those configurations in my image file I previously updated... Do I have to create a new image for it.

Moreover, what about Databases which we run over the EC2 environment.



Thanks...

[Dane]

Did you have to pay for the rightscale image package? I see rightscale takes most of the pain out of setting up a server, but it's cost is steep at $500 per month.

[GM22]

Hi -Is there a reason why when I rebooted the EC2 instance after installing ddclient, all my files were missing? I tried again,.. after rebooting, all my server changes were gone.. I must be missing something..

Thanks.GM

[Ajf 4]

Wow, it is really great.I am very pleased that that i'm standing at an excellent blog of my life, i'm really glad to get my comment here in very decent topic. Thanks to you!

[cheap jordan shoes]

we used a register mechanism whereby they shared the same ec3 credentials and wrote a file in a particular directory with their ip address, then the controller would ls the directory every minute to see if any new m/c's had come up and assign work accordingly. wfm :-)

[replica jerseys]

Thanks for the post! I'm wrangling with EC2 now, and this helps out a lot.

[jordan 13]

I am satified with your posts,and many people almost agree with you,so better then,I will follow your future posts.come on!

[gucci outlet online]

I'm looking into EC2 right now, and it's great to have identified the above at this stage than further down the line.

[Brianlayman]

UPDATE: Amazon addressed this issue in a month after this excellent article was written by introducing the concept of Amazon Elastic IPs. Elastic IPs are static IP addresses assigned to the account and not instances. Elastic IPs are free for the time they are assigned to a specific instance. Elastic IP Addresses are able to be mapped to a different instance without the delay of DNS propagation.

Techniques described in this article seem to have formed the foundation for more recent scripts that allow the host name to be defined for instances upon launch and other techniques. One example I've just found is http://blog.ternarylabs.com/2010/09/15/automatically-configure-hostname-for-new-ec2-instances/

[birkenstock outlet]

Well done, thak you for your great posts, i learned more from here.

[Microsoft outlook 2010 Online]

Becoming a passionate blogger, I'm happy to determine other men and women taking gumption and adding on the neighborhood. I just desired to remark to display my understanding for a submit because it is especially inviting, and lots of writers do not get the credit score they den this from you.

[Microsoft outlook 2010 Online]

Pretty good post. I just found your site and wanted to say that I have really enjoyed browsing your posts.In any case I'll be subscribing to your blog and I hope you post again sooner

[Christian Louboutin Slingback]

Fantastic Read! Looking forward to more! Bookmarked the site and will be back again!
Christian Louboutin Slingback
Christian Louboutin Slingback
[url=http://www.christianlouboutinshoesale.org/ ]Christian Louboutin Slingback[/url]
[url="http://www.christianlouboutinshoesale.org/ " title="Christian Louboutin Slingback"]Christian Louboutin Slingback[/url]

[Christian Louboutin D'orsay]

Nice post.Thank you for taking the time to publish this information very useful
Christian Louboutin D'orsay
Christian Louboutin D'orsay
[url=http://www.christianlouboutinshoesale.org/ ]Christian Louboutin D'orsay[/url]
[url="http://www.christianlouboutinshoesale.org/ " title="Christian Louboutin D'orsay"]Christian Louboutin D'orsay[/url]

[Christian Louboutin Hot Red]

Perhaps this is one of the most interesting blogs that I have ever seen. Interesting article, Funny comment. Keep it up!
Christian Louboutin Hot Red
Christian Louboutin Hot Red
[url=http://www.christianlouboutinshoesale.org/ ]Christian Louboutin Hot Red[/url]
[url="http://www.christianlouboutinshoesale.org/ " title="Christian Louboutin Hot Red"]Christian Louboutin Hot Red[/url]

[Christian Louboutin Sandals]

I’m still waiting for some interesting thoughts from your side in your next post thanks
Christian Louboutin Sandals
Christian Louboutin Sandals
[url=http://www.christianlouboutinshoesale.org/ ]Christian Louboutin Sandals[/url]
[url="http://www.christianlouboutinshoesale.org/ " title="Christian Louboutin Sandals"]Christian Louboutin Sandals[/url]

[karen millen outlet]

thanks for sharing great article with us
karen millen outlet
http://www.karenmillenes.co.uk

[Microsoft outlook 2010 Online]

Pretty good post. I just found your site and wanted to say that I have really enjoyed browsing your posts.In any case I'll be subscribing to your blog and I hope you post again sooner
Coach Outlet
Coach Outlet Stores
Coach Outlet Online Stores

[microsoft outlook 2010]

Becoming a passionate blogger, I'm happy to determine other men and women taking gumption and adding on the neighborhood. I just desired to remark to display my understanding for a submit because it is especially inviting, and lots of writers do not get the credit score they den this from you.

[Coach Outlet Stores]

I like it.but The meaning of the last paragraphCoach Leather Handbags
a bit puzzled.It’s always good to have passions in life to keep yourself from going down the negative path and work towards staying positive.I really enjoyed this.
Coach HandbagsYou can look your article comments. This information has really been helpful for most of the readers.