The following story I find really interesting.
Bhaskar Sunkara, VP Product Experience at AppDynamics, presents "Performance on Amazon AWS".
Among other things, AppDynamics is known to be used by NetFlix to monitor thousands of JVMs on Amazon AWS.

The presentation tells you about how Netflix partnered with App Dynamics to monitor their systems in a (more than) dynamic cloud infrastructure by AWS.
A large part of the presentation explains about what the important design principles are to be aware of when developing for the cloud.

Main message: machines will come and go, systems will go down. Application availability is in the hands of the developer. Develop for availability.