Over at the SmugMug Sorcery blog I posted about how we scale puppet in Amazon EC2: http://sorcery.smugmug.com/2013/01/14/scaling-puppet-in-ec2/. You should definitely take a look.
ec2-run-instances ami-1234de7b --kernel aki-5037dd39 --instance-type t1.micro --region us-east-1 --key test1 --user-data-file ~/svn/ec2/init-instance.sh --group default --instance-initiated-shutdown-behavior stop
The instance started as usual. Once it was up and configured (via Puppet), I was able to log in and start poking around.
During the course of my testing, I upgraded some packages to the latest versions (I ran
sudo apt-get update ; sudo apt-get upgrade) and rebooted the instance. After the reboot, I was unable to log in. As I dug into the problem, I noticed the following on the console:
$ ec2-get-console-output i-87705bed | tail [ 0.850326] devtmpfs: mounted [ 0.850368] Freeing unused kernel memory: 216k freed [ 0.852042] Write protecting the kernel text: 4328k [ 0.852509] Write protecting the kernel read-only data: 1336k init: console-setup main process (63) terminated with status 1 %Ginit: plymouth-splash main process (215) terminated with status 2 init: plymouth main process (45) killed by SEGV signal cloud-init running: Thu, 16 Sep 2010 22:00:31 +0000. up 7.09 seconds mountall: Disconnected from Plymouth
It appeared that there was a problem with the mounts. On another clean instance, I noticed that the
/etc/fstab file had a reference to
/mnt, which isn’t valid on t1.micro instances. All other instance types include ephemeral storage, but t1.micro instances do not.
Once I removed the conflicting line in
/etc/fstab, the problem instance was able to boot normally.
For the curious, here’s my
#!/bin/sh apt-get update apt-get --yes install puppet wget --output-document=/etc/puppet/puppet.conf http://example.com/puppet.conf perl -pi -e 's/START=no/START=yes/' /etc/default/puppet /etc/init.d/puppet restart
After discovering this problem with micro instances, I added the following to the
if [ "`curl -s http://169.254.169.254/latest/meta-data/instance-type`" = "t1.micro" ] ; then sed -i -e '5d' /etc/fstab fi
This deletes the line in
/etc/fstab that mounts the ephemeral storage.