Puppet SysAdmin

Scaling Puppet in EC2

Over at the SmugMug Sorcery blog I posted about how we scale puppet in Amazon EC2: You should definitely take a look.


Playing with EC2 Micro Instances

Last night I decided to give the t1.micro instance size a try on Amazon EC2. I used the Ubuntu 32-bit EBS AMI for the test.

--kernel aki-5037dd39 
--instance-type t1.micro 
--region us-east-1 
--key test1 
--user-data-file ~/svn/ec2/ 
--group default 
--instance-initiated-shutdown-behavior stop

The instance started as usual. Once it was up and configured (via Puppet), I was able to log in and start poking around.

During the course of my testing, I upgraded some packages to the latest versions (I ran sudo apt-get update ; sudo apt-get upgrade) and rebooted the instance. After the reboot, I was unable to log in. As I dug into the problem, I noticed the following on the console:

$ ec2-get-console-output i-87705bed | tail
[    0.850326] devtmpfs: mounted
[    0.850368] Freeing unused kernel memory: 216k freed
[    0.852042] Write protecting the kernel text: 4328k
[    0.852509] Write protecting the kernel read-only data: 1336k
init: console-setup main process (63) terminated with status 1
%Ginit: plymouth-splash main process (215) terminated with status 2
init: plymouth main process (45) killed by SEGV signal
cloud-init running: Thu, 16 Sep 2010 22:00:31 +0000. up 7.09 seconds
mountall: Disconnected from Plymouth

It appeared that there was a problem with the mounts. On another clean instance, I noticed that the /etc/fstab file had a reference to /mnt, which isn’t valid on t1.micro instances. All other instance types include ephemeral storage, but t1.micro instances do not.

Once I removed the conflicting line in /etc/fstab, the problem instance was able to boot normally.

For the curious, here’s my script:

apt-get update
apt-get --yes install puppet
wget --output-document=/etc/puppet/puppet.conf
perl -pi -e 's/START=no/START=yes/' /etc/default/puppet
/etc/init.d/puppet restart

After discovering this problem with micro instances, I added the following to the script:

if [ "`curl -s`" = "t1.micro" ] ; then
    sed -i -e '5d' /etc/fstab

This deletes the line in /etc/fstab that mounts the ephemeral storage.