Automatic Volume Snapshots using Amazon EC2 and EBS

The server that I've been using to host my other site, oc-technology.com, has been hosted on an Amazon Web Services EC2 server since October of 2008. This tutorial / software assumes you have an Amazon Web Services (AWS) account, and are running a debian-like EC2 server, mounting an Elastic Block Store (EBS). If any of this sounds like martian to you, check out http://aws.amazon.com for an overview.

I've wrote a small script (two files, volsnapshot.ksh and volsnapshot.pl) that automatically takes snapshots of my Elastic Block Store (EBS). This script runs as a cron job and does makes nightly snapshots of my Elastic Block Store. I did this so that, in the event of a catastrophic problem, I could restore my EBS to a previous state.

This script also keeps any number of previous backups, deleting "expired" snapshots (ones that are too old). This allows one to automate backups without having to worry about massive Amazon S3 usage.

This backup requires 2 files: vol-snapshot.ksh and vol-snapshot.pl.

Finally, you'll need to create a cron job at the end.

#!/bin/bash
#
# File: vol-snapshot.ksh
# Author: John O'Connor
# Copyright (C) 2009, John O'Connor
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; version 2 of the License.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARARANTY; without even the implied warranty of
# MERCHANTIBILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.

export EC2_HOME='<path_to_amazon_API_tools>'  # Make sure you use the API tools, not the AMI tools

export EC2_BIN=$EC2_HOME/bin
export EC2_PRIVATE_KEY=<path_to_key_directory>/pk-<privateKey>.pem
export EC2_CERT=<path_to_certificate_directory>/cert-<certificate>.pem
export PATH=$PATH:$EC2_BIN
#
# To find the current location of JAVA_HOME, try env | grep JAVA_HOME
# It's necessary to put this environment variable in here because
# cron will not have access to your standard environment variables.
export JAVA_HOME=<path_to_your_java_runtime>

./vol-snapshot.pl > logs/snapshot_`date +%Y%m%d_%H%M%S`.log
# You can also do the following if you don't wish to keep the logs
# ./vol-snapshot.pl

And for the perl file, make sure you keep it in the same directory as the shell script, or otherwise change the paths above to point to the correct location.

#!/usr/bin/perl
#
# File: vol-snapshot.pl
# Author: John O'Connor
# Copyright (C) 2009, John O'Connor
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; version 2 of the License.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARARANTY; without even the implied warranty of
# MERCHANTIBILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#!/usr/bin/perl -w

##############################
# Global Variables
##############################
EC2_HOME='<path_to_amazon_API_tools>'  # Make sure you use the API tools, not the AMI tools

my $EC2_HOME = '<path_to_amazon_API_tools>';
my $EC2_BIN = "$EC2_HOME/bin";
my $EC2_PRIVATE_KEY = '<path_to_key_directory>/pk-<privateKey>.pem';
my $EC2_CERT = '<path_to_certificate_directory>/cert-<certificate>.pem';
my @volumes = ();
my @snapshots = ();
my $SNAPSIZE_MAX = 2; # This is the number of snapshots to save.
my $DEBUG = 1; # Set this to 0 to get less output.  I recommend leaving this alone.

##############################
# Subroutine Prototypes
##############################
sub getVolumes();
sub getSnapshots();

##############################
# Main
##############################
if (@ARGV)
  {
    @volumes = @ARGV;
  }
else
  {
    @volumes = getVolumes();
  }
my $volsize = @volumes;
foreach (@volumes)
  {
    print "$_\n" if $DEBUG;
    `$EC2_BIN/ec2-create-snapshot $_`;
  }

$volsize == 0 ? print "No volumes found.\n" : $volsize == 1 ? print "$volsize volume found.\n" : print "$volsize volumes found.\n";

@snapshots = getSnapshots();
my $snapsize = @snapshots;
print "$snapsize number of snapshots\n" if $DEBUG;
if ($snapsize > $SNAPSIZE_MAX)
  {
    print "$snapsize, $SNAPSIZE_MAX\n" if $DEBUG;
    for (my $i = 0; $i < $SNAPSIZE_MAX; $i++)
      {
    my $snapshot = $snapshots[$i];
    print "$_\n" if $DEBUG;
    print "$snapsize snapshots left\n" if $DEBUG;
    `$EC2_BIN/ec2-delete-snapshot $snapshot`;
      }
  }


##############################
# Subroutines
##############################
sub getVolumes()
{
    my $volstring = `$EC2_BIN/ec2-describe-volumes`;
    my @volumes = split(/\n/, $volstring);
    my @vol = ();
    my $volcount = 0;
    foreach my $volume (@volumes)
    {
      if ($volume =~ /VOLUME\s+(vol-(?:\S+))/)
    {
      ($vol[$volcount]) = $volume =~ /VOLUME\s+(vol-(?:\S+))/;
      $volcount++;
    }
    }
    return @vol;
}

sub getSnapshots()
  {
    my $snapstring = `$EC2_BIN/ec2-describe-snapshots`;
    my @snapshots = split(/\n/, $snapstring);
    my @snap = ();
    my $snapcount = 0;
    foreach my $snapshot (@snapshots)
      {
    if ($snapshot =~ /SNAPSHOT\s+(snap-(?:\S+))/)
      {
        ($snap[$snapcount]) = $snapshot =~ /SNAPSHOT\s+(snap-(?:\S+))/;
        $snapcount++;
      }
      }
    return @snap
  }

Ok -- I know what you're thinking: You put the paths in both files. Why don't you just snag the environment variables you just set in the .ksh file?

I don't remember -- it didn't work last time I tried it, and I'm too lazy to figure out why. If someone else out there tries it and gets it to work, please do post it here! I'll change it in the code.

Finally, you need to create a cron job that runs the Vol-snapshot.ksh file.

In your terminal, type:

crontab -e

Then, add the following line:

30 0 * * * <your_scripts_dir>/vol-snapshot.ksh

This runs the cron job at 12:30am every day. For more information on how to use crontab, (including how to change when the cron job is run), check out http://www.adminschoice.com/docs/crontab.htm

It's not perfect, but it's worked fine for me so far, and has definitely saved me more than a few headaches.

Good luck, and let me know how this works out for you.