It’s automatic, no manual!
If you are reading this article, then, it’s possible that you normally store data in the Amazon EBS volumes. As you know, it is professional to take snapshots of the EBS volumes in so as to back up the data on Amazon S3. If you have decided to follow this professional standard, it means you have possibly realized the need to manage the number of snapshots that you keep for a certain EBS volume and delete the older snapshots that are no longer needed. Doing this will definitely help in achieving disaster recovery as well as saving storage cost.
You may also have policies stating that the backups need to be kept a particular number of miles away as part of the disaster recovery plan. In order to fulfill these requirements, you will need to copy your EBS snapshots to the disaster recovery region and perform the snapshot management there.
This logic involves a lot of components that are absolutely different. First of all, you need to tag your snapshots so that you can manage them. Then, find out how many snapshots you have for a particular EBS volume at present and make an assessment of the value against the retention rule. Should it happen that the number of snapshots is more than that of your retention value, then, you will delete the old snapshots. All these procedures are clear examples of a simple and basic snapshot management workflow. The question you now ask is, how do you automate something like this in the AWS? You do not need to worry as that already been ironed out in this article. All that is required from your own end is to read this to the end, learn the nitty-gritty, and go ahead to apply it. If you are ready,
How does it work?
For any snapshot to be taken, the API calls need to be provided with necessary permission. At this point, the AWS will create a default role that is called AWS Data Life cycle Manager Service Role but with a policy that it should have the ability to create a snapshot on all the regions.
This will thereafter take the snapshots based on tags. This means that all your EBS volumes must be given a tag before you enable this policy.
The limitations that you will encounter with this include the following
- You will need to wait for some time after creating the policy
- If one of the tags is already part of any lifecycle policy, it becomes a problem because you cannot use the same tag for another policy. Doing that will bring an error.
- Should you decide to remove your policy or make a change to the tags from the policy, the snapshots which are already taken will not be deleted. You can still make use of them.
The highest retention period that you can have per volume is 1000 days. If you provide more days than this, then the snapshots will be automatically removed after 1000 days while those that are not up to 1000 days will be kept.
Creating the Snapshot Lifecycle Policy
- Go to the EC2 Console
- Under the category of Elastic Block Store, you will see the Lifecycle Manager.
- Click on Create snapshot policy option
- For description, provide the name you want for your policy
- Target Volume with tags: Provide your tag (Key: Value) or you can just choose it from the drop-down list available.
- Schedule name: Provide a name that you will like to give your schedule
- Create Snapshots every: You can select either 12 hr or 24 hr
- Snapshot Creation Start Time: You can set the start time in Universal Time Coordinated (UTC).
Policy status after creation
You can decide to put the policy into action immediately or later. If you want it to start operating now, select ENABLE. However, if you want it to operate later, choose DISABLE. You have the opportunity to enable it later, here we go!
Creating IAM Role. Login to AWS management console, Goto Services, navigate IAM and create a user for automatic snapshot
Enter username and select access type: Programmatic access and then click next
- Note down the AccessKey ID and Secret access key.
Now Select the user that you just created and grant the permission, For that go to the permission tab and select Add Inline policy
Then Custom policy, give policy name and add following JSON code
{
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:CreateSnapshot",
"ec2:CreateTags",
"ec2:DeleteSnapshot",
"ec2:Describe*",
"ec2:DescribeTags",
],
"Resource":[
"*"
]
}
]
}Login to AWS instance via ssh and install aws-cli(aws commend line interface) and install aws-cli, you should ask to enter the information like AWS Access key ID, AWS Secret Access Key, Default region name and the default output format should be JSON.
sudo apt-get install awscli aws configure --profile=backup
Create a script file with extension .py (Python script) for example shapshotscript.py and paste below python code and save the file.
!/usr/bin/env python3
# backup.py --volume-ids=vol-6b25bca2 --volume-ids=vol-6b25bca2 --expiry-days=7#
import argparse
import subprocess
import json
import logging
import time, datetime, dateutil.parser
profile = 'backup' # Your AWS CLI profile
region = 'us-east-1' # replace this region to your EC2 instance region
def bash(command):
process = subprocess.Popen(command, stdout=subprocess.PIPE)
return process.communicate()[0].decode('utf-8')
def getOurSnapshots():
"""
Return a list of snapshot Dicts created with this plugin.
"""
return json.loads(bash([
"aws", "ec2", "describe-snapshots",
"--filters", "Name=tag-key,Values=Group", "Name=tag-value,Values=backup",
"--profile", profile,
"--region", region,
"--output=json"
]))['Snapshots']
def createSnapshots(volumeIds):
"""
Return True if snapshots of the given volumes are created, else False
Keyword arguments:
volumeIds -- List of EBS volume IDs
"""
# Create the snapshots
snapshots = []
for volumeId in volumeIds:
snapshots.append(createSnapshotForVolume(volumeId))
# Add Name and Group tags to the snapshot
if len(snapshots):
snapshotIds = []
date = time.strftime("%Y-%m-%d")
for snapshot in snapshots:
snapshotIds.append(snapshot['SnapshotId'])
# create-tags returns no output now so just perform the command and
# return True
bash([
"aws", "ec2", "create-tags",
"--resources", ' '.join(snapshotIds),
"--tags", "Key=Name,Value='Snapshot "+date+"'", "Key=Group,Value=backup",
"--profile", profile,
"--region", region,
"--output=json"
])
return True
return False
def createSnapshotForVolume(volumeId):
"""
Return a Dict of a created snapshot for the given EBS volume
Keyword arguments:
volumeId -- An EBS volume ID
"""
date = time.strftime("%Y-%m-%d")
message = "Creating snapshot for volume "+volumeId+"..."
response = json.loads(bash([
"aws", "ec2", "create-snapshot",
"--volume-id", volumeId,
"--description", "Volume Snapshot"+date,
"--profile", profile,
"--region", region,
"--output=json"
]))
message += response['SnapshotId']
logging.info(message)
return response
def deleteOldSnapshots(snapshots, max_age):
"""
Delete all listed snapshots older than max_age
"""
snapshotIds = []
date = datetime.datetime.now()
for snapshot in snapshots:
snapshotDate = dateutil.parser.parse(snapshot['StartTime']).replace(tzinfo=None)
dateDiff = date - snapshotDate
if dateDiff.days >= max_age:
message = "Deleting snapshot "+snapshot['SnapshotId']+" ("+str(dateDiff.days)+" days old)..."
# delete-snapshot no longer returns any output
bash([
"aws", "ec2", "delete-snapshot",
"--snapshot-id", snapshot['SnapshotId'],
"--profile", profile,
"--region", region,
"--output=json"
])
message += "done"
logging.info(message)
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='ADD YOUR DESCRIPTION HERE')
parser.add_argument('-i','--volume-ids', help='EBS volume ID', required=True)
parser.add_argument('-d','--delete-old', help='Delete old snapshots?', required=False, type=bool,default=True)
parser.add_argument('-x','--expiry-days', help='Number of days to keep snapshots', required=False, type=int, default=7)
args = parser.parse_args()
logging.basicConfig(filename='backup.log', level=logging.DEBUG, format='%(asctime)s:%(message)s', datefmt='%Y-%m-%d %I:%M:%S%p')
# Get all active volumes
volumeIds = args.volume_ids.split(',')
# Create the snapshots
if len(volumeIds):
snapshots = createSnapshots(volumeIds)
pass
# Delete snapshots older than expiry-days
if args.delete_old:
deleteOldSnapshots(getOurSnapshots(), args.expiry_days)Note: Change the profile name and region in the script to your EC2 instance region
Give the read and execute permission to script file for example 0755
Run the below command to take the snapshot
python3 /path/of/backup/script/ --volume-ids-yourvolid --expiry-days=no-of-days
For example, if your script name snapshot.py and located at /home/ubuntu/aws/ and your instance volume id is vol-0b7a9b59f536d5015 and the retention period will be 7 days then it should look like.
python3 /home/ubuntu/aws/shapshot.py --volume-ids-vol-0b7a9b59f536d5015 --expiry-days=7
Set this up in a cron job to trigger every day or whatever you want to run the cron job.
Moving Forward
You can Follow below steps that you can apply. Follow them and have some fun, happy snapshotting.
Linux/Windows System and SQL Server Database Administrator with more than 4.5 years of experience in Linux Servers administration, Windows Administration, Network Administration, SQL Server Administration, MySQL Administration and SQL Programming.