Introducing Backblaze B2 Cloud Storage Lifecycle Rules

Lifecycle Rules enable the Backblaze B2 Cloud Storage service to automatically hide and/or delete older versions of files stored in Backblaze B2. This capability is available for the Web GUI, CLI, and API. Lifecycle Rules are applied to some or all of the files in a Backblaze B2 Bucket with each bucket having its own Lifecycle Rules. Let’s take a look at how you can use this capability to help manage your Backblaze B2 storage space.

Overview

By default, Backblaze B2 will keep all the files you upload and all of the different versions of a file you upload. While B2 Cloud Storage will keep the different versions, it will “hide” older versions to keep the file list easier to view as shown in the B2 Cloud Storage file list (from the Web GUI) below.

There are two indicators that we have multiple versions of the same file. First the file name has a “2” at the end. This is the file version count and in this case means there are two versions of the file. The date/time stamp shown is for the most recent version of the file. The second indicator is the “missing” file information icon on the right.

When you click on the file version count (i.e. the “2”), the list will expand to show all of the file versions.

The older version, V2 in the example, is hidden. All versions of a given file are available to download, delete, or just keep around as desired. If the file names are different at all, they are stored as different files, so myfile-version1.doc is not the same as myfile-version2.doc, even if the content is identical.

There is no limit to the number of versions you can have of a given file. Also, by default, the length of time older versions stay in Backblaze B2 is forever, unless you delete them or you stop using the B2 Cloud Storage service.

Since all versions of a given file are kept around, you can use Lifecycle Rules to help manage the older versions. Let’s see how.

Lifecycle Rules and the Web GUI

To use Lifecycle Rules you will to have an active Backblaze B2 account and at least one bucket. It would help to have a few files in your Backblaze B2 account as well. After logging into your Backblaze B2 account and clicking “Buckets” on the B2 Cloud Storage menu, you’ll see a list of your buckets. For the bucket of interest select the “Bucket Settings” link as shown in the example below.

The “Bucket Settings” pop up screen, similar to the one below will appear.

Previously, this screen just contained the settings for making a bucket public or private. The additional items are for managing Lifecycle Rules.

Let’s cover the more common ways to use Lifecycle Rules.

File Versions

The File Versions rules apply to all the files in the bucket, unless otherwise specified. The four options for File Versions work as follows:

• Keep All Versions (Default): Keeps all versions of a given file.
• Keep Only Last Version: This keeps only the most current version of a file. The previous version of the file is “hidden” for one day and then deleted.
• Keep Prior Versions for a Number of Days: Older versions of a file are deleted after the specified number of days. The number of days can be one or greater. The most current version of a file is always kept unless it is explicitly deleted.
• Lifecycle Rules Below: Allows you to construct a rule that applies to one or more files. The three parameters are covered in the next section.

Most common situations can be handled using one of the first three options.

Constructing a Lifecycle Rule in the Web GUI

Before you start constructing Lifecycle Rules, here are a few things to know:

1. You are not allowed to create two or more rules that apply to the same file or files.
2. Remember that the old version of a file is automatically hidden when a new version of the same name is uploaded.
3. Read the File Versions documentation to understand how file versioning works in Backblaze B2.
4. Only use this option if you know what you are doing. You are giving B2 Cloud Storage the ability to hide and then permanently delete your files. Make sure this is something you want to do. Once deleted, files can not be restored by Backblaze. Please be careful.

There are three variables you need to enter to construct a Lifecycle Rule. All three are required for each rule added.

• fileNamePrefix: This specifies which files in the bucket the rules applies to. For example, a value of “workinprocess/” will apply to all files in the “workinprocess” directory/folder. A value of “” (open quote, close quote, no space) means the rule applies to all the files in the bucket.
• daysFromUploadingtoHiding: Setting this value causes the specified files to be hidden automatically after the specified number of days. This applies to all copies of the file, even the most current version. For example, setting this variable to 30 will hide all of the files in the specified directory/folder 30 days after they have been uploaded—this includes the most current version of the file. Only use this option if you want to hide all versions of the files specified. Valid values are “null” or numbers one and greater. Null means no files are hidden based on this rule.
• daysFromHidingtoDeleting: Setting this value causes the specified hidden files to be deleted automatically after the specified number of days. Remember that files are automatically hidden when they are replaced with a newer version. For example, if you set this variable to 10, then 10 days after a newer version of the file is uploaded, the older version will be deleted. Valid values are “null” or numbers one and greater. Null means no files are deleted based on this rule.

Lifecycle Rules Examples

Let’s assume that you back up your “log” files once a day to Backblaze B2 and that these files are stored in a directory/folder named log. Below are sample Lifecycle Rules you could set up to manage the log files in Backblaze B2 based on your data retention policies.

Example 1:

fileNamePrefix: log/
daysFromUploadingtoHiding: 7
daysFromHidingtoDeleting: null

Result: The files in the log/ folder will be hidden seven days after they are uploaded to Backblaze B2. This means the current version, as well as any previous versions of every file in the log/ folder, will be hidden after seven days. The hidden files are still available for download using their specific file_id and continue to count as part of your Backblaze B2 storage amount.

Example 2:

fileNamePrefix: log/
daysFromUploadingtoHiding: 7
daysFromHidingtoDeleting: 30

Result: The files in the log/ folder will be hidden seven days after they are uploaded to Backblaze B2. After the files have been hidden for 30 days, they will be permanently deleted. Until they are deleted, the files are still available for download and count as part of your Backblaze B2 storage amount.

Example 3:

fileNamePrefix: log/
daysFromUploadingtoHiding: null
daysFromHidingtoDeleting: 180

Result: The files in the log/ folder will be deleted 180 days after they have been either 1) explicitly hidden, or 2) been replaced by a file of the same name (i.e. they are an older version of the file—remember older file versions are automatically hidden). Until the files are deleted, all versions of a file are available for download and count as part of your Backblaze B2 storage amount.

Lifecycle Rules for the CLI and API

Both the Backblaze B2 CLI and Backblaze B2 API can utilize Lifecycle Rules.

For the Backblaze B2 API, the following calls work with Lifecycle Rules:

• b2_create_bucket
• b2_update_bucket
• b2_list_buckets

Each API call has an optional parameter named “lifecycleRules” that requires the three variables noted above in the following JSON format:
{
"daysFromHidingToDeleting": 30,
"daysFromUploadingToHiding": null,
"fileNamePrefix": "backup/"
}

In addition, b2_delete_bucket will return the list of lifecycleRules for the bucket being deleted.

Backblaze B2 Integrations and Lifecycle Rules

Application developers are encouraged to look at Lifecycle Rules to use in their applications that integrate into B2 Cloud Storage. This is especially true for applications that perform sync or backup operations to B2 Cloud Storage. For example, the user interface of the application could allow the user/admin to specify the number of days older versions of files should be kept in support of data retention rules in an organization.

Backblaze B2 Command-Line Tool

The Backblaze B2 Command-Line Tool provides command line access to the B2 Cloud Storage service. To use Lifecycle Rules with the Backblaze B2 CLI, you will need to be running version 0.7.0 of the Backblaze B2 Command-Line Tool. To upgrade to this version you can follow the Get Command-Line Tool instructions on the Backblaze website. The following commands utilize Lifecycle Rules in the CLI.

• b2 create-bucket [--bucketInfo <json>] [--lifecycleRules <json>] <bucketName> [allPublic | allPrivate]
• b2 update-bucket [--bucketInfo <json>] [--lifecycleRules <json>] <bucketName> [allPublic | allPrivate]
• b2 get-bucket <bucketName>

The b2 get-bucket command will return all the variables for the named bucket.

Summary

Lifecycle Rules are very handy in helping manage multiple versions of files you are storing in Backblaze B2. These rules can help save storage space by automatically deleting older, unneeded versions of files and can make the B2 Cloud Storage file list easier to navigate and manage. Since Lifecycle Rules will delete older file versions, one recommendation is to start with a large number of days for “daysFromHidingtoDeleteing” and reduce this number as you become comfortable with this feature. The easiest way to do this is to use the “Keep Prior Versions a Number of Days” option in the Web GUI.

There is more information on Lifecycle Rules in the Backblaze B2 documentation section of the Backblaze website. The website also has code examples for the API calls, information on the Backblaze B2 Command-Line tool, and more. Let us know if you find anything there we need to update or you feel is missing.

[Editor’s note: Updated on 1/11/2016 with reference to file version documentation along with enhancing specifics regarding hidden files].