Contents x
Use S5cmd with Backblaze B2
    • Print
    • Dark
      Light
    Contents

    Use S5cmd with Backblaze B2

      • Print
      • Dark
        Light
      Article summary

      s5cmd is a free, open-source tool written in the Go language. It supports a wide range of object management tasks, both for cloud storage services like Backblaze B2 Cloud Storage and local file systems. It also supports parallel tasking and multiple threads for very fast throughput. s5cmd provides binaries for most operating systems on the releases page in the git repository.

      You can use s5cmd with Backblaze B2 by configuring s5cmd to connect with your Backblaze B2 account, using your preferred command-line interface (CLI).

      Enable Backblaze B2

      Before you begin: You must have a Backblaze B2 Cloud Storage account. If you already have a Backblaze account and the left navigation menu contains a B2 Cloud Storage section, your account is already enabled for Backblaze B2.

      1. Sign in to the Backblaze web console.
      2. In the user menu in the upper-right corner of the page, select My Settings.
      3. Under Enabled Products, select the checkbox to enable B2 Cloud Storage.
      4. Review the Terms and Conditions, and click OK to accept them. 

      Create an Application Key

      Application keys control access to your Backblaze B2 Cloud Storage account and the buckets that are contained in your account.
      1. Sign in to the Backblaze web console.
      2. In the left navigation menu under B2 Cloud Storage, click Application Keys.
      3. Click Add a New Application Key, and enter an app key name. Please note: You cannot search an app key by this name; therefore, app key names are not required to be globally unique. Key names are limited to 100 characters and can contain letters, numbers, and "-", but not I18N characters, such as é, à, and ü.
      4. Select All or a specific bucket in the Allow Access to Bucket(s) dropdown menu.
      5. Optionally, select your access type (Read and Write, Read Only, or Write Only).
      6. Optionally, select the Allow List All Bucket Names checkbox (required for the B2 Native API b2_list_buckets and the S3-Compatible API S3 List Buckets operations).
      7. Optionally, enter a file name prefix to restrict application key access only to files with that prefix. Depending on what you selected in step #4, this limits application key access to files with the specified prefix for all buckets or just the selected bucket.
      8. Optionally, enter a positive integer to limit the time, in seconds, before the application key expires. The value must be less than 1000 days (in seconds).
      9. Click Create New Key, and note the resulting keyID and applicationKey values.
      Note
      When you create a new app key, the response contains the actual key string, for example N2Zug0evLcHDlh_L0Z0AJhiGGdY. You can always find the keyID on this page, but for security, the applicationKey appears only once. Make sure you copy and securely save this value elsewhere.

      Set Up s5cmd with Backblaze B2

      1. Use the following command to create a directory that will contain your credentials file:
        mkdir ~/.aws
      2. Use the following command to create your credentials file:
        vi ~/.aws/credentials
      3. When prompted, complete the following fields:
        • aws_access_key_id (your Backblaze B2 Key ID)
        • aws_secret_access_key (your Backblaze B2 application key)
      4. Use the following command to create an alias that defines your endpoint URL:
        alias s5cmd='s5cmd --endpoint-url https://s3.us-west-002.backblazeb2.com'
        This step helps speed up your work by automatically defining the endpoint URL. For example, you can add this command to your .bashrc file in Linux.

      Use s5cmd with Backblaze B2

      After you set up s5cmd to use your Backblaze B2 endpoint, you can work with some simple usage patterns. The following examples assume that you created an alias to define your endpoint URL, as described in the "Set Up s5cmd with Backblaze B2" task.

      If you did not create the alias, then you must manually enter the URL when you use these patterns.

      List All Buckets

      Use this command to list all of your buckets:

      s5cmd ls

      If you did not create an alias to define your endpoint URL, use the following example, submitting your URL, to list all of your buckets:

      s5cmd -endpoint-url https://s3.us-west-002.backblazeb2.com ls

      List the Contents of a Bucket

      Use the following command to list the contents of your bucket:

      s5cmd ls s3://mydatabucket-1/

      Create a Bucket

      Use the following command to create your bucket:

      s5cmd mb s3://mydatabucketus1/

      Copy a Local File into a Bucket

      Use the following command to copy a local file into your bucket:

      s5cmd cp my-pets2020.jpg s3://mydatabucket-1/

      Performance Optimization for High-Speed Links

      In most circumstances, you can use the default number of threads for copy operations. If you have a fast Internet connection, you can manually increase the number of threads being used with the cp --concurrency command.

      s5cmd --endpoint-url https://s3.us-west-002.backblazeb2.com cp --concurrency 50 /tmp/sample.txt s3://smcmdus1/test/

      If you used the alias command from the "Set Up s5cmd with Backblaze B2" task, use the following command to optimize your performance:

      s5cmd cp --concurrency 50 /tmp/sample.txt s3://smcmdus1/test/

      The more threads that you use, the more memory is used. You should monitor your system memory as you test for the optimal number of threads. On a system with 8 CPUs and 24 GB of RAM, testing showed that cp --concurrency 100 achieved the fastest transfer rate. Your own environment may require different settings.

      Was this article helpful?
      What's Next