Using Object Storage with s3cmd

• Object-Storage
• s3cmd
• CORS
• buckets

In this tutorial you will learn how to use s3cmd as a client for Scaleway Object Storage. S3cmd is a tool that allows you to create, list and delete buckets from the command line, as well as to download, upload, and delete objects to/from your buckets.

Before you start

To complete the actions presented below, you must have:

• A Scaleway account logged into the console
• Owner status or IAM permissions allowing you to perform actions in the intended Organization
• A valid API key

Installing S3cmd

You can install S3cmd frm the command line using your Linux package manager.

1. Enter the following command to update and upgrade your packages:

   apt update && apt upgrade -y
   

2. Enter the following command to install s3cmd:

   apt install s3cmd
   

Configuring S3cmd

Carry out the following commands from your terminal.

1. Create a configuration file named .s3cfg in your home directory:

   cd $HOME
   touch .s3cfg
   

2. Open the file with a text editor, e.g. nano:

   nano .s3cfg
   

3. Copy and paste the following text into the file, then save and close. Make sure you use the host-base, host-bucket and bucket_location details of your own bucket (visible in the Bucket settings tab of the console). You will also need to replace <ACCESS_KEY> and <SECRET_KEY> with your own API credentials.

   [default]
   # Object Storage Region NL-AMS
   host_base = s3.nl-ams.scw.cloud
   host_bucket = %(bucket).s3.nl-ams.scw.cloud
   bucket_location = nl-ams
   use_https = True
   # Login credentials
   access_key = <ACCESS_KEY>
   secret_key = <SECRET_KEY>
   

   Tip

   Alternatively you can use the configuration file generator by running s3cmd --configure to generate a configuration file.

Using Multiple Configuration Files

You can configure s3cmd to use different configuration files on the same computer to manage buckets in multiple regions. Here, we assume that your default configuration file was for a bucket in nl-ams, and we create a new configuration file for buckets in fr-par.

Carry out the following steps from the terminal.

1. Create a default configuration file as indicated in the previous step.
2. Create a new file named .s3cfg-fr-par in your home directory.

   cd $HOME
   touch .s3cfg-fr-par
   

3. Copy and paste the following text into the file, then save and close. Make sure you replace <ACCESS_KEY> and <SECRET_KEY> with your own API credentials.

   [default]
   signature = s3v4
   # Object Storage Region FR-PAR
   bucket_location = fr-par
   host_base = https://s3.fr-par.scw.cloud
   host_bucket = https://s3.fr-par.scw.cloud
   # Login credentials
   access_key = <ACCESS_KEY>
   secret_key = <SECRET_KEY>
   

4. Use s3cmd by specifying the new configuration file in the command:

   s3cmd ls -c ~/.s3cfg-fr-par
   

Using Object Storage with S3cmd

Creating a new bucket

Enter the following command to create a bucket with S3cmd. The bucket name must contain only alphanumeric and lowercase characters. Here, we call it mynewbucket:

s3cmd mb s3://mynewbucket

You should see an output similar to the following:

Bucket 's3://mynewbucket/' created

Uploading objects into the bucket

Enter the following command to upload files into a bucket. Here, we upload the files movie1.avi and photo1.jpg into mynewbucket:

s3cmd put movie1.avi photo1.jpg s3://mynewbucket

You should see an output similar to the following:

movie1.avi -> s3://mynewbucket/movie1.avi [1 of 2]
0 of 0 0% in 0s 0.00 B/s done
photo1.jpg -> s3://mynewbucket/photo1.jpg [2 of 2]
0 of 0 0% in 0s 0.00 B/s done

ERROR: Cannot retrieve any response status before encountering an EPIPE or ECONNRESET exception
WARNING: Upload failed: /bucket_name?partNumber=1&uploadId=ZWY5ZWRhNzItMjE3NS00ZTY4LWFkOTMtYTFhODRkYTRkNzdl ([Errno 32] Broken pipe)

Listing objects inside a bucket

Enter the following command to list files inside a bucket. Here, we list the files in the bucket mynewbucket:

s3cmd ls s3://mynewbucket

You should see an output similar to the following, showing the files inside your bucket:

2014-09-10 08:44 0 s3://mynewbucket/movie1.avi
2014-09-10 08:44 0 s3://mynewbucket/photo1.jpg

Downloading an object from a bucket

Enter the following command to download an object from your bucket to your local machine. Here, we download the object movie1.avi from the bucket mynewbucket:

s3cmd get s3://mynewbucket/movie1.avi movie1.avi

You should see an output similar to the following:

s3://mynewbucket/movie1.avi -> movie.avi [1 of 1]
0 of 0 0% in 0s 0.00 B/s done

Removing objects from a bucket

Enter the following command to remove an object from your bucket. Here, we remove the object movie1.avi from the bucket mynewbucket:

s3cmd del s3://mynewbucket/movie1.avi

You should see an output similar to the followng:

File s3://mynewbucket/movie1.avi deleted

Removing a bucket

Enter the following command to remove a bucket. Here, we remove the bucket mynewbucket:

john@localhost:~$ s3cmd rb s3://mynewbucket

You should see an output similar to the followng:

Bucket 's3://mynewbucket/' removed

Configuring CORS

The CORS standard provides browsers with a way to request resources from remote domains when they have permission. This lets you configure how your buckets can be accessed from browser-based applications and websites.

Setting a CORS configuration on a bucket

1. Create a CORS configuration file and open it in a text editor:

   nano cors.xml
   

2. Put your configuration in the file, save it and exit the editor:

   <CORSConfiguration>
     <CORSRule>
       <AllowedOrigin>http://www.example.com</AllowedOrigin>
       <AllowedMethod>PUT</AllowedMethod>
       <AllowedMethod>POST</AllowedMethod>
       <AllowedMethod>DELETE</AllowedMethod>
       <AllowedHeader>*</AllowedHeader>
       <MaxAgeSeconds>3000</MaxAgeSeconds>
       <ExposeHeader>x-server-side-encryption</ExposeHeader>
     </CORSRule>
   </CORSConfiguration>
   

3. Set the information on the bucket. Make sure to replace bucketname with the name of your bucket:

   s3cmd setcors cors.xml s3://bucketname
   

Deleting the CORS configuration of a bucket

Run the following command to delete the CORS configuration of a bucket. Replace bucketname with the name of your bucket:

s3cmd delcors s3://bucketname

Going further

For more information about the different s3cmd commands, refer to the official documentation by running s3cmd --help.