Managing the lifecycle of objects
You can configure the lifecycle of a group of objects to manage their storage costs effectively. A lifecycle configuration is a set of rules that defines actions applied to a group of objects stored on Object Storage.
The Amazon S3 API allows you to automatically manage the lifecycle of your object by setting lifecycle rules. These rules allow you to automatically:
- Delete your objects and their versions by setting expiration durations
- Transition objects and their versions to a colder storage class
- Abort incomplete multipart uploads
Lifecycle management on Object Storage is available on every Amazon S3 compliant tool (sdk, aws-cli, boto, etc), as well as from the Scaleway console.
Before you start
To complete the actions presented below, you must have:
- Owner status or IAM permissions allowing you to perform actions in the intended Organization
- An Object Storage bucket
- Installed and initialized the AWS CLI
Creating a lifecycle rule using the AWS CLI
-
Create a new
my-lifecycle-rule.jsonfile. -
Add the required rules to it using the rule elements detailed in the section about XML tokens.
{ "Rules": [ { "ID": "TransitionToGlacierAfter90Days", "Status": "Enabled", "Filter": { "Prefix": "" }, "Transitions": [ { "Days": 90, "StorageClass": "GLACIER" } ] } ] } -
Run the following command to apply the lifecycle rule to the specified bucket. Do not forget to replace the placeholders with the appropriate values.
aws s3api put-bucket-lifecycle-configuration \ --bucket my-bucket \ --lifecycle-configuration file://my-lifecycle-rule.jsonNo output is returned.
-
Run the following command to make sure the rule was properly applied:
aws s3api get-bucket-lifecycle-configuration \ --bucket my-bucket \The content of your rules displays.
Lifecycle specification
A lifecycle configuration is an XML file, containing a set of rules with predefined actions that you want Object Storage to perform on objects during their lifetime:
<LifecycleConfiguration>
<Rule>
...
</Rule>
<Rule>
...
</Rule>
...
</LifecycleConfiguration>Each Rule consists of the following:
Filteridentifying a subset of objects to which the rule applies.Statuswhether the rule is in effect (a rule can be present butDisabled)- Lifecycle transition(s) to perform on the selected objects
Available XML tokens
Rule
- Description
- Container for a lifecycle rule. There is a limit of one thousand (1000) rules per bucket.
- Parent
LifecycleConfiguration
Filter
- Description
- Container for elements that describes the subset of the object the rule applies for. If the content of the tag is empty, the rule applies to all objects in the bucket.
- Parent
Rule
And
- Description
- Container for chaining rule filters.
- Parent
Rule
Sample
<And>
<Prefix>myprefix</Prefix>
<Tag>
<Key>mytagkey1</Key>
<Value>mytagvalue1</Value>
</Tag>
</And>This applies the Prefix rule and the Tag.
Prefix
- Description
- Object key prefix identifying one or more objects to which the rule applies. Object Storage limits this token to 1 per
FilterRule. - Parent
FilterorAnd
Tag
- Description
- Container for specifying a
Keyand aValue. - Parent
FilterorAnd
Status
- Description
- Describes whether or not a rule is enabled.
- Parent
Rule- Values
Enabled,Disabled
Transition
- Description
- Specifies a period and a destination for an object's lifetime
- Parent
Rule
Day
- Description
- Specifies the number of days after object creation when the rule takes effect.
- Parent
Transitionand/orExpiration
StorageClass
- Description
- Specifies the storage class to which the object shall be transferred. On the Scaleway platform the storage classes
STANDARD,GLACIERandONEZONE_IAare available. - Parent
Transition
Expiration
- Description
- Describes the expiration of the object lifetime. If versioning is enabled, this rule only deletes the current version of an object.
- Parent
Rule
ID
- Description
- Unique identifier for the rule. This value is limited to 255 characters.
- Parent
Rule
NoncurrentVersionExpiration
- Description
- Describes the lifetime of non-current versions before they expire. Expired objects are deleted automatically.
- Parent
Rule
NoncurrentVersionTransitions
- Description
- Describes the lifetime of non-current versions before being transitioned to the specified storage class. See Transition for allowed transitions.
- Parent
Rule
NoncurrentDays
Description The number of days after which non-current versions are expired or transitioned.
- Parent
NoncurrentVersionTransitions,NoncurrentVersionExpiration
Configuration example
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<ID>id-1</ID>
<Expiration>
<Days>1</Days>
</Expiration>
<Filter>
<And>
<Prefix>myprefix</Prefix>
<Tag>
<Key>mytagkey1</Key>
<Value>mytagvalue1</Value>
</Tag>
<Tag>
<Key>mytagkey2</Key>
<Value>mytagvalue2</Value>
</Tag>
</And>
</Filter>
<Status>Enabled</Status>
</Rule>
</LifecycleConfiguration>The lifecycle configuration above translates to the following effective rules:
- on the rule named
id-1, we set the objects with themyprefixPrefix - the tags
mytagkey1,mytagkey2sets tomytagvalue1 mytagvalue2is set to expire after one day.
Setting rules for incomplete multipart uploads
When using the awscli to perform a multipart upload, it is possible to set a bucket lifecycle rule that allows you to automate the stoppage of uploads if they are not completed. You can define a deadline for the expected completion with a lifecycle rule. If the multipart upload is incomplete by the end of the defined timeframe, it will be stopped and the parts associated with it will be deleted.
You can automate the removal of incomplete multipart uploads by adding or uploading a lifecycle configuration to your bucket. The setup can be done via an API call or an AWS-CLI command.
Send a PutBucketLifecycleConfiguration request with the following XML payload:
<?xml version="1.0" encoding="UTF-8"?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<ID>Remove uncompleted uploads</ID>
<Filter>
<Prefix></Prefix>
</Filter>
<Status>Enabled</Status>
<AbortIncompleteMultipartUpload>
<DaysAfterInitiation>1</DaysAfterInitiation>
</AbortIncompleteMultipartUpload>
</Rule>
</LifecycleConfiguration>CLI command
- Create a file called
incomplete-upload-lifecycle-policy.jsonwith the following content.{ "Rules": [{ "ID": "Remove incomplete multipart uploads", "Status": "Enabled", "Filter": { "Prefix": "" }, "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 1 } }] } - In a terminal, run the command below to apply your lifecycle configuration to the desired bucket:
aws s3api put-bucket-lifecycle-configuration --bucket my-bucket --lifecycle-configuration file://incomplete-upload-lifecycle-policy.json