Optimize your Object Storage performance

Scaleway Object Storage is a highly resilient and versatile service that guarantees the reliability and accessibility of your data, while being fully Amazon S3-compatible and user-friendly.

Even though it is designed to provide best-in-class latency and throughput, user infrastructure plays a predominant role in achieving optimum efficiency, as many different factors can have an impact on performance, such as your hardware, your software stack, or the way you manage your objects.

These guidelines aim to provide recommendations and methods to help you get the most out of your Object Storage.

Hardware recommendations

Network

To avoid network-related bottlenecks in your infrastructure, allow as much bandwidth as possible to your Object Storage deployment.

Adapt the throughput and latency of your bandwidth according to your most critical use cases:

• Favor throughput over latency for concurrent requests on large multipart requests to maximize efficiency. A smaller number of objects means fewer per-request connection setups and Secure Socket Layer (SSL) handshakes, which generate latency for each part, but have little-to-no impact on throughput thanks to parallelization.

• Favor latency over throughput for smaller-size objects (less than 1 MB), such as logs or website assets. Avoid per-request connection to remove TCP slow starts and SSL handshakes on each request. A lower latency will also reduce your Time To First Byte (TTFB) for larger objects. The tradeoff is that ListObject operations will take longer to complete as they will return a greater number of objects.

CPU

The CPU plays a major role in your Object Storage infrastructure, as it will determine your concurrency capacity. A good order of magnitude is 16 cores per socket to handle concurrent requests efficiently.

Try various operations with a single request at a time and measure your CPU usage. The operations that have the highest CPU usage are likely to throttle your machine and create bottlenecks. These operations can be used to determine the concurrency capacity of your Object Storage deployment.

For example, if the most CPU-intensive operation uses 20% of your CPU, you can execute up to five requests concurrently, to reach 100% of your processing capacity.

Geographic location

The physical distance to the hardware hosting your Object Storage can also have an impact on performance, especially on latency. Make sure to benchmark the different regions where Object Storage is available to compare latency on your mission-critical operations.

For instance, media and content distribution are often heavily affected by the physical distance between the host and the client, as objects are usually large in this scenario.

Software recommendations

Now that your hardware is properly set up, make sure that your object transfers are properly configured. The way you manage your object operations also has a major impact on performance, as well as the tools you use.

Multipart uploads

The most impactful feature is multipart uploads. Large and small objects have different constraints, and adapting your processes to your “raw material” is quintessential for a seamless Object Storage deployment. Breaking bigger objects, such as uncompressed 4K videos, into multiple parts will allow you to leverage the full potential of Scaleway Object Storage.

Multipart uploads must respect the following technical specifications:

• 1 to 1000 parts per object
• 5 MB to 5 GB per part (except for the last part)
• Object size cannot exceed 5 TB

Breaking your objects into very small parts is counter productive, as many operations are executed on each part, thus increasing the latency and reducing the throughput.

Multipart upload allows you to leverage a significant concurrency capacity, because:

• you can parallelize part upload for a better throughput.

• you can reduce the risk of failed uploads due to a network issue by using smaller parts, as it allows to retry the upload of the failed part and not the whole object.

• you can pause your upload and resume later, and upload parts over a desired period of time to reduce stress linked to upload on your infrastructure.

• you can upload object parts as you are creating them, without knowing the size of the object or the number of parts.

• you can request different parts simultaneously using a single cURL request. GetObject requests allow you to know the number of parts of an object.

• you can automate the removal of incomplete multipart uploads to clear the parts that are no longer needed if an upload fails.

Tools and features

Try benchmarking several clients to upload objects, such as MinIO CLI, awscli, Rclone, or s5cmd.

Leverage the different storage classes offered by Scaleway, and benchmark them according to your needs to find the right configuration for your deployment.

Enable Scaleway Cockpit to access detailed and in-depth monitoring and logs to identify potential bottlenecks in your Object Storage deployment.

Enable Scaleway Edge Services to achieve better performance on GET operations by caching your objects closer to your users. This feature is particularly suited to CDN applications.

Do not hesitate in reaching out or getting in contact with other Object Storage users on our Community Slack.