What is Scaleway Managed Inference?Link to this anchor

Scaleway’s Managed Inference is a fully managed service that allows you to deploy, run, and scale AI models in a dedicated environment. It provides optimized infrastructure, customizable deployment options, and secure access controls to meet the needs of enterprises and developers looking for high-performance inference solutions.

Where are the inference servers located?Link to this anchor

All models are currently hosted in a secure data center located in Paris, France, operated by OPCORE. This ensures low latency for European users and compliance with European data privacy regulations.

What is the difference between Managed Inference and Generative APIs?Link to this anchor

• Managed Inference: Allows deployment of curated or custom models with chosen quantization and instances, offering predictable throughput and enhanced security features like private network isolation and access control. Managed Inference is billed by hourly usage, whether provisioned capacity is receiving traffic or not.
• Generative APIs: A serverless service providing access to pre-configured AI models via API, billed per token usage.

Where can I find information regarding the data, privacy, and security policies applied to Scaleway’s AI services?Link to this anchor

You can find detailed information regarding the policies applied to Scaleway’s AI services in our Data, privacy, and security for Scaleway’s AI services documentation.

Is Managed Inference compatible with Open AI APIs?Link to this anchor

Managed Inference aims to achieve seamless compatibility with OpenAI APIs. Find detailed information in the Scaleway Managed Inference as drop-in replacement for the OpenAI APIs documentation.

What are the SLAs applicable to Managed Inference?Link to this anchor

We are currently working on defining our SLAs for Managed Inference. We will provide more information on this topic soon.

What are the performance guarantees (vs. Generative APIs)?Link to this anchor

Managed Inference provides dedicated resources, ensuring predictable performance and lower latency compared to Generative APIs, which are a shared, serverless offering optimized for infrequent traffic with moderate peak loads. Managed Inference is ideal for workloads that require consistent response times, high availability, custom hardware configurations or generate extreme peak loads during a narrow period of time. Compared to Generative APIs, no usage quota is applied to the number of tokens per second generated, since the output is limited by the GPU Instances size and number of your Managed Inference Deployment.

What types of models can I deploy with Managed Inference?Link to this anchor

You can deploy a variety of models, including:

• Large language models (LLMs)
• Image processing models
• Audio recognition models
• Custom AI models (through API only yet) Managed Inference supports both open-source models and your own uploaded proprietary models.

How do I deploy a model using Managed Inference?Link to this anchor

Deployment is done through Scaleway’s console or API. You can choose a model from Scaleway’s selection or import your own directly from Hugging Face’s repositories, configure Instance types, set up networking options, and start inference with minimal setup.

Can I fine-tune or retrain my models within Managed Inference?Link to this anchor

Managed Inference is primarily designed for deploying and running inference workloads. If you need to fine-tune or retrain models, you may need to use a separate training environment, such as Scaleway’s GPU Instances, and then deploy the trained model in Managed Inference.

What Instance types are available for inference?Link to this anchor

Managed Inference offers different Instance types optimized for various workloads from Scaleway’s GPU Instances range. You can select the Instance type based on your model’s computational needs and compatibility.

How is Managed Inference billed?Link to this anchor

Billing is based on the Instance type and usage duration. Unlike Generative APIs, which are billed per token, Managed Inference provides predictable costs based on the allocated infrastructure. Pricing details can be found on the Scaleway pricing page.

Can I run inference on private models?Link to this anchor

Yes, Managed Inference allows you to deploy private models with access control settings. You can restrict access to specific users, teams, or networks.

Does Managed Inference support model quantization?Link to this anchor

Yes, Scaleway Managed Inference supports model quantization to optimize performance and reduce inference latency. You can select different quantization options depending on your accuracy and efficiency requirements.

Is Managed Inference suitable for real-time applications?Link to this anchor

Yes, Managed Inference is designed for low-latency, high-throughput applications, making it suitable for real-time use cases such as chatbots, recommendation systems, fraud detection, and live video processing.

Can I use Managed Inference with other Scaleway services?Link to this anchor

Absolutely. Managed Inference integrates seamlessly with other Scaleway services, such as Object Storage for model hosting, Kubernetes for containerized applications, and Scaleway IAM for access management.

Do model licenses apply when using Managed Inference?Link to this anchor

Yes, model licenses need to be complied with when using Managed Inference. Applicable licenses are available for each model in our documentation.

• For models provided in the Scaleway catalog, you need to accept licenses (including potential EULA) before creating any Managed Inference deployment.
• For custom models you choose to import on Scaleway, you are responsible for complying with model licenses (as with any software you choose to install on a GPU Instance for example).