How to query language models

Scaleway’s Generative APIs service allows users to interact with powerful language models hosted on the platform.

There are several ways to interact with language models:

• The Scaleway console provides complete playground, aiming to test models, adapt parameters, and observe how these changes affect the output in real-time.
• Via the Chat API

Before you start

To complete the actions presented below, you must have:

• A Scaleway account logged into the console
• Owner status or IAM permissions allowing you to perform actions in the intended Organization
• A valid API key for API authentication
• Python 3.7+ installed on your system

Accessing the Playground

Scaleway provides a web playground for instruct-based models hosted on Generative APIs.

1. Navigate to Generative APIs under the AI section of the Scaleway console side menu. The list of models you can query displays.
2. Click the name of the chat model you want to try. Alternatively, click «See more Icon» next to the chat model, and click Try model in the menu.

The web playground displays.

Using the playground

1. Enter a prompt at the bottom of the page, or use one of the suggested prompts in the conversation area.
2. Edit the hyperparameters listed on the right column, for example the default temperature for more or less randomness on the outputs.
3. Switch models at the top of the page, to observe the capabilities of chat models offered via Generative APIs.
4. Click View code to get code snippets configured according to your settings in the playground.

Querying language models via API

The Chat API is an OpenAI-compatible REST API for generating and manipulating conversations.

You can query the models programmatically using your favorite tools or languages. In the following example, we will use the OpenAI Python client.

Installing the OpenAI SDK

Install the OpenAI SDK using pip:

pip install openai

Initializing the client

Initialize the OpenAI client with your base URL and API key:

from openai import OpenAI
# Initialize the client with your base URL and API key
client = OpenAI(
    base_url="https://api.scaleway.ai/v1",  # Scaleway's Generative APIs service URL
    api_key="<SCW_SECRET_KEY>"  # Your unique API secret key from Scaleway
)

Generating a chat completion

You can now create a chat completion, for example with the llama-3.1-8b-instruct model:

# Create a chat completion using the 'llama-3.1-8b-instruct' model
response = client.chat.completions.create(
    model="llama-3.1-8b-instruct",
    messages=[{"role": "user", "content": "Describe a futuristic city with advanced technology and green energy solutions."}],
    temperature=0.2,  # Adjusts creativity
    max_tokens=100,   # Limits the length of the output
    top_p=0.7         # Controls diversity through nucleus sampling. You usually only need to use temperature.
)
# Print the generated response
print(response.choices[0].message.content)

This code sends a message to the model and returns an answer based on your input. The temperature, max_tokens, and top_p parameters control the response’s creativity, length, and diversity, respectively.

A conversation style may include a default system prompt. You may set this prompt by setting the first message with the role system. For example:

[
  {
  	"role": "system",
  	"content": "You are Xavier Niel."
  },
  {
  	"role": "user",
  	"content": "Hello, what is your name?"
  }
]

Model parameters and their effects

The following parameters will influence the output of the model:

• messages: A list of message objects that represent the conversation history. Each message should have a role (e.g., “system”, “user”, “assistant”) and content.
• temperature: Controls the output’s randomness. Lower values (e.g., 0.2) make the output more deterministic, while higher values (e.g., 0.8) make it more creative.
• max_tokens: The maximum number of tokens (words or parts of words) in the generated output.
• top_p: Recommended for advanced use cases only. You usually only need to use temperature. top_p controls the diversity of the output, using nucleus sampling, where the model considers the tokens with top probabilities until the cumulative probability reaches top_p.
• stop: A string or list of strings where the model will stop generating further tokens. This is useful for controlling the end of the output.

Streaming

By default, the outputs are returned to the client only after the generation process is complete. However, a common alternative is to stream the results back to the client as they are generated. This is particularly useful in chat applications, where it allows the client to view the results incrementally as each token is produced. Following is an example using the chat completions API:

from openai import OpenAI
client = OpenAI(
    base_url="https://api.scaleway.ai/v1",  # Scaleway's Generative APIs service URL
    api_key="<SCW_API_KEY>"  # Your unique API key from Scaleway
)
response = client.chat.completions.create(
  model="llama-3.1-8b-instruct",
  messages=[{
    "role": "user",
    "content": "Sing me a song",
  }],
  stream=True,
)
for chunk in response:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

Async

The service also supports asynchronous mode for any chat completion.

import asyncio
from openai import AsyncOpenAI
client = AsyncOpenAI(
    base_url="https://api.scaleway.ai/v1",  # Scaleway's Generative APIs service URL
    api_key="<SCW_API_KEY>"  # Your unique API key from Scaleway
)
async def main():
    stream = await client.chat.completions.create(
        model="llama-3.1-8b-instruct",
        messages=[{
        "role": "user",
        "content": "Sing me a song",
        }],
        stream=True,
    )
    async for chunk in stream:
        if chunk.choices and chunk.choices[0].delta.content:
            print(chunk.choices[0].delta.content, end="")
asyncio.run(main())