Was this page helpful?

How to use structured outputs

Reviewed on 30 October 2024 • Published on 17 September 2024

Structured outputs allow users to get consistent, machine-readable JSON format responses from language models. JSON, as a widely-used format, enables seamless integration with a variety of platforms and applications. Its interoperability is crucial for developers aiming to incorporate AI functionality into their current systems with minimal adjustments.

By specifying a response format when using the Chat Completions API, you can ensure that responses are returned in a JSON structure. There are two main modes for generating JSON: Object Mode (schemaless) and Schema Mode (deterministic, structured output).

There are several ways to interact with language models:

The Scaleway console provides a complete playground, aiming to test models, adapt parameters, and observe how these changes affect the output in real-time.
Via the Chat API

Before you startLink to this anchor

To complete the actions presented below, you must have:

A Scaleway account logged into the console
Owner status or IAM permissions allowing you to perform actions in the intended Organization
A valid API key for API authentication
Python 3.7+ installed on your system

Types of structured outputsLink to this anchor

Structured outputs (schema mode):
- Type {"type": "json_schema"}
- This mode enforces a strict schema format, where the output adheres to the predefined structure.
- Supports complex types and validation mechanisms as per the JSON schema specification, including nested schemas composition (anyOf, allOf, oneOf etc), $ref, all types, and regular expressions.
JSON mode (Legacy method):
- Type: {"type": "json_object"}
- This mode is non-deterministic and allows the model to output a JSON object without strict validation.
- Useful for flexible outputs when you expect the model to infer a reasonable structure based on your prompt.
- JSON mode is older and has been used by developers since early API implementations, but lacks reliability in response formats.

Note

All LLMs in the Scaleway library support Structured outputs and JSON mode. However, a schemaless JSON mode will produce lower quality results and is not recommended.

Code examplesLink to this anchor

Tip

Before diving into the code examples, ensure you have the necessary libraries installed:

pip install openai pydantic

The following Python examples demonstrate how to use Structured outputs to generate structured responses.

We are using the base code below to send our LLM a voice note transcript to structure:

import json
from openai import OpenAI
from pydantic import BaseModel, Field
# Set your preferred model
MODEL = "llama-3.1-8b-instruct"
# Set your API key
API_KEY = "<SCW_API_KEY>"
client = OpenAI(
    base_url="https://api.scaleway.ai/v1",
    api_key=API_KEY,
)
# Define the schema for the output using Pydantic
class VoiceNote(BaseModel):
    title: str = Field(description="A title for the voice note")
    summary: str = Field(description="A short one sentence summary of the voice note.")
    actionItems: list[str] = Field(description="A list of action items from the voice note")
# Transcript to use for the output
TRANSCRIPT = ( 
    "Good evening! It's 6:30 PM, and I'm just getting home from work. I have a few things to do " 
    "before I can relax. First, I'll need to water the plants in the garden since they've been in the sun all day. " 
    "Then, I'll start preparing dinner. I think a simple pasta dish with some garlic bread should be good. " 
    "While that's cooking, I'll catch up on a couple of phone calls I missed earlier."
)

Using structured outputs with JSON schema (Pydantic)Link to this anchor

Using Pydantic, users can define the schema as a Python class and enforce the model to return results adhering to this schema.

extract = client.chat.completions.create(
    messages=[
        {
            "role": "system",
            "content": "The following is a voice message transcript. Only answer in JSON using '{' as the first character.",
        },
        {
            "role": "user",
            "content": TRANSCRIPT,
        },
    ],
    model=MODEL,
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "VoiceNote",
            "schema": VoiceNote.model_json_schema(),
        }
    },
)
output = json.loads(extract.choices[0].message.content)
print(json.dumps(output, indent=2))

Output example:

{
  "title": "To-Do List",
  "summary": "Returning from work, need to complete tasks before relaxing",
  "actionItems": [
    "Water garden",
    "Prepare dinner: pasta dish with garlic bread",
    "Catch up on missed phone calls"
  ]
}

Using structured outputs with JSON schema (manual definition)Link to this anchor

Alternatively, users can manually define the JSON schema inline when calling the model.

extract = client.chat.completions.create(
    messages=[
        {
            "role": "system",
            "content": "The following is a voice message transcript. Only answer in JSON using '{' as the first character.",
        },
        {
            "role": "user",
            "content": TRANSCRIPT,
        },
    ],
    model=MODEL,
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "VoiceNote", 
            "schema": {
                "type": "object",
                "properties": {
                    "title": {"type": "string"},
                    "summary": {"type": "string"},
                    "actionItems": {
                        "type": "array",
                        "items": {"type": "string"}
                    }
                },
                "additionalProperties": False,
                "required": ["title", "summary", "actionItems"]
            }
        }
    }
)
output = json.loads(extract.choices[0].message.content)
print(json.dumps(output, indent=2))

Output example:

{
  "title": "Evening Routine",
  "actionItems": [
    "Water the plants",
    "Cook dinner (pasta and garlic bread)",
    "Make phone calls"
  ],
  "summary": "Made a list of tasks to accomplish before relaxing tonight"
}

Tip

When using the OpenAI SDKs like in the examples above, you are expected to set additionalProperties to false, and to specify all your properties as required.

Using JSON mode (schemaless, Legacy method)Link to this anchor

Important

When using the OpenAI SDKs as in the examples above, you are expected to set additionalProperties to false, and to specify all your properties as required.
JSON mode: It is important to explicitly ask the model to generate a JSON output either in the system prompt or user prompt. To prevent infinite generations, model providers most often encourage users to ask the model for short JSON objects. Prompt example: Only answer in JSON using '{' as the first character..

In JSON mode, you can prompt the model to output a JSON object without enforcing a strict schema.

extract = client.chat.completions.create(
    messages=[
        {
            "role": "system",
            "content": "The following is a voice message transcript. Only answer in JSON using '{' as the first character.",
        },
        {
            "role": "user",
            "content": TRANSCRIPT,
        },
    ],
    model=MODEL,
    response_format={
        "type": "json_object",
    },
)
output = json.loads(extract.choices[0].message.content)
print(json.dumps(output, indent=2))

Output example:

{
  "current_time": "6:30 PM",
  "tasks": [
    {
      "task": "water the plants in the garden",
      "priority": "high"
    },
    {
      "task": "prepare dinner (pasta with garlic bread)",
      "priority": "high"
    },
    {
      "task": "catch up on phone calls",
      "priority": "medium"
    }
  ]
}

ConclusionLink to this anchor

Using structured outputs with LLMs can significantly improve their reliability, especially to implement agentic use cases.

Structured outputs provide strict adherence to a predefined schema, ensuring consistency.
JSON mode (Legacy Method) is flexible but less predictable.

We recommend using structured outputs (json_schema) for most use cases.