Understanding the Llama-3.1-70b-instruct model

Model overview

Model names

meta/llama-3.1-70b-instruct:fp8
meta/llama-3.1-70b-instruct:bf16

Compatible Instances

Model introduction

Released July 23, 2024, Meta’s Llama 3.1 is an iteration of the open-access Llama family. Llama 3.1 was designed to match the best proprietary models, outperform many of the available open source on common industry benchmarks.

Why is it useful?

It is great to see Meta continuing its commitment to open(weight) AI, and we are excited to fully support their mission with integration in the Scaleway ecosystem.

• Llama 3.1 was optimized for multilingual dialogue use cases, with many supported languages: English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai.
• Llama 3.1 brings a context window up to 128K tokens, a sharp increase compared to its predecessor (Llama 3 was 8192).
• Llama 3.1 supports tool calling, enabling the model to answer a given prompt using tool(s) it knows about, making it possible to interact with the outside world.

How to use it

Sending Managed Inference requests

To perform inference tasks with your Llama-3.1 deployed at Scaleway, use the following command:

curl -s \
-H "Authorization: Bearer <IAM API key>" \
-H "Content-Type: application/json" \
--request POST \
--url "https://<Deployment UUID>.ifr.fr-par.scaleway.com/v1/chat/completions" \
--data '{"model":"meta/llama-3.1-70b-instruct:fp8", "messages":[{"role": "user","content": "There is a llama in my garden, what should I do?"}], "max_tokens": 500, "temperature": 0.7, "stream": false}'

Make sure to replace <IAM API key> and <Deployment UUID> with your actual IAM API key and the Deployment UUID you are targeting.

Receiving Inference responses

Upon sending the HTTP request to the public or private endpoints exposed by the server, you will receive inference responses from the managed Managed Inference server. Process the output data according to your application’s needs. The response will contain the output generated by the LLM model based on the input provided in the request.