Understanding the Molmo-72b-0924 model

Model overview

Model name

allenai/molmo-72b-0924:fp8

Compatible Instances

Model introduction

Molmo 72B is the powerhouse of the Molmo family, multimodal models developed by the renowned research lab Allen Institute for AI. Vision-language models like Molmo can analyze an image and offer insights from visual content alongside text. This multimodal functionality creates new opportunities for applications that need both visual and textual comprehension.

Molmo is open-weight and distributed under the Apache 2.0 license. All artifacts (code, data set, evaluations) are also expected to be fully open-source. Its base model is Qwen2-72B (Twonyi Qianwen license).

Why is it useful?

• Molmo-72b allows you to process real world and high resolution images, unlocking capacities such as transcribing handwritten files or payment receipts, extracting information from graphs, captioning images, etc.
• This model achieves the highest academic benchmark scores and ranks second on human evaluation at the time of writing (September 2024)

How to use it

Sending inference requests

To perform inference tasks with your Molmo-72b model deployed at Scaleway, use the following command:

curl -s \
-H "Authorization: Bearer <IAM API key>" \
-H "Content-Type: application/json" \
--request POST \
--url "https://<Deployment UUID>.ifr.fr-par.scw.cloud/v1/chat/completions" \
--data '{
       "model": "allenai/molmo-72b-0924:fp8",
       "messages": [
        {
          "role": "user",
          "content": [
              {"type" : "text", "text": "Describe this image in detail please."},
              {"type": "image_url", "image_url": {"url": "https://picsum.photos/id/32/512/512"}}
          ]
        }
       ],
       "top_p": 1, 
       "temperature": 0.7, 
       "stream": false
}'

Make sure to replace <IAM API key> and <Deployment UUID> with your actual IAM API key and the Deployment UUID you are targeting.

Known limitations

• Molmo-72b was reported to struggle with transparent images. The official recommandation is to add white or dark background to images for the time being.
• Molmo-72b chat template does not support the system role. Ensure that the messages array is properly formatted with user role and assistant role only.
• Molmo-72b is not able to generate structured outputs (response_format parameter not supported).
• Molmo-72b cannot do function calling (tools parameter not supported).

Passing images to Molmo-72b

Image URLs

If the image is available online, you can just include the image URL in your request as demonstrated above. This approach is simple and does not require any encoding.

Base64 encoded image

Base64 encoding is a standard way to transform binary data, like images, into a text format, making it easier to transmit over the internet.

The following Python code sample shows you how to encode an image in base64 format and pass it to your request payload.

import base64
from io import BytesIO
from PIL import Image
def encode_image(img):
    buffered = BytesIO()
    img.save(buffered, format="JPEG")
    encoded_string = base64.b64encode(buffered.getvalue()).decode("utf-8")
    return encoded_string
img = Image.open("path_to_your_image.jpg")
base64_img = encode_image(img)
payload = {
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What is this image?"},
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/jpeg;base64,{base64_img}"},
                },
            ],
        }
    ],
    ... # other parameters
}

Receiving Managed Inference responses

Upon sending the HTTP request to the public or private endpoints exposed by the server, you will receive inference responses from the managed Managed Inference server. Process the output data according to your application’s needs. The response will contain the output generated by the visual language model based on the input provided in the request.

Frequently Asked Questions

What types of images are supported by Molmo-72b?

• Bitmap (or raster) image formats, meaning storing images as grids of individual pixels, are supported: PNG, JPEG, WEBP, and non-animated GIFs in particular.
• Vector image formats (SVG, PSD) are not supported.

Are other file types supported?

Only bitmaps can be analyzed by Molmo. PDFs and videos are not supported.

Is there a limit to the size of each image?

The only limitation is the context window (1 token for each 16x16 pixel).

What is the maximum amount of images per conversation?

One conversation can handle a maximum of 1 image (per request). Sending more than one image will return a 400 error.