Structured Outputs

In this section, we provide detailed guidance on working with structured outputs when using Tromero models. Structured outputs are essential for applications that require specific data formats, such as JSON responses, tool integrations, or regex pattern matching.

As an example, imagine you are using a model to extract user information from a call transcript. You ask the model to tell you name and age, but every time you call it, the format is different. Sometimes it says "Here it is. Name: Lola, Age: 23", and others it says: "Name is Lola, age is 23".

Structured outputs will help you standardize the output format, making it easier to process and extract the information you need consistently.

This section will cover the following inference parameters:

• output_format: Generate JSON outputs.
• guided_schema: Define the expected structure of the output.
• tools: Incorporate tool usage within model outputs.
• guided_regex: Apply regex patterns to enforce specific output formats.

JSON Outputs

Tromero models can be configured to generate outputs in JSON format, which is useful for applications that require structured data. These are the two keywords that can be passed at inference time to generate JSON outputs:

• output_format: Set this keyword to {"type":"json_object"} to specify that the model output should be structured as a JSON object, but does not enforce a specific structure.

# Generate JSON output
response = client.chat.completions.create(
    model="your-model-name",
    messages=[
        {"role": "system", "content": "You are an API that returns responses in JSON format."},
        {"role": "user", "content": "Please provide the weather report for New York in JSON format."}
    ],
    output_format={"type": "json_object"}
)
print(response['choices'][0]['message']['content'])

• guided_schema: This keyword accepts a JSON Schema that defines the expected structure of the output. The model will generate outputs that adhere to this schema.

A JSON Schema is a standarized format for specifying templates for JSON objects. You can find more information here.

# Generate JSON output with guided schema
response = client.chat.completions.create(
    model="your-model-name",
    messages=[
        {"role": "system", "content": "You are an API that returns responses in JSON format."},
        {"role": "user", "content": "Please provide the weather report for New York in JSON format."}
    ],
    guided_schema={
        "type": "object",
        "properties": {
            "location": {"type": "string", "description": "The location for the weather report"},
            "temperature": {"type": "number", "description": "The temperature in Celsius"},
            "conditions": {"type": "string", "description": "The weather conditions"}
        },
        "required": ["location", "temperature", "conditions"]
    }
)

Tool Usage

Another use case of structured outputs is tool usage. Tools can be passed as a list at inference time, and the model will select the appropriate tool for the context, and generate parameters to call the tool.

For example, say we had the following tools available:

tools = [
    {
        "type": "function",
        "function": {
            "name": "calculator",
            "description": "Function to calculate the result of a mathematical expression.",
            "parameters": {
                "expression": {"type": "string", "description": "The mathematical expression to evaluate"}
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Function to get the weather report for a specific location.",
            "parameters": {
                "location": {"type": "string", "description": "The location to get the weather report for"}
            }
        }
    }
]

If we now prompt the model for the solution to "83*57", and pass this list in the "tools" parameter, it would respond like so: [{"name": "calculator", "parameters": {"expression": "83*57"}}] (for example, Mistral models). Some models (such as Llama models) may not return a list and respond like so {"name": "calculator", "parameters": {"expression": "83*57"}}. The user can now easily extract the tool name and parameters to call the tool and get the result.

Certain models have been trained to accept the tool response back as input. This feature is not implemented yet, so contact our team to express your interest.

response = client.chat.completions.create(
    model="your-model-name",
    messages=[
        {"role": "system", "content": "You are an API that can perform specialized tasks."},
        {"role": "user", "content": "Calculate 83*57."}
    ],
    tools=tools
)
# response: [{"name": "calculator", "parameters": {"expression": "83*57"}}]

Regex Guidance

The structure of the output can also be enforced using regex patterns. For example, we could pass the regex patter "Michael Jackson was born on \d4-\d2-\d2" to ensure the model output adheres to the format "Michael Jackson was born on YYYY-MM-DD", and we can extract the date from the output, and easily parse it.

Make sure the pattern you pass is a valid regex pattern. You can test your regex patterns using online tools like regex101.

import re

date_pattern = r"\d{4}-\d{2}-\d{2}"

response = client.chat.completions.create(
    model="your-model-name",
    messages=[
        {"role": "system", "content": "Generate a date in the format YYYY-MM-DD."},
        {"role": "user", "content": "What is today's date?"}
    ],
    guided_regex=date_pattern
)

# Extract the date from the output
date = re.search(date_pattern, response['choices'][0]['message']['content']).group()
print(date)