InferenceRequestSchema

messages object[]required

Conversation history as an ordered array of messages. Each message has a role and content. For image-to-text models, content may be an array of text and image parts.

Possible values: >= 1

Array [

content object[]nullable

The message content. For image-to-text models, pass an array of content parts, each a text part or an image_url part (shown below). For plain text-only messages, you can also pass content as a single string.

Array [

typestringrequired

The content part type.

Possible values: [text, image_url]

textstring

The text of this part. Provide when type is text.

image_url object

The image for this part. Provide when type is image_url.

urlstringrequired

The image as a data URI (base64-encoded) or a publicly reachable URL.

detailstring

How much detail the model uses when processing the image.

Possible values: [auto, high, low]

Default: auto

]

rolestringrequired

The role of the message author: system, user, or assistant.

]

modelstringrequired

Model identifier. Use the model_name from List base models for base models, or the model_name from List owned models for fine-tuned or imported models.

Example: Llama-3.3-70B-Instruct

streamboolean

When true, returns the response as a stream of server-sent events, yielding tokens as they are generated. When false (default), waits for the full completion before returning.

Default: false

Example: false

stream_optionsobject

Options that control streaming behaviour, such as whether to include token usage in the final stream chunk.

Default: {}

max_tokensinteger

Maximum number of tokens to generate. Defaults to the model's maximum if omitted.

Possible values: >= 1 and <= 4095

temperaturenumber

Sampling temperature controlling randomness. Lower values produce more deterministic output; higher values increase variety.

Possible values: >= 0 and <= 2

Default: 1

top_pnumber

Nucleus sampling threshold. Only tokens comprising the top top_p probability mass are considered. Set to null to disable.

Possible values: >= 0 and <= 1

top_kinteger

Limits sampling to the top_k most probable tokens at each step. Set to -1 to disable.

Possible values: >= -1 and <= 200

Default: -1

presence_penaltynumber

Penalizes tokens that have already appeared in the output, encouraging topic variety.

Possible values: >= -2 and <= 2

Default: 0

repetition_penaltynumber

Multiplier applied to the logit of previously generated tokens to reduce repetition. Values above 1.0 penalize repeats; values below 1.0 encourage them.

Possible values: >= 0.01 and <= 2

Default: 1

InferenceRequestSchema

{
  "model": "Llama-3.3-70B-Instruct",
  "messages": [
    {
      "role": "user",
      "content": "Reply with exactly: OK"
    }
  ],
  "stream": false
}