Inference API quickstart

Make your first inference request

Gigatoken exposes OpenAI-compatible inference endpoints backed by provider devices serving catalog models. These public docs are readable without signing in.

Get an inference key

Generate inference keys from the Inference console API keys page. You must sign in to create or revoke keys.

Copy the key when it is generated. Gigatoken shows the full key value only once, at generation time, and later screens show key metadata instead of the secret.

Base URL and authentication

Use the production inference API base URL https://coordinator.gigatoken.baremetallabs.ai/v1. The inference API is OpenAI-compatible, so OpenAI client SDKs can point at this base URL.

Send your inference key as an HTTP bearer token: Authorization: Bearer $GIGATOKEN_INFERENCE_KEY. Only inference keys authorize inference requests. Provider keys are rejected by inference endpoints.

export GIGATOKEN_BASE_URL="https://coordinator.gigatoken.baremetallabs.ai/v1"
export GIGATOKEN_INFERENCE_KEY="gt_your_inference_key"
export GIGATOKEN_MODEL="qwen/qwen3.5-0.8b"

Anthropic SDK clients

Gigatoken also serves POST /v1/messages and POST /v1/messages/count_tokens for Anthropic-compatible clients. Use the official Anthropic SDK with a coordinator base URL override, an inference key, and exact Gigatoken catalog ids.

See Connect Anthropic SDK for the end-to-end SDK guide, including streaming, tool use, extended thinking, image input, token counting, and current limitations.

Discover available models

The catalog below is generated from the Gigatoken model catalog source of truth. The examples use qwen/qwen3.5-0.8b, a current catalog alias.

Served catalog models run with vendor-recommended sampling defaults sourced from their official model cards.

Catalog metadata records each model's full context length and per-tier VRAM requirements. Provider daemons auto-select the highest served context tier their detected VRAM supports for the active model.

For live availability, call GET /v1/models. That endpoint returns models currently routable on online provider devices only, so it can be empty or a subset of the catalog. Each returned model includes max_served_context_length, the largest currently servable context window across online devices for that model.

Alias	Image input	Context tiers	Min VRAM	License
google/gemma-4-31b-it	Yes	32k / 64k / 128k / 256k	36 GiB	Apache-2.0
google/gemma-4-26b-a4b-it	Yes	32k / 64k / 128k / 256k	28 GiB	Apache-2.0
google/gemma-4-26b-a4b-it-q4	Yes	32k / 64k / 128k / 256k	19 GiB	Apache-2.0
google/gemma-4-12b-it	Yes	32k / 64k / 128k / 256k	9 GiB	Apache-2.0
qwen/qwen3.5-0.8b	No	32k / 64k / 128k / 256k	4 GiB	Apache-2.0
qwen/qwen3.6-27b	Yes	32k / 64k / 128k / 256k	31 GiB	Apache-2.0
qwen/qwen3.6-35b-a3b	Yes	32k / 64k / 128k / 256k	37 GiB	Apache-2.0
qwen/qwen3.6-35b-a3b-q4	Yes	32k / 64k / 128k / 256k	23 GiB	Apache-2.0

curl "$GIGATOKEN_BASE_URL/models" \
  -H "Authorization: Bearer $GIGATOKEN_INFERENCE_KEY"

{
  "object": "list",
  "data": [
    {
      "id": "google/gemma-4-31b-it",
      "object": "model",
      "max_served_context_length": 131072
    }
  ]
}

Reasoning effort

Reasoning is a per-request choice. Omit the field or send null to use medium on reasoning-capable models, pass none to disable reasoning for that request, and expect non-reasoning models to resolve to zero reasoning tokens when their catalog rung is zero. Allowed lowercase values are none, low, medium, high, and xhigh. Unknown, mixed-case, empty, or non-string values are rejected with HTTP 400 before a request is routed to a provider device.

For qwen/qwen3.5-0.8b, the starter ladder is 0, 2,048, 4,096, 8,192, and 16,384 reasoning tokens. max_output_tokens caps visible output for Responses, and gigatoken prompt --max-tokens caps visible output for the CLI prompt surface. Gigatoken adds the selected reasoning-token budget when it asks the provider runtime to generate.

CLI prompt

gigatoken prompt "Compare local and hosted inference." --reasoning-effort high

Responses API

{
  "model": "qwen/qwen3.5-0.8b",
  "input": "Write one sentence about local inference.",
  "reasoning": {
    "effort": "high"
  },
  "max_output_tokens": 512
}

Responses API

Start with the OpenAI-compatible Responses API for new inference clients. Gigatoken accepts a prompt through input and returns the provider response from the selected model.

Responses requests may override individual sampler fields by including temperature, top_p, top_k, min_p, or presence_penalty.

Gigatoken does not support server-side Responses persistence or previous-response references. Do not set store: true, and omit previous_response_id or leave it null.

The SDK examples intentionally use raw-response methods because convenience accessors such as output_text may not parse Gigatoken's current Responses shape.

curl

curl "$GIGATOKEN_BASE_URL/responses" \
  -H "Authorization: Bearer $GIGATOKEN_INFERENCE_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen/qwen3.5-0.8b",
    "input": "Write one sentence about local inference.",
    "reasoning": {
      "effort": "high"
    }
  }'

Python SDK

Install or update the OpenAI Python SDK in your application:

python -m pip install --upgrade openai

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["GIGATOKEN_INFERENCE_KEY"],
    base_url=os.environ.get(
        "GIGATOKEN_BASE_URL",
        "https://coordinator.gigatoken.baremetallabs.ai/v1",
    ),
)

response = client.responses.with_raw_response.create(
    model=os.environ.get("GIGATOKEN_MODEL", "qwen/qwen3.5-0.8b"),
    input="Write one sentence about local inference.",
    reasoning={"effort": "high"},
)

print(response.text)

JavaScript and TypeScript SDK

Install or update the OpenAI JavaScript/TypeScript SDK in your application:

npm install openai@latest

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.GIGATOKEN_INFERENCE_KEY,
  baseURL:
    process.env.GIGATOKEN_BASE_URL ??
    "https://coordinator.gigatoken.baremetallabs.ai/v1",
});

const response = await client.responses
  .create({
    model: process.env.GIGATOKEN_MODEL ?? "qwen/qwen3.5-0.8b",
    input: "Write one sentence about local inference.",
    reasoning: { effort: "high" },
  })
  .asResponse();

console.log(await response.text());

Image input

Responses image requests use a user message whose content array contains an input_image content part with an image_url field. The placeholder URL and base64 data URI below must be replaced with a real image before the snippets return useful results.

Use a vision-capable catalog alias such as google/gemma-4-26b-a4b-it-q4. Current image-capable aliases are:

google/gemma-4-31b-it
google/gemma-4-26b-a4b-it
google/gemma-4-26b-a4b-it-q4
google/gemma-4-12b-it
qwen/qwen3.6-27b
qwen/qwen3.6-35b-a3b
qwen/qwen3.6-35b-a3b-q4

Inline base64 image bytes count against the existing 16 MiB request body cap. Base64 encoding adds roughly 33% overhead, so images larger than about 12 MiB exceed the cap once encoded.

Remote image URLs do not add the image bytes to the request body. Oversized request bodies return HTTP 413 with error code responses_payload_too_large.

Sending image content to qwen/qwen3.5-0.8b, the current text-only catalog model, returns HTTP 400 with error code model_does_not_support_image_input on the Responses path.

Image input composes with the existing streaming, function calling, and routing patterns without extra image-specific fields.

Remote image URL

curl "$GIGATOKEN_BASE_URL/responses" \
  -H "Authorization: Bearer $GIGATOKEN_INFERENCE_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-26b-a4b-it-q4",
    "input": [
      {
        "type": "message",
        "role": "user",
        "content": [
          { "type": "input_text", "text": "Describe this image in one sentence." },
          { "type": "input_image", "image_url": "https://example.com/sample.jpg" }
        ]
      }
    ]
  }'

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["GIGATOKEN_INFERENCE_KEY"],
    base_url=os.environ.get(
        "GIGATOKEN_BASE_URL",
        "https://coordinator.gigatoken.baremetallabs.ai/v1",
    ),
)

response = client.responses.with_raw_response.create(
    model="google/gemma-4-26b-a4b-it-q4",
    input=[
        {
            "type": "message",
            "role": "user",
            "content": [
                {"type": "input_text", "text": "Describe this image in one sentence."},
                {"type": "input_image", "image_url": "https://example.com/sample.jpg"},
            ],
        }
    ],
)

print(response.text)

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.GIGATOKEN_INFERENCE_KEY,
  baseURL:
    process.env.GIGATOKEN_BASE_URL ??
    "https://coordinator.gigatoken.baremetallabs.ai/v1",
});

const response = await client.responses
  .create({
    model: "google/gemma-4-26b-a4b-it-q4",
    input: [
      {
        type: "message",
        role: "user",
        content: [
          { type: "input_text", text: "Describe this image in one sentence." },
          { type: "input_image", image_url: "https://example.com/sample.jpg" },
        ],
      },
    ],
  })
  .asResponse();

console.log(await response.text());

Inline base64 data URI

curl "$GIGATOKEN_BASE_URL/responses" \
  -H "Authorization: Bearer $GIGATOKEN_INFERENCE_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-26b-a4b-it-q4",
    "input": [
      {
        "type": "message",
        "role": "user",
        "content": [
          { "type": "input_text", "text": "Describe this image in one sentence." },
          { "type": "input_image", "image_url": "data:image/png;base64,iVBORw0KGgo..." }
        ]
      }
    ]
  }'

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["GIGATOKEN_INFERENCE_KEY"],
    base_url=os.environ.get(
        "GIGATOKEN_BASE_URL",
        "https://coordinator.gigatoken.baremetallabs.ai/v1",
    ),
)

response = client.responses.with_raw_response.create(
    model="google/gemma-4-26b-a4b-it-q4",
    input=[
        {
            "type": "message",
            "role": "user",
            "content": [
                {"type": "input_text", "text": "Describe this image in one sentence."},
                {
                    "type": "input_image",
                    "image_url": "data:image/png;base64,iVBORw0KGgo...",
                },
            ],
        }
    ],
)

print(response.text)

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.GIGATOKEN_INFERENCE_KEY,
  baseURL:
    process.env.GIGATOKEN_BASE_URL ??
    "https://coordinator.gigatoken.baremetallabs.ai/v1",
});

const response = await client.responses
  .create({
    model: "google/gemma-4-26b-a4b-it-q4",
    input: [
      {
        type: "message",
        role: "user",
        content: [
          { type: "input_text", text: "Describe this image in one sentence." },
          {
            type: "input_image",
            image_url: "data:image/png;base64,iVBORw0KGgo...",
          },
        ],
      },
    ],
  })
  .asResponse();

console.log(await response.text());

Function Calling

Send function tools with the request, execute the requested function in your client, then send the full transcript back with a matching function_call_output.

curl "$GIGATOKEN_BASE_URL/responses" \
  -H "Authorization: Bearer $GIGATOKEN_INFERENCE_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen/qwen3.5-0.8b",
    "input": "What is the weather in Boston?",
    "tools": [
      {
        "type": "function",
        "name": "get_weather",
        "description": "Get current weather for a city.",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {
              "type": "string"
            }
          },
          "required": ["location"],
          "additionalProperties": false
        },
        "strict": true
      }
    ],
    "tool_choice": "auto"
  }'

{
  "output": [
    {
      "type": "function_call",
      "id": "fc_weather",
      "call_id": "call_weather",
      "name": "get_weather",
      "arguments": "{\"location\":\"Boston\"}",
      "status": "completed"
    }
  ]
}

curl "$GIGATOKEN_BASE_URL/responses" \
  -H "Authorization: Bearer $GIGATOKEN_INFERENCE_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen/qwen3.5-0.8b",
    "input": [
      {
        "type": "message",
        "role": "user",
        "content": "What is the weather in Boston?"
      },
      {
        "type": "function_call",
        "id": "fc_weather",
        "call_id": "call_weather",
        "name": "get_weather",
        "arguments": "{\"location\":\"Boston\"}",
        "status": "completed"
      },
      {
        "type": "function_call_output",
        "call_id": "call_weather",
        "output": "{\"temperature\":\"72 F\",\"conditions\":\"sunny\"}"
      }
    ]
  }'

Streaming Responses

Set stream: true to receive server-sent events as the model generates output.

curl -N "$GIGATOKEN_BASE_URL/responses" \
  -H "Authorization: Bearer $GIGATOKEN_INFERENCE_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen/qwen3.5-0.8b",
    "input": "Write one sentence about local inference.",
    "stream": true
  }'

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["GIGATOKEN_INFERENCE_KEY"],
    base_url=os.environ.get(
        "GIGATOKEN_BASE_URL",
        "https://coordinator.gigatoken.baremetallabs.ai/v1",
    ),
)

with client.responses.with_streaming_response.create(
    model=os.environ.get("GIGATOKEN_MODEL", "qwen/qwen3.5-0.8b"),
    input="Write one sentence about local inference.",
    stream=True,
) as response:
    for line in response.iter_lines():
        if line:
            print(line)

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.GIGATOKEN_INFERENCE_KEY,
  baseURL:
    process.env.GIGATOKEN_BASE_URL ??
    "https://coordinator.gigatoken.baremetallabs.ai/v1",
});

const response = await client.responses
  .create({
    model: process.env.GIGATOKEN_MODEL ?? "qwen/qwen3.5-0.8b",
    input: "Write one sentence about local inference.",
    stream: true,
  })
  .asResponse();

if (!response.body) {
  throw new Error("Missing response body");
}

const decoder = new TextDecoder();
for await (const chunk of response.body) {
  process.stdout.write(decoder.decode(chunk, { stream: true }));
}

Optional routing fields

Default routing omits routing fields and lets Gigatoken select an eligible online provider device for the requested model.

provider_id scopes Responses routing to a specific provider owner pool.
target_device_id is currently supported on Responses requests only, pins routing to one device, and requires provider_id.

Troubleshooting

These are the common first-call failures and how to resolve them.

Status	Cause	Resolution
401	The request has no bearer token, a malformed key, a revoked key, or an unknown key.	Send `Authorization: Bearer ...` with an active inference key.
403	A provider key was used. Provider keys manage provider workflows and are rejected for inference requests.	Generate or use an inference key from /inference/keys.
400 / 404	The requested model alias is not in the catalog. Responses returns HTTP 400 for unknown aliases; Anthropic Messages returns HTTP 404 `not_found_error` for unknown catalog ids and Claude model names.	Use an exact catalog alias, or call `GET /v1/models` and choose one of the live aliases returned.
400	Image content was sent to a catalog model that does not accept image input. Responses uses error code `model_does_not_support_image_input`; Anthropic Messages returns HTTP 400 `invalid_request_error` with a message indicating the target model does not support image input.	Pick a vision-capable catalog alias from the image-input list/table above. After choosing a vision-capable alias, call `GET /v1/models` only to confirm that alias is currently routable.
503 / 529	The alias is valid, but no routable online provider device is currently serving it. Responses returns HTTP 503; Anthropic endpoints return HTTP 529 `overloaded_error`.	Retry later, or choose a model returned by `GET /v1/models`.