Inference

Inference serves a model behind a live, OpenAI-compatible HTTPS endpoint running on dedicated GPUs. Deploy a checkpoint from one of your fine-tuning jobs, or any supported open model from Hugging Face, and get a URL, an API key, and a production dashboard in minutes. The serving engine supports 160+ architectures (dense, Mixture-of-Experts, and vision-language) across four task shapes: chat, completion, embedding, and reranking.

What Is Inference

Training produces a model; inference runs that model so applications can send it requests. Each endpoint provisions its own GPU pod, downloads the weights, launches the inference server, and exposes an endpoint secured by a per-endpoint key. You are billed per second of GPU time only while the endpoint is running.

Deploying a Model

There are two ways to start serving a model. From Inference → New Inference, or by clicking the Deploy button next to any of your fine-tuned models on the Models page (which carries the source checkpoint into the wizard for you). The flow has two screens:

1. Choose a source. Pick from Your fine-tuned models (checkpoints, ordered best-first) or From Hugging Face (search by name, filter by size or type, optionally scoped to a connected HF account for gated models). The platform validates that the architecture is supported before it lets you continue and blocks anything the engine cannot serve, with the exact reason.
2. Configure and deploy. A summary card shows the resolved model facts: architecture, parameter count, dense vs MoE, quantization, and native context length. Every serving option is pre-filled with a sensible default. Review, adjust anything you like, and click Deploy. Your endpoint begins provisioning immediately.

After you deploy, the endpoint walks a status pipeline: Provisioning → Downloading weights → Loading model → Ready. Weight download shows a byte-accurate progress bar. Large models (tens to hundreds of GB) take longer, and the provisioning deadline scales with model size so a big model is never killed for being slow to start.

Serving Configuration

Every field has a working default. You can deploy without touching any of them; adjust only what you need.

Each model type maps to one OpenAI-compatible route:

GPU Sizing

The platform sizes GPUs for inference from three things: the weights (which must all be resident; for Mixture-of-Experts models that means every expert, not just the active ones), the KV-cache budget for your chosen context length and concurrency, and runtime overhead. As you change context length, quantization, or count, a live VRAM bar shows the weights / KV-cache / overhead split so you can see the fit.

• Minimum: the smallest GPU count that fits is enforced; counts below it are disabled with the reason.
• Recommended: a starred count with headroom for real concurrency; you can go higher, up to 8 GPUs in a single pod.
• Hardware gating: GPUs that cannot run the chosen quantization (for example fp8 on older cards) are shown disabled with an explanation.

Calling Your Endpoint

Every endpoint speaks the OpenAI API. Point any OpenAI SDK at your endpoint's Base URL, pass your inference key as the Bearer token, and use the endpoint id as the model name. The API key is shown once at creation (and again if you regenerate it), so copy it then.

curl https://api.usbios.ai/v1/chat/completions \
  -H "Authorization: Bearer <api_key_or_access_token>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-inference-id",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Or with the OpenAI Python SDK:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.usbios.ai/v1",
    api_key="sk-usf-your-inference-key",
)

resp = client.chat.completions.create(
    model="your-inference-id",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,          # streaming recommended for chat & completion
)
for chunk in resp:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

The API / Docs tab on the inference page renders these snippets pre-filled with your real Base URL, model name, and route in cURL, Python, and Node, including the streaming variant for chat and completion models.

• Requests without a valid Authorization: Bearer header are rejected, as are requests from a workspace with an unpaid balance.
• Streaming is recommended for chat and completion, since long non-streaming generations can time out at the proxy. Embeddings and reranks return in a single response.
• To call a different model, change the id in the request (or point base_url at another endpoint). The API shape is identical everywhere.

Monitoring & Metrics

The inference detail page has six tabs. The Overview tab shows cumulative-forever cards: total requests, generated and prompt tokens, cache-hit rate, average time-to-first-token, average end-to-end latency, aborted requests, uptime, and total spend. These totals never reset, even when the engine or pod restarts.

• Metrics: time-series graphs (requests/min, tokens/s in and out, TTFT and end-to-end latency percentiles, running and queued requests, cache-hit rate, GPU utilization and memory) with 1h / 24h / 7d / 30d windows.
• API / Docs: copy-paste OpenAI snippets, the Base URL, and key regeneration.
• Events: the full lifecycle timeline: created, downloading, ready, restarts, crashes with a classified cause, re-provisions, and stops.

Playground

The Playground tab lets you chat with (or prompt) a running endpoint directly in the dashboard, with streaming responses, without writing any code. It adapts to the model type (a chat box for chat models, a prompt box for completion models) and uses the inference key on your behalf. It is the fastest way to sanity-check a model right after it goes Ready.

Managing Inference

From the Settings tab you can stop, resume, restart, regenerate the API key, or delete an endpoint. An endpoint runs continuously until you stop it, your balance runs out, or it fails. It moves through these states:

Billing

Inference is billed per second of GPU time from the same wallet as training, at the GPU rate shown in the wizard. Every endpoint runs on dedicated GPUs and keeps running (and billing) until you stop it, so keep these points in mind:

• Booking gate: you need a balance of at least twice the hourly cost to create or resume an endpoint, and a one-hour deposit is held up front.
• Stop-on-zero: if your balance reaches zero, you get a short grace warning, then the endpoint moves to Paused (low balance) rather than running up a debt. Top up and resume.
• Resume re-downloads weights: every stop/resume provisions a fresh pod and downloads the weights again, which is billed GPU time. For very large models this warm-up can take a while; the resume dialog shows the estimate.