Skip to content
Registered users
Theme

Backends

Connect the model servers convo writes and judges with, check they answer, and compare model prices before you pick one.

A backend is a model server convo calls: Ollama on this machine, Ollama Cloud, or any server that speaks the OpenAI chat API, such as vLLM, llama.cpp or LM Studio. Convo ships with two built-in backends, "Ollama Cloud" and "Ollama local". Backends is under Settings in the sidebar. Anyone can list and test backends; adding or changing one needs an admin.

Add an Ollama server on another machine

You have a GPU box on your network running Ollama, and you want convo to generate on it for free instead of paying for cloud tokens.

  1. Step 1.

    Open Backends and click "New backend". The form opens on the right. Nothing is saved until you press "Create".

    The Backends list with the New backend button highlighted
  2. Step 2.

    Fill in "Name" and "Slug", for example Ollama on the GPU box and gpu-box, leave "Kind" on ollama, and set "Base URL" to the server's address, for example http://192.168.1.40:11434. The slug is a short stable id in lowercase letters, digits and dashes.

    The New backend form with name, slug, kind and base URL filled in
  3. Step 3.

    Leave "API key" on "None", since a server on your own network usually has no key.

    The API key choices, with None selected
  4. Step 4.

    Check the limits. "Concurrency" is how many requests run at once, and "Timeout (ms)" is how long one request may take. Leave "Max context" blank to use the model's own.

    The Concurrency, Max context, Max stop sequences and Timeout fields
  5. Step 5.

    Click "Create". Saving calls no model and costs nothing.

    The Create button at the bottom of the form, highlighted
  6. Step 6.

    The new row's "Where" column says "LAN", for a server on your own network. Click "List models" on it to see what the server has. Its own models read "Local ยท free". Click "Set as default" on one to make it the backend's "Default model", or "Use for test" to try it.

    The model list for a backend, with Use for test and Set as default on each model
  7. Step 7.

    Click the arrow at the start of the row to open the test panel, then "Test connection". It sends one tiny request and shows "Connection ok" with the latency, the context size, the reply and the token counts. A cold local model can take up to a minute.

    An open test panel showing a successful test result

Your run configs can now pick this backend under "Backend" or "Judge backend". See Run configs.

Next storyCompare model prices before choosing one

Compare model prices before choosing one

You're setting up a run on Ollama Cloud and want to see what each model costs per token, and which ones convo recommends, before you choose.

  1. Step 1.

    On the "Ollama Cloud" row, click "List models". Each model shows "Cloud" or "Local" and its price per MTok (million tokens), input then output. Recommended models are starred, listed first, and say what they're recommended for and why.

    The model list on Ollama Cloud, with prices and recommended models at the top
  2. Step 2.

    Type in "Search models, capabilities or roles" to narrow the list, for example judge to see the models recommended for judging.

    The model list filtered by a search
  3. Step 3.

    The same prices show in the "Model" and "Judge model" lists on Run configs, so you can compare there while you edit a config.

    The Model field on a run config, highlighted
  4. Step 4.

    If you're an admin, click "Refresh prices" on the row. Convo reads Ollama's published price list and shows each model's stored and published price. New and changed prices start ticked, unless the stored price was typed in by hand. Only the rows you tick change, and only when you click the apply button, which counts them, for example "Apply 1 price".

    The price comparison drawer with one new model ticked and the stored prices unchanged
    Example prices. Your refresh shows Ollama's current published rates.

Before a run starts, the start form uses these prices for its cost estimate. See Runs.

Next storyConnect a server that needs an API key

Connect a server that needs an API key

You want to add an OpenAI-compatible server that needs a key, without the key ever showing up in the browser again.

  1. Step 1.

    Click "New backend" and set "Kind" to openai_compatible. Fill in "Name", "Slug", "Base URL" and "Default model".

    The New backend form with openai_compatible picked as the kind
  2. Step 2.

    Under "API key", choose "Paste key" and paste it into "Key". Convo stores it encrypted. If the key is set as a variable on the server instead, choose "Environment variable" and type the "Variable name"; the value never reaches the browser.

    The API key choices with Paste key selected and the Key field below
  3. Step 3.

    Click "Create". From now on the row's "Key" column shows only the last 4 characters.

    A backend row whose Key column shows a stored key by its last 4 characters
  4. Step 4.

    To change the key later, click the pencil ("Edit") on the row. The form shows which key is saved by its last 4 characters; leave "Key" blank to keep it, or paste a new one. Choosing "None" shows "Saving removes the stored key."

    The Edit form for a backend with a stored key

Deleting a backend stops every run config that uses it, and the built-in backends can't be deleted at all.

Next pageSegments