trickroom / Docs
Sign in
API reference/assistants

Run a suite against pinned assistant versions

Application-scoped assistant control plane. External pipelines, dataset ingestion, and training run in the builder's infrastructure. Eval suites run fixture tool-mode jobs or disposable offline Linux desktops through Temporal. Metrics/artifacts are externally reported; reference URLs are never fetched. Model credentials are write-only. Pull subscriptions replay unacknowledged events. Resource lists return up to 200 recent records; trigger lists return an ID next_cursor for after, and message and delivery feeds use sequence cursors.

POST/v1/applications/{id}/eval-runs

Authorization

Authorizationstringrequired

Bearer tak_… — an application API key. Keep it on your backend; all resources are scoped to that application.

Create an application key →

Path parameters

idstringrequired

Min length: 1

Body application/json

suite_idstringrequired

Min length: 1 · Max length: 200

harness_version_idstring

Min length: 1 · Max length: 200

assistant_version_idsarray<string>

Min items: 1 · Max items: 2

variantsarray<object>

Min items: 1 · Max items: 4

Item fields
labelstring

Min length: 1 · Max length: 60

assistant_version_idstringrequired

Min length: 1 · Max length: 200

harness_version_idstring

Min length: 1 · Max length: 200

modelstring

Min length: 1 · Max length: 200

reasoning_effortstring

Allowed: "none""minimal""low""medium""high""xhigh""max"

samplingobject
Object fields
temperaturenumber

Minimum: 0 · Maximum: 2

top_pnumber

Maximum: 1

seedinteger

Minimum: -9007199254740991 · Maximum: 9007199254740991

logprobsboolean
top_logprobsinteger

Minimum: 0 · Maximum: 20

return_token_idsboolean
splitstring

Pattern: ^[a-zA-Z0-9_.-]{1,100}$

task_keysarray<string>

Min items: 1 · Max items: 2000

repetitionsinteger

Default: 1 · Minimum: 1 · Maximum: 16

concurrencyinteger

Default: 1 · Minimum: 1 · Maximum: 8

case_timeout_secondsinteger

Default: 120 · Minimum: 5 · Maximum: 600

max_stepsinteger

Default: 20 · Minimum: 1 · Maximum: 100

max_cost_usdnumber

Maximum: 1000

labelstring

Min length: 1 · Max length: 100

Response

202Run a suite against pinned assistant versions
X-Request-Idstring

Server-generated request identifier for logs and support.

application/json

idstringrequired

Min length: 1

created_atstringrequired

Format: date-time · Pattern: ^(?:(?:\d\d[2468][048]|\d\d[13579][26]|\d\d0[48]|[02468][048]00|[13579][26]00)-02-29|\d{4}-(?:(?:0[13578]|1[02])-(?:0[1-9]|[12]\d|3[01])|(?:0[469]|11)-(?:0[1-9]|[12]\d|30)|(?:02)-(?:0[1-9]|1\d|2[0-8])))T(?:(?:[01]\d|2[0-3]):[0-5]\d:[0-5]\d(?:\.\d+)?(?:Z|([+-](?:[01]\d|2[0-3]):[0-5]\d)))$

application_idstringrequired

Min length: 1

suite_idstringrequired

Min length: 1

definition_jsonobjectrequired

Additional keys: any

result_jsonobject | nullrequired
rewardsobject
Object fields
variantsarray<object>required
Item fields
variant_indexintegerrequired

Minimum: -9007199254740991 · Maximum: 9007199254740991

samplesintegerrequired

Minimum: -9007199254740991 · Maximum: 9007199254740991

scoredintegerrequired

Minimum: -9007199254740991 · Maximum: 9007199254740991

unscoredintegerrequired

Minimum: -9007199254740991 · Maximum: 9007199254740991

meannumber | nullrequired
sdnumber | nullrequired
ci95array<any> | nullrequired
histogramarray<integer>required

Min items: 10 · Max items: 10

pass_at_kobjectrequired

Additional keys: number

scorersarray<object>required
Item fields
kindstringrequired
scorerstringrequired
meannumber | nullrequired
scoredintegerrequired

Minimum: -9007199254740991 · Maximum: 9007199254740991

unscoredintegerrequired

Minimum: -9007199254740991 · Maximum: 9007199254740991

Additional keys: any

Additional keys: any

comparisonsarray<object>required
Item fields
variant_indexintegerrequired

Minimum: -9007199254740991 · Maximum: 9007199254740991

baseline_indexintegerrequired

Minimum: -9007199254740991 · Maximum: 9007199254740991

pairedintegerrequired

Minimum: -9007199254740991 · Maximum: 9007199254740991

mean_differencenumber | nullrequired
ci95array<any> | nullrequired

Additional keys: any

Additional keys: any

4XX400 invalid input; 401 missing/revoked credentials; 403 disallowed operation/application; 404 missing or inaccessible resource; 409 conflicting submission/version/state; 413 body too large; 415 unsupported content type. Inspect error.code, not message text.
X-Request-Idstring

Server-generated request identifier for logs and support.

application/json

errorobjectrequired
Object fields
codestringrequired
messagestringrequired
request_idstringrequired

Additional keys: any

5XX500 unexpected failure or 503 unavailable dependency. A failed response does not prove that a submission was not committed. Retry job/flow admission with the same key and body.
X-Request-Idstring

Server-generated request identifier for logs and support.

application/json

errorobjectrequired
Object fields
codestringrequired
messagestringrequired
request_idstringrequired

Additional keys: any