Endpoint sentinel

This grew out of The Instrument Moves. The gui-reasoning prompt below is that paper’s determinism probe, and the first two endpoints were its pinned endpoint and its control.

The same prompts, at temperature=0 with a fixed seed, sent to each endpoint every day in three windows eight hours apart, with the provider and quantization pinned and the served provider read back. Two probes: a reasoning prompt (gui-reasoning) on every endpoint, and a tool-calling prompt (gui-tool) that offers a small GUI tool schema and asks for one call, on five of them. Each panel is one endpoint-probe: the per-day median completion length with its interquartile band, the share of a day’s probes that repeated an earlier probe byte for byte, and where a single step in the series would explain the most variance.

Record closed. Probed 2026-09-04 to 2026-10-13, 12,553 probes; scheduled probing stopped 2026-10-22. From 10-08 17:53 UTC the router answered every request with HTTP 402 (credits exhausted), so the last answered days are 10-09/10 for the Kimi endpoints and 10-13 for the gpt-oss pair; those final days show as "unavailable" below for that reason, not because the endpoints went away. The closing analysis is in the Closing report.

Loading data…

Alerts

What the drift detector decided against each endpoint's previous week: availability, output length, latency. A day that shifts on three or more endpoints at once is attributed to the client or the router, not the endpoints.

Endpoints

Endpoints compared

All days in the selected range, per endpoint. Compare within one model only: different models produce different lengths by design. Identity rate is the share of a day's probes that repeat an earlier probe of that day byte for byte; 0 means every completion differed.

Closing report

Six weeks, fourteen endpoints, two probes, 7,764 answered requests. What the record showed, in order of weight:

1. The audit’s endpoint moved again, with both edges caught. AtlasCloud int4 Kimi-K2.5, the endpoint whose mid-run change the paper documents, stepped on 2026-09-20: median completion 1,650–1,800 → 1,414 tokens, reasoning 1,500–1,700 → 1,254, latency 41–46 s → 28–30 s, and a single never-before-seen completion became the modal answer, returned 5–12 times out of 24 on each of the next eighteen days. On 10-08/09 it went back: median 1,640–1,750, latency 46 s, modal completion gone. Shorter, faster and more repetitive at once, for eighteen days, then reverted, on a model id that never changed. The detector flagged the first day; a rolling seven-day baseline then absorbed the new level, which is the case such a baseline is worst at. The per-day panel above shows it plainly.
2. Endpoints disappear and do not come back. Three of the eight providers serving Kimi-K2.5 on 09-04 delisted it within thirteen days (DeepInfra 09-08, DigitalOcean 09-09, Phala 09-17); none returned in the following five weeks. The paper’s control endpoint stopped existing eleven days after submission.
3. Determinism at temperature 0 is a state, not a trait. DeepInfra’s bf16 gpt-oss-120b returned one completion to more than half its probes on 9 of 37 days and 18–24 distinct completions on the others; its modal completions recur across weeks, so the weights did not change. The fp4 sibling at CoreWeave never repeated a completion in 47 days. Kimi thinking endpoints repeated 0–13% of the time, except AtlasCloud inside its 09-20 regime.
4. Decisions are nondeterministic, not just wording. On the tool probe the first action never settled: gpt-oss at bf16 chose swipe 81% of the time and was unanimous on 10 of 32 days; at fp4 72%, unanimous 3 of 31; AtlasCloud Kimi-K2.5 split open_app 55% / tap 45%, unanimous 1 day of 28. Same prompt, same tools, temperature 0. This is the mechanism behind flipped verdicts in agent benchmarks, seen at step one.
5. Same model id, different product. Across providers labelled the same quantization, Kimi-K2.5’s reasoning length ranged 1.4k–2.7k tokens and latency 3 s (Bedrock, which never thinks) to 130 s. Quantization labels explained none of it.
6. Throttling is its own event. DeepInfra’s Kimi-K2.6 returned 429 to most of a day’s probes on six days, three of them consecutive (10-02 to 10-04), while the router listed it throughout.
7. Latency belongs to the client, the endpoint and the hour together. Moving the client from a laptop to GitHub runners shifted medians ±20–30% in both directions; within a day the three windows differed by up to 70%. It was demoted to recorded-not-alarmed on 09-29, which the significant-day lists vindicate.
8. The detector did not cry wolf. 19,168 shuffled tests tripped the per-test level 0.0–0.3% of the time; every real alarm maps to an event in the list below.

What the record cannot say: six weeks on one router is a set of observations, not rates; the tool-probe identity figure in the table is an artefact (tool-call rows carry no text); nothing here separates a provider’s own change from the router’s routing, though the served provider matched the pin on every answered probe. Method, raw record and the full report: the repository is linked from the paper page.

Events

Serving-side changes the sentinel has caught. Each gives the last probe that succeeded and the first that failed; until 2026-09-28 the sentinel probed once a day, so the window is up to 24 hours wide; from 09-29, eight.

• 2026-09-07 → 09-08: DeepInfra stopped serving moonshotai/kimi-k2.5 (fp4). Last success 09-07 01:01 UTC; from 09-08 00:34 UTC every pinned request returns 404, “no allowed providers are available”, and the model’s endpoint listing no longer includes DeepInfra. DeepInfra still serves kimi-k2.6 and gpt-oss-120b. This was the control endpoint of an agent-benchmark audit submitted eleven days earlier: a reproduction pinned to that paper’s provider and quantization can no longer be run.

• 2026-09-08 → 09-09: DigitalOcean stopped serving moonshotai/kimi-k2.5. Last success 09-08 01:32 UTC; 404 with the same text from 09-09 02:55 UTC. kimi-k2.5 went from 8 providers on the router to 6 in four days.

• Not events: 09-09 CoreWeave gpt-oss-120b, 23 of 24 probes rate-limited (429), fine the day before and after; 09-10 Novita kimi-k2.6, 4 local network failures; 09-08 to 09-10 latency rose on three endpoints a day at once, which the detector labels matrix-wide: those were the days this client’s requests hung.

• 2026-09-16 → 09-17: Phala stopped serving moonshotai/kimi-k2.5. Last success 09-16 05:02 UTC; 404 from 09-17. Three providers have dropped the model in ten days; the router lists five where it listed eight on 09-04.

• 2026-09-09 → 09-12: DeepInfra’s bf16 gpt-oss-120b collapsed to near-determinism, then left it. For four days 17–20 of 24 probes returned the identical completion (2,217 tokens, interquartile range zero); from 09-13 the same endpoint returned 18–23 distinct completions a day and the detector flagged the length shift. The modal completions of 09-15 and 09-16 are byte-for-byte the ones that topped 09-06 and 09-07, so the model did not change; how deterministic its serving is did. The fp4 sibling at CoreWeave returned 24 distinct completions every single day. Determinism at temperature=0 is a property of the endpoint’s state, not of the endpoint.

• 2026-09-20 → 10-08: AtlasCloud’s int4 Kimi-K2.5 ran in a different regime for eighteen days. Completion median 1,414 tokens (from 1,650–1,800), reasoning 1,254 (from 1,500–1,700), latency 28–30 s (from 41–46 s), and one new completion returned 5–12 times a day. Reverted 10-08/09. The first and last changes of a serving configuration, both inside the record.

• DeepInfra fp4 Kimi-K2.6 rate-limited on 09-12, 09-18, 09-20 and 10-02 → 10-04 (429 to most of a day’s probes; listed on the router throughout). Reported as rate-limited, not unavailable.

• 2026-10-08 17:53 UTC: credits exhausted. HTTP 402 on every request from then on; the record effectively ends 10-09/10 (Kimi) and 10-13 (gpt-oss). Not an endpoint event.

The dead endpoints stayed in the matrix to the end. A 0/24 day is the record that the endpoint is gone.

How to read it

• A step in the median means the endpoint’s behaviour changed. What changed on the serving side is not visible from outside. The screen reports where a single step would explain the most variance; that marks where to look for an identifiable event and is not evidence of one.
• Identity rate at temperature 0 is how far an endpoint is from deterministic. Two endpoints serving the same weights can differ from “never repeats” to “repeats half the time”. Since 2026-09-24 it is a tested metric: the day’s repeats against the previous week’s, Fisher’s exact test. An endpoint that starts or stops repeating itself has changed how it serves, even when length and latency hold. For tool probes identity means the same tool call with the same arguments.
• Latency is recorded, not alarmed. Over the first 25 days it raised 28 of the 36 significant tests and three of them lasted a second day: endpoints swing between throughput levels from one day to the next, which is load. Since 2026-09-29 length and identity decide an endpoint’s status, and the day’s probes are spread over three windows so that a day is not one hour.
• Errors are data. A day of timeouts is a finding about the endpoint.
• The record before 2026-09-07 is oversampled: a scheduling defect let up to four rounds overlap on 09-05 and 09-06. Per-probe values stand; day-level counts do not. 09-10 ran 47 hours on hung requests and rate limits and 09-11 was skipped; request deadlines, per-endpoint time budgets and per-endpoint parallelism date from 09-11.

Method, probe text, and the raw record are in the repository, linked from the paper page.