CALEDAPT

17 Nov 2026

Caledapt Signal | 17 Nov 2026 - New frameworks improve LLM safety and local agent reliability

Two preprints this week attack the same weak point from opposite ends: a security benchmark that could not be reproduced, and a tool call that invents a value it was never given. Both now describe machinery you can run locally.

Correction Corrected 18 November 2026: every source link now points at the cited item rather than the publisher homepage, and each date is the item's own submission or release date. VAST was submitted on 14 November and System 1 on 16 November, not 17 November; the Ollama entry is a release candidate, not a stable release. No other editorial change.

Today's AI Signal

VAST compiles vulnerability scenarios into reproducible test ranges

What happened. The authors describe a compiler that turns vulnerability scenarios into multi-step evaluation environments, with a workflow-hardening layer kept separate from the scenario itself. In a budget-matched 270-episode contrast, three frontier models solved 12 of 90 instances when given only the scenario contract, and 76 of 90 when the full workflow protocol was supplied (cluster sign p at or below 0.00391). This is a preprint.

Why it mattersA security test is only useful if a failure can be reproduced and diagnosed. Environments that can be compiled and re-run make it possible to tell whether a model got worse, or whether the test changed underneath it.

What Caledapt recommends: Take one workflow you already test by hand and write the scenario as a declarative file, so the same run can be repeated after a model or prompt change.

Read the VAST preprint · Preprint · submitted 14 Nov 2026

System 1 agents choose tools without generating text

What happened. A deterministic compiler turns each tool's schema and the incoming request into multiple-choice questions, and a one-pass decision model answers them. On 1,952 requests to 157 biomedical tools, the best System 1 agent chose the right tool or correctly declined for 97.0% of requests, against 95.2% for the best of 18 local LLM-agent settings. None of its tool calls contained invented content, compared with 5.5 to 19.7 percent of LLM calls. This is a preprint, and the result is specific to that benchmark.

Why it mattersAn invented argument in a tool call propagates through every later step, which is exactly the failure that makes local agents risky around patient or client records.

What Caledapt recommends: Before connecting a local model to a record system, check whether the task needs free text at all: if the choice is between a few known tools and fields, a constrained decision step is easier to test than a generation step.

Read the System 1 preprint · Preprint · submitted 16 Nov 2026

The Future Build Lab

Intermediate

Local Data Validator

The problem: A monthly spreadsheet reaches a decision-maker with a missing reference and a total that does not add up, and nobody notices until the figures are already in a report.

The solution: A CLI that runs a small set of approved rules over a CSV, reports each failure with its row number, and exits non-zero without modifying the source file. The rules are drafted once, approved by a person, and then run deterministically.

Example input: A CSV of invoices where row 14 has no VAT number and row 27's total is 40.00 higher than its line items.

Expected output: Two lines naming row 14 and row 27, a non-zero exit code, and the original CSV unchanged.

Stack: Python or Rust, a CSV parser, and a rules file in JSON or TOML

Why it matters: It puts a deterministic check in front of a judgement call, so the person reviewing the figures spends their attention on the exceptions rather than on arithmetic.

First step: Write five rules for one real spreadsheet, run them against last month's file, and compare the output with the errors a person actually found.

Tools Worth Exploring

Ollama v0.40.17-rc1 Pre-release

A pre-release of Ollama's local runtime. The notes list a fix to ChatGPT CLI restore and graceful restart, and thinking metadata for OpenCode settings.

Why try it: Worth testing on a spare machine if you script the CLI, but it is a release candidate: keep your working install until the notes show a stable tag.

View the Ollama v0.40.17-rc1 release notes (pre-release) · v0.40.17-rc1 · Released 10 Nov 2026

langchain-openai 1.8.1

A patch release. Its changelog entry recovers function-call arguments from streamed done events, following the 1.8.0 release.

Why try it: If you stream tool calls through LangChain and occasionally see empty arguments, this is the fix to test against your own traffic.

View the langchain-openai 1.8.1 release notes · 1.8.1 · Released 17 Nov 2026

One Thing to Try Today

Ask for the retention statement in writing

Before a local model touches client records, ask the vendor or your own team where the data is stored, for how long, and who can read it. File the answer with the workflow it belongs to.

Want to make AI useful in your business?

Book a free 30-minute consultation with Caledapt.

Book a free consultation