Naysay research
Evidence for AI-agent security evaluation.
Working paper v0.6 / Not peer reviewed
Auditing Agent Actions with Customer-Retained Evidence
Report Integrity, Outcome Verification, and Economic Design
Author: Howie Xu.
Abstract
An agent's report may omit or alter its tool actions. We investigate whether records retained independently by the customer improve detection of these discrepancies. A customer-operated HTTP gateway records tool requests and responses and produces a signed receipt for subsequent verification.
In a local simulated business environment, twelve agent workflows generated 27 tool calls. Nine workflows supported a comparison of four evidence sources. Model reviewers assessed authentic reports and deliberately altered versions in separate requests, without access to the alteration labels.
Study 1: Report integrity
| Evidence available to the reviewer | Detected |
|---|---|
| Final answer | 13 / 26 |
| Submitted logs | 12 / 26 |
| Verified gateway records | 26 / 26 |
| Independently retained ordinary logs | 24 / 26 |
The comparison contains 176 eligible reviewer judgments after excluding one ineffective alteration across all four interfaces. No authentic control was falsely rejected in 18 control judgments per interface; these include repeated inputs.
The results support further study of independently retained execution evidence. The small, correlated sample does not establish a general advantage for signatures over ordinary retained logs, production reliability, or a token's economic value.
Study 2: Action outcomes
A second implementation verifies bounded Git and SQLite updates through a customer-controlled HTTP service. It distinguishes whether a registered action occurred from whether its intended value still holds, including after a lost response, retry, rollback or later overwrite.
Across 26 authored fault scenarios, the service returned 18 correct execution decisions, 8 unresolved decisions and no incorrect decisions. All 26 current-state judgments matched the backend snapshots. Retained native logs using the same reconciliation algorithm produced identical results.
These are local engineering scenarios, separate from the model-reviewer pilot. The signed interface supports offline authentication of customer observations; investigation savings and customer adoption have not yet been measured. The replication files include both implementations and the complete results.
Proposed provider network
The revised paper proposes a provider network: customers fund bounded jobs in USDG, independent operators commit capital against defined obligations, and participants exercise specified protocol-control rights. It sets out a paid-operator pilot and comparisons with conventional contracts and stablecoin collateral.
Provider staking, penalties and shared governance are proposals, not deployed features. The technical results are unchanged; no paid provider-network experiment has been conducted.
Released .
Read the paper (PDF) / Download replication files (ZIP)
Try Naysay / Return to the homepage