Ecaveo Working Papers, AI Adoption: The Higher-Order Issues
Knowledge With an Expiry Date
evidence, Evaluation and Decision-Making When the AI System Keeps Changing
Fractional Head of AI, Ecaveo
Abstract
Every claim an organisation makes about an AI system has a half-life, because the model, the data, the prompts, the interface and the users' own habits all move after the evidence is gathered, largely on the supplier's schedule and mostly out of the buyer's sight. A pilot can be accurate on the day it closes and misleading by the day the board reads it. This paper separates five sources of decay and observes that the two least visible, supplier updates and user adaptation, are precisely the two a short pilot cannot catch. It proposes a decay-rate model sorting claims into five tiers of durability, from system-specific findings that expire on any model update, through configuration and mechanism findings, to design and governance principles that outlast them. A four-part evidence record adapted from model cards and datasheets fixes what was observed, under what configuration, by whom, and when it expires. The shelf lives are offered as judgements for discussion rather than measured values, and the paper says so. Its claim is modest and structural: impermanence made visible is the precondition for governing it.
Read the paper
This is the reading edition, complete and unabridged. The designed edition, with the full illustrated layout, is available from the original publication page.
Cite this paper
Download RIS Download BibTeX RIS opens in Zotero, Mendeley, EndNote and RefWorks
BibTeX, inline
@techreport{forrest2022knowledgewit,
author = {Forrest, Paul},
title = {Knowledge With an Expiry Date: evidence, Evaluation and Decision-Making When the AI System Keeps Changing},
institution = {Ecaveo Services Ltd},
series = {Ecaveo Working Papers, AI Adoption: The Higher-Order Issues},
year = {2022},
month = {12},
pages = {26},
url = {https://papers.ecaveo.com/knowledge-with-an-expiry-date-evidence-when-the-ai-system-changes/}
}