← All papers

Ecaveo Working Papers, AI Adoption: The Higher-Order Issues

Knowledge With an Expiry Date

evidence, Evaluation and Decision-Making When the AI System Keeps Changing

Paul Forrest

Fractional Head of AI, Ecaveo

Published
December 2022
Pages
26
Topic
Evidence and evaluation
Type
Working paper

Abstract

Every claim an organisation makes about an AI system has a half-life, because the model, the data, the prompts, the interface and the users' own habits all move after the evidence is gathered, largely on the supplier's schedule and mostly out of the buyer's sight. A pilot can be accurate on the day it closes and misleading by the day the board reads it. This paper separates five sources of decay and observes that the two least visible, supplier updates and user adaptation, are precisely the two a short pilot cannot catch. It proposes a decay-rate model sorting claims into five tiers of durability, from system-specific findings that expire on any model update, through configuration and mechanism findings, to design and governance principles that outlast them. A four-part evidence record adapted from model cards and datasheets fixes what was observed, under what configuration, by whom, and when it expires. The shelf lives are offered as judgements for discussion rather than measured values, and the paper says so. Its claim is modest and structural: impermanence made visible is the precondition for governing it.

Read the paper

Download PDF 0.7 MB · 26 pages · no registration

This is the reading edition, complete and unabridged. The designed edition, with the full illustrated layout, is available from the original publication page.

Cite this paper

Forrest, P. (2022). Knowledge With an Expiry Date: evidence, Evaluation and Decision-Making When the AI System Keeps Changing. Ecaveo Working Papers, AI Adoption: The Higher-Order Issues. Ecaveo Services Ltd. https://papers.ecaveo.com/knowledge-with-an-expiry-date-evidence-when-the-ai-system-changes/

Download RIS Download BibTeX RIS opens in Zotero, Mendeley, EndNote and RefWorks

BibTeX, inline
@techreport{forrest2022knowledgewit,
  author      = {Forrest, Paul},
  title       = {Knowledge With an Expiry Date: evidence, Evaluation and Decision-Making When the AI System Keeps Changing},
  institution = {Ecaveo Services Ltd},
  series      = {Ecaveo Working Papers, AI Adoption: The Higher-Order Issues},
  year        = {2022},
  month       = {12},
  pages       = {26},
  url         = {https://papers.ecaveo.com/knowledge-with-an-expiry-date-evidence-when-the-ai-system-changes/}
}