Litigation over AI systems: what can genuinely be established
Two opposite and equally false claims circulate in this field: that a generative system is a black box about which nothing can be established, and that tools exist which can tell whether a text was written by a machine. It is worth setting out what can be demonstrated and what cannot, before a case is built on the wrong expectation.
A system is not a model
The first confusion to clear up is between the model and the system built around it. The model is the statistically opaque part: trained numerical weights whose behaviour cannot be deduced by inspection. The system that queries that model, however, is ordinary software, and as such can be inspected piece by piece:
- which model is queried and in which version;
- which system instructions are placed before the user's request — the part that determines behaviour more than people assume;
- which documents are retrieved and supplied to the model at request time, in retrieval-augmented architectures, and on what selection criteria;
- which external tools the system may invoke, and with what permissions;
- which controls sit on the output before it reaches the user;
- which parameters govern randomness in generation.
In most supply disputes the answer is here and not in the model: not in statistical behaviour, but in what the supplier built and in what it had said it would build. That is an examination of code and configuration — the most traditional of expert tasks.
Inference logs
A serious system keeps a record of interactions, because without one it cannot be operated or improved. Where they exist, those records allow every individual request to be reconstructed precisely: the text the user sent, the context added automatically, the response produced, the model and version used, any tools invoked, often cost and latency.
This is ordinary forensic material: it is acquired, its integrity is verified, and it is read like any other log. The problem is not technical but temporal: many suppliers keep such records for short windows, and retention is configurable. In a case about a generative output the first useful step is to have them frozen, before the questions are even drafted.
It should also be verified, not assumed, that the logs existed at the time of the events. The absence of logging in a system making consequential decisions is itself a finding, and when the standard of care is assessed it can weigh heavily.
Reproducibility: the delicate point
«Can you show that this system, given that question, answered in that way?» The honest answer is: one can show that it can answer that way, not that it necessarily did at the time.
There are two reasons, and both belong in the report.
The first is that generative models are non-deterministic by construction: for the same input the answer may vary, because generation includes a random component governed by parameters. Configurations exist that reduce or remove that variability, but they have to be set, and in production systems often are not.
The second is that the system changes over time, and in ways that are rarely versioned publicly: the supplier updates the model, system instructions get rewritten, the documents retrieved from the archive differ because the archive differs.
An asymmetry follows, and it is best put in writing: a successful reproduction is strong evidence, because it shows the disputed output falls within the system's behaviour. A failed reproduction proves far less than it appears to, because it may reflect an update in the meantime. Anyone presenting the second as proof that the output was never produced is forcing the conclusion.
Detectors of machine-written text are not evidence
Tools exist that promise to establish whether a text was written by a language model. They turn up in disciplinary and academic proceedings and occasionally in court. On this it is best to be blunt: they are not usable as evidence.
They work on statistical properties of text — how predictable the word choices are, how much sentence structure varies — that are not peculiar to machine generation. A plainly written technical text, a translation, a document drafted to a template, and above all a text written by someone whose first language it is not, display the same characteristics. Measured error rates are high and, more seriously, systematically stacked against exactly those groups: a false positive is not randomly distributed, it lands predictably on some people more than others.
An expert who founds a conclusion on these tools exposes the instructing party to an easy and well-grounded challenge. Metadata and technical marks are a different matter: some generation systems embed provenance information or watermarks in the files they produce, and the document a text was written in almost always retains a revision history showing whether it was typed progressively or pasted in as a block. Those are real circumstantial elements, acquired and weighed like any other data.
Training data
This is the hardest question and must be approached without illusions. Establishing from the outside whether a particular work was used to train a model is, in the general case, not technically ascertainable: the weights do not contain the documents, they have metabolised them. Research techniques exist that attempt to infer whether a sample belonged to the training set, but they yield probabilistic and fragile results that do not transfer into a courtroom as findings.
Two different, and often more useful, things can be established well. The first is the similarity of the output to the protected work: a comparison of texts or images by settled methods, which answers the legally relevant question in most cases. The second is the supplier's documentation: declared sources, data licensing agreements, internal records of corpus acquisition. In disputes between businesses the answer is almost always there rather than in the model.
On the data-protection side the reasoning is analogous: if the system returns information relating to an individual, that shows the information is in some form available to the system — whether through training or through retrieval at request time is itself a matter for examination, and the distinction is not a minor one.
What to ask for, in practice
First requests, in order of urgency
- Preservation of interaction logs for the relevant period, before rotation erases them.
- Production of the system configuration as at the date of the events: model and version, system instructions, generation parameters, output controls.
- Contractual and technical documentation of what was promised, including any agreed quality thresholds and how they were to be measured.
- Change records for the system over the period in question: who changed what, and when.
Questions framed around these objects produce verifiable answers. A question framed as «whether the artificial intelligence made a mistake» produces none.
Artificial intelligence systems: how I can help → · Back to insights