Everything Delivered. No One Can Fully Process It.

At HistoriaMP, the central question is shifting.

For a long time, the focus was: how do you get from a manuscript image to a justified, verifiable reading without smoothing the visible finding into fluent language?

That question is not fully solved, but it has become methodically tractable. At the same time, a second question emerges:

What happens to everything that is produced during the analysis?

The problem is not that too little is produced. On the contrary. It is precisely the volume and variety of the results that reveals how far from self-evident their handover actually is.

More than a text

A source-bound manuscript analysis does not consist only of an image and a transcription.

Between the source and an accepted reading lie numerous layers: image versions, layout areas, segments, visible traces, sign candidates, abbreviation findings, alternative readings, uncertainties, human decisions and technical processing steps.

These layers must not be mixed together.

A visible trace is not yet a sign. A sign candidate is not yet a reading. A linguistically plausible reading is not visual evidence. And a technical log is not a scholarly justification.

This separation produces data that a classic edition often does not preserve at all: the image crop actually processed, the position of a segment, discarded reading proposals, uncertainty assessments, human review decisions, tool and model versions, or earlier states of an analysis object.

This is intentional. It is a core part of traceability.

At the end of an analysis, therefore, there is not just one result, but a web of interconnected results.

The results multiply - and can contradict each other

The scale becomes visible already at an early stage: layout detection.

If the same manuscript page is examined with several established layout tools, the results do not necessarily agree. One tool might detect a few dozen regions and lines, another significantly more. A third divides the same page differently again.

None of these results has to be fundamentally wrong. Different procedures can capture and segment the same visible structure differently.

So even at this stage there is no single, unambiguous finding. Competing results emerge that must be documented, compared and assessed.

With every additional analysis layer, this web grows. A single page turns into numerous states, variants, dependencies and references.

What matters is the status

The sheer amount of data is not the actual problem.

What matters is that every piece of information keeps its scholarly status.

For every statement, it must remain recognizable:

• Is it an observation on the image?
• Is it one candidate among several?
• Is it uncertain, and what does that uncertainty refer to?
• Has it already been reviewed?
• Was it discarded, and for what reason?
• Was it confirmed, by whom and on what basis?

A discarded candidate is not a faulty record that could simply be deleted. Its rejection is part of the documented decision. It shows which possibility was examined and why it did not enter the reading.

Uncertainty is not generic either. Uncertainty about the visible letter body differs from uncertainty about resolving an abbreviation. A confidence value remains nearly meaningless unless it is known whether it came from a model, a scholarly reviewer, or a later-formed consensus.

This status is an essential part of the scholarly statement.

It is what distinguishes a traceable analysis from a smoothed transcription.

And it is exactly this status that risks getting lost in the handover.

It is not that standards are missing

One might assume that established procedures for exchanging such information already exist. In part, that is true.

For digital editions, TEI is the obvious standard. It can record that a sign is unclear, that several readings exist, or that a transcription refers to a specific image area. For the published reading and its editorially relevant uncertainties, TEI is capable.

IIIF provides standardized structures to make images, crops and annotations accessible across system boundaries.

PAGE-XML and METS, as used for example in the OCR-D environment, can technically organize regions, lines, files and processing states.

For provenance, PROV-O offers a general model. It can describe that an artifact resulted from a specific processing activity applied to a concrete input.

So there is no fundamental lack of standards. Each of these standards was developed for a specific task and is proven within that scope.

The problem is that they describe different slices.

TEI represents an edition, but not necessarily the entire technical genesis process. IIIF provides images and annotations, but does not model a complete, multi-stage analysis pipeline. PAGE-XML organizes layout and recognition results, but does not capture the source-critical argument from visible finding to a humanly accountable reading. PROV-O can represent derivations and activities, but without additional domain definitions it does not know the meaning of a glyph finding or an editorial rejection.

The gap, therefore, does not lie within any single standard.

It lies between the standards.

Transferring a file is not the same as understanding it

This can be shown with a simple scenario.

Suppose HistoriaMP output its results in formally correct form: the edition as TEI, the image references via IIIF, the genesis history as a provenance graph, and the layout as PAGE-XML.

Several established standards would then be used correctly. A scholarly institution could take over the entire package.

Yet decisive questions would remain open.

• Does the receiving software recognize whether an object represents an observation, a candidate, or an accepted reading?
• Does it understand why a discarded proposal has to be preserved?
• Can it distinguish what an uncertainty actually refers to?
• Does it recognize the difference between a machine assessment, a human decision and a later-formed consensus?
• Or does it simply see several formally linked files?

This is where the difference between syntactic and semantic interoperability lies.

Syntactic interoperability means that a system can technically parse a format.

Semantic interoperability requires that it also understands the meaning of the contained terms, roles and relationships.

A file can therefore be transferred technically flawlessly while part of its scholarly content is lost.

Two kinds of status

For a handover, a further distinction between two kinds of status is needed.

Scholarly or epistemic status describes what role a statement plays in the process of inquiry: observation, candidate, uncertainty, confirmation, rejection or accepted reading.

Technical or procedural status, by contrast, describes what happened to an artifact within the system: created, reviewed, versioned, replaced, reprocessed or failed.

Both levels are relevant. But they are not the same thing.

A failed processing step can be technically significant without itself containing a scholarly statement. Conversely, a discarded reading candidate can remain scholarly relevant even though it was not adopted in the further process.

Exactly this distinction has to survive a handover.

Where HistoriaMP currently stands

For HistoriaMP, this is not an abstract question about the future. It is the point the project is currently arriving at.

The analysis produces artifact-based intermediate states. The path from source to finding is documented. Uncertainties stay visible. Candidates do not silently become readings. Earlier states and discarded possibilities can be preserved.

The basic result structures are therefore in place.

What is missing is a broadly supported exchange profile in which these results retain their status and their relationships.

It is not enough to provide a large number of files. As long as no widely used scholarly tool understands the full chain

Source
→ Image version
→ Segment
→ Visible finding
→ Candidate
→ Review
→ Decision
→ Technical provenance

as one coherent analysis, the data remain available, but only partly reusable.

This also changes the question of transparency.

Until now it was:

Can we disclose how a reading came about?

That question can, in principle, be answered yes.

The harder question is:

Can another scholarly institution actually continue working with this information?

No reason to document less

This should not lead to a call for less documentation.

The problem is not that HistoriaMP records too much detail. Precise documentation merely reveals how many observations, dependencies, uncertainties and decisions stand behind a seemingly simple transcription.

Reducing everything to the final reading would simplify the handover, but it would remove exactly the information that makes later review possible.

Nor would yet another all-encompassing project format automatically be a solution.

A proprietary schema could fully describe HistoriaMP's internal structure. But as long as other institutions have no software that understands that schema, it would only create a new project-specific dependency.

The data would be fully documented, but still not usable without adaptation.

The problem would not be solved. It would only be relocated.

Transparency needs a recipient

Discussions about scholarly transparency usually focus on the project that produces the data.

It is expected to disclose its methods, preserve intermediate steps, mark uncertainties and document technical dependencies.

That is necessary. But it is not enough.

Transparency only works fully once a recipient can actually make meaningful use of the information provided.

A file that can only be interpreted with substantial project-specific knowledge is formally open, yet practically restricted in access.

The same applies to a data package whose individual components are each standard-compliant, but which no widely used scholarly application understands as one coherent analysis.

The challenge, therefore, is not only to disclose results.

It is to ensure that their scholarly status and their relationships survive outside the original system.

Everything delivered - and still not fully usable

HistoriaMP can preserve and make accessible image versions, observations, candidates, decisions and technical processing steps. The analysis does not have to end as a closed black box.

But a second problem arises at the point of handover - one that disclosure alone does not solve.

The individual components can be described with existing standards. The data can be provided completely, traceably and in formally correct form.

What is missing is a broadly shared domain model for what kind of statement is being transferred, how the layers relate to each other, and how their status should be interpreted by other software.

Equally missing is a widely used tooling environment that can take over these relationships without project-specific adaptation.

HistoriaMP can therefore deliver what scholarship demands. External software, however, cannot readily capture and further process this information in its full meaning.

This is why research data can be fully available and still only partly reusable.

Everything has been delivered.

It still cannot be fully processed.