Re-Indexing A Source Can Leave Stale Chunks Active

Bug

When a source is re-indexed with changed or shorter content, the current chunk creation logic only upserts chunks that exist in the new payload. It does not remove, archive, or supersede chunks from a previous version that are no longer present.

Impact

Retrieval and entity extraction can continue to use stale chunks from an older version of the file. This becomes more likely once source replacement and manual re-indexing become first-class workflows.

Current Status

Still open. Recent cleanup changed success signaling and some wording from "chunks created" to "queued for indexing", but it did not remove, archive, supersede, or hide stale chunks.

Proposed Solution

Define source-scoped cleanup and version activation semantics before changing deletion behavior.

Recommended future shape:

• Treat each indexing attempt as a version/run.
• Mark old chunks as superseded when a newer run becomes active.
• Make retrieval ignore superseded chunks by default.
• Keep failed or superseded artifacts available for debugging until explicit cleanup/deletion is implemented.
• When document deletion is redesigned, delete or archive source-scoped chunks, vectors, attribution, and graph artifacts consistently.

Do not implement partial cleanup without also defining how source deletion, failed indexing, retry, and graph attribution cleanup should work.