RAG Queued Indexing Can Race With Source Replacement

Bug

Queued RAG jobs receive only a knowledge_source_id and later read metadata.indexing_payload from the knowledge_sources row.

KnowledgeSourceManager::register() updates an existing source row for the same logical source. If a user uploads/replaces the same source before the first queued chain runs, the second upload can overwrite the payload that the first chain was supposed to index.

Example:

1. Upload A writes payload A to source 123 and queues chain A.
2. Upload B replaces the same source, writes payload B to source 123, and queues chain B.
3. Chain A starts later, loads source 123, and reads payload B.

Impact

Older queued work can index the wrong version, duplicate indexing for the newest version, or overwrite status/errors for a newer upload. This is especially confusing once replacement, retry, deletion, and cleanup semantics are formalized.

Current Status

Still open. Recent cleanup fixed queue routing, embedding model/provider consistency, E2E tenant setup, and the addEntry() queued-work return value, but it did not change source replacement or indexing version semantics.

This was re-raised in the follow-up Copilot review as the mutable queued indexing payload/source replacement race.

Proposed Solution

Introduce explicit source versioning or indexing runs.

Recommended future shape:

• Keep knowledge_sources as the logical document/source.
• Add an immutable source indexing run record for each upload/re-index attempt.
• Store the content snapshot and selected settings on the run, not on the mutable source row.
• Pass indexing_run_id through the queue chain.
• Let jobs check that the run is still current before writing active chunks/status.
• Mark older runs as superseded when a newer upload replaces the source.

Simpler alternative:

• Create a new knowledge_sources row per upload/version.
• Mark previous sources for the same logical key as superseded/inactive.
• Ensure retrieval and graph views ignore superseded source rows by default.