38 KiB
project_name, document_type, version, status, date, inputDocuments
| project_name | document_type | version | status | date | inputDocuments | |||
|---|---|---|---|---|---|---|---|---|
| Automatic PDF Renamer | epics-and-stories | 1.0 | draft | 2026-10-13 |
|
Automatic PDF Renamer — Epics and Stories
1. Overview
Product vision
Build a personal, fully offline application that watches a self-hosted Garage S3 bucket for new French text-based PDFs, extracts document data, classifies documents, applies configurable naming rules, and moves files into an organized bucket structure. A local dashboard lets the owner review and correct uncertain results. JSON sidecars are the source of truth; SQLite is a rebuildable query index.
Product principles
- Offline by design: no cloud APIs, telemetry, or runtime downloads. Documents remain on self-hosted infrastructure.
- No silent wrong renames: ambiguous, incomplete, or invalid results must be routed for human review.
- Recoverability: renames are traceable and reversible; sidecars are authoritative and SQLite can be rebuilt from them.
- Incremental delivery: first deliver deterministic extraction with regular expressions and configured lookup lists; add ML as a separate capability without replacing the stable ingestion pipeline.
- Configuration over code: adding a document template should not require changing application code.
- Single-user v1: Nextcloud handles user-facing file access and sharing. Dashboard authentication and multi-user management are outside the v1 scope unless explicitly re-approved.
Delivery strategy
- Deliver the technical foundation and durable metadata model.
- Deliver the ingestion and storage foundation, with classification outcomes delegated to local ML models.
- Add local ML inference for all extraction and classification decisions, then training and model lifecycle management.
- Deliver the dashboard for search, review, correction, validation, and template visibility.
- Complete consistency checks, recovery, deployment hardening, and operational verification.
2. Requirements Inventory
2.1 Functional requirements
The following inventory consolidates the requirements from the project brief and PRD. The original FR identifiers are retained where practical for traceability.
- FR-ING-01 — Polling and discovery: poll a configurable Garage S3
inbox/prefix at a configurable interval; consider only PDFs and ignore temporary, hidden, and lock files. - FR-ING-02 — Upload stability: process a PDF only after its size and ETag remain stable across consecutive listings.
- FR-ING-03 — Restart recovery: on startup, recover interrupted technical operations and discover eligible inbox objects that have not completed processing; reconcile
pending/without automatically reprocessing documents awaiting manual correction. - FR-TXT-01 — Text extraction: extract all available text from every page of a PDF using a local library; no OCR in v1.
- FR-TXT-02 — Empty text handling: route PDFs with empty or insufficient extracted text to human review with an explicit reason.
- FR-TXT-03 — Text persistence: persist extracted text in the sidecar and SQLite replica so normal review and training workflows do not need to re-parse the PDF.
- FR-CLS-01 — Ordered ML-assisted extraction: use local ML to extract emitting company, document type, and emission date; use another ML model to select additional-field groups from company and document type, then one shared token-level ML model to extract their values; independently predict template and period format from company and document type; finally calculate period and storage key deterministically. No predefined mapping selects these outcomes.
- FR-CLS-02 — Confidence and unknown class (post-v1): ML classification exposes confidence and routes results below a configurable threshold to review. The deterministic initial release uses explicit rule outcomes and does not generate artificial confidence scores.
- FR-FLD-01 — Common fields: every document has template, emission date, emitting company, document type, and period.
- FR-FLD-02 — Optional fields: templates define optional fields and reusable field groups with formats, allowed values, and extraction rules.
- FR-FLD-03 — Deterministic extraction: the light release uses configured regular expressions, date parsing, company gazetteers, and fuzzy matching where useful.
- FR-FLD-04 — ML-assisted extraction: the ML release can use trained models to help identify configured values where deterministic rules are insufficient.
- FR-FLD-05 — Period computation: derive period from emission date and template/company/document-type rules, including granularity, formatting, and offset.
- FR-FLD-06 — Validation: validate dates, companies, document types, periods, and optional field formats before applying a rename. Invalid or missing required values must be reviewed.
- FR-STO-01 — Garage storage: use a versioned Garage S3 bucket and the
inbox/,pending/, andfiles/key areas. - FR-STO-02 — Sidecar source of truth: store a JSON sidecar for each PDF with SHA-256 as its unique document identifier, current and original keys, status, extracted fields/text, confidence where applicable, timestamps, and event history. Any byte-level PDF content change creates a new identity; a rename/move preserving bytes does not.
- FR-STO-03 — SQLite replica: mirror sidecar data into SQLite for fast listing, filtering, search, and consistency diagnostics.
- FR-STO-04 — Safe moves: move objects using copy-before-delete, resolve destination collisions with an incremental suffix, and record the actual final key.
- FR-STO-05 — Operation audit trail: preserve sufficient history and key information to trace file moves and renames. A user-facing undo/rollback feature for file renames is out of scope.
- FR-DASH-01 — File list: list all known files and show emission date, period, current key/name, status, template, document type, company, and confidence where available.
- FR-DASH-02 — Search and filters: filter by status, template, document type, company, period, and date range; sort columns and search object names and metadata, and full extracted text across every PDF page.
- FR-DASH-04 — PDF detail and preview: stream the PDF from Garage through the backend and show it alongside extracted metadata.
- FR-DASH-05 — Correction and validation: allow the owner to change the template and field values, preview the resulting target key, and validate the final result.
- FR-DASH-06 — Lifecycle: support the business statuses
pending,processed, andvalidated; validated records are eligible for training. - FR-TPL-01 — Template configuration: define templates, fields, extraction and validation rules, period rules, destination prefixes, and key patterns in configuration.
- FR-TPL-02 — Configuration file editing: retrieve and update the JSONC configuration through dedicated endpoints; validate changes before activating them.
- FR-ML-01 — Training corpus: train from the initial labeled corpus and files validated by the owner.
- FR-ML-02 — Evaluation: show evaluation metrics, including accuracy and confusion matrix, before model activation.
- FR-ML-03 — Model lifecycle: version model artifacts, allow activation or discard, keep one active model, and activate without restarting the service.
- FR-REC-01 — Consistency scan: detect missing objects, missing sidecars, orphaned records, and sidecar/SQLite drift.
- FR-REC-02 — Rebuild: reconstruct SQLite from sidecars through both a CLI command and a dashboard/API action.
- FR-REC-03 — Repair and resume: expose anomalies and safe repair actions; re-scan pending work after restart.
- FR-OPS-01 — Local deployment: run the API, poller, dashboard, and background jobs in a container next to Garage and Nextcloud.
- FR-OPS-02 — Nextcloud compatibility: renamed/moved files remain visible at their new keys without reconfiguring the Nextcloud integration.
2.2 Non-functional requirements
- NFR-01 Privacy: no network calls except the local Garage bucket, local dashboard, and required local infrastructure. No runtime downloads or telemetry.
- NFR-02 Performance: process a standard PDF in under 5 seconds on the target server, including the S3 download. Dashboard list loads in under 1 second for 10,000 records; the PRD also calls out a 1,000-file acceptance check.
- NFR-03 Reliability: no silent invalid renames; ambiguous or invalid results are routed to review.
- NFR-04 Durability: bucket versioning is enabled; SQLite uses WAL; sidecar writes are atomic; crash recovery repairs replica drift; retries are idempotent and technical failures remain distinct from business statuses.
- NFR-05 Language: French documents in v1; the UI may use one locale, English or French.
- NFR-06 Deployment/configuration: containerized deployment, environment variables for secrets and endpoints, and a single documented template configuration format.
- NFR-07 Maintainability: Python/FastAPI, Garage-compatible S3 client, SQLite, React dashboard, local PDF text extraction, and scikit-learn for ML.
- NFR-08 Testing: unit tests plus targeted integration/end-to-end tests for Garage behavior, recovery, offline operation, and critical user journeys.
- NFR-09 Encryption: at-rest encryption is explicitly out of scope for v1; document the decision and revisit if the data sensitivity changes.
2.3 Additional requirements and planning decisions
- Existing correctly named documents provide the initial labeled corpus; v1 processes new incoming files only, not the historical backlog.
- OCR, scanned PDFs, multilingual documents, cloud services, and automatic periodic retraining are out of scope for v1.
- The three business statuses must not be overloaded with technical states. Transient technical failures use a separate retry/error state, with attempt count, next retry time, and error details. Retry up to 3 times after the initial attempt, with delays of 5 seconds, 30 seconds, and 2 minutes. After the final failed retry, stop processing that item without another retry or review workflow; only document/business issues requiring manual correction use
pending. - S3 object operations, sidecar updates, and SQLite writes do not form one distributed transaction. Each state transition must be recoverable and verifiable rather than relying on a claim of cross-system atomicity.
3. Functional Requirements Coverage Map
| Requirement group | Covered by epic(s) |
|---|---|
| FR-ING-01 to FR-ING-03: discovery, stability, restart scan | Epic 2; recovery aspects in Epic 5 |
| FR-TXT-01 to FR-TXT-03: text extraction and persistence | Epic 2 |
| FR-CLS-01 to FR-CLS-02: rules-first classification and ML confidence | Epics 2 and 3 |
| FR-FLD-01 to FR-FLD-06: fields, deterministic/ML extraction, periods, validation | Epics 2 and 3; review overrides in Epic 4 |
| FR-STO-01 to FR-STO-05: S3, sidecars, SQLite, safe and traceable moves | Epics 1 and 2; recovery in Epic 5 |
| FR-DASH-01 to FR-DASH-06: list, search, preview, correction, validation, lifecycle | Epic 4 |
| FR-TPL-01 to FR-TPL-02: templates and template visibility | Epics 2 and 4 |
| FR-ML-01 to FR-ML-03: training, evaluation, model lifecycle | Epic 3 |
| FR-REC-01 to FR-REC-03: consistency, rebuild, repair, resume | Epic 5 |
| FR-OPS-01 to FR-OPS-02: deployment and Nextcloud compatibility | Epics 1 and 5 |
| NFR-01, NFR-05 to NFR-09: privacy, language, configuration, maintainability, testing, encryption decision | All epics; verification primarily in Epic 5 |
| NFR-02 to NFR-04: performance, correctness, durability | Epics 1, 2, 4, and 5 |
4. Epic List
- Technical Foundation, Storage, and Metadata — establish the offline service, Garage access, sidecar schema, and rebuildable SQLite index.
- Rules-Based PDF Ingestion and Classification (Light Release) — deliver a complete deterministic pipeline using text extraction, regular expressions, configured lookup data, validation, and safe renaming; no ML.
- Machine Learning for PDF Ingestion — add local model training and inference for document classification and configured-value extraction, integrated with the existing pipeline.
- Dashboard, Human Review, and Configuration Editing — make results searchable and correctable, support PDF preview and validation, and retrieve/update the JSONC configuration file.
- Consistency, Recovery, and Operational Readiness — detect and repair drift, recover safely from interruptions, verify offline behavior and performance, and document deployment/operations.
Epic 1: Technical Foundation, Storage, and Metadata
Goal: Provide the reliable offline foundation required by all user-facing workflows. PDF objects and sidecars live in Garage; SQLite is a rebuildable replica, never the authority.
Requirements covered: FR-STO-01 to FR-STO-03, FR-OPS-01, NFR-01, NFR-04, NFR-06 to NFR-09.
Story 1.1: Run the service in a local container
As the owner, I want the application to run as a container alongside Garage and Nextcloud, So that I can deploy and operate the tool on my own infrastructure without cloud services.
Acceptance Criteria:
Given the required local configuration and Garage endpoint are available
When I start the application container
Then the FastAPI service starts and exposes its local API and static dashboard
And startup does not download models, packages, or other assets from the internet
And missing or invalid configuration is reported clearly without exposing secrets.
Story 1.2: Connect to Garage and define storage key conventions
As the application, I want a configured S3-compatible storage adapter and consistent key conventions, So that every ingestion and file operation uses the same Garage interface.
Acceptance Criteria:
Given valid Garage credentials, endpoint, and bucket configuration
When the application performs a storage health check
Then it can list, inspect, read, write, copy, and delete objects using the supported Garage API
And key helpers consistently recognize the inbox/, pending/, and files/ areas
And credentials are supplied through environment variables and are never written into sidecars or logs.
Story 1.3: Persist authoritative PDF sidecars
As the application, I want a versioned JSON sidecar for every managed PDF, So that processing decisions and file history survive restarts and can be audited.
Acceptance Criteria:
Given a PDF is first accepted for processing
When its metadata record is created
Then the sidecar records schema version, SHA-256, original key, current key, status, template and extracted fields when known, extracted text, timestamps, and event history
And subsequent state changes append a timestamped event and update the current state
And sidecar writes use a temporary key and a safe replacement procedure rather than exposing partially written JSON
And the sidecar remains the authoritative record when SQLite disagrees.
Story 1.4: Mirror sidecar data into SQLite
As the owner, I want sidecar metadata indexed in SQLite, So that dashboard queries remain fast without making SQLite authoritative.
Acceptance Criteria:
Given a sidecar is created or updated
When the replica synchronization completes
Then the corresponding SQLite record reflects its queryable metadata
And SQLite uses WAL mode and a versioned schema/migration mechanism
And a replica failure is logged and can be repaired from the sidecar without losing the authoritative record.
Story 1.5: Rebuild the SQLite index from sidecars
As the owner, I want to recreate SQLite from the bucket's sidecars, So that a damaged or deleted local database does not require reprocessing the PDF files.
Acceptance Criteria:
Given the SQLite database is missing or unusable and sidecars remain available in Garage
When the rebuild operation is run
Then SQLite is recreated from the sidecars, including statuses, fields, extracted text, and event-derived metadata available in the schema
And the rebuilt index supports the same core list and detail queries as before
And malformed or unreadable sidecars are reported rather than silently discarded.
Epic 2: Rules-Based PDF Ingestion and Classification (Light Release)
Goal: Deliver a complete end-to-end workflow for new PDFs using deterministic rules only. The application polls Garage, extracts text from every PDF page, fills fields with regexes and configured lookup data, validates the result, and safely renames or routes the file for review. This epic must not depend on a machine-learning model.
Requirements covered: FR-ING-01 to FR-ING-03, FR-TXT-01 to FR-TXT-03, deterministic part of FR-CLS-01, FR-FLD-01 to FR-FLD-03 and FR-FLD-05 to FR-FLD-06, FR-STO-04 to FR-STO-05, FR-TPL-01, FR-OPS-02, NFR-02 to NFR-04.
Story 2.1: Discover stable incoming PDF files
As the owner, I want the application to detect complete PDFs in the configured inbox, So that newly uploaded documents enter the pipeline automatically.
Acceptance Criteria:
Given a PDF is uploaded to the configured inbox/ prefix
When its size and ETag are unchanged across two consecutive listings
Then it becomes eligible for processing
And non-PDF files, hidden files, .part files, and lock files are ignored
And a file that is still being uploaded is not eligible for processing prematurely
And a poller error for one object does not stop discovery of other objects
And transient technical failures are retried up to 3 times after the initial attempt, with delays of 5 seconds, 30 seconds, and 2 minutes, and recorded separately from business status
And a technical failure alone never moves a document to pending/.
Story 2.2: Extract and persist full PDF text
As the application, I want to extract and persist all available text from every page of each eligible PDF, So that downstream classification and field extraction use the document's content.
Acceptance Criteria:
Given an eligible PDF contains extractable text
When the extraction step runs
Then the configured local PDF library extracts text from every page of the PDF
And the text is saved in the sidecar and mirrored to SQLite
And the application does not call an external OCR or document-processing service
And if the text is empty or below the configured minimum length, the file is routed to review with reason empty text.
Story 2.3: Define templates and deterministic document rules
As the owner, I want each template to declare its fields, extraction rules, validation rules, period rules, and destination key pattern, So that I can adapt document handling without changing application code.
Acceptance Criteria:
Given a valid template configuration is loaded
When the template registry starts or reloads
Then each template exposes its required common fields, optional fields, regex rules, company/document-type mappings, period rules, target prefix, and key pattern
And invalid configuration is rejected with a useful error and does not partially replace the active configuration
And template configuration contains no credentials or secrets
And the configuration format is the project-wide format selected in the Additional Requirements section.
Story 2.4: Extract common and optional fields using rules
As the application, I want to fill document fields using configured regular expressions and reference data, So that the light release can classify and name common document types without ML.
Acceptance Criteria:
Given full-document extracted text and a configured set of templates are available
When deterministic extraction runs
Then it extracts candidate values for emission date, emitting company, document type, and configured optional fields using the applicable regexes and lookup lists
And date parsing uses a local date parser and company matching uses the configured gazetteer, with fuzzy matching only where configured
And the selected template and all extracted candidates are recorded in metadata
And fields that cannot be extracted are left explicitly missing rather than guessed.
Story 2.5: Compute periods and validate the extracted data
As the application, I want to compute the configured period and validate all required fields before renaming, So that only internally consistent metadata can drive an automatic move.
Acceptance Criteria:
Given an emission date and the applicable company/document-type period rules are available
When period computation runs
Then the period is derived using the configured granularity, format, and offset
And the implementation supports the PRD granularities day, week, month, quarter, semester, and year, and offsets current, previous, and following
And validation checks date parseability and plausible range, recognized company, allowed document type, period consistency, and configured optional-field formats
And any missing or invalid required value routes the document to review with the failing fields and reasons recorded
And no invalid result is automatically renamed.
Story 2.6: Route files and apply safe, traceable renames
As the owner, I want valid documents to be renamed and moved automatically while uncertain documents wait for review, So that routine files are organized without risking silent data loss.
Acceptance Criteria:
Given all required fields are valid and the deterministic rules produce a target key
When the pipeline completes
Then the object is copied to the target key before the source is deleted
And destination collisions are resolved with an incremental suffix and the final key is recorded
And the sidecar records the original key, current key, extracted values, decision, status, and rename events
And the file receives status processed only after the required state updates have been persisted or recoverably recorded
And when confidence is not applicable in this rules-only release, the decision is based on explicit rule outcomes and field validation rather than a fabricated ML confidence score
And ambiguous, incomplete, or invalid files are moved to pending/ and receive status pending with a review reason
And the operation can be audited from its recorded history; no user-facing rename reversal is required.
Story 2.7: Make ingestion idempotent and restart-safe
As the owner, I want ingestion to resume safely after interruptions, So that retries do not create duplicate processing or lose track of a file.
Acceptance Criteria:
Given the service restarts while files exist in inbox/ or pending/
When the recovery scan runs
Then objects without a completed sidecar state are discovered and resumed or flagged for review
And objects already completed are not processed again in a way that creates duplicate renames or events
And if a crash occurs between copy and delete, the recovery logic detects both keys and resolves the state from the sidecar and object hashes
And transient technical failures are recorded separately from the three business statuses
And retryable failures are retried safely up to 3 times after the initial attempt, with delays of 5 seconds, 30 seconds, and 2 minutes
And after retries are exhausted, processing of that item stops without launching another retry or manual-review workflow
And a technical failure alone does not move the document to pending/.
Epic 3: Machine Learning for PDF Ingestion
Goal: Improve document classification and configured-value extraction with locally trained models, while retaining the deterministic pipeline and human-review safety net. ML must be an optional capability layered behind stable classification and extraction interfaces.
Requirements covered: ML extension of FR-CLS-01 and FR-CLS-02, FR-FLD-04, FR-ML-01 to FR-ML-03, FR-DASH-05 training feedback, NFR-01 to NFR-04, NFR-07 to NFR-08.
Story 3.1: Build a labeled training corpus from validated documents
As the owner, I want initial labeled examples and validated documents to form the training corpus, So that the model learns from known document types and my corrections.
Acceptance Criteria:
Given the initial labeled corpus and sidecars for validated files are available
When the training dataset is assembled
Then eligible samples are mapped to their validated template/document labels and relevant field labels
And unvalidated or inconsistent records are excluded unless explicitly selected through a documented policy
And duplicate content is detected by hash to avoid unintended train/test leakage
And the dataset build is reproducible and records which samples were used.
Story 3.2: Train and evaluate a local document classifier
As the owner, I want to train a local classifier and review its quality before using it, So that ML-based classification is measurable rather than blindly trusted.
Acceptance Criteria:
Given a valid training corpus contains enough labeled examples
When I trigger model training
Then a TF-IDF plus Logistic Regression classifier is trained locally
And evaluation metrics include accuracy and a confusion matrix using a held-out evaluation set where the dataset permits it
And the training run records timestamp, dataset/model version, metrics, and warnings about insufficient or imbalanced data
And training failure leaves the currently active model unchanged
And no runtime or training data is sent to external services.
Story 3.3: Use ML predictions during ingestion
As the owner, I want the ingestion pipeline to use the active classifier to propose a document template, So that documents whose layouts or wording vary can be recognized more reliably.
Acceptance Criteria:
Given an active model is available and text extraction succeeds
When a PDF enters classification
Then the classifier returns ranked template candidates and scores
And a configurable threshold determines whether the top prediction is eligible for automatic processing
And a low-confidence or unknown prediction is routed to human review with the candidates and scores stored in metadata
And the pipeline can fall back to deterministic rules when the model is unavailable or disabled
And downstream field validation and safe rename rules remain mandatory regardless of classifier confidence.
Story 3.4: Add ML-assisted extraction for configured field groups
As the owner, I want ML to help select values from configured field groups when regex extraction is insufficient, So that the application can resolve more document-specific values without hard-coded per-document code.
Acceptance Criteria:
Given a template maps a period format/emitting company/document type/additional fields to a predefined field group with allowed values
When ML-assisted extraction is enabled for that field
Then the extractor proposes a value from the configured candidate set and records the method and score where available
And regex and deterministic lookup results remain available as inputs or fallback paths
And values outside the configured schema are never accepted automatically
And low-confidence, contradictory, or invalid results are routed to review
And the selected value and its provenance are persisted in the sidecar.
Story 3.5: Version, activate, discard, and roll back models
As the owner, I want to control which trained model is active, So that model improvements can be adopted safely and reversed if quality regresses.
Acceptance Criteria:
Given a completed training run has produced a candidate model and metrics
When I activate the candidate
Then the model artifacts are stored in a versioned local registry and the active-model pointer is switched atomically without restarting the service
And only one model version is active at a time
And I can discard a candidate without affecting the active model
And I can restore a previous version
And each file's metadata records the model version used for its prediction when ML was enabled.
Epic 4: Dashboard, Human Review, and Configuration Editing
Goal: Give the owner a local interface to find documents, understand automated decisions, correct metadata, validate files, and inspect template configuration. The dashboard is the human safety net for both the rules-only and ML-enabled ingestion paths.
Requirements covered: FR-DASH-01 to FR-DASH-06, FR-TPL-01 to FR-TPL-02, review aspects of FR-FLD-06, FR-ML-01, NFR-02, NFR-05.
Story 4.1: Browse and filter all known documents
As the owner, I want a searchable list of all files the application has seen, So that I can find documents and understand their processing status.
Acceptance Criteria:
Given file records are present in SQLite
When I open the dashboard file list
Then each row shows emission date, period, current object name/key, status, template, document type, emitting company, and confidence where available
And I can filter by status, template, document type, company, period, and date range
And I can sort by supported columns and search object names, metadata fields, and the full extracted text across all pages
And status badges distinguish pending, processed, and validated
And the list is served from SQLite rather than performing a full S3 scan for every request.
Story 4.2: Inspect a PDF and its processing decision
As the owner, I want to open a file's detail page with its PDF and extracted metadata, So that I can understand and correct the system's decision.
Acceptance Criteria:
Given a file appears in the list
When I open its detail page
Then the page displays the PDF streamed from Garage through the backend alongside its extracted fields
And it displays the current status, metadata, extracted text, decision/review reason, and ML candidates/scores when available
And a missing or unavailable PDF is reported clearly without breaking the rest of the dashboard.
Story 4.3: Correct metadata and preview the target key
As the owner, I want to change a template and edit extracted fields before validation, So that the final file name and destination reflect the correct document data.
Acceptance Criteria:
Given I am viewing a document detail page
When I change the template
Then the page keeps the five common fields, reloads the optional fields for the selected template, and performs best-effort re-extraction without silently overwriting user edits
And the period is recomputed from the emission date and the new template rules and is read-only
And the target key and destination prefix update as I edit fields
And invalid or missing required fields are highlighted and prevent final validation until corrected or explicitly handled by a defined rule.
Story 4.4: Validate a file and preserve the correction history
As the owner, I want to confirm or correct a file's metadata and validate the result, So that the file is moved to its final location and my correction can improve future processing.
Acceptance Criteria:
Given I have reviewed a document and its required fields are valid
When I select Validate
Then the application applies the safe rename/move if the target key changed and sets the business status to validated
And if the key did not change, validation does not perform an unnecessary rename
And the corrected values and validation event are written to the sidecar before SQLite is synchronized
And the validated record becomes eligible for the ML training corpus
And the operation is idempotent and its history remains auditable.
Story 4.5: Retrieve and update the JSONC configuration file
As the owner, I want to retrieve and edit the JSONC configuration file from the dashboard, So that I can change templates and extraction rules without editing application code.
Acceptance Criteria:
Given the current configuration file is available
When I open the configuration editor
Then the dashboard retrieves the complete JSONC file through GET /config
And I can submit the complete modified file through PUT /config
And invalid JSONC or schema errors are reported without replacing the active configuration
And a valid file is saved atomically and the in-memory configuration is reloaded.
Epic 5: Consistency, Recovery, and Operational Readiness
Goal: Make the application diagnosable and safe to operate by detecting storage/index inconsistencies, providing recoverable repair paths, and verifying offline, performance, and deployment requirements.
Requirements covered: FR-REC-01 to FR-REC-03, FR-OPS-01 to FR-OPS-02, NFR-01 to NFR-04, NFR-06, NFR-08 to NFR-09.
Story 5.1: Scan for storage and metadata inconsistencies
As the owner, I want a consistency job to compare PDF objects, sidecars, and SQLite, So that missing or divergent records are detected before they cause silent errors.
Acceptance Criteria:
Given the bucket and SQLite contain managed records
When a consistency scan runs on schedule or on demand
Then it detects PDFs without sidecars, sidecars whose current object is missing, orphaned records, and sidecar/SQLite drift
And each anomaly is recorded with its type, affected key/record, detection time, and a suggested safe action
And the scan does not silently delete or overwrite data to make the stores appear consistent.
Story 5.2: Repair or rebuild inconsistent state safely
As the owner, I want safe repair actions for detected anomalies, So that I can restore a consistent index without risking document loss.
Acceptance Criteria:
Given consistency anomalies are available
When I request an available repair or a full SQLite rebuild
Then repairs follow the sidecar-as-source-of-truth rule and record an event or diagnostic result
And SQLite can be rebuilt from sidecars using both a CLI command and the dashboard/API
And ambiguous cases that cannot be repaired safely are flagged for manual investigation
And repair actions are repeatable and do not duplicate objects or overwrite unrelated files.
Story 5.3: Resume interrupted processing after restart
As the owner, I want the application to recover work interrupted by a crash, So that files are not stranded in an unknown state.
Acceptance Criteria:
Given the service starts with objects in inbox/ or pending/ and possibly incomplete prior operations
When the startup recovery scan runs
Then it identifies files without a completed processing state and resumes or flags them for review
And it detects copy-before-delete intermediate states using sidecar history and object identity
And it repairs SQLite drift from authoritative sidecars
And it never deletes the only known copy of a PDF as an unverified recovery shortcut.
5. Cross-Epic Definition of Done
A story is complete only when all applicable items below are satisfied:
- Acceptance criteria are implemented and verified with automated tests where practical.
- Errors and ambiguous outcomes are explicit; no invalid document is silently renamed. Transient technical failures are retried separately and never mislabeled as
pending. - Metadata changes follow the sidecar-first, SQLite-replica model and can be recovered.
- No external network dependency is introduced.
- Configuration, logs, and documentation are updated where behavior changes.
- The change does not break the rules-only pipeline when ML is unavailable or disabled.
- Any architecture or requirement discrepancy discovered during implementation is documented and resolved before the affected behavior is considered complete.
6. Release Boundaries
Release A — Rules-based light release
Includes Epics 1 and 2 plus the minimum dashboard capabilities from Epic 4 needed to inspect, correct, and validate pending files. It must provide a working end-to-end workflow without ML.
Release B — ML-assisted ingestion
Adds Epic 3 and its integration with the review workflow. ML is an enhancement, not a hard dependency for storage, review, or safe rename behavior.
Release C — Operationally complete v1
Completes the remaining Epic 4 capabilities and Epic 5, including consistency repair, rebuild, recovery, offline verification, and deployment documentation.
7. Open Decisions to Resolve Before the Affected Stories
- ML field extraction: specify which field groups use ML in v1 and what evaluation threshold is required before enabling the shared extractor. Classification and deterministic validation must remain usable independently.
New dashboard epic — Configurable list and bulk metadata editing
Story: Configure visible columns
As the owner, I want to choose which metadata columns appear on the home page, including configured additional fields, so that I can tailor the table to the task at hand.
Acceptance criteria:
- Default visible order is Status, Document (filename), Company, Document type, Period, Last modified.
- Users can add/remove available columns, including additional fields identified by stable field keys.
- Empty values are rendered consistently; hiding a column does not delete or change its data.
Story: Select documents
As the owner, I want to select rows or every document matching the current filters, including across pagination, so that I can perform a batch operation efficiently.
Acceptance criteria:
- Each row has a selection control; page selection and all-filtered selection are distinct.
- The UI displays the selected count and clearly communicates the selection scope.
- Filter changes cannot leave a misleading stale selection.
Story: Bulk edit metadata
As the owner, I want to change company, document type, and additional-field values for multiple selected documents in one operation, so that I can correct repeated classification mistakes efficiently.
Acceptance criteria:
- Each editable field defaults to unchanged; additional fields support set/replace and clear.
- A confirmation step summarizes the operation and affected count.
- The server validates each document using the single-document rules, recomputes derived period and target key, and safely handles rename collisions.
- Sidecars remain authoritative, SQLite is synchronized, and an event is recorded for every successful change.
- Partial failures are reported per document; successful documents are not silently reprocessed on retry.