diff --git a/automatic-pdf-renamer/brief.md b/automatic-pdf-renamer/brief.md index 8f2920b..efbad74 100644 --- a/automatic-pdf-renamer/brief.md +++ b/automatic-pdf-renamer/brief.md @@ -1,93 +1,99 @@ - # Project Brief — Automatic PDF Renamer +# Project Brief — Automatic PDF Renamer - ## 1. Vision +## 1. Vision - A personal, 100% offline tool that watches a folder on a server, automatically identifies the type of each incoming PDF (invoice, purchase order, contract, etc.) from its content, applies the corresponding naming template by extracting the fields (date, counterparty, reference, amount), and moves the file into an organized folder structure. Low-confidence decisions go through human review in a local web dashboard. Every operation is logged and reversible. +A personal, 100% offline tool that watches an S3 bucket hosted on a self-hosted Garage storage cluster, automatically identifies the type of each incoming PDF (invoice, purchase order, contract, etc.) from its content, applies the corresponding naming template by extracting the fields (date, counterparty, reference, amount), and moves the object into an organized S3 prefix structure. Users upload, download, and share files through Nextcloud, which mounts the bucket as external storage. Low-confidence decisions go through human review in a local web dashboard. Every operation is logged in JSON sidecar files (source of truth) and replicated to SQLite, and every rename is reversible. - ## 2. Problem and Context +## 2. Problem and Context - PDFs arrive continuously in a watched folder (emails, downloads, exports). Current naming is manual or inconsistent, making documents hard to find and archive. The user owns 50 to 500 files already correctly named according to 1 to 5 existing templates, which will serve as training and reference corpus. +PDFs arrive continuously in an inbox S3 prefix (dropped via Nextcloud by the user or by upload automations). Current naming is manual or inconsistent, making documents hard to find and archive. The user owns 50 to 500 files already correctly named according to 1 to 5 existing templates, which will serve as training and reference corpus. - ## 3. Objectives +## 3. Objectives - - Classify each incoming PDF against a known naming template, based on the file content (first-page text, header/footer), not the current filename. - - Fill the fields of the selected template (date, counterparty, reference, amount) from the content. - - Rename and move the file into an organized tree structure depending on the document type. - - High confidence: automatic processing. Low confidence or incomplete extraction: queued for human review in the dashboard. - - Log every decision (SQLite) with rollback capability and retraining from corrections. - - Run fully offline, with no cloud APIs whatsoever. +- Classify each incoming PDF against a known naming template, based on the file content (first-page text, header/footer), not the current object name. +- Fill the fields of the selected template (date, counterparty, reference, amount) from the content. +- Rename and move the object within the bucket into an organized prefix structure depending on the document type. +- High confidence: automatic processing. Low confidence or incomplete extraction: queued for human review in the dashboard. +- Write metadata as JSON sidecar objects in the bucket (source of truth) and replicate every decision into SQLite for fast queries, with rollback capability and retraining from corrections. +- Run fully offline on self-hosted infrastructure: no cloud APIs, no data ever leaves the server. - ## 4. Scope +## 4. Scope - ### In scope (v1) +### In scope (v1) - - Folder watching (watcher) for new files only. - - PDF text extraction (pdfplumber or PyMuPDF). All files are text-based, no OCR in v1. - - Template classification: TF-IDF + Logistic Regression (scikit-learn) on the extracted text, with a configurable confidence threshold. - - Per-template slot filling: regex + dateutil for dates, amounts, references; gazetteer + fuzzy matching (rapidfuzz) for counterparties, with spaCy/CamemBERT NER as fallback if needed. - - Templates and extractors declared in configuration (YAML/JSON), not in code. - - SQLite database: rename history, decisions, confidence scores, rollback target. - - Local web dashboard: all processed files with status (not processed / processed / validated), file detail page with PDF preview, template and field correction, validation, history, statistics. - - On-demand classifier retraining, using the existing corpus plus every file with "validated" status. +- Ingestion by polling the `inbox/` prefix of a Garage S3 bucket for new objects only. Nextcloud (External Storage app) is the user-facing interface to upload, download, and share files. +- PDF text extraction (pdfplumber or PyMuPDF). All files are text-based, no OCR in v1. +- Template classification: TF-IDF + Logistic Regression (scikit-learn) on the extracted text, with a configurable confidence threshold. +- Per-template slot filling: regex + dateutil for dates, amounts, references; gazetteer + fuzzy matching (rapidfuzz) for counterparties, with spaCy/CamemBERT NER as fallback if needed. +- Templates and extractors declared in configuration (YAML/JSON), not in code. +- Storage: Garage (self-hosted S3-compatible, https://garagehq.deuxfleurs.fr/) with a single versioned bucket; objects addressed by S3 keys, no local filesystem for documents. +- Metadata: one JSON sidecar object per PDF in the bucket, holding template, fields, confidence, status, and history — the source of truth. SQLite acts as a replicated index for dashboard queries, rebuildable from the sidecars. +- Local web dashboard: all processed files with status (not processed / processed / validated), file detail page with PDF preview streamed from S3, template and field correction, validation, history, statistics. +- On-demand classifier retraining, using the existing corpus plus every file with "validated" status. - ### Out of scope (v1) +### Out of scope (v1) - - OCR of scanned documents (planned for v2 if the mix changes). - - Backlog processing of existing files — new files only. - - Remote language models, cloud APIs, telemetry. - - Multi-user management, authentication. - - Merging, splitting, or modifying PDF content. +- OCR of scanned documents (planned for v2 if the mix changes). +- Backlog processing of existing files — new files only. +- Cloud S3 providers, remote language models, cloud APIs, telemetry. +- At-rest encryption: the documents are not sensitive; bucket and volumes are unencrypted in v1. +- Multi-user management, authentication on the dashboard (Nextcloud handles sharing and access control on files). +- Merging, splitting, or modifying PDF content. - ## 5. User and Usage +## 5. User and Usage - Single user (personal use), technically proficient. Nominal flow: +Single user (personal use), technically proficient. Nominal flow: - 1. A PDF arrives in the watched folder. - 2. The watcher waits for the write to finish, extracts the text, classifies, extracts the fields, validates. - 3. High confidence and valid fields: the file is renamed and moved; the event is logged. - 4. Otherwise: the file appears in the dashboard review queue; the user corrects or validates; the correction feeds the training corpus. +1. A PDF is uploaded into the `inbox/` prefix via Nextcloud (or any S3 client). +2. The poller detects the new object, waits for write completion (stable ETag/content-length on repeated listing), downloads it, extracts the text, classifies, extracts the fields, validates. +3. High confidence and valid fields: the object is copied to its renamed key in the target prefix and the inbox copy deleted; metadata is written to the sidecar and replicated to SQLite; the event is logged. +4. Otherwise: the file appears in the dashboard review queue; the user corrects or validates; the correction feeds the training corpus. - ## 6. Functional Requirements +## 6. Functional Requirements - - FR1: detect new PDFs in a configurable folder, waiting for file stability. - - FR2: classify the document into one of the configured templates, with a confidence score and an "unknown" class below the threshold. - - FR3: fill each template field according to its schema (type, format, validity range). - - FR4: validate fields before renaming (parseable date, numeric amount, recognized or flagged counterparty); any failed validation routes to review. - - FR5: rename and move to the template's target folder, resolving collisions (incremental suffix). - - FR6: log every operation in SQLite with an old name → new name → decision mapping, and per-file status (not processed / processed / validated), enabling rollback and selecting training data. - - FR7: local web dashboard (FastAPI + React): file list, detail page, review, correction, validation, history with undo, retraining trigger. - - FR7a: the dashboard shows every processed file with its status: "not processed" (confidence was too low, rename pending), "processed" (automatically renamed and moved), or "validated" (the user confirmed or corrected the result — such files are eligible for future model retraining). - - FR7b: clicking a file opens a detail page displaying the PDF, where the user can change the template if misclassified, correct the extracted field values, and validate the result. Validation sets the status to "validated" and feeds the corrected data into the training corpus. - - FR8: adding a new template is done through a configuration entry (YAML) plus example files, with no code changes. +- FR1: detect new PDFs in a configurable `inbox/` S3 prefix by polling the bucket listing at a configurable interval, ignoring non-PDF keys and temporary uploads (e.g. Nextcloud `.part` files). +- FR2: classify the document into one of the configured templates, with a confidence score and an "unknown" class below the threshold. +- FR3: fill each template field according to its schema (type, format, validity range). +- FR4: validate fields before renaming (parseable date, numeric amount, recognized or flagged counterparty); any failed validation routes to review. +- FR5: rename and move within the bucket via S3 copy + delete to the template's target prefix, resolving key collisions (incremental suffix). +- FR6: write a JSON sidecar per file (old key → new key → decision mapping, per-file status not processed / processed / validated) and replicate to SQLite, enabling rollback and selecting training data. +- FR7: local web dashboard (FastAPI + React): file list, detail page, review, correction, validation, history with undo, retraining trigger. PDFs are streamed from the bucket for preview. +- FR7a: the dashboard shows every processed file with its status: "not processed" (confidence was too low, rename pending), "processed" (automatically renamed and moved), or "validated" (the user confirmed or corrected the result — such files are eligible for future model retraining). +- FR7b: clicking a file opens a detail page displaying the PDF, where the user can change the template if misclassified, correct the extracted field values, and validate the result. Validation sets the status to "validated" and feeds the corrected data into the training corpus. +- FR8: adding a new template is done through a configuration entry (YAML) plus example files, with no code changes. - ## 7. Non-Functional Requirements +## 7. Non-Functional Requirements - - Privacy: 100% offline, no data ever leaves the server. - - Performance: a standard-size PDF processed in under a few seconds. - - Reliability: zero silent invalid renames — every ambiguous case goes to review. - - Reversibility: every rename can be undone from the history. - - Document language: French only in v1. - - Stack: Python (FastAPI) backend, React frontend, SQLite, scikit-learn, spaCy/fr, pdfplumber or PyMuPDF, rapidfuzz, watchdog. - - Portability: containerized deployment or systemd service on the user's server. +- Privacy: 100% offline on self-hosted infrastructure (Garage). No data ever leaves the server. +- Encryption: none at rest in v1; documents are not sensitive. Network access to the bucket stays on the local network. +- Performance: a standard-size PDF processed in under a few seconds, including the S3 download. +- Reliability: zero silent invalid renames — every ambiguous case goes to review. +- Reversibility: every rename can be undone from the history (bucket versioning enabled; undo copies the previous version back if needed). +- Document language: French only in v1. +- Stack: Python (FastAPI) backend, boto3-compatible S3 client (Garage), React frontend, SQLite, scikit-learn, spaCy/fr, pdfplumber or PyMuPDF, rapidfuzz. Nextcloud with External Storage (S3) as the user-facing file interface. +- Portability: containerized deployment or systemd service on the user's server, alongside the Garage deployment. - ## 8. Success Criteria +## 8. Success Criteria - - ≥ 95% correct template classification on typical files after initial training on the existing corpus. - - ≥ 80% of files processed without human intervention (the rest in review, never a silent wrong rename). - - Zero non-reversible renames. - - Adding a new template in under 30 minutes without writing code. - - Working v1 in under one month. +- ≥ 95% correct template classification on typical files after initial training on the existing corpus. +- ≥ 80% of files processed without human intervention (the rest in review, never a silent wrong rename). +- Zero non-reversible renames. +- Adding a new template in under 30 minutes without writing code. +- SQLite can be fully rebuilt from the bucket sidecars at any time. +- Working v1 in under one month. - ## 9. Risks and Mitigations +## 9. Risks and Mitigations - - Limited training corpus (50–500 files): start simple (TF-IDF + LogReg), optionally sentence-transformers embeddings if accuracy is insufficient; the review loop continuously enriches the corpus. - - Templates too weakly distinguished in the text: enrich features (header, issuer, business keywords per template). - - Poorly extracted or empty text files: systematic detection and routing to review. - - Sensitive files moved by mistake: high default confidence threshold + strict validation + rollback. +- Limited training corpus (50–500 files): start simple (TF-IDF + LogReg), optionally sentence-transformers embeddings if accuracy is insufficient; the review loop continuously enriches the corpus. +- Templates too weakly distinguished in the text: enrich features (header, issuer, business keywords per template). +- Poorly extracted or empty text files: systematic detection and routing to review. +- Partial uploads (Nextcloud chunked uploads): poll-only keys with a stable size across two listing cycles and a `.part`/`.~lock` exclusion list. +- Sidecar/object drift (sidecar missing or out of sync with SQLite): sidecars win; a consistency job rebuilds SQLite and flags orphaned or missing sidecars. +- Garage version of S3 API: pin a Garage release and run a Week 1 compatibility spike (listing, multipart, copy-object, versioning). - ## 10. Milestones +## 10. Milestones - 1. Week 1: text extraction, parsing the existing corpus into (template, fields), declarative template configuration. - 2. Week 2: classifier + extractors + validation + SQLite logging + watcher. - 3. Week 3: React dashboard (file list with statuses, detail page with PDF preview and correction, validation, history, rollback, retraining). - 4. Week 4: hardening (thresholds, collisions, tests), deployment, production go-live on the watched folder. +1. Week 1: text extraction, Garage bucket provisioning and S3 client spike, parsing the existing corpus into (template, fields), declarative template configuration. +2. Week 2: classifier + extractors + validation + sidecar/SQLite logging + inbox poller. +3. Week 3: React dashboard (file list with statuses, detail page with PDF preview and correction, validation, history, rollback, retraining) + Nextcloud external storage mounting. +4. Week 4: hardening (thresholds, collisions, consistency job, tests), deployment, production go-live on the bucket inbox. \ No newline at end of file diff --git a/automatic-pdf-renamer/prd.md b/automatic-pdf-renamer/prd.md index ed66aef..23246da 100644 --- a/automatic-pdf-renamer/prd.md +++ b/automatic-pdf-renamer/prd.md @@ -1,53 +1,77 @@ # PRD — Automatic PDF Renamer -Product Requirements Document. Derived from the project brief (BMAD). Personal tool, single user, 100% offline. +Product Requirements Document. Derived from the project brief (BMAD). Personal tool, single user, 100% offline. Storage is a self-hosted Garage (S3-compatible) bucket; Nextcloud is the user-facing file interface; JSON sidecars are the source of truth; SQLite is a replicated index. ## 1. Product Overview -A server-side service with a local web dashboard that watches a folder for incoming PDFs, identifies the document type from its content, applies the matching naming template, fills the template fields, renames and moves the file into an organized folder tree, and tracks every file through a three-state lifecycle (not processed / processed / validated). Low-confidence or invalid files wait for human correction; validated files feed future model retraining. +A server-side service with a local web dashboard that polls an `inbox/` prefix of a Garage S3 bucket for incoming PDFs, identifies the document type from its content, applies the matching naming template, fills the template fields, moves the object to an organized prefix tree within the bucket, and tracks every file through a three-state lifecycle (not processed / processed / validated). Metadata lives in per-file JSON sidecars (source of truth), replicated to SQLite for queries. Low-confidence or invalid files wait for human correction; validated files feed future model retraining. Users upload, download, and share documents through Nextcloud, which mounts the bucket as external storage. ## 2. Personas Single user: the owner. Technically proficient, wants minimal manual work but zero silent errors. -## 3. Document Lifecycle and Statuses +## 3. Storage and Metadata Architecture + +### 3.1 Object storage + +- Garage (https://garagehq.deuxfleurs.fr/), self-hosted, S3 API. One versioned bucket, e.g. `pdf-renamer`. +- Key layout: `inbox/` (entry point), `pending/` (not-processed files moved out of inbox to avoid re-processing), and one prefix per template (e.g. `invoices/2026/...`). +- Sidecars: `{object-key}.meta.json` next to each PDF. +- No encryption at rest in v1 (documents not sensitive); bucket served on the local network only. +- Renames are copy-object + delete within the bucket; collisions resolved with an incremental suffix. + +### 3.2 Metadata: sidecars as source of truth, SQLite as replica + +- Each PDF has a sidecar JSON object holding: sha256, original key, current key, status, template, document type, emission date, emitting company, all optional fields, confidence, candidates, and the full event history (classified, renamed, validated, undone, corrected). +- Sidecar is updated atomically on every state change: write the new sidecar to a temp key, copy over the final sidecar key. +- SQLite mirrors the sidecar content in queryable tables for the dashboard (lists, filters, statistics). It is a cache/replica, never authoritative. Any sidecar↔SQLite divergence is resolved in favor of the sidecar. +- A rebuild command reconstructs the full SQLite database by scanning all sidecars; a consistency job detects PDFs without sidecars (orphaned) and sidecars without PDFs (missing object) and flags them in the dashboard. + +### 3.3 Nextcloud integration + +- Nextcloud connects to the bucket via the External Storage (S3/Swift) app, mounted read-write on `inbox/` and read-write on template prefixes, giving upload, download, preview, and sharing without touching the dashboard. +- The renamer service uses its own S3 credentials distinct from Nextcloud's; Nextcloud users never need renamer credentials. +- The renamer never touches Nextcloud's own storage or database; the only shared surface is the bucket. + +## 4. Document Lifecycle and Statuses | Status | Meaning | |---|---| -| not processed | Classification confidence below threshold, or field validation failed. File untouched in the watched folder (or a pending subfolder), waiting for review. | +| not processed | Classification confidence below threshold, or field validation failed. Object moved to the `pending/` prefix, waiting for review. | | processed | Automatically classified, fields validated, renamed and moved. Eligible for undo. | -| validated | The user reviewed the result (confirmed or corrected template/fields). Eligible for model retraining. Terminal state for the review flow; the file was renamed and moved. | +| validated | The user reviewed the result (confirmed or corrected template/fields). Eligible for model retraining. Terminal state for the review flow; the object was renamed and moved. | Transitions: -- new file → not processed (low confidence or invalid fields) -- new file → processed (high confidence, valid fields) +- new object in `inbox/` → not processed (low confidence or invalid fields; object copied to `pending/`) +- new object in `inbox/` → processed (high confidence, valid fields) - not processed → validated (user corrects and validates in the detail page; rename then applied) - processed → validated (user opens the detail page and confirms; no changes needed) -- processed or validated → previous state restorable via undo (rollback of the rename) +- processed or validated → previous state restorable via undo (copy of the previous version back to the old key) -## 4. Functional Requirements +## 5. Functional Requirements -### Epic 1 — Ingestion and watching +### Epic 1 — Ingestion -- FR1.1 Watch a configurable folder for new PDF files only. Ignore other extensions and temporary files (e.g. `.tmp`, partial downloads). -- FR1.2 Wait for file stability (size unchanged over a configurable interval) before processing. -- FR1.3 On service restart, detect PDFs that arrived while the service was down but were never processed (no record in the database). -- FR1.4 Configurable poll interval or OS-level file system events (watchdog). +- FR1.1 Poll the configurable `inbox/` prefix at a configurable interval; process only keys ending in `.pdf`. Ignore other extensions and temporary files (Nextcloud `.part`, `.~lock`, hidden files). +- FR1.2 Wait for object stability: process a key only after its size and ETag are unchanged across two consecutive listings (handles chunked uploads and partial writes). +- FR1.3 On service restart, detect objects present in `inbox/` (or `pending/`) with no sidecar record and process them. +- FR1.4 Optionally reduce latency with bucket notification hooks if the Garage deployment supports them; polling is the v1 baseline. +- FR1.5 After processing, the inbox copy is deleted (processed files) or the object is copied to `pending/` and the inbox copy deleted (not-processed files), so `inbox/` always converges to empty. ### Epic 2 — Text extraction -- FR2.1 Extract text with pdfplumber or PyMuPDF (choice fixed at implementation, documented in config). +- FR2.1 Download the object from S3 and extract text with pdfplumber or PyMuPDF (choice fixed at implementation, documented in config). - FR2.2 Extract the full first page plus header and footer zones of all pages. - FR2.3 If extracted text is empty or below a minimum length, route the file to not processed with reason "empty text". -- FR2.4 Store the extracted text in the database for the file's lifetime (enables detail page display and retraining without re-parsing). +- FR2.4 Store the extracted text in the sidecar (and replicated to SQLite) for the file's lifetime, so the detail page and retraining never need to re-download and re-parse the object. ### Epic 3 — Template classification - FR3.1 Classify the document into one of the configured templates using TF-IDF + Logistic Regression on the extracted text. - FR3.2 Output a confidence score per template; the top score below a configurable threshold (default 0.8) yields "unknown" and routes the file to not processed with reason "low confidence". - FR3.3 Display the top-N template candidates with scores on the detail page. -- FR3.4 Retrain the classifier on demand from: the initial corpus + all files with validated status. Retraining is a manual action from the dashboard, with the training date and resulting evaluation metrics stored and displayed. +- FR3.4 Retrain the classifier on demand from: the initial corpus + all files with validated status. Retraining is a manual action from the dashboard, with the training date and resulting evaluation metrics stored in the sidecar-adjacent model registry and displayed. ### Epic 4 — Document data model, slot filling and validation @@ -56,104 +80,124 @@ Transitions: Every document, regardless of template, carries at least these four common fields: - Common field 1 — Template: the selected template (i.e. the document family: invoice, purchase order, contract, bank statement, rent receipt, etc.). -- Common field 2 — Emission date: the date the document was emitted, extracted from its content (not the file modification date). +- Common field 2 — Emission date: the date the document was emitted, extracted from its content (not the object modification date). - Common field 3 — Emitting company: the company or organization that emitted the document, matched against the counterparty gazetteer. - Common field 4 — Document type: the fine-grained type of document within the family (e.g. debit advice vs monthly statement for bank statements; quotes vs order confirmations for purchase documents). -The filename pattern of each template is composed from these common fields, plus the template's optional fields. +The object key pattern of each template is composed from these common fields, plus the template's optional fields. #### 4.2 Optional fields (template-specific) -- FR4.1 Each template declares in YAML its optional fields beyond the common four: name, type, format, extraction rules, validation rules, and whether the field is rendered in the filename, the target folder path, or stored only. Examples: bank statements carry account owner and statement type; rent receipts carry the concerned premises; contracts carry the counterparty signatory. +- FR4.1 Each template declares in YAML its optional fields beyond the common four: name, type, format, extraction rules, validation rules, and whether the field is rendered in the object key, the target prefix, or stored only. Examples: bank statements carry account owner and statement type; rent receipts carry the concerned premises; contracts carry the counterparty signatory. - FR4.2 Fill common and optional fields using per-template regexes, dateutil for dates, and a counterparty gazetteer with rapidfuzz matching for the emitting company (similarity threshold configurable; below threshold the company is "unrecognized"). - FR4.3 Validate fields before renaming: emission date parses and is within a plausible range, emitting company recognized, document type among the template's allowed values, each optional field matches its declared format (empty optional fields are allowed). Any failure routes to not processed with the failing fields identified. -- FR4.4 Collisions in the target folder are resolved with an incremental suffix; the final name is always the one logged. +- FR4.4 Collisions in the target prefix are resolved with an incremental suffix; the final key is always the one logged. ### Epic 5 — Renaming, moving, logging, undo -- FR5.1 Rename and move the file to the template's target folder according to its filename pattern. -- FR5.2 Every event is logged in SQLite: file hash, old path, new path, template, extracted fields, confidence, decision, status, timestamps. -- FR5.3 Undo restores the previous path and sets the status back to its prior value, keeping the full audit trail. -- FR5.4 Deleting or moving a file on disk that is not in a terminal state is detected and the record is flagged as orphaned. +- FR5.1 Rename and move the object within the bucket (S3 copy to the target key per the template pattern, then delete of the source key). Update the sidecar and SQLite in the same transaction boundary: sidecar first, then SQLite; a crash between the two is repaired by the consistency job. +- FR5.2 Every event is logged in the sidecar history and replicated to SQLite: file hash, old key, new key, template, extracted fields, confidence, decision, status, timestamps. +- FR5.3 Undo copies the object back to its previous key (using bucket versioning if the previous version was overwritten), restores the sidecar history entry, and sets the status back to its prior value, keeping the full audit trail. +- FR5.4 Objects missing from the bucket while not in a terminal state are detected by the consistency job and the record is flagged as orphaned. ### Epic 6 — Dashboard: file list -- FR6.1 A single list shows every file the system has seen, with columns: emission date, current filename, status, template (document family), document type, emitting company, confidence. -- FR6.2 Filter by status, template, document type, emitting company, and date range; sort by any column; free-text search on filename and emitting company. +- FR6.1 A single list shows every file the system has seen, with columns: emission date, current object name, status, template (document family), document type, emitting company, confidence. Served from SQLite. +- FR6.2 Filter by status, template, document type, emitting company, and date range; sort by any column; free-text search on object name and emitting company. - FR6.3 Status is visually distinguishable (color or badge): not processed, processed, validated. -- FR6.4 The list shows counters: files per status, automation rate (processed / total), validation rate. +- FR6.4 The list shows counters: files per status, automation rate (processed / total), validation rate, plus a consistency indicator (orphaned objects, missing sidecars). ### Epic 7 — Dashboard: file detail page - FR7.1 Clicking a file in the list opens a detail page. -- FR7.2 The detail page displays the PDF (embedded viewer) alongside the extracted data. +- FR7.2 The detail page displays the PDF (embedded viewer), streamed from the bucket by the backend, alongside the extracted data. - FR7.3 The user can change the template (dropdown of configured templates, showing the classifier's top candidates and scores). - FR7.4 The detail page always shows the four common fields (template, emission date, emitting company, document type) plus the current template's optional fields; the user can correct each value inline; changing the template reloads the optional fields with the new template's schema and best-effort re-extraction, keeping the common field values. -- FR7.5 A preview shows the target filename and destination folder resulting from the current values, updating live as fields change. +- FR7.5 A preview shows the target object key and destination prefix resulting from the current values, updating live as fields change. - FR7.6 A validate action applies the rename/move (if not already done), sets the status to validated, and records the corrected values as training data. - FR7.7 An undo action (for processed or validated files) rolls back the rename. +- FR7.8 A direct link opens the file in Nextcloud (external storage share) for download or sharing. ### Epic 8 — Template management - FR8.1 Templates are defined in YAML configuration files, hot-reloaded without service restart. - FR8.2 Adding a template = YAML entry + example files; no code change. -- FR8.3 The dashboard lists configured templates with their slot definitions and the count of corpus files per template. +- FR8.3 The dashboard lists configured templates with their slot definitions, target prefix, and the count of corpus files per template. ### Epic 9 — Retraining - FR9.1 Manual "Retrain" action in the dashboard: trains on initial corpus + validated files, shows evaluation metrics (accuracy, confusion matrix) before activation. -- FR9.2 The user can activate or discard the new model. Only one model is active at a time; model files are versioned on disk. +- FR9.2 The user can activate or discard the new model. Only one model is active at a time; model files are versioned on disk (and optionally mirrored under a `models/` prefix in the bucket). -## 5. Non-Functional Requirements +### Epic 10 — Consistency and recovery -- NFR1 Privacy: fully offline. No network calls except serving the local dashboard. Dependencies must not require downloads at runtime. -- NFR2 Performance: end-to-end processing of a standard PDF under 5 seconds on the target server; dashboard file list loads under 1 s for 10,000 records. +- FR10.1 A periodic consistency job verifies, for every sidecar: object exists at current key, sidecar in sync with SQLite; and for every PDF object: sidecar exists. Anomalies are listed in the dashboard with repair actions. +- FR10.2 Full SQLite rebuild from sidecars is available as a dashboard action and a CLI command. +- FR10.3 The pending prefix is re-scanned at startup to resume interrupted processing. + +## 6. Non-Functional Requirements + +- NFR1 Privacy: fully offline on self-hosted infrastructure. No network calls except the local Garage bucket, the local Nextcloud instance, and serving the local dashboard. Dependencies must not require downloads at runtime. +- NFR2 Performance: end-to-end processing of a standard PDF under 5 seconds on the target server, including the S3 download; dashboard file list loads under 1 s for 10,000 records (served from SQLite). - NFR3 Reliability: zero silent wrong renames. Every ambiguous case routes to not processed. -- NFR4 Durability: SQLite with WAL; a crash at any point leaves no half-renamed state (rename is atomic on the same filesystem; cross-filesystem moves use copy+verify+delete). +- NFR4 Durability: bucket versioning enabled; SQLite with WAL. A crash at any point leaves no half-renamed state: the S3 copy is a non-destructive copy-then-delete, the sidecar is written atomically, and the consistency job repairs SQLite-sidecar drift. - NFR5 Reversibility: every rename has an undo. - NFR6 Language: documents in French; UI in English or French (single locale acceptable in v1). -- NFR7 Deployment: runs as a systemd service or container; configuration via a single YAML file and environment variables. -- NFR8 Stack: Python (FastAPI, watchdog, scikit-learn, pdfplumber/PyMuPDF, rapidfuzz, spaCy fr model optional), React (Vite, a lightweight PDF viewer such as react-pdf), SQLite. +- NFR7 Deployment: runs as a systemd service or container next to the Garage deployment and the Nextcloud server; configuration via a single YAML file and environment variables (S3 endpoint, credentials, bucket, prefixes). +- NFR8 Stack: Python (FastAPI, boto3 or minio S3 client, scikit-learn, pdfplumber/PyMuPDF, rapidfuzz, spaCy fr model optional), React (Vite, a lightweight PDF viewer such as react-pdf), SQLite, Garage for S3 storage, Nextcloud with External Storage (S3) as the user-facing file interface. +- NFR9 No encryption at rest in v1: documents are not sensitive; this decision is recorded explicitly and can be revisited. -## 6. Data Model (indicative) +## 7. Data Model -- files: id, sha256, original_path, current_path, status, template_id, confidence, emission_date, emitting_company, document_type, created_at, updated_at, extracted_text. +Sidecar JSON (source of truth), one per PDF at `{key}.meta.json`: + +- schema_version, sha256, original_key, current_key, status, template_id, confidence, emission_date, emitting_company, document_type, extracted_text, created_at, updated_at, events[] (type: classified / renamed / validated / undone / corrected, payload, timestamp). + +SQLite replica (rebuilt from sidecars): + +- files: id, sha256, original_key, current_key, status, template_id, confidence, emission_date, emitting_company, document_type, created_at, updated_at, extracted_text. - fields: id, file_id, slot_name, value, is_common (bool), was_corrected (bool), validated_by. -- events: id, file_id, type (classified, renamed, validated, undone), payload_json, created_at. -- templates: mirrored from YAML at load: id, name, document_types, pattern, target_folder, common_fields_json, optional_fields_json. +- events: id, file_id, type, payload_json, created_at. +- templates: mirrored from YAML at load: id, name, document_types, key_pattern, target_prefix, common_fields_json, optional_fields_json. - models: id, trained_at, metrics_json, active (bool). +- consistency_flags: id, file_id, kind (orphaned_object, missing_sidecar, drift), details_json, created_at, resolved_at. -## 7. API Surface (indicative) +## 8. API Surface (indicative) -- GET /files?status=&template=&q=&page= — list with filters. -- GET /files/{id} — detail: text, fields, candidates, current path. -- GET /files/{id}/pdf — stream the PDF for the viewer. +- GET /files?status=&template=&q=&page= — list with filters (from SQLite). +- GET /files/{id} — detail: text, fields, candidates, current key, nextcloud_url. +- GET /files/{id}/pdf — stream the PDF from the bucket for the viewer. - POST /files/{id}/validate — body: template_id, fields; applies rename if needed, sets validated. - POST /files/{id}/undo — rolls back the rename. - GET /templates — configured templates. - POST /retrain — starts training; GET /retrain/status; POST /retrain/{id}/activate. +- GET /consistency — list anomalies; POST /consistency/rebuild — rebuild SQLite from sidecars. -## 8. Acceptance Criteria (v1) +## 9. Acceptance Criteria (v1) -- AC1 A new PDF with high-confidence classification and valid fields appears as processed within the poll interval, renamed and moved correctly, and is undoable. -- AC2 A PDF below the confidence threshold appears as not processed; correcting and validating it on the detail page renames it and sets validated. +- AC1 A new PDF dropped in `inbox/` via Nextcloud with high-confidence classification and valid fields appears as processed within the poll interval, renamed and moved in the bucket, and is undoable. +- AC2 A PDF below the confidence threshold appears as not processed, is moved to `pending/`, and appears in Nextcloud under `pending/`; correcting and validating it on the detail page renames it and sets validated. - AC3 Opening a processed file and validating without changes sets validated without re-renaming. - AC4 The file list filters by status, template, document type, emitting company, and date; counters match the data. -- AC5 Changing the template on the detail page keeps the common fields, reloads the optional fields per the new template schema, and shows the live target filename preview. +- AC5 Changing the template on the detail page keeps the common fields, reloads the optional fields per the new template schema, and shows the live target key preview. - AC5b Every file in every state carries the four common fields (template, emission date, emitting company, document type); a document whose emitting company cannot be recognized is routed to not processed rather than renamed. - AC6 Retraining uses initial corpus + validated files, shows metrics, and activation swaps the model without service restart. - AC7 Adding a template via YAML (with example files) makes it available for classification and in the detail page dropdown, with no code change and no restart. -- AC8 Killing the service mid-processing leaves no half-renamed file; restart recovers and re-processes pending files. -- AC9 No network traffic other than local dashboard serving (verifiable with a firewall audit). -- AC10 1,000-file list view renders with filters in under 1 second. +- AC8 Killing the service mid-processing leaves no half-renamed object: the copy-then-delete order and the sidecar-first write guarantee a consistent state on restart, and the consistency job repairs SQLite. +- AC9 Deleting the SQLite database entirely and running the rebuild command restores a fully functional dashboard from the sidecars alone. +- AC10 No network traffic other than the local Garage bucket, local Nextcloud, and local dashboard serving (verifiable with a firewall audit). +- AC11 1,000-file list view renders with filters in under 1 second. +- AC12 Nextcloud sees renamed and moved files at their new location without reconfiguration; sharing a renamed file from Nextcloud keeps working. -## 9. Release Plan +## 10. Release Plan -- v1 (this PRD): new files only, text-based French PDFs, three statuses, detail page with correction and validation, manual retrain, undo. -- v2 candidates: OCR for scanned documents, backlog ingestion of historical files, automatic periodic retraining, multilingual documents, automatic counterparty learning from validated files. +- v1 (this PRD): new files only, text-based French PDFs, three statuses, Garage S3 storage, sidecar source of truth with SQLite replica, Nextcloud as file interface, detail page with correction and validation, manual retrain, undo, consistency job. +- v2 candidates: OCR for scanned documents, backlog ingestion of historical files, automatic periodic retraining, multilingual documents, automatic counterparty learning from validated files, bucket notification hooks to replace polling, at-rest encryption if the document mix becomes sensitive. -## 10. Open Questions +## 11. Open Questions - Choice between pdfplumber and PyMuPDF (license and performance tradeoff) — decide in Week 1 spike. -- Whether not-processed files stay in the watched folder or move to a pending subfolder (recommendation: pending subfolder to avoid re-processing loops). +- S3 client library: boto3 vs minio-py against Garage — covered by the Week 1 Garage compatibility spike (listing, copy-object, multipart, versioning behavior). - Exact confidence threshold and per-template overrides — calibrate on the initial corpus. +- Whether Nextcloud mounts the whole bucket or one mount per prefix (inbox vs template prefixes) — depends on the Nextcloud External Storage S3 mount semantics; decide during Week 3 integration. +- Poll interval tradeoff between responsiveness and Garage listing load. \ No newline at end of file