Manually corrected the brief and prd

automatic-pdf-renamer/brief.md

@@ -6,7 +6,7 @@ A personal, 100% offline tool that watches an S3 bucket hosted on a self-hosted
 
 ## 2. Problem and Context
 
-PDFs arrive continuously in an inbox S3 prefix (dropped via Nextcloud by the user or by upload automations). Current naming is manual or inconsistent, making documents hard to find and archive. The user owns 50 to 500 files already correctly named according to 1 to 5 existing templates, which will serve as training and reference corpus.
+PDFs arrive continuously in an inbox S3 prefix (dropped via Nextcloud by the user or by upload automations). Current naming is manual or inconsistent, making documents hard to find and archive. The user owns around 500 files already correctly named according to 4 existing templates, which will serve as training and reference corpus.
 
 ## 3. Objectives
 
@@ -22,14 +22,14 @@ PDFs arrive continuously in an inbox S3 prefix (dropped via Nextcloud by the use
 ### In scope (v1)
 
 - Ingestion by polling the `inbox/` prefix of a Garage S3 bucket for new objects only. Nextcloud (External Storage app) is the user-facing interface to upload, download, and share files.
-- PDF text extraction (pdfplumber or PyMuPDF). All files are text-based, no OCR in v1.
+- PDF text extraction. All files are text-based, no OCR in v1.
 - Template classification: TF-IDF + Logistic Regression (scikit-learn) on the extracted text, with a configurable confidence threshold.
 - Per-template slot filling: regex + dateutil for dates, amounts, references; gazetteer + fuzzy matching (rapidfuzz) for counterparties, with spaCy/CamemBERT NER as fallback if needed.
 - Templates and extractors declared in configuration (YAML/JSON), not in code.
 - Storage: Garage (self-hosted S3-compatible, https://garagehq.deuxfleurs.fr/) with a single versioned bucket; objects addressed by S3 keys, no local filesystem for documents.
 - Metadata: one JSON sidecar object per PDF in the bucket, holding template, fields, confidence, status, and history — the source of truth. SQLite acts as a replicated index for dashboard queries, rebuildable from the sidecars.
 - Local web dashboard: all processed files with status (not processed / processed / validated), file detail page with PDF preview streamed from S3, template and field correction, validation, history, statistics.
-- On-demand classifier retraining, using the existing corpus plus every file with "validated" status.
+- On-demand classifier retraining, using the existing corpus based on all the files with "validated" status.
 
 ### Out of scope (v1)
 
@@ -57,7 +57,7 @@ Single user (personal use), technically proficient. Nominal flow:
 - FR4: validate fields before renaming (parseable date, numeric amount, recognized or flagged counterparty); any failed validation routes to review.
 - FR5: rename and move within the bucket via S3 copy + delete to the template's target prefix, resolving key collisions (incremental suffix).
 - FR6: write a JSON sidecar per file (old key → new key → decision mapping, per-file status not processed / processed / validated) and replicate to SQLite, enabling rollback and selecting training data.
-- FR7: local web dashboard (FastAPI + React): file list, detail page, review, correction, validation, history with undo, retraining trigger. PDFs are streamed from the bucket for preview.
+- FR7: local web dashboard (FastAPI + React): file list, detail page, review, correction, validation, history, retraining trigger. PDFs are streamed from the bucket for preview.
 - FR7a: the dashboard shows every processed file with its status: "not processed" (confidence was too low, rename pending), "processed" (automatically renamed and moved), or "validated" (the user confirmed or corrected the result — such files are eligible for future model retraining).
 - FR7b: clicking a file opens a detail page displaying the PDF, where the user can change the template if misclassified, correct the extracted field values, and validate the result. Validation sets the status to "validated" and feeds the corrected data into the training corpus.
 - FR8: adding a new template is done through a configuration entry (YAML) plus example files, with no code changes.
@@ -68,10 +68,9 @@ Single user (personal use), technically proficient. Nominal flow:
 - Encryption: none at rest in v1; documents are not sensitive. Network access to the bucket stays on the local network.
 - Performance: a standard-size PDF processed in under a few seconds, including the S3 download.
 - Reliability: zero silent invalid renames — every ambiguous case goes to review.
-- Reversibility: every rename can be undone from the history (bucket versioning enabled; undo copies the previous version back if needed).
 - Document language: French only in v1.
 - Stack: Python (FastAPI) backend, boto3-compatible S3 client (Garage), React frontend, SQLite, scikit-learn, spaCy/fr, pdfplumber or PyMuPDF, rapidfuzz. Nextcloud with External Storage (S3) as the user-facing file interface.
-- Portability: containerized deployment or systemd service on the user's server, alongside the Garage deployment.
+- Portability: containerized deployment, alongside the Garage deployment.
 
 ## 8. Success Criteria
 
@@ -80,20 +79,12 @@ Single user (personal use), technically proficient. Nominal flow:
 - Zero non-reversible renames.
 - Adding a new template in under 30 minutes without writing code.
 - SQLite can be fully rebuilt from the bucket sidecars at any time.
-- Working v1 in under one month.
 
 ## 9. Risks and Mitigations
 
-- Limited training corpus (50–500 files): start simple (TF-IDF + LogReg), optionally sentence-transformers embeddings if accuracy is insufficient; the review loop continuously enriches the corpus.
+- Limited training corpus (~500 files): start simple (TF-IDF + LogReg), optionally sentence-transformers embeddings if accuracy is insufficient; the review loop continuously enriches the corpus.
 - Templates too weakly distinguished in the text: enrich features (header, issuer, business keywords per template).
 - Poorly extracted or empty text files: systematic detection and routing to review.
 - Partial uploads (Nextcloud chunked uploads): poll-only keys with a stable size across two listing cycles and a `.part`/`.~lock` exclusion list.
 - Sidecar/object drift (sidecar missing or out of sync with SQLite): sidecars win; a consistency job rebuilds SQLite and flags orphaned or missing sidecars.
-- Garage version of S3 API: pin a Garage release and run a Week 1 compatibility spike (listing, multipart, copy-object, versioning).
+- Garage version of S3 API: pin a Garage release and run a compatibility spike (listing, multipart, copy-object, versioning).
-
-## 10. Milestones
-
-1. Week 1: text extraction, Garage bucket provisioning and S3 client spike, parsing the existing corpus into (template, fields), declarative template configuration.
-2. Week 2: classifier + extractors + validation + sidecar/SQLite logging + inbox poller.
-3. Week 3: React dashboard (file list with statuses, detail page with PDF preview and correction, validation, history, rollback, retraining) + Nextcloud external storage mounting.
-4. Week 4: hardening (thresholds, collisions, consistency job, tests), deployment, production go-live on the bucket inbox.

automatic-pdf-renamer/prd.md

@@ -15,39 +15,38 @@ Single user: the owner. Technically proficient, wants minimal manual work but ze
 ### 3.1 Object storage
 
 - Garage (https://garagehq.deuxfleurs.fr/), self-hosted, S3 API. One versioned bucket, e.g. `pdf-renamer`.
-- Key layout: `inbox/` (entry point), `pending/` (not-processed files moved out of inbox to avoid re-processing), and one prefix per template (e.g. `invoices/2026/...`).
+- Key layout: `inbox/` (entry point), `pending/` (not-processed files moved out of inbox to avoid re-processing), and `files/` for the processes files (validated or not).
 - Sidecars: `{object-key}.meta.json` next to each PDF.
 - No encryption at rest in v1 (documents not sensitive); bucket served on the local network only.
 - Renames are copy-object + delete within the bucket; collisions resolved with an incremental suffix.
 
 ### 3.2 Metadata: sidecars as source of truth, SQLite as replica
 
-- Each PDF has a sidecar JSON object holding: sha256, original key, current key, status, template, document type, emission date, emitting company, all optional fields, confidence, candidates, and the full event history (classified, renamed, validated, undone, corrected).
+- Each PDF has a sidecar JSON object holding: sha256, original key, current key, status, template, document type, emission date, emitting company, all optional fields, confidence, and the full event history (classified, renamed, validated, undone, corrected).
 - Sidecar is updated atomically on every state change: write the new sidecar to a temp key, copy over the final sidecar key.
 - SQLite mirrors the sidecar content in queryable tables for the dashboard (lists, filters, statistics). It is a cache/replica, never authoritative. Any sidecar↔SQLite divergence is resolved in favor of the sidecar.
 - A rebuild command reconstructs the full SQLite database by scanning all sidecars; a consistency job detects PDFs without sidecars (orphaned) and sidecars without PDFs (missing object) and flags them in the dashboard.
 
 ### 3.3 Nextcloud integration
 
-- Nextcloud connects to the bucket via the External Storage (S3/Swift) app, mounted read-write on `inbox/` and read-write on template prefixes, giving upload, download, preview, and sharing without touching the dashboard.
+- Nextcloud connects to the bucket via the External Storage (S3/Swift) app, mounted read-write on `inbox/` and read-only on other prefixes, giving upload, download, preview, and sharing without touching the dashboard.
 - The renamer service uses its own S3 credentials distinct from Nextcloud's; Nextcloud users never need renamer credentials.
 - The renamer never touches Nextcloud's own storage or database; the only shared surface is the bucket.
 
 ## 4. Document Lifecycle and Statuses
 
-| Status | Meaning |
+| Status        | Meaning |
-|---|---|
+|---            |---|
 | not processed | Classification confidence below threshold, or field validation failed. Object moved to the `pending/` prefix, waiting for review. |
-| processed | Automatically classified, fields validated, renamed and moved. Eligible for undo. |
+| processed     | Automatically classified, fields validated, renamed and moved. |
-| validated | The user reviewed the result (confirmed or corrected template/fields). Eligible for model retraining. Terminal state for the review flow; the object was renamed and moved. |
+| validated     | The user reviewed the result (confirmed or corrected template/fields). Eligible for model retraining. Terminal state for the review flow; the object was renamed and moved. |
 
 Transitions:
 
 - new object in `inbox/` → not processed (low confidence or invalid fields; object copied to `pending/`)
 - new object in `inbox/` → processed (high confidence, valid fields)
 - not processed → validated (user corrects and validates in the detail page; rename then applied)
-- processed → validated (user opens the detail page and confirms; no changes needed)
+- processed → validated (user confirms or corrects and validates in the detail page; rename only if the user maid some changes)
-- processed or validated → previous state restorable via undo (copy of the previous version back to the old key)
 
 ## 5. Functional Requirements
 
@@ -62,7 +61,7 @@ Transitions:
 ### Epic 2 — Text extraction
 
 - FR2.1 Download the object from S3 and extract text with pdfplumber or PyMuPDF (choice fixed at implementation, documented in config).
-- FR2.2 Extract the full first page plus header and footer zones of all pages.
+- FR2.2 Extract the full first page only.
 - FR2.3 If extracted text is empty or below a minimum length, route the file to not processed with reason "empty text".
 - FR2.4 Store the extracted text in the sidecar (and replicated to SQLite) for the file's lifetime, so the detail page and retraining never need to re-download and re-parse the object.
 
@@ -93,18 +92,17 @@ The object key pattern of each template is composed from these common fields, pl
 - FR4.3 Validate fields before renaming: emission date parses and is within a plausible range, emitting company recognized, document type among the template's allowed values, each optional field matches its declared format (empty optional fields are allowed). Any failure routes to not processed with the failing fields identified.
 - FR4.4 Collisions in the target prefix are resolved with an incremental suffix; the final key is always the one logged.
 
-### Epic 5 — Renaming, moving, logging, undo
+### Epic 5 — Renaming, moving, logging
 
 - FR5.1 Rename and move the object within the bucket (S3 copy to the target key per the template pattern, then delete of the source key). Update the sidecar and SQLite in the same transaction boundary: sidecar first, then SQLite; a crash between the two is repaired by the consistency job.
-- FR5.2 Every event is logged in the sidecar history and replicated to SQLite: file hash, old key, new key, template, extracted fields, confidence, decision, status, timestamps.
+- FR5.2 Every event is logged in the sidecar history: file hash, old key, new key, template, extracted fields, confidence, decision, status, timestamps.
-- FR5.3 Undo copies the object back to its previous key (using bucket versioning if the previous version was overwritten), restores the sidecar history entry, and sets the status back to its prior value, keeping the full audit trail.
+- FR5.3 Objects missing from the bucket while not in a terminal state are detected by the consistency job and the record is flagged as orphaned.
-- FR5.4 Objects missing from the bucket while not in a terminal state are detected by the consistency job and the record is flagged as orphaned.
 
 ### Epic 6 — Dashboard: file list
 
 - FR6.1 A single list shows every file the system has seen, with columns: emission date, current object name, status, template (document family), document type, emitting company, confidence. Served from SQLite.
-- FR6.2 Filter by status, template, document type, emitting company, and date range; sort by any column; free-text search on object name and emitting company.
+- FR6.2 Filter by status, template, document type, emitting company, and date range; sort by any column; free-text search on object name, any field and optionnally first-page content.
-- FR6.3 Status is visually distinguishable (color or badge): not processed, processed, validated.
+- FR6.3 Status is visually distinguishable (badge): not processed, processed, validated.
 - FR6.4 The list shows counters: files per status, automation rate (processed / total), validation rate, plus a consistency indicator (orphaned objects, missing sidecars).
 
 ### Epic 7 — Dashboard: file detail page
@@ -115,19 +113,17 @@ The object key pattern of each template is composed from these common fields, pl
 - FR7.4 The detail page always shows the four common fields (template, emission date, emitting company, document type) plus the current template's optional fields; the user can correct each value inline; changing the template reloads the optional fields with the new template's schema and best-effort re-extraction, keeping the common field values.
 - FR7.5 A preview shows the target object key and destination prefix resulting from the current values, updating live as fields change.
 - FR7.6 A validate action applies the rename/move (if not already done), sets the status to validated, and records the corrected values as training data.
-- FR7.7 An undo action (for processed or validated files) rolls back the rename.
-- FR7.8 A direct link opens the file in Nextcloud (external storage share) for download or sharing.
 
 ### Epic 8 — Template management
 
-- FR8.1 Templates are defined in YAML configuration files, hot-reloaded without service restart.
+- FR8.1 Templates are defined in YAML configuration files.
 - FR8.2 Adding a template = YAML entry + example files; no code change.
 - FR8.3 The dashboard lists configured templates with their slot definitions, target prefix, and the count of corpus files per template.
 
 ### Epic 9 — Retraining
 
 - FR9.1 Manual "Retrain" action in the dashboard: trains on initial corpus + validated files, shows evaluation metrics (accuracy, confusion matrix) before activation.
-- FR9.2 The user can activate or discard the new model. Only one model is active at a time; model files are versioned on disk (and optionally mirrored under a `models/` prefix in the bucket).
+- FR9.2 The user can activate or discard the new model. Only one model is active at a time; model files are versioned on disk.
 
 ### Epic 10 — Consistency and recovery
 
@@ -137,61 +133,55 @@ The object key pattern of each template is composed from these common fields, pl
 
 ## 6. Non-Functional Requirements
 
-- NFR1 Privacy: fully offline on self-hosted infrastructure. No network calls except the local Garage bucket, the local Nextcloud instance, and serving the local dashboard. Dependencies must not require downloads at runtime.
+- NFR1 Privacy: fully offline on self-hosted infrastructure. No network calls except the local Garage bucket, and serving the local dashboard. Dependencies must not require downloads at runtime.
 - NFR2 Performance: end-to-end processing of a standard PDF under 5 seconds on the target server, including the S3 download; dashboard file list loads under 1 s for 10,000 records (served from SQLite).
 - NFR3 Reliability: zero silent wrong renames. Every ambiguous case routes to not processed.
 - NFR4 Durability: bucket versioning enabled; SQLite with WAL. A crash at any point leaves no half-renamed state: the S3 copy is a non-destructive copy-then-delete, the sidecar is written atomically, and the consistency job repairs SQLite-sidecar drift.
-- NFR5 Reversibility: every rename has an undo.
+- NFR5 Language: documents in French; UI in English or French (single locale acceptable in v1).
-- NFR6 Language: documents in French; UI in English or French (single locale acceptable in v1).
+- NFR6 Deployment: runs as a container next to the Garage deployment and the Nextcloud server; configuration via a single YAML file (templates) and environment variables (S3 endpoint, credentials, bucket).
-- NFR7 Deployment: runs as a systemd service or container next to the Garage deployment and the Nextcloud server; configuration via a single YAML file and environment variables (S3 endpoint, credentials, bucket, prefixes).
+- NFR7 Stack: Python (FastAPI, boto3 or minio S3 client, scikit-learn, pdfplumber/PyMuPDF, rapidfuzz, spaCy fr model optional), React (Vite, a lightweight PDF viewer such as react-pdf), SQLite, Garage for S3 storage, Nextcloud with External Storage (S3) as the user-facing file interface.
-- NFR8 Stack: Python (FastAPI, boto3 or minio S3 client, scikit-learn, pdfplumber/PyMuPDF, rapidfuzz, spaCy fr model optional), React (Vite, a lightweight PDF viewer such as react-pdf), SQLite, Garage for S3 storage, Nextcloud with External Storage (S3) as the user-facing file interface.
 - NFR9 No encryption at rest in v1: documents are not sensitive; this decision is recorded explicitly and can be revisited.
 
 ## 7. Data Model
 
 Sidecar JSON (source of truth), one per PDF at `{key}.meta.json`:
 
-- schema_version, sha256, original_key, current_key, status, template_id, confidence, emission_date, emitting_company, document_type, extracted_text, created_at, updated_at, events[] (type: classified / renamed / validated / undone / corrected, payload, timestamp).
+- schema_version, sha256, original_key, current_key, status, template_name, confidence, emission_date, emitting_company, document_type, optional_field1, optional_field2, extracted_text, created_at, updated_at, events[] (type: classified / renamed / validated / corrected, payload, timestamp).
 
 SQLite replica (rebuilt from sidecars):
 
-- files: id, sha256, original_key, current_key, status, template_id, confidence, emission_date, emitting_company, document_type, created_at, updated_at, extracted_text.
+- files: id, sha256, original_key, current_key, status, template_id, confidence, emission_date, emitting_company, document_type, optional_field1, optional_field2, created_at, updated_at, extracted_text, created_at, updated_at.
-- fields: id, file_id, slot_name, value, is_common (bool), was_corrected (bool), validated_by.
+- consistency_flags: id, file_id, kind (orphaned_object, missing_sidecar, drift).
-- events: id, file_id, type, payload_json, created_at.
-- templates: mirrored from YAML at load: id, name, document_types, key_pattern, target_prefix, common_fields_json, optional_fields_json.
-- models: id, trained_at, metrics_json, active (bool).
-- consistency_flags: id, file_id, kind (orphaned_object, missing_sidecar, drift), details_json, created_at, resolved_at.
 
 ## 8. API Surface (indicative)
 
 - GET /files?status=&template=&q=&page= — list with filters (from SQLite).
-- GET /files/{id} — detail: text, fields, candidates, current key, nextcloud_url.
+- GET /files/{id} — detail: text, fields, candidates, current key.
 - GET /files/{id}/pdf — stream the PDF from the bucket for the viewer.
 - POST /files/{id}/validate — body: template_id, fields; applies rename if needed, sets validated.
-- POST /files/{id}/undo — rolls back the rename.
 - GET /templates — configured templates.
 - POST /retrain — starts training; GET /retrain/status; POST /retrain/{id}/activate.
 - GET /consistency — list anomalies; POST /consistency/rebuild — rebuild SQLite from sidecars.
 
 ## 9. Acceptance Criteria (v1)
 
-- AC1 A new PDF dropped in `inbox/` via Nextcloud with high-confidence classification and valid fields appears as processed within the poll interval, renamed and moved in the bucket, and is undoable.
+- AC1 A new PDF dropped in `inbox/` via Nextcloud with high-confidence classification and valid fields appears as processed within the poll interval, renamed and moved in the bucket.
-- AC2 A PDF below the confidence threshold appears as not processed, is moved to `pending/`, and appears in Nextcloud under `pending/`; correcting and validating it on the detail page renames it and sets validated.
+- AC2 A PDF below the confidence threshold appears as not processed, is moved to `pending/`; correcting and validating it on the detail page renames it and sets validated.
 - AC3 Opening a processed file and validating without changes sets validated without re-renaming.
 - AC4 The file list filters by status, template, document type, emitting company, and date; counters match the data.
 - AC5 Changing the template on the detail page keeps the common fields, reloads the optional fields per the new template schema, and shows the live target key preview.
-- AC5b Every file in every state carries the four common fields (template, emission date, emitting company, document type); a document whose emitting company cannot be recognized is routed to not processed rather than renamed.
+- AC5b Every file in every state carries the four common fields (template, emission date, emitting company, document type); a document whose fields cannot be recognized is routed to not processed rather than renamed.
 - AC6 Retraining uses initial corpus + validated files, shows metrics, and activation swaps the model without service restart.
-- AC7 Adding a template via YAML (with example files) makes it available for classification and in the detail page dropdown, with no code change and no restart.
+- AC7 Adding a template via YAML (with example files) makes it available for classification and in the detail page dropdown, with no code change.
 - AC8 Killing the service mid-processing leaves no half-renamed object: the copy-then-delete order and the sidecar-first write guarantee a consistent state on restart, and the consistency job repairs SQLite.
 - AC9 Deleting the SQLite database entirely and running the rebuild command restores a fully functional dashboard from the sidecars alone.
 - AC10 No network traffic other than the local Garage bucket, local Nextcloud, and local dashboard serving (verifiable with a firewall audit).
 - AC11 1,000-file list view renders with filters in under 1 second.
-- AC12 Nextcloud sees renamed and moved files at their new location without reconfiguration; sharing a renamed file from Nextcloud keeps working.
+- AC12 Nextcloud sees renamed and moved files at their new location without reconfiguration.
 
 ## 10. Release Plan
 
-- v1 (this PRD): new files only, text-based French PDFs, three statuses, Garage S3 storage, sidecar source of truth with SQLite replica, Nextcloud as file interface, detail page with correction and validation, manual retrain, undo, consistency job.
+- v1 (this PRD): new files only, text-based French PDFs, three statuses, Garage S3 storage, sidecar source of truth with SQLite replica, Nextcloud as file interface, detail page with correction and validation, manual retrain, consistency job.
 - v2 candidates: OCR for scanned documents, backlog ingestion of historical files, automatic periodic retraining, multilingual documents, automatic counterparty learning from validated files, bucket notification hooks to replace polling, at-rest encryption if the document mix becomes sensitive.
 
 ## 11. Open Questions
@@ -199,5 +189,4 @@ SQLite replica (rebuilt from sidecars):
 - Choice between pdfplumber and PyMuPDF (license and performance tradeoff) — decide in Week 1 spike.
 - S3 client library: boto3 vs minio-py against Garage — covered by the Week 1 Garage compatibility spike (listing, copy-object, multipart, versioning behavior).
 - Exact confidence threshold and per-template overrides — calibrate on the initial corpus.
-- Whether Nextcloud mounts the whole bucket or one mount per prefix (inbox vs template prefixes) — depends on the Nextcloud External Storage S3 mount semantics; decide during Week 3 integration.
 - Poll interval tradeoff between responsiveness and Garage listing load.