From 732ba6de0a3d03a1ae84f2069a68f10e699ccd39 Mon Sep 17 00:00:00 2001 From: napnap75 Date: Thu, 1 Oct 2026 14:14:00 +0200 Subject: [PATCH] Initial project specifications for PDF renamer (#7) --- automatic-pdf-renamer/brief.pdf | 93 +++++++++++++++++++ automatic-pdf-renamer/prd.pdf | 159 ++++++++++++++++++++++++++++++++ 2 files changed, 252 insertions(+) create mode 100644 automatic-pdf-renamer/brief.pdf create mode 100644 automatic-pdf-renamer/prd.pdf diff --git a/automatic-pdf-renamer/brief.pdf b/automatic-pdf-renamer/brief.pdf new file mode 100644 index 0000000..7117f67 --- /dev/null +++ b/automatic-pdf-renamer/brief.pdf @@ -0,0 +1,93 @@ +# Project Brief — Automatic PDF Renamer + +## 1. Vision + +A personal, 100% offline tool that watches a folder on a server, automatically identifies the type of each incoming PDF (invoice, purchase order, contract, etc.) from its content, applies the corresponding naming template by extracting the fields (date, counterparty, reference, amount), and moves the file into an organized folder structure. Low-confidence decisions go through human review in a local web dashboard. Every operation is logged and reversible. + +## 2. Problem and Context + +PDFs arrive continuously in a watched folder (emails, downloads, exports). Current naming is manual or inconsistent, making documents hard to find and archive. The user owns 50 to 500 files already correctly named according to 1 to 5 existing templates, which will serve as training and reference corpus. + +## 3. Objectives + +- Classify each incoming PDF against a known naming template, based on the file content (first-page text, header/footer), not the current filename. +- Fill the fields of the selected template (date, counterparty, reference, amount) from the content. +- Rename and move the file into an organized tree structure depending on the document type. +- High confidence: automatic processing. Low confidence or incomplete extraction: queued for human review in the dashboard. +- Log every decision (SQLite) with rollback capability and retraining from corrections. +- Run fully offline, with no cloud APIs whatsoever. + +## 4. Scope + +### In scope (v1) + +- Folder watching (watcher) for new files only. +- PDF text extraction (pdfplumber or PyMuPDF). All files are text-based, no OCR in v1. +- Template classification: TF-IDF + Logistic Regression (scikit-learn) on the extracted text, with a configurable confidence threshold. +- Per-template slot filling: regex + dateutil for dates, amounts, references; gazetteer + fuzzy matching (rapidfuzz) for counterparties, with spaCy/CamemBERT NER as fallback if needed. +- Templates and extractors declared in configuration (YAML/JSON), not in code. +- SQLite database: rename history, decisions, confidence scores, rollback target. +- Local web dashboard: all processed files with status (not processed / processed / validated), file detail page with PDF preview, template and field correction, validation, history, statistics. +- On-demand classifier retraining, using the existing corpus plus every file with "validated" status. + +### Out of scope (v1) + +- OCR of scanned documents (planned for v2 if the mix changes). +- Backlog processing of existing files — new files only. +- Remote language models, cloud APIs, telemetry. +- Multi-user management, authentication. +- Merging, splitting, or modifying PDF content. + +## 5. User and Usage + +Single user (personal use), technically proficient. Nominal flow: + +1. A PDF arrives in the watched folder. +2. The watcher waits for the write to finish, extracts the text, classifies, extracts the fields, validates. +3. High confidence and valid fields: the file is renamed and moved; the event is logged. +4. Otherwise: the file appears in the dashboard review queue; the user corrects or validates; the correction feeds the training corpus. + +## 6. Functional Requirements + +- FR1: detect new PDFs in a configurable folder, waiting for file stability. +- FR2: classify the document into one of the configured templates, with a confidence score and an "unknown" class below the threshold. +- FR3: fill each template field according to its schema (type, format, validity range). +- FR4: validate fields before renaming (parseable date, numeric amount, recognized or flagged counterparty); any failed validation routes to review. +- FR5: rename and move to the template's target folder, resolving collisions (incremental suffix). +- FR6: log every operation in SQLite with an old name → new name → decision mapping, and per-file status (not processed / processed / validated), enabling rollback and selecting training data. +- FR7: local web dashboard (FastAPI + React): file list, detail page, review, correction, validation, history with undo, retraining trigger. +- FR7a: the dashboard shows every processed file with its status: "not processed" (confidence was too low, rename pending), "processed" (automatically renamed and moved), or "validated" (the user confirmed or corrected the result — such files are eligible for future model retraining). +- FR7b: clicking a file opens a detail page displaying the PDF, where the user can change the template if misclassified, correct the extracted field values, and validate the result. Validation sets the status to "validated" and feeds the corrected data into the training corpus. +- FR8: adding a new template is done through a configuration entry (YAML) plus example files, with no code changes. + +## 7. Non-Functional Requirements + +- Privacy: 100% offline, no data ever leaves the server. +- Performance: a standard-size PDF processed in under a few seconds. +- Reliability: zero silent invalid renames — every ambiguous case goes to review. +- Reversibility: every rename can be undone from the history. +- Document language: French only in v1. +- Stack: Python (FastAPI) backend, React frontend, SQLite, scikit-learn, spaCy/fr, pdfplumber or PyMuPDF, rapidfuzz, watchdog. +- Portability: containerized deployment or systemd service on the user's server. + +## 8. Success Criteria + +- ≥ 95% correct template classification on typical files after initial training on the existing corpus. +- ≥ 80% of files processed without human intervention (the rest in review, never a silent wrong rename). +- Zero non-reversible renames. +- Adding a new template in under 30 minutes without writing code. +- Working v1 in under one month. + +## 9. Risks and Mitigations + +- Limited training corpus (50–500 files): start simple (TF-IDF + LogReg), optionally sentence-transformers embeddings if accuracy is insufficient; the review loop continuously enriches the corpus. +- Templates too weakly distinguished in the text: enrich features (header, issuer, business keywords per template). +- Poorly extracted or empty text files: systematic detection and routing to review. +- Sensitive files moved by mistake: high default confidence threshold + strict validation + rollback. + +## 10. Milestones + +1. Week 1: text extraction, parsing the existing corpus into (template, fields), declarative template configuration. +2. Week 2: classifier + extractors + validation + SQLite logging + watcher. +3. Week 3: React dashboard (file list with statuses, detail page with PDF preview and correction, validation, history, rollback, retraining). +4. Week 4: hardening (thresholds, collisions, tests), deployment, production go-live on the watched folder. diff --git a/automatic-pdf-renamer/prd.pdf b/automatic-pdf-renamer/prd.pdf new file mode 100644 index 0000000..ed66aef --- /dev/null +++ b/automatic-pdf-renamer/prd.pdf @@ -0,0 +1,159 @@ +# PRD — Automatic PDF Renamer + +Product Requirements Document. Derived from the project brief (BMAD). Personal tool, single user, 100% offline. + +## 1. Product Overview + +A server-side service with a local web dashboard that watches a folder for incoming PDFs, identifies the document type from its content, applies the matching naming template, fills the template fields, renames and moves the file into an organized folder tree, and tracks every file through a three-state lifecycle (not processed / processed / validated). Low-confidence or invalid files wait for human correction; validated files feed future model retraining. + +## 2. Personas + +Single user: the owner. Technically proficient, wants minimal manual work but zero silent errors. + +## 3. Document Lifecycle and Statuses + +| Status | Meaning | +|---|---| +| not processed | Classification confidence below threshold, or field validation failed. File untouched in the watched folder (or a pending subfolder), waiting for review. | +| processed | Automatically classified, fields validated, renamed and moved. Eligible for undo. | +| validated | The user reviewed the result (confirmed or corrected template/fields). Eligible for model retraining. Terminal state for the review flow; the file was renamed and moved. | + +Transitions: + +- new file → not processed (low confidence or invalid fields) +- new file → processed (high confidence, valid fields) +- not processed → validated (user corrects and validates in the detail page; rename then applied) +- processed → validated (user opens the detail page and confirms; no changes needed) +- processed or validated → previous state restorable via undo (rollback of the rename) + +## 4. Functional Requirements + +### Epic 1 — Ingestion and watching + +- FR1.1 Watch a configurable folder for new PDF files only. Ignore other extensions and temporary files (e.g. `.tmp`, partial downloads). +- FR1.2 Wait for file stability (size unchanged over a configurable interval) before processing. +- FR1.3 On service restart, detect PDFs that arrived while the service was down but were never processed (no record in the database). +- FR1.4 Configurable poll interval or OS-level file system events (watchdog). + +### Epic 2 — Text extraction + +- FR2.1 Extract text with pdfplumber or PyMuPDF (choice fixed at implementation, documented in config). +- FR2.2 Extract the full first page plus header and footer zones of all pages. +- FR2.3 If extracted text is empty or below a minimum length, route the file to not processed with reason "empty text". +- FR2.4 Store the extracted text in the database for the file's lifetime (enables detail page display and retraining without re-parsing). + +### Epic 3 — Template classification + +- FR3.1 Classify the document into one of the configured templates using TF-IDF + Logistic Regression on the extracted text. +- FR3.2 Output a confidence score per template; the top score below a configurable threshold (default 0.8) yields "unknown" and routes the file to not processed with reason "low confidence". +- FR3.3 Display the top-N template candidates with scores on the detail page. +- FR3.4 Retrain the classifier on demand from: the initial corpus + all files with validated status. Retraining is a manual action from the dashboard, with the training date and resulting evaluation metrics stored and displayed. + +### Epic 4 — Document data model, slot filling and validation + +#### 4.1 Common fields (mandatory for every document) + +Every document, regardless of template, carries at least these four common fields: + +- Common field 1 — Template: the selected template (i.e. the document family: invoice, purchase order, contract, bank statement, rent receipt, etc.). +- Common field 2 — Emission date: the date the document was emitted, extracted from its content (not the file modification date). +- Common field 3 — Emitting company: the company or organization that emitted the document, matched against the counterparty gazetteer. +- Common field 4 — Document type: the fine-grained type of document within the family (e.g. debit advice vs monthly statement for bank statements; quotes vs order confirmations for purchase documents). + +The filename pattern of each template is composed from these common fields, plus the template's optional fields. + +#### 4.2 Optional fields (template-specific) + +- FR4.1 Each template declares in YAML its optional fields beyond the common four: name, type, format, extraction rules, validation rules, and whether the field is rendered in the filename, the target folder path, or stored only. Examples: bank statements carry account owner and statement type; rent receipts carry the concerned premises; contracts carry the counterparty signatory. +- FR4.2 Fill common and optional fields using per-template regexes, dateutil for dates, and a counterparty gazetteer with rapidfuzz matching for the emitting company (similarity threshold configurable; below threshold the company is "unrecognized"). +- FR4.3 Validate fields before renaming: emission date parses and is within a plausible range, emitting company recognized, document type among the template's allowed values, each optional field matches its declared format (empty optional fields are allowed). Any failure routes to not processed with the failing fields identified. +- FR4.4 Collisions in the target folder are resolved with an incremental suffix; the final name is always the one logged. + +### Epic 5 — Renaming, moving, logging, undo + +- FR5.1 Rename and move the file to the template's target folder according to its filename pattern. +- FR5.2 Every event is logged in SQLite: file hash, old path, new path, template, extracted fields, confidence, decision, status, timestamps. +- FR5.3 Undo restores the previous path and sets the status back to its prior value, keeping the full audit trail. +- FR5.4 Deleting or moving a file on disk that is not in a terminal state is detected and the record is flagged as orphaned. + +### Epic 6 — Dashboard: file list + +- FR6.1 A single list shows every file the system has seen, with columns: emission date, current filename, status, template (document family), document type, emitting company, confidence. +- FR6.2 Filter by status, template, document type, emitting company, and date range; sort by any column; free-text search on filename and emitting company. +- FR6.3 Status is visually distinguishable (color or badge): not processed, processed, validated. +- FR6.4 The list shows counters: files per status, automation rate (processed / total), validation rate. + +### Epic 7 — Dashboard: file detail page + +- FR7.1 Clicking a file in the list opens a detail page. +- FR7.2 The detail page displays the PDF (embedded viewer) alongside the extracted data. +- FR7.3 The user can change the template (dropdown of configured templates, showing the classifier's top candidates and scores). +- FR7.4 The detail page always shows the four common fields (template, emission date, emitting company, document type) plus the current template's optional fields; the user can correct each value inline; changing the template reloads the optional fields with the new template's schema and best-effort re-extraction, keeping the common field values. +- FR7.5 A preview shows the target filename and destination folder resulting from the current values, updating live as fields change. +- FR7.6 A validate action applies the rename/move (if not already done), sets the status to validated, and records the corrected values as training data. +- FR7.7 An undo action (for processed or validated files) rolls back the rename. + +### Epic 8 — Template management + +- FR8.1 Templates are defined in YAML configuration files, hot-reloaded without service restart. +- FR8.2 Adding a template = YAML entry + example files; no code change. +- FR8.3 The dashboard lists configured templates with their slot definitions and the count of corpus files per template. + +### Epic 9 — Retraining + +- FR9.1 Manual "Retrain" action in the dashboard: trains on initial corpus + validated files, shows evaluation metrics (accuracy, confusion matrix) before activation. +- FR9.2 The user can activate or discard the new model. Only one model is active at a time; model files are versioned on disk. + +## 5. Non-Functional Requirements + +- NFR1 Privacy: fully offline. No network calls except serving the local dashboard. Dependencies must not require downloads at runtime. +- NFR2 Performance: end-to-end processing of a standard PDF under 5 seconds on the target server; dashboard file list loads under 1 s for 10,000 records. +- NFR3 Reliability: zero silent wrong renames. Every ambiguous case routes to not processed. +- NFR4 Durability: SQLite with WAL; a crash at any point leaves no half-renamed state (rename is atomic on the same filesystem; cross-filesystem moves use copy+verify+delete). +- NFR5 Reversibility: every rename has an undo. +- NFR6 Language: documents in French; UI in English or French (single locale acceptable in v1). +- NFR7 Deployment: runs as a systemd service or container; configuration via a single YAML file and environment variables. +- NFR8 Stack: Python (FastAPI, watchdog, scikit-learn, pdfplumber/PyMuPDF, rapidfuzz, spaCy fr model optional), React (Vite, a lightweight PDF viewer such as react-pdf), SQLite. + +## 6. Data Model (indicative) + +- files: id, sha256, original_path, current_path, status, template_id, confidence, emission_date, emitting_company, document_type, created_at, updated_at, extracted_text. +- fields: id, file_id, slot_name, value, is_common (bool), was_corrected (bool), validated_by. +- events: id, file_id, type (classified, renamed, validated, undone), payload_json, created_at. +- templates: mirrored from YAML at load: id, name, document_types, pattern, target_folder, common_fields_json, optional_fields_json. +- models: id, trained_at, metrics_json, active (bool). + +## 7. API Surface (indicative) + +- GET /files?status=&template=&q=&page= — list with filters. +- GET /files/{id} — detail: text, fields, candidates, current path. +- GET /files/{id}/pdf — stream the PDF for the viewer. +- POST /files/{id}/validate — body: template_id, fields; applies rename if needed, sets validated. +- POST /files/{id}/undo — rolls back the rename. +- GET /templates — configured templates. +- POST /retrain — starts training; GET /retrain/status; POST /retrain/{id}/activate. + +## 8. Acceptance Criteria (v1) + +- AC1 A new PDF with high-confidence classification and valid fields appears as processed within the poll interval, renamed and moved correctly, and is undoable. +- AC2 A PDF below the confidence threshold appears as not processed; correcting and validating it on the detail page renames it and sets validated. +- AC3 Opening a processed file and validating without changes sets validated without re-renaming. +- AC4 The file list filters by status, template, document type, emitting company, and date; counters match the data. +- AC5 Changing the template on the detail page keeps the common fields, reloads the optional fields per the new template schema, and shows the live target filename preview. +- AC5b Every file in every state carries the four common fields (template, emission date, emitting company, document type); a document whose emitting company cannot be recognized is routed to not processed rather than renamed. +- AC6 Retraining uses initial corpus + validated files, shows metrics, and activation swaps the model without service restart. +- AC7 Adding a template via YAML (with example files) makes it available for classification and in the detail page dropdown, with no code change and no restart. +- AC8 Killing the service mid-processing leaves no half-renamed file; restart recovers and re-processes pending files. +- AC9 No network traffic other than local dashboard serving (verifiable with a firewall audit). +- AC10 1,000-file list view renders with filters in under 1 second. + +## 9. Release Plan + +- v1 (this PRD): new files only, text-based French PDFs, three statuses, detail page with correction and validation, manual retrain, undo. +- v2 candidates: OCR for scanned documents, backlog ingestion of historical files, automatic periodic retraining, multilingual documents, automatic counterparty learning from validated files. + +## 10. Open Questions + +- Choice between pdfplumber and PyMuPDF (license and performance tradeoff) — decide in Week 1 spike. +- Whether not-processed files stay in the watched folder or move to a pending subfolder (recommendation: pending subfolder to avoid re-processing loops). +- Exact confidence threshold and per-template overrides — calibrate on the initial corpus.