Project Brief — Automatic PDF Renamer

1. Vision

Document identity: the SHA-256 digest of the exact PDF bytes is the unique identifier. Any byte-level change to a PDF creates a new document identity; moving or renaming an unchanged PDF does not. Two imports with identical PDF bytes resolve to the same document identity, not two independent document records.

A personal, 100% offline tool that watches an S3 bucket hosted on a self-hosted Garage storage cluster, automatically identifies the type of each incoming PDF (invoice, purchase order, contract, etc.) from its content, applies the corresponding naming template by extracting the fields (date, counterparty, reference, amount), and moves the object into an organized S3 prefix structure. Users upload, download, and share files through Nextcloud, which mounts the bucket as external storage. Uncertain or incomplete results go through human review in a local web dashboard. Every operation is logged in JSON sidecar files (source of truth) and replicated to SQLite.

2. Problem and Context

PDFs arrive continuously in an inbox S3 prefix (dropped via Nextcloud by the user or by upload automations). Current naming is manual or inconsistent, making documents hard to find and archive. The user owns around 500 files already correctly named according to 4 existing templates, which will serve as training and reference corpus.

3. Objectives

Extract the emitting company, document type, additional-field values, and emission date from the full text content of every page (including headers and footers), not from the current object name.
Use ML to select additional-field groups from emitting company and document type, then use one shared token-level extraction model to identify all requested field values in the text. Independently use ML to predict template and period format; do not use predefined mappings for these decisions.
Calculate the period and storage key only after the required extracted fields and configured mapping have been validated.
Rename and move the object within the bucket into an organized prefix structure depending on the document type.
Unambiguous deterministic rules and valid fields: automatic processing. If rules cannot determine a valid result or extraction is incomplete, queue the file with status pending for human review in the dashboard.
Write metadata as JSON sidecar objects in the bucket (source of truth) and replicate every decision into SQLite for fast queries, with retraining from corrections.
Run fully offline on self-hosted infrastructure: no cloud APIs, no data ever leaves the server.

4. Scope

In scope (v1)

Ingestion by polling the inbox/ prefix of a Garage S3 bucket for new objects only. Nextcloud (External Storage app) is the user-facing interface to upload, download, and share files.
Extract all available text from every page of each PDF. All files are text-based; OCR for scanned/image-only documents is out of scope for v1.
Deterministic template classification in v1 using configured rules, extracted text, and reference data; uncertain or insufficient results are routed to pending/ for manual correction. TF-IDF + Logistic Regression is a post-v1 enhancement, not a v1 dependency.
Per-template slot filling: regex + dateutil for dates, amounts, references; gazetteer + fuzzy matching (rapidfuzz) for counterparties, with spaCy/CamemBERT NER as fallback if needed.
Templates and extractors declared in configuration (JSONC format), not in code.
Storage: Garage (self-hosted S3-compatible, https://garagehq.deuxfleurs.fr/) with a single versioned bucket; objects addressed by S3 keys, no local filesystem for documents.
Metadata: one JSON sidecar object per PDF in the bucket, holding template, fields, confidence, status, and history — the source of truth. SQLite acts as a replicated index for dashboard queries, rebuildable from the sidecars.
Local web dashboard: all processed files with status (pending / processed / validated), file detail page with PDF preview streamed from S3, template and field correction, validation, history, and full-text search across the entire extracted document in addition to metadata/field search.
On-demand classifier retraining, using the existing corpus based on all the files with "validated" status.

Out of scope (v1)

OCR of scanned documents (planned for v2 if the mix changes).
Backlog processing of existing files — new files only.
Cloud S3 providers, remote language models, cloud APIs, telemetry.
At-rest encryption: the documents are not sensitive; bucket and volumes are unencrypted in v1.
Multi-user management, authentication on the dashboard (Nextcloud handles sharing and access control on files).
Merging, splitting, or modifying PDF content.

5. User and Usage

Single user (personal use), technically proficient. Nominal flow:

A PDF is uploaded into the inbox/ prefix via Nextcloud (or any S3 client).
The poller detects the new object, waits for write completion (stable ETag/content-length on repeated listing), downloads it, extracts text from every page, applies deterministic classification and field-extraction rules, then validates.
A deterministic result with valid fields: the object is copied to its renamed key in the target prefix; the sidecar is updated, then SQLite is synchronized, and the inbox copy is deleted only after the sidecar records the new key.
If deterministic rules cannot produce a valid result: the file is moved to pending/ and appears in the dashboard review queue; the user corrects or validates it. Validated corrections may feed a future ML training corpus.

6. Functional Requirements

FR1: detect new PDFs in a configurable inbox/ S3 prefix by polling the bucket listing at a configurable interval, ignoring non-PDF keys and temporary uploads (e.g. Nextcloud .part files).
FR2: classify the document using configured deterministic rules and reference data. If the evidence is insufficient or contradictory, do not guess; set status pending and queue the file for manual correction. Confidence scores and an "unknown" class belong to the post-v1 ML enhancement.
FR3: fill each template field according to its schema (type, format, validity range).
FR4: validate fields before renaming (parseable date, numeric amount, recognized or flagged counterparty); any failed validation routes to review.
FR5: rename and move within the bucket via S3 copy + delete to the template's target prefix, resolving key collisions (incremental suffix).
FR6: write a JSON sidecar per file (key mapping, decisions, and business status) and replicate to SQLite, enabling auditability and selecting training data.
FR7: local web dashboard (FastAPI + React): file list, detail page, review, correction, validation, history, retraining trigger. PDFs are streamed from the bucket for preview.
FR7a: the dashboard shows each document's status: pending (deterministic classification or required-field validation needs manual correction), processed (automatically renamed and moved), or validated (the user confirmed or corrected the result — eligible for future model retraining).
FR7b: clicking a file opens a detail page displaying the PDF, where the user can change the template if misclassified, correct the extracted field values, and validate the result. Validation sets the status to "validated" and feeds the corrected data into the training corpus.
FR8: adding a new template is done by editing the JSONC configuration file plus example files, with no code changes. Dedicated endpoints retrieve the current configuration file and upload a validated modified version.

7. Non-Functional Requirements

Privacy: 100% offline on self-hosted infrastructure (Garage). No data ever leaves the server.
Encryption: none at rest in v1; documents are not sensitive. Network access to the bucket stays on the local network.
Performance: a standard-size PDF processed in under a few seconds, including the S3 download.
Reliability: zero silent invalid renames — every ambiguous case goes to review.
Document language: French only in v1.
Stack: Python (FastAPI) backend, boto3-compatible S3 client (Garage), React frontend, SQLite, scikit-learn, spaCy/fr, pdfplumber or PyMuPDF, rapidfuzz. Nextcloud with External Storage (S3) as the user-facing file interface.
Portability: containerized deployment, alongside the Garage deployment.

8. Success Criteria

≥ 95% correct template classification on typical files after initial training on the existing corpus.
≥ 80% of files processed without human intervention (the rest in review, never a silent wrong rename).
Every rename is logged and traceable; no user-facing undo or rollback of file renames is required.
Adding a new template in under 30 minutes without writing code.
SQLite can be fully rebuilt from the bucket sidecars at any time.

9. Risks and Mitigations

Limited training corpus (~500 files): start with deterministic rules; evaluate TF-IDF + Logistic Regression as a post-v1 enhancement; the review loop continuously enriches the corpus.
Templates too weakly distinguished in the text: enrich features (header, issuer, business keywords per template).
Poorly extracted or empty text files: systematic detection and routing to review; extraction and search must cover all pages, not only page one.
Partial uploads (Nextcloud chunked uploads): poll-only keys with a stable size across two listing cycles and a .part/.~lock exclusion list.
Sidecar/object drift (sidecar missing or out of sync with SQLite): sidecars win; a consistency job rebuilds SQLite and flags orphaned or missing sidecars.
Garage version of S3 API: pin a Garage release and run a compatibility spike (listing, multipart, copy-object, versioning).

Dashboard list usability and bulk editing (change proposal)

The home page is a configurable document table. Its default visible columns, in order, are Status, Document (current filename), Company, Document type, Period, and Last modified. “Last modified” is the document metadata's latest modification timestamp, not the PDF emission date or filesystem creation time.

Users can show or hide available columns, including any configured additional field. Column configuration affects presentation only: it must not alter metadata or filters. Additional-field columns are labelled from their configured field definitions and render missing values clearly. The list retains existing filtering, search, sorting, and consistency indicators.

Each row has a selection control. Users can select individual visible rows, select all rows on the current page, or select all rows matching the active filters across pagination. The UI must state which selection mode is active and the number of affected documents. A bulk-edit action applies explicitly chosen changes to the selected set: company, document type, and configured additional fields (set, replace, or clear a value). Unchanged fields are preserved. The user reviews the proposed changes and affected count before confirmation. Validation failures and per-document outcomes are reported; one invalid document must not silently hide failures for the others.

Bulk edits use the existing metadata and validation rules, recompute derived period and destination key when relevant fields change, and safely rename/move objects when the target key changes. Sidecars remain authoritative; SQLite is synchronized after durable sidecar updates. Every change is auditable in the document event history. The feature must not silently invent company/type values or overwrite colliding objects.

12 KiB Raw Permalink Blame History