multiarch-docker-images/automatic-pdf-renamer/prd.md

# PRD — Automatic PDF Renamer

Product Requirements Document. Derived from the project brief (BMAD). Personal tool, single user, 100% offline. Storage is a self-hosted Garage (S3-compatible) bucket; Nextcloud is the user-facing file interface; JSON sidecars are the source of truth; SQLite is a replicated index.

## 1. Product Overview

A server-side service with a local web dashboard that polls an `inbox/` prefix of a Garage S3 bucket for incoming PDFs, identifies the document type from its content, applies the matching naming template, fills the template fields, moves the object to an organized prefix tree within the bucket, and tracks every file through a three-state lifecycle (`pending` / `processed` / `validated`). Metadata lives in per-file JSON sidecars (source of truth), replicated to SQLite for queries. Low-confidence or invalid files wait for human correction; validated files feed future model retraining. Users upload, download, and share documents through Nextcloud, which mounts the bucket as external storage.

## 2. Personas

Single user: the owner. Technically proficient, wants minimal manual work but zero silent errors.

## 3. Storage and Metadata Architecture

### 3.1 Object storage

- Garage ([https://garagehq.deuxfleurs.fr/](https://garagehq.deuxfleurs.fr/)), self-hosted, S3 API. One versioned bucket, e.g. `pdf-renamer`.
- Key layout: `inbox/` (entry point), `pending/` (files awaiting manual correction, moved out of inbox to avoid re-processing), and `files/` for the processes files (validated or not).
- Sidecars: `{object-key}.meta.json` next to each PDF.
- No encryption at rest in v1 (documents not sensitive); bucket served on the local network only.
- Renames are copy-object + delete within the bucket; collisions resolved with an incremental suffix.

### 3.2 Metadata: sidecars as source of truth, SQLite as replica

- Each PDF has a sidecar JSON object holding: sha256, original key, current key, status, template, document type, emission date, period, emitting company, all optional fields, confidence, and the full event history (classified, renamed, validated, corrected).
- Sidecar is updated atomically on every state change: write the new sidecar to a temp key, copy over the final sidecar key.
- SQLite mirrors the sidecar content in queryable tables for the dashboard (lists, filters, search, and consistency diagnostics). It is a cache/replica, never authoritative. Any sidecar↔SQLite divergence is resolved in favor of the sidecar.
- A rebuild command reconstructs the full SQLite database by scanning all sidecars; a consistency job detects PDFs without sidecars (orphaned) and sidecars without PDFs (missing object) and flags them in the dashboard.

### 3.3 Nextcloud integration

- Nextcloud connects to the bucket via the External Storage (S3/Swift) app, mounted read-write on `inbox/` and read-only on other prefixes, giving upload, download, preview, and sharing without touching the dashboard.
- The renamer service uses its own S3 credentials distinct from Nextcloud's; Nextcloud users never need renamer credentials.
- The renamer never touches Nextcloud's own storage or database; the only shared surface is the bucket.

## 4. Document Lifecycle and Statuses


| Status        | Meaning                                                                                                                                                                     |
| ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| pending | Deterministic rules cannot determine a valid classification or required field, or validation fails. Object moved to the `pending/` prefix, waiting for review.                                           |
| processed     | Automatically classified, fields validated, renamed and moved.                                                                                                              |
| validated     | The user reviewed the result (confirmed or corrected template/fields). Eligible for model retraining. Terminal state for the review flow; the object was renamed and moved. |


Transitions:

- new object in `inbox/` → pending (deterministic rules cannot determine a valid result; object copied to `pending/`)
- new object in `inbox/` → processed (deterministic rules yield an unambiguous result and all required fields are valid)
- pending → validated (user corrects and validates in the detail page; rename then applied)
- processed → validated (user confirms or corrects and validates in the detail page; rename only if the user maid some changes)

## 5. Functional Requirements

### Epic 1 — Ingestion

- FR1.1 Poll the configurable `inbox/` prefix at a configurable interval; process only keys ending in `.pdf`. Ignore other extensions and temporary files (Nextcloud `.part`, `.~lock`, hidden files).
- FR1.2 Wait for object stability: process a key only after its size and ETag are unchanged across two consecutive listings (handles chunked uploads and partial writes).
- FR1.3 On service restart, recover eligible objects in `inbox/` and resume interrupted technical operations. Do not automatically reprocess a `pending` document that is awaiting manual correction unless the user requests processing after correction.
- FR1.4 Optionally reduce latency with bucket notification hooks if the Garage deployment supports them; polling is the v1 baseline.
- FR1.5 After successful processing, the inbox copy is deleted. Documents requiring manual correction are copied to `pending/` and removed from `inbox/`. Transient technical failures remain eligible for retry/recovery and must not be mislabeled as `pending`.

### Epic 2 — Text extraction

- FR2.1 Download the object from S3 and extract text with pdfplumber or PyMuPDF (choice fixed at implementation, documented in config).
- FR2.2 Extract all available text from every page of each PDF; extraction must not stop after the first page. OCR of scanned/image-only documents remains out of scope for v1.
- FR2.3 If extracted text is empty or below a minimum length, route the file to `pending` with reason "empty text".
- FR2.4 Store the complete extracted text from all pages in the sidecar and replicate it to SQLite for the file's lifetime, so search, the detail page, and retraining do not need to re-download and re-parse the object.

### Epic 3 — ML-assisted field extraction and classification

- FR3.1 Process the full extracted document text in ordered stages. Stage 1 extracts `emitting_company`, `document_type`, `additional_fields`, and `emission_date`; emission date extraction runs in the same stage as the other text-derived fields.
- FR3.2 Use local ML to predict applicable additional-field groups from `(emitting_company, document_type)`, then use one shared local token-level extraction model to extract all selected values from document text. No predefined mapping selects these groups. Do not select or consume the template and period format before this extraction stage completes.
- FR3.3 Stage 2 resolves `template_name` and `period_format` from the configured mapping using the first-stage fields. Stage 3 calculates `period` from `emission_date` and `period_format`, then generates and validates the target storage key.
- FR3.4 Ambiguous or missing indispensable values, unknown values, unresolved mappings, and invalid calculations must route to `pending`; the pipeline must never substitute a default company or document type or fabricate a candidate.
- FR3.5 ML is an assistive candidate-proposal/ranking layer, not an authority that bypasses configuration. Deterministic rules and configured allow-lists remain available as fallback and validation boundaries. ML must be disabled or unavailable without breaking the rules-only path. Exact model strategy and confidence thresholds remain an explicit calibration decision.
- FR3.6 Retrain on demand from the initial labeled corpus plus eligible validated files. Record the training date and evaluation metrics in the local model registry; require evaluation before activation.

### Epic 4 — Document data model, slot filling and validation

#### 4.1 Common fields (mandatory for every document)

Every document, regardless of template, carries at least these five common fields:

- Common field 1 — Template: the selected template (i.e. the document family: invoice, purchase order, contract, bank statement, rent receipt, etc.).
- Common field 2 — Emission date: the date the document was emitted, extracted from its content (not the object modification date).
- Common field 3 — Emitting company: the company or organization that emitted the document, matched against the companies list in configuration.
- Common field 4 — Document type: the fine-grained type of document within the family (e.g. debit advice vs monthly statement for bank statements; quotes vs order confirmations for purchase documents).
- Common field 5 u2014 Period: the period the document refers to, in human-readable form, derived from the emission date according to the period format configuration. The period format is selected per (company, document_type) mapping and includes:
  - Granularity: `day`, `week`, `month`, `quarter`, `semester`, or `year`
  - Format: custom template string with placeholders (e.g., `{month_name} {year}`, `Q{quarter} {year}`)
  - Offset: `current` (date in middle of period), `previous` (arrives after period), `following` (arrives before period)

The object key pattern of each template is composed from these common fields, plus the template's optional fields.

#### 4.2 Optional fields (template-specific)\n\n**Required processing order:** (1) extract emitting company, document type, additional-field values, and emission date from text; (2) resolve template and period format using the first-stage fields and configured mapping; (3) calculate period and storage key. The first stage must not select a default class. If an indispensable field is missing or ambiguous, route the document to `pending`.

- FR4.1 Each template declares in JSONC its optional fields beyond the common five. Additional fields are organized in predefined field groups (e.g., `bank_accounts`, `properties`, `contract_references`, `customer_numbers`) with their possible values. The (company, document_type) mapping specifies which field groups to extract for each document. Examples: bank statements extract from `bank_accounts` group; rent receipts extract from `properties` group.
- FR4.2 Extract `emitting_company`, `document_type`, `additional_fields`, and `emission_date` from the document text in the first processing stage. Predict additional-field groups with a local ML model from `(emitting_company, document_type)`, then extract their values using one shared token-level model trained from validated field spans in extracted text. No predefined mapping or allow-list determines these decisions. Only after this stage resolves required fields may the system select `template_name` and `period_format`. Compute `period` server-side from `emission_date` and the selected period format, then generate the storage key from the validated fields and template pattern. Recompute period and key whenever a relevant source field or mapping changes; period is not client-editable.
- FR4.3 Validate fields before renaming: emission date parses and is within a plausible range, period is non-empty and consistent with the emission date and the template's granularity/offset, emitting company recognized, document type among the template's allowed values, each optional field matches its declared format (empty optional fields are allowed). Any failure routes to `pending` with the failing fields identified.
- FR4.4 Collisions in the target prefix are resolved with an incremental suffix; the final key is always the one logged.

### Epic 5 — Renaming, moving, logging

- FR5.1 Rename and move the object within the bucket using a recoverable sequence: copy to the target key, persist the sidecar with the new current key and event, delete the source key only after the sidecar is durable, then synchronize SQLite. These stores do not share a transaction. A crash or replica failure is recovered by the consistency job. If a transient technical operation fails, retry it safely for up to 3 times after the initial attempt, with delays of 5 seconds, 30 seconds, and 2 minutes. Do not assign business status `pending` solely because of technical failure.
- FR5.2 Every event is logged in the sidecar history: file hash, old key, new key, template, extracted fields, confidence, decision, status, timestamps.
- FR5.3 Objects missing from the bucket while not in a terminal state are detected by the consistency job and the record is flagged as orphaned.

### Epic 6 — Dashboard: file list

- FR6.1 A single list shows every file the system has seen, with columns: emission date, period, current object name, status, template (document family), document type, emitting company, and confidence where applicable. Served from SQLite.
- FR6.2 Filter by status, template, document type, emitting company, period, and date range; sort by any column; free-text search on object name, metadata fields, and the complete extracted text from every page of the PDF.
- FR6.3 Status is visually distinguishable (badge): `pending`, `processed`, `validated`.
- FR6.4 The list exposes a consistency indicator for known anomalies (orphaned objects, missing sidecars), without a separate statistics dashboard or aggregate KPI counters.

### Epic 7 — Dashboard: file detail page

- FR7.1 Clicking a file in the list opens a detail page.
- FR7.2 The detail page displays the PDF (embedded viewer), streamed from the bucket by the backend, alongside the extracted data.
- FR7.3 The user can change the template (dropdown of configured templates, showing the classifier's top candidates and scores).
- FR7.4 The detail page always shows the five common fields (template, emission date, emitting company, document type, period) plus the current template's optional fields; the user can correct editable values inline. Changing the template reloads the optional fields with the new template's schema and best-effort re-extraction, keeps the other common field values, and recomputes the period from the emission date and the selected rules. The period is read-only, calculated by the server, and never accepted as user input.
- FR7.5 A preview shows the target object key and destination prefix resulting from the current values, updating live as fields change.
- FR7.6 A validate action applies the rename/move (if not already done), sets the status to validated, and records the corrected values as training data.

### Epic 8 — Configuration file management

- FR8.1 Templates and extraction rules are defined in a JSONC configuration file; there is no separate template-management API or template registry endpoint.
- FR8.2 Adding or changing a template means editing the JSONC configuration file (including period granularity and offset) plus example files; no code change is required.
- FR8.3 The dashboard configuration editor retrieves the current JSONC file and uploads a modified version through dedicated configuration endpoints. The server validates the file before replacing the active configuration and reloading the in-memory template registry.

### Epic 9 — Retraining

- FR9.1 Manual "Retrain" action in the dashboard: trains on initial corpus + validated files, shows evaluation metrics (accuracy, confusion matrix) before activation.
- FR9.2 The user can activate or discard the new model. Only one model is active at a time; model files are versioned on disk.

### Epic 10 — Consistency and recovery

- FR10.1 A periodic consistency job verifies, for every sidecar: object exists at current key, sidecar in sync with SQLite; and for every PDF object: sidecar exists. Anomalies are listed in the dashboard with repair actions.
- FR10.2 Full SQLite rebuild from sidecars is available as a dashboard action and a CLI command.
- FR10.3 At startup, recover interrupted technical operations from recorded state. Re-scan `pending/` to reconcile metadata and detect anomalies, but do not automatically repeat classification or rename for documents awaiting manual correction.

## 6. Non-Functional Requirements

- NFR1 Privacy: fully offline on self-hosted infrastructure. No network calls except the local Garage bucket, and serving the local dashboard. Dependencies must not require downloads at runtime.
- NFR2 Performance: end-to-end processing of a standard PDF under 5 seconds on the target server, including the S3 download; dashboard file list loads under 1 s for 10,000 records (served from SQLite).
- NFR3 Reliability: zero silent wrong renames. Every ambiguous case routes to `pending`.
- NFR4 Durability: bucket versioning enabled; SQLite with WAL. Operations must be recoverable after crashes using copy-then-delete, atomic sidecar writes, idempotent retries, and consistency repair for SQLite-sidecar drift. For transient technical failures, perform at most 3 retries after the initial attempt, with delays of 5 seconds, 30 seconds, and 2 minutes. After the 3 retries are exhausted, stop processing that item; do not launch another retry or manual-review workflow, and do not change its business status to `pending` solely because of the technical failure.
- NFR5 Language: documents in French; UI in English or French (single locale acceptable in v1).
- NFR6 Deployment: runs as a container next to the Garage deployment and the Nextcloud server; configuration via a single JSONC file (templates) and environment variables (S3 endpoint, credentials, bucket).
- NFR7 Stack: Python (FastAPI, boto3 or minio S3 client, scikit-learn, pdfplumber/PyMuPDF, rapidfuzz, spaCy fr model optional), React (Vite, a lightweight PDF viewer such as react-pdf), SQLite, Garage for S3 storage, Nextcloud with External Storage (S3) as the user-facing file interface.
- NFR9 No encryption at rest in v1: documents are not sensitive; this decision is recorded explicitly and can be revisited.

## 7. Data Model

Sidecar JSON (source of truth), one per PDF at `{key}.meta.json`. The `sha256` field is the unique document identifier and must match the digest of the exact PDF bytes. Any byte-level change creates a new document record; a rename/move without content changes preserves the identifier. Identical PDF bytes map to the same record, so ingestion must detect an existing SHA-256 and handle it as a duplicate rather than create a second row.

- schema_version, sha256, original_key, current_key, status, template_name, confidence, emission_date, period, emitting_company, document_type, optional_fields (dict of additional field values), extracted_text (full text from all pages), created_at, updated_at, events[] (type: classified / renamed / validated / corrected, payload, timestamp).

SQLite replica (rebuilt from sidecars):

- files: sha256 (primary key), original_key, current_key, status, template_id, confidence, emission_date, period, emitting_company, document_type, optional_fields (JSON), created_at, updated_at, extracted_text.
- consistency_flags: id, sha256 (nullable reference to the affected PDF digest), kind (orphaned_object, missing_sidecar, drift).

## 8. API Surface (indicative)

- GET /files?status=&template=&q=&page= — list with filters (from SQLite).
- GET /files/{sha256} — detail: text, fields, candidates, current key.
- GET /files/{sha256}/pdf — stream the PDF from the bucket for the viewer.
- POST /files/{sha256}/validate — body: template_id, fields; applies rename if needed, sets validated.
- GET /config — retrieve the current JSONC configuration file.
- PUT /config — validate and save a replacement JSONC configuration file, then reload the configuration.
- POST /retrain — starts training and returns a generic job ID; GET /jobs/{job_id} reports status and result.
- GET /consistency — list anomalies; POST /consistency/rebuild — rebuild SQLite from sidecars.

## 9. Acceptance Criteria (v1)

- AC1 A new PDF dropped in `inbox/` via Nextcloud whose deterministic rules yield an unambiguous template and valid fields appears as processed within the poll interval, renamed and moved in the bucket.
- AC2 A PDF that cannot be classified unambiguously by deterministic rules, or has missing/invalid required fields, appears as `pending` and is moved to `pending/`; correcting and validating it on the detail page renames it and sets validated.
- AC3 Opening a processed file and validating without changes sets validated without re-renaming.
- AC4 The file list filters by status, template, document type, emitting company, period, and date; the consistency indicator matches the known anomalies.
- AC5 Changing the template on the detail page keeps the other common fields, reloads the optional fields per the new template schema, recomputes the read-only period per the new template's granularity and offset, and shows the live target key preview.
- AC5b Every file in every state carries the five common fields (template, emission date, emitting company, document type, period); a document whose fields cannot be recognized is routed to `pending` rather than renamed.
- AC5c Period computation per template rules: a document dated 2026-09-20 under a monthly template with offset current yields "September 2026"; under a quarterly template with offset next it yields "Q4 2026"; under a yearly template it yields "2026".
- AC6 Retraining uses the initial corpus + validated files, runs as a generic asynchronous job queried through `GET /jobs/{job_id}`, shows metrics when complete, and activation swaps the model without service restart.
- AC7 Editing and uploading the JSONC configuration (with example files) makes a new template available for classification and correction; the server validates the replacement before activating it, with no code change.
- AC8 Killing the service mid-processing leaves a recoverable state: startup recovery and the consistency job reconcile object keys, sidecars, and SQLite; retries do not duplicate renames or delete the only recoverable copy.
- AC9 Deleting the SQLite database entirely and running the rebuild command restores a fully functional dashboard from the sidecars alone.
- AC10 No network traffic other than the local Garage bucket, local Nextcloud, and local dashboard serving (verifiable with a firewall audit).
- AC11 1,000-file list view renders with filters in under 1 second.
- AC12 Nextcloud sees renamed and moved files at their new location without reconfiguration.

## 10. Release Plan

- v1 (this PRD): new files only, text-based French PDFs, three statuses, Garage S3 storage, sidecar source of truth with SQLite replica, Nextcloud as file interface, detail page with correction and validation, manual retrain, consistency job.
- v2 candidates: OCR for scanned documents, backlog ingestion of historical files, automatic periodic retraining, multilingual documents, automatic counterparty learning from validated files, bucket notification hooks to replace polling, at-rest encryption if the document mix becomes sensitive.

## 11. Open Questions

- Choice between pdfplumber and PyMuPDF (license and performance tradeoff) — decide in Week 1 spike.
- S3 client library: boto3 vs minio-py against Garage — covered by the Week 1 Garage compatibility spike (listing, copy-object, multipart, versioning behavior).
- Post-v1 ML decision: exact model confidence threshold and per-template overrides, to be calibrated on the initial corpus before enabling ML.
- Poll interval tradeoff between responsiveness and Garage listing load.
- Exact human-readable period formats (French month names, quarter notation) and their rendering in the object key — to be fixed in the template JSONC spec.


### ML-only inference contract

The runtime uses local model artifacts for emitting-company, document-type and emission-date extraction; additional-field-group selection from emitting company and document type; shared token-level additional-field span extraction; and independent `template_name` and `period_format` prediction from emitting company and document type. Configuration may define template key patterns and period arithmetic parameters, but it must not select classification outcomes. Missing models, invalid predictions, or confidence below `CONFIDENCE_THRESHOLD` route the document to `pending`. The `period` and final storage key remain deterministic calculations after model inference.

## Change requirements — configurable document list and bulk editing

### FR6.5 — Default columns
The home page MUST show these columns by default and in this order: status, document (current filename), company (emitting company), document type, period, and last modified. “Last modified” MUST be a persisted metadata modification timestamp and MUST NOT be confused with emission date or object creation time.

### FR6.6 — Configurable columns
Users MUST be able to add and remove columns without changing the underlying records or filters. Available columns include all common/list fields and configured additional fields. Column IDs must be stable (additional fields use their field key, not their display label). A field with no value displays a consistent empty marker. The default column set is used when no user preference exists; if preferences are persisted, unavailable fields are safely ignored.

### FR6.7 — Row and filtered-result selection
Users MUST be able to select individual rows, clear selection, select all rows on the current page, and select all records matching the current filters across pagination. “Select all filtered” MUST use the server-side filter query rather than only the rows loaded in the browser. The UI MUST distinguish page selection from all-filtered selection and show the affected count. Changing filters MUST invalidate or explicitly reconcile a prior selection.

### FR6.8 — Bulk metadata edit
Users MUST be able to apply one or more changes to the selected documents: emitting company, document type, and any configured additional field. Each field in the bulk form supports “leave unchanged” as the default; additional fields can be set/replaced or explicitly cleared. No field changes unless the user explicitly chooses an operation. Show a review/confirmation step with selection scope, count, and proposed changes.

### FR6.9 — Validation, persistence, and feedback
Bulk changes MUST use the same validation and business rules as the single-document workflow. Company and document type must be configured/allowed values. Recompute derived period and destination key whenever relevant source fields change; period remains server-calculated and read-only. If a key changes, use the existing crash-safe copy → durable sidecar → delete-source sequence, collision policy, and sidecar-first SQLite synchronization. Preserve unchanged fields, record an event per document, and return per-document success/failure details. Partial failures must be visible and retryable without reapplying successful items unintentionally.

### Acceptance criteria
- Fresh users see exactly the six default columns above in the specified order.
- A configured additional field can be added as a column and removed again.
- “Select all filtered” includes matching records beyond the current page and excludes records outside the active filters.
- Bulk editing can change company/type and set, replace, or clear an additional field while preserving all other values.
- Derived period/key are recomputed where necessary; invalid edits do not corrupt sidecars, PDFs, or the SQLite replica.
- The UI reports the number of selected records and per-document outcomes.