thomas/multiarch-docker-images
1
0
Fork 0
mirror of https://github.com/napnap75/multiarch-docker-images.git synced 2026-11-11 00:22:12 +01:00
Code Issues Projects Releases Wiki Activity

Add architecture document and user story for paperless-ngx import

Browse Source
This commit is contained in:
thomas
2026-10-01 22:54:42 +02:00
parent 36df49048c
commit 2d26876bf5
3 changed files with 326 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
automatic-pdf-renamer/arch.md
+212
View File
@@ -0,0 +1,212 @@
# Architecture — Automatic PDF Renamer
Architecture document derived from the PRD, following the BMAD method. Personal tool, single user, 100% offline, self-hosted.
## 1. Architecture Overview
A single Python service (FastAPI) hosts the API, the inbox poller, and the static React dashboard. It talks to a Garage S3 bucket (source of truth for objects and JSON sidecars) and mirrors metadata into a local SQLite replica.
Everything runs in one container, deployed next to the existing Garage and Nextcloud deployments. There is no message broker, no external database, and no runtime network access beyond Garage and local dashboard serving.
### 1.1 High-level view
```mermaid
flowchart TD
nc["Nextcloud (External Storage S3)"]
user["Owner (browser)"]
dash["React dashboard (static build)"]
api["FastAPI service (single container)"]
poll["Inbox poller (asyncio task)"]
s3["Garage S3 bucket"]
db[("SQLite replica (WAL)")]
worker["Subprocess jobs: retrain / consistency / rebuild"]
model[("Model registry on disk")]
user --> dash
dash --> api
api --> db
api --> s3
api --> worker
worker --> db
worker --> model
worker --> s3
poll --> s3
poll --> db
nc --> s3
user --> nc
```
## 2. Architecture Goals and Constraints
- Offline-first: no runtime downloads, no external services (NFR1, AC10).
- Zero silent wrong renames: every ambiguous path routes to not processed (NFR3).
- Crash safety: copy-then-delete renames, atomic sidecar writes, SQLite repairable from sidecars alone (NFR4, AC8, AC9).
- Minimal operational surface: one container, one YAML template config, env-var credentials (NFR6).
- Single user, local network: performance targets are modest (NFR2) but correctness targets are strict.
## 3. Tech Stack and Key Decisions
| Concern | Decision | Rationale |
|---|---|---|
| Language/runtime | Python 3.11+, single service | NFR7; ecosystem for ML and S3. |
| Web framework | FastAPI + Uvicorn | NFR7; async I/O for S3 polling and PDF streaming. |
| Text extraction | PyMuPDF | Fastest option; AGPL acceptable for a personal offline tool. Wrapped behind a small extractor interface so it remains swappable (PRD open question resolved). |
| S3 client | boto3 | Most complete API coverage (copy-object, versioning, ETag), mature Garage compatibility. Wrapped behind a thin storage interface; v1 ships one implementation. |
| SQLite access | SQLAlchemy ORM, WAL mode | Declared replica schema, typed models, migrations via Alembic. Sidecar remains authoritative; ORM layer only reads/writes the mirror. |
| ML classification | scikit-learn TF-IDF + Logistic Regression | PRD FR3.1; small corpus is sufficient, no deep learning needed. |
| Fuzzy matching | rapidfuzz | PRD FR4.2, counterparty gazetteer. |
| Dates | python-dateutil | PRD FR4.2, emission date parsing and period computation. |
| Frontend | React + Vite, react-pdf viewer | NFR7; built to static assets served by FastAPI. |
| Process model | API, poller, and background jobs in one container | Single-user tool; a process manager is unnecessary complexity. |
| Inbox poller | asyncio background task inside the FastAPI process | Lightweight, no queue infrastructure; restart scan (FR1.3) covers crash recovery. |
| Long jobs (retrain, consistency, rebuild) | Subprocesses launched and monitored by the API | Keeps the event loop responsive; status exposed via the existing endpoints (FR9, FR10). |
| Dashboard auth | Username + password (single account, env-var credentials, session cookie) | Local network but not fully trusted; cheap to implement. |
| Testing | Unit tests only, pytest | Single-user offline tool; the storage layer is behind interfaces so mocks are cheap. Integration/CI coverage deferred. |
## 4. Components
### 4.1 Service process (single container)
One Python process runs:
- FastAPI app: serves the API, the static dashboard, and the PDF streaming proxy.
- Poller task: asyncio loop, configurable interval, lists the `inbox/` prefix.
- Job supervisor: spawns subprocesses for retrain, consistency scan, and SQLite rebuild; tracks their status in memory and in SQLite.
On startup, the process runs the recovery scan (FR1.3, FR10.3): objects in `inbox/` and `pending/` without sidecar records are enqueued for processing.
### 4.2 Module layout
```
app/
main.py # FastAPI app, lifespan: recovery scan + poller start
config.py # env vars + YAML template loading
storage/ # S3 interface (boto3), key helpers, collision suffix
sidecar.py # atomic sidecar write (temp key + copy), schema
replica/ # SQLAlchemy models, sync, rebuild
pipeline/ # extract (PyMuPDF), classify, fill fields, validate
period.py # period computation from emission date + template rules
gazetteer.py # counterparty matching (rapidfuzz)
templates.py # YAML template registry (slots, period rules, key patterns)
rename.py # copy-then-delete move + event logging
jobs/ # poller, consistency scan, retrain subprocess, rebuild
api/ # routes: files, templates, retrain, consistency, auth
web/ # React + Vite app (built to static assets)
models/ # versioned model registry (initial + trained)
```
### 4.3 Processing pipeline
One ingestion run, fully synchronous within the poller task:
```mermaid
flowchart TD
in["Stable object in inbox/"] --> dl["Download PDF"]
dl --> tx["Extract first-page text (PyMuPDF)"]
tx --> empty{"Text empty?"}
empty -- yes --> pending["Move to pending/ (reason: empty text)"]
empty -- no --> cls["Classify (TF-IDF + LogReg)"]
cls --> conf{"Confidence >= threshold?"}
conf -- no --> pending
conf -- yes --> fill["Fill fields: regexes, gazetteer, period from emission date"]
fill --> valid{"Fields valid?"}
valid -- no --> pending
valid -- yes --> rename["Compute key, copy-then-delete, write sidecar + SQLite"]
rename --> done["Status: processed"]
pending --> wait["Status: not processed (sidecar with failing fields)"]
```
Every terminal step writes a sidecar event (classified, renamed, corrected, validated) per FR5.2. Sidecar first, SQLite second (FR5.1); the consistency job repairs any drift after a crash.
### 4.4 Storage layer
- boto3 client pinned to the Garage endpoint, credentials from env vars.
- Interface operations: list prefix, head object, get, put, copy, delete. Rename = copy + delete, collisions resolved with an incremental suffix before the copy.
- Sidecar write: put JSON to a temp key, then copy over the final sidecar key, then delete the temp key. Never an in-place overwrite.
### 4.5 Metadata and replica
- Sidecar JSON is the sole source of truth; schema per PRD section 7 (now including the period field).
- SQLAlchemy models `files` and `consistency_flags` mirror it. `extracted_text` is stored in SQLite for search; it is a replica, and the rebuild command repopulates it from sidecars.
- `POST /consistency/rebuild` and the CLI rebuild command both clear and repopulate SQLite from all sidecars (AC9).
- Divergence resolution is always sidecar-wins (PRD 3.2); the consistency job flags drift rather than guessing.
### 4.6 Period computation
A pure function of (emission date, template period rules): granularity in {day, month, quarter, year} and offset in {current, next}. Produces the human-readable period string (e.g. "Q3 2026", "September 2026", "2026"). Recomputed whenever the emission date or template changes on the detail page; the user can override the suggested value, and validation checks consistency between period, date, and template rules (FR4.2, FR4.3, AC5c).
### 4.7 Frontend
Single-page React app, built with Vite into static assets served by FastAPI. Views: file list (filters, counters, badges), detail page (embedded react-pdf viewer, five common fields, optional fields, template dropdown with candidate scores, live key preview), templates list, retrain and consistency pages. Auth: login form, session cookie, all API routes behind the auth dependency except the PDF stream which shares it.
## 5. Data Flow: Renaming and Validation
```mermaid
sequenceDiagram
participant U as Owner
participant API as FastAPI
participant S3 as Garage
participant DB as SQLite
U->>API: POST /files/{id}/validate (template, fields)
API->>API: Validate fields, compute target key
alt key changed
API->>S3: Copy object to target key
API->>S3: Delete old key
end
API->>S3: Atomic sidecar update (status: validated, event)
API->>DB: Mirror update
API-->>U: 200, new key
```
## 6. Jobs and Scheduling
| Job | Trigger | Execution |
|---|---|---|
| Inbox poll | Interval (configurable, default from PRD) | asyncio task, in-process |
| Stability check | Two consecutive listings, same size + ETag | Poller state kept in memory |
| Consistency scan | Periodic (configurable) + on demand | Subprocess |
| Retrain | Dashboard action | Subprocess; writes a new versioned model dir; activation is an atomic pointer swap, no restart (AC6) |
| SQLite rebuild | Dashboard action or CLI | Subprocess |
Subprocesses report progress and metrics by writing a status JSON consumed by the API's status endpoints. Only one job of each kind runs at a time; the API refuses a duplicate start.
## 7. Configuration and Secrets
- Templates: single YAML file (or a directory of YAML files) defining, per template: document family, document types, period rules, optional field schema, extraction regexes, validation rules, key pattern, target prefix.
- Environment: S3 endpoint, access key, secret key, bucket name, poll interval, confidence threshold, dashboard credentials, paths (SQLite file, model registry, template YAML).
- No secrets in the YAML; no secrets in sidecars.
## 8. Error Handling and Crash Safety
- Ingestion: any unexpected exception marks the file not processed with the reason in the sidecar; the poller continues.
- Rename: copy strictly before delete; a crash between the two leaves both keys, and the startup scan re-evaluates (the sidecar's current key is authoritative).
- Sidecar/SQLite: sidecar first; drift repaired by the consistency job; rebuild is always possible from sidecars alone.
- Retrain: trained models land in a new versioned directory; activation swaps an active-model pointer; a failed training run leaves the current model untouched.
- All object-not-found conditions during processing are treated as consistency anomalies and flagged (FR5.3).
## 9. Deployment
- One Docker image: Python backend + prebuilt static frontend assets + model registry. Garage and Nextcloud are pre-existing deployments on the same network; the compose file of this project only declares the renamer service.
- SQLite file and model registry on a mounted volume; container restarts are stateless otherwise.
- Offline dependency policy: all Python wheels and the frontend build are baked into the image build; the spaCy French model, if used, is vendored at build time.
## 10. Testing Strategy
- Unit tests only (pytest), per the stack decision.
- Mocked seams: storage interface (fake in-memory S3), extractor interface, classifier (fixture model), clock.
- Priority coverage: period computation (granularity/offset matrix), field validation, collision suffix logic, sidecar atomicity ordering, rebuild determinism, key pattern rendering.
- The pipeline is exercised end-to-end in unit tests with mocked S3 to validate the not-processed routing reasons.
## 11. Architectural Risks and Mitigations
| Risk | Mitigation |
|---|---|
| PyMuPDF AGPL incompatibility if the tool is ever shared | Extractor interface keeps the swap to pdfplumber cheap. |
| Garage quirks in copy-object/versioning | Week 1 spike per PRD; storage interface isolates the client. |
| Corpus too small for reliable classification | Low-confidence routing to pending; retrain on validated files grows the corpus. |
| SQLite drift after crash | Sidecar-first ordering + consistency job + full rebuild path. |
| Single-process blast radius | Long jobs are subprocesses; the poller catches per-file exceptions. |
## 12. Traceability to PRD
Every epic maps to a component: ingestion (jobs/poller), extraction and classification (pipeline), data model and validation (templates, period, gazetteer, pipeline), renaming and logging (rename, sidecar), dashboards (web + api), template management (templates.py + config), retraining (jobs/retrain + models), consistency and recovery (jobs/consistency, replica/rebuild). Acceptance criteria AC1-AC12 are covered by the components above; AC8-AC10 by section 8, AC9 by section 4.5.
automatic-pdf-renamer/prd.md
+32 -20
View File
@@ -14,7 +14,7 @@ Single user: the owner. Technically proficient, wants minimal manual work but ze
### 3.1 Object storage
- Garage (https://garagehq.deuxfleurs.fr/), self-hosted, S3 API. One versioned bucket, e.g. `pdf-renamer`.
- Garage ([https://garagehq.deuxfleurs.fr/](https://garagehq.deuxfleurs.fr/)), self-hosted, S3 API. One versioned bucket, e.g. `pdf-renamer`.
- Key layout: `inbox/` (entry point), `pending/` (not-processed files moved out of inbox to avoid re-processing), and `files/` for the processes files (validated or not).
- Sidecars: `{object-key}.meta.json` next to each PDF.
- No encryption at rest in v1 (documents not sensitive); bucket served on the local network only.
@@ -22,7 +22,7 @@ Single user: the owner. Technically proficient, wants minimal manual work but ze
### 3.2 Metadata: sidecars as source of truth, SQLite as replica
- Each PDF has a sidecar JSON object holding: sha256, original key, current key, status, template, document type, emission date, emitting company, all optional fields, confidence, and the full event history (classified, renamed, validated, undone, corrected).
- Each PDF has a sidecar JSON object holding: sha256, original key, current key, status, template, document type, emission date, period, emitting company, all optional fields, confidence, and the full event history (classified, renamed, validated, undone, corrected).
- Sidecar is updated atomically on every state change: write the new sidecar to a temp key, copy over the final sidecar key.
- SQLite mirrors the sidecar content in queryable tables for the dashboard (lists, filters, statistics). It is a cache/replica, never authoritative. Any sidecar↔SQLite divergence is resolved in favor of the sidecar.
- A rebuild command reconstructs the full SQLite database by scanning all sidecars; a consistency job detects PDFs without sidecars (orphaned) and sidecars without PDFs (missing object) and flags them in the dashboard.
@@ -35,12 +35,14 @@ Single user: the owner. Technically proficient, wants minimal manual work but ze
## 4. Document Lifecycle and Statuses
| Status | Meaning |
|--- |---|
| not processed | Classification confidence below threshold, or field validation failed. Object moved to the `pending/` prefix, waiting for review. |
| processed | Automatically classified, fields validated, renamed and moved. |
| Status | Meaning |
| ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| not processed | Classification confidence below threshold, or field validation failed. Object moved to the `pending/` prefix, waiting for review. |
| processed | Automatically classified, fields validated, renamed and moved. |
| validated | The user reviewed the result (confirmed or corrected template/fields). Eligible for model retraining. Terminal state for the review flow; the object was renamed and moved. |
Transitions:
- new object in `inbox/` → not processed (low confidence or invalid fields; object copied to `pending/`)
@@ -76,20 +78,28 @@ Transitions:
#### 4.1 Common fields (mandatory for every document)
Every document, regardless of template, carries at least these four common fields:
Every document, regardless of template, carries at least these five common fields:
- Common field 1 — Template: the selected template (i.e. the document family: invoice, purchase order, contract, bank statement, rent receipt, etc.).
- Common field 2 — Emission date: the date the document was emitted, extracted from its content (not the object modification date).
- Common field 3 — Emitting company: the company or organization that emitted the document, matched against the counterparty gazetteer.
- Common field 4 — Document type: the fine-grained type of document within the family (e.g. debit advice vs monthly statement for bank statements; quotes vs order confirmations for purchase documents).
- Common field 5 — Period: the period the document refers to, in human-readable form, derived from the emission date according to the template's period rules. The granularity and the offset relative to the emission date are declared per template:
- Granularity `day`: the emission date itself, e.g. "20 September 2026" (a simple invoice).
- Granularity `month`: e.g. "September 2026" (a monthly rent receipt).
- Granularity `quarter`: e.g. "Q3 2026" (quarterly building-charge invoice).
- Granularity `year`: e.g. "2026" (yearly tax document).
- Offset `current`: the period containing the emission date (document sent after the start of the period; e.g. "Q3 2026" for 2026-09-20).
- Offset `next`: the period following the emission date's period (document sent in advance; e.g. "Q4 2026" for 2026-09-20).
- Offset is only meaningful for granularities coarser than a day; for `day` the period always equals the emission date.
The object key pattern of each template is composed from these common fields, plus the template's optional fields.
#### 4.2 Optional fields (template-specific)
- FR4.1 Each template declares in YAML its optional fields beyond the common four: name, type, format, extraction rules, validation rules, and whether the field is rendered in the object key, the target prefix, or stored only. Examples: bank statements carry account owner and statement type; rent receipts carry the concerned premises; contracts carry the counterparty signatory.
- FR4.2 Fill common and optional fields using per-template regexes, dateutil for dates, and a counterparty gazetteer with rapidfuzz matching for the emitting company (similarity threshold configurable; below threshold the company is "unrecognized").
- FR4.3 Validate fields before renaming: emission date parses and is within a plausible range, emitting company recognized, document type among the template's allowed values, each optional field matches its declared format (empty optional fields are allowed). Any failure routes to not processed with the failing fields identified.
- FR4.1 Each template declares in YAML its optional fields beyond the common five: name, type, format, extraction rules, validation rules, and whether the field is rendered in the object key, the target prefix, or stored only. Examples: bank statements carry account owner and statement type; rent receipts carry the concerned premises; contracts carry the counterparty signatory.
- FR4.2 Fill common and optional fields using per-template regexes, dateutil for dates, and a counterparty gazetteer with rapidfuzz matching for the emitting company (similarity threshold configurable; below threshold the company is "unrecognized"). The period is computed from the emission date using the template's declared granularity and offset (current vs next period); it is recomputed whenever the emission date or template changes.
- FR4.3 Validate fields before renaming: emission date parses and is within a plausible range, period is non-empty and consistent with the emission date and the template's granularity/offset, emitting company recognized, document type among the template's allowed values, each optional field matches its declared format (empty optional fields are allowed). Any failure routes to not processed with the failing fields identified.
- FR4.4 Collisions in the target prefix are resolved with an incremental suffix; the final key is always the one logged.
### Epic 5 — Renaming, moving, logging
@@ -100,8 +110,8 @@ The object key pattern of each template is composed from these common fields, pl
### Epic 6 — Dashboard: file list
- FR6.1 A single list shows every file the system has seen, with columns: emission date, current object name, status, template (document family), document type, emitting company, confidence. Served from SQLite.
- FR6.2 Filter by status, template, document type, emitting company, and date range; sort by any column; free-text search on object name, any field and optionnally first-page content.
- FR6.1 A single list shows every file the system has seen, with columns: emission date, period, current object name, status, template (document family), document type, emitting company, confidence. Served from SQLite.
- FR6.2 Filter by status, template, document type, emitting company, period, and date range; sort by any column; free-text search on object name, any field and optionnally first-page content.
- FR6.3 Status is visually distinguishable (badge): not processed, processed, validated.
- FR6.4 The list shows counters: files per status, automation rate (processed / total), validation rate, plus a consistency indicator (orphaned objects, missing sidecars).
@@ -110,15 +120,15 @@ The object key pattern of each template is composed from these common fields, pl
- FR7.1 Clicking a file in the list opens a detail page.
- FR7.2 The detail page displays the PDF (embedded viewer), streamed from the bucket by the backend, alongside the extracted data.
- FR7.3 The user can change the template (dropdown of configured templates, showing the classifier's top candidates and scores).
- FR7.4 The detail page always shows the four common fields (template, emission date, emitting company, document type) plus the current template's optional fields; the user can correct each value inline; changing the template reloads the optional fields with the new template's schema and best-effort re-extraction, keeping the common field values.
- FR7.4 The detail page always shows the five common fields (template, emission date, emitting company, document type, period) plus the current template's optional fields; the user can correct each value inline; changing the template reloads the optional fields with the new template's schema and best-effort re-extraction, keeping the common field values, and recomputes the period from the new template's granularity and offset. The period is editable and displayed as a free-text field with a suggested value derived from the emission date.
- FR7.5 A preview shows the target object key and destination prefix resulting from the current values, updating live as fields change.
- FR7.6 A validate action applies the rename/move (if not already done), sets the status to validated, and records the corrected values as training data.
### Epic 8 — Template management
- FR8.1 Templates are defined in YAML configuration files.
- FR8.2 Adding a template = YAML entry + example files; no code change.
- FR8.3 The dashboard lists configured templates with their slot definitions, target prefix, and the count of corpus files per template.
- FR8.2 Adding a template = YAML entry (including the period granularity and offset) + example files; no code change.
- FR8.3 The dashboard lists configured templates with their slot definitions, period rules, target prefix, and the count of corpus files per template.
### Epic 9 — Retraining
@@ -146,11 +156,11 @@ The object key pattern of each template is composed from these common fields, pl
Sidecar JSON (source of truth), one per PDF at `{key}.meta.json`:
- schema_version, sha256, original_key, current_key, status, template_name, confidence, emission_date, emitting_company, document_type, optional_field1, optional_field2, extracted_text, created_at, updated_at, events[] (type: classified / renamed / validated / corrected, payload, timestamp).
- schema_version, sha256, original_key, current_key, status, template_name, confidence, emission_date, period, emitting_company, document_type, optional_field1, optional_field2, extracted_text, created_at, updated_at, events[] (type: classified / renamed / validated / corrected, payload, timestamp).
SQLite replica (rebuilt from sidecars):
- files: id, sha256, original_key, current_key, status, template_id, confidence, emission_date, emitting_company, document_type, optional_field1, optional_field2, created_at, updated_at, extracted_text, created_at, updated_at.
- files: id, sha256, original_key, current_key, status, template_id, confidence, emission_date, period, emitting_company, document_type, optional_field1, optional_field2, created_at, updated_at, extracted_text, created_at, updated_at.
- consistency_flags: id, file_id, kind (orphaned_object, missing_sidecar, drift).
## 8. API Surface (indicative)
@@ -168,9 +178,10 @@ SQLite replica (rebuilt from sidecars):
- AC1 A new PDF dropped in `inbox/` via Nextcloud with high-confidence classification and valid fields appears as processed within the poll interval, renamed and moved in the bucket.
- AC2 A PDF below the confidence threshold appears as not processed, is moved to `pending/`; correcting and validating it on the detail page renames it and sets validated.
- AC3 Opening a processed file and validating without changes sets validated without re-renaming.
- AC4 The file list filters by status, template, document type, emitting company, and date; counters match the data.
- AC5 Changing the template on the detail page keeps the common fields, reloads the optional fields per the new template schema, and shows the live target key preview.
- AC5b Every file in every state carries the four common fields (template, emission date, emitting company, document type); a document whose fields cannot be recognized is routed to not processed rather than renamed.
- AC4 The file list filters by status, template, document type, emitting company, period, and date; counters match the data.
- AC5 Changing the template on the detail page keeps the common fields, reloads the optional fields per the new template schema, recomputes the period per the new template's granularity and offset, and shows the live target key preview.
- AC5b Every file in every state carries the five common fields (template, emission date, emitting company, document type, period); a document whose fields cannot be recognized is routed to not processed rather than renamed.
- AC5c Period computation per template rules: a document dated 2026-09-20 under a monthly template with offset current yields "September 2026"; under a quarterly template with offset next it yields "Q4 2026"; under a yearly template it yields "2026".
- AC6 Retraining uses initial corpus + validated files, shows metrics, and activation swaps the model without service restart.
- AC7 Adding a template via YAML (with example files) makes it available for classification and in the detail page dropdown, with no code change.
- AC8 Killing the service mid-processing leaves no half-renamed object: the copy-then-delete order and the sidecar-first write guarantee a consistent state on restart, and the consistency job repairs SQLite.
@@ -190,3 +201,4 @@ SQLite replica (rebuilt from sidecars):
- S3 client library: boto3 vs minio-py against Garage — covered by the Week 1 Garage compatibility spike (listing, copy-object, multipart, versioning behavior).
- Exact confidence threshold and per-template overrides — calibrate on the initial corpus.
- Poll interval tradeoff between responsiveness and Garage listing load.
- Exact human-readable period formats (French month names, quarter notation) and their rendering in the object key — to be fixed in the template YAML spec.
automatic-pdf-renamer/user-story-0-paperless-ngx-initial-import.md
+81
View File
@@ -0,0 +1,81 @@
# User Story 0 — Paperless-ngx Initial Import
BMAD user story. Precedes v1 epics; provides the initial corpus and the pre-existing organized files in the bucket.
## Story
As the owner, I want a standalone Python script that imports my existing paperless-ngx documents into the empty Garage bucket with their metadata as sidecars, so that the renamer service starts with a populated, correctly named file tree and a training corpus, without any document being reclassified by hand.
## Background and Context
- The renamer service assumes an empty v1 scope (new files only). This story backfills the historical corpus from paperless-ngx before the service is ever started.
- The script runs once, offline, from the owner's machine or the server. It is not part of the renamer container and has no dashboard surface.
- Sidecars created here follow the exact same schema as the service's (PRD section 7), with status validated, since the metadata comes from a system the owner already curated.
- After the import, the service's rebuild command (FR10.2) populates SQLite from these sidecars. No code in the service is needed for this story beyond schema compatibility.
## Requirements
### FR0.1 Connection
- Connect to the paperless-ngx REST API (`PAPERLESS_URL` + API token from env vars).
- Connect to the target Garage bucket with boto3 (S3 endpoint, credentials, bucket from env vars). The bucket must be empty on the `files/` prefix; the script refuses to run if any object already exists there (dry-run flag lists what would happen instead).
### FR0.2 Listing and mapping
- List all documents from paperless via the API (paginated `/api/documents/`).
- For each document, fetch the document metadata and map it to the sidecar fields:
- template ← the first path segment of the paperless storage path (the document family, e.g. `Banque`, `Factures`).
- emission_date ← paperless document date (`created` field, the content date).
- emitting_company ← the document's correspondent name.
- document_type ← the document type name.
- period ← the last space-separated part(s) of the original filename, parsed as a period: a month-year ("août 2025"), a quarter ("Q3 2026"), a year ("2026"), or the date itself for daily documents. If no period can be parsed, fall back to the period computed from emission_date and the template's granularity/offset rules; if no template rule matches, leave the period equal to the emission date's month-year and log a warning.
- extracted_text ← the first page text of the downloaded PDF, extracted with the same library as the service (PyMuPDF), stored in the sidecar exactly as the service would (PRD FR2.2, FR2.4). This makes the imported files immediately usable as the classifier corpus without any later re-download or re-parse.
- optional fields ← sourced from the document's paperless labels. Each label becomes an optional field entry in the sidecar: the label name is the field name, the label value is the field value (boolean labels are stored as true). Labels that collide with a common field name are ignored with a warning. The template YAML's optional field schemas are not enforced at import time; the consistency job or a later review surfaces any mismatch.
- original_key ← the paperless archive filename and path as stored by paperless (`storage_path` + `original filename`).
- status ← validated, confidence ← 1.0, with a single event `imported` in the events array recording the source (paperless document id and URL).
- sha256 computed from the downloaded PDF binary.
### FR0.3 Upload
- Download the PDF through the paperless API (`/api/documents/{id}/download/`).
- Store it in the bucket at `files/{template}/{original filename}` — i.e. reuse the path and filename paperless already uses, so the tree stays familiar and no renames occur during import. Collision: incremental suffix per PRD FR4.4.
- Write the sidecar `{key}.meta.json` with the atomic temp-key + copy pattern (same as the service) so the file tree is consistent if the script is interrupted.
### FR0.4 Resumability and safety
- The script is resumable: a document whose PDF key and sidecar already exist (matching sha256) is skipped.
- A summary is printed at the end: imported, skipped, failed, period fallbacks used, per-template counts.
- Failures (API errors, download errors) are logged with the paperless document id and do not abort the run; the script exits non-zero if any failure occurred.
- The script never writes to `inbox/` or `pending/`.
## Acceptance Criteria
- AC0.1 Running the script against the paperless instance and an empty bucket results in every paperless document present under `files/` with its paperless path and filename, each with a valid sidecar.
- AC0.2 Every sidecar carries the five common fields with correct provenance: template from storage path, emission_date from document date, emitting_company from correspondent, document_type from document type, period parsed from the filename tail (e.g. "août 2025" for "2025-08-31 Relevé de compte Compte LMNP août 2025.pdf"), plus the first-page extracted_text and one optional field per paperless label.
- AC0.3 All imported sidecars have status validated, a computed sha256, and an imported event; running the service's SQLite rebuild afterwards yields a fully populated dashboard with zero not-processed files from the import.
- AC0.4 Interrupting the script and rerunning it completes the import without duplicates (resumability), and the bucket never contains a sidecar without its PDF or vice versa for more than one in-flight file.
- AC0.5 The script performs no network calls other than the paperless API and the Garage endpoint.
## Tasks
- T1 Standalone script skeleton: env-var config (paperless URL, token, S3 endpoint, credentials, bucket), boto3 client, argument parsing (dry-run, resume default, limit for a trial run).
- T2 Paperless client: paginated document listing, metadata fetch (correspondent, document type, storage path, dates), PDF download, sha256 computation.
- T3 Period parser: month-year / quarter / year / date from the filename tail; fallback to template-rule computation from emission_date.
- T3b First-page extractor: PyMuPDF, same extraction parameters as the service; empty text stored as an empty string with a warning (the document stays validated, it is imported metadata, not classified).
- T3c Label mapper: paperless labels → optional fields dictionary; name-collision guard against the common field names.
- T4 Sidecar builder: PRD section 7 schema, imported event, status validated.
- T5 Uploader: key construction from storage path + filename, collision suffix, atomic sidecar write, skip-if-present resumability check.
- T6 Summary reporting and exit codes.
- T7 Unit tests (pytest): period parser matrix, sidecar schema, key construction, collision suffix, resume logic with a mocked S3 and mocked paperless API.
## Interactions and Affected Components
- No changes to the renamer service code. Shared elements: sidecar schema and the atomic write pattern (the script duplicates the small helper, deliberately, to stay standalone).
- Output feeds: the classifier's initial corpus (template + first-page text later re-extracted by the service is not needed here; validated status makes these files eligible for the first retrain) and the dashboard after rebuild.
## Notes and Open Points
- Sidecars now include extracted_text (first page, PyMuPDF, same parameters as the service), so no post-import re-extraction backfill is needed and the corpus is immediately usable for the first retrain.
- Optional fields come solely from paperless labels. Labels are free-form in paperless; the template YAML schemas are not enforced at import, and labels colliding with common field names are dropped with a warning.
- Document subtype mapping (paperless document types to the renamer template YAML document types) is identity on the name; if the template YAML later diverges, the consistency job surfaces mismatches.
- The script lives in the repo as `scripts/import_paperless.py` with its own minimal dependencies (boto3, requests, python-dateutil, PyMuPDF), not in the container image.