mirror of
https://github.com/napnap75/multiarch-docker-images.git
synced 2026-11-11 00:22:12 +01:00
Add epics and stories / Make specifications consistent
This commit is contained in:
@@ -0,0 +1,456 @@
|
||||
# API — Automatic PDF Renamer
|
||||
|
||||
Version: 1.0 (design specification)
|
||||
Base path: `/api/v1`
|
||||
Backend: FastAPI; storage: Garage S3-compatible bucket; metadata source of truth: per-file sidecars; query index: SQLite.
|
||||
|
||||
This document defines the HTTP API consumed by the local web dashboard and the operational API used to observe ingestion, recover the index, and run maintenance jobs. The API is a local, same-origin interface; it does not expose the S3 credentials or bypass the sidecar-as-source-of-truth rule.
|
||||
|
||||
## 1. API principles
|
||||
|
||||
- **Nextcloud/S3 remains the upload interface.** A user uploads a PDF to the configured `inbox/` prefix through Nextcloud or an S3 client. The API does not accept PDF uploads in v1.
|
||||
- **Ingestion is server-side and automatic.** The poller discovers stable PDF objects in `inbox/`, extracts text from every page, applies deterministic rules, validates fields, and either renames the file or moves it to `pending/`.
|
||||
- **Sidecars are authoritative.** For a mutation, persist the sidecar first, then synchronize SQLite. SQLite is used for list/search and consistency diagnostics, and can be rebuilt from sidecars.
|
||||
- **Business statuses are exactly** `pending`, `processed`, and `validated`. Technical job state is separate and must not be represented as a business status.
|
||||
- **The initial release is deterministic.** API responses may include a nullable confidence/model section for forward compatibility, but the rules-only release must not fabricate ML scores.
|
||||
- **No rename undo endpoint exists.** Renames are recorded for audit, not reversed by a user-facing API.
|
||||
- The API and processing pipeline remain offline except for the configured local Garage endpoint.
|
||||
|
||||
## 2. Conventions
|
||||
|
||||
### 2.1 JSON and identifiers
|
||||
|
||||
Requests and responses use UTF-8 JSON unless an endpoint streams a PDF or returns an HTTP error. Timestamps are ISO 8601 UTC strings. Dates use `YYYY-MM-DD`. S3 object keys are opaque strings, not filesystem paths.
|
||||
|
||||
`sha256` is the canonical unique identifier for a managed PDF, computed from the exact PDF bytes. All document endpoints use this digest instead of a separate `file_id` or database `id`. Renaming/moving a PDF without changing its bytes preserves the identifier. Any byte-level content change creates a new document record with a new SHA-256; references to the previous digest must not be silently rewritten. If identical bytes are imported again, ingestion must detect the existing digest and treat the import as a duplicate rather than create a second document record. Compute and verify the digest during ingestion and consistency checks.
|
||||
|
||||
### 2.2 Pagination, sorting, filtering
|
||||
|
||||
List endpoints use page-based pagination:
|
||||
|
||||
- `page`: integer, default `1`, minimum `1`.
|
||||
- `page_size`: integer, default `50`, maximum `200`.
|
||||
- `sort_by`: allowlisted field name; default `updated_at`.
|
||||
- `sort_order`: `asc` or `desc`; default `desc`.
|
||||
|
||||
Unknown filters or unsupported sort fields return `422 Unprocessable Entity`. Search and filters operate on SQLite's replica/index; they must not cause a full S3 listing on every dashboard request.
|
||||
|
||||
### 2.3 Standard error response
|
||||
|
||||
HTTP errors return a simple human-readable text message in the response body, with `Content-Type: text/plain; charset=utf-8`. Error responses do not use a JSON envelope or include field-level details.
|
||||
|
||||
Example response body:
|
||||
|
||||
```text
|
||||
The emission date must use YYYY-MM-DD format.
|
||||
```
|
||||
|
||||
HTTP status codes distinguish error categories:
|
||||
- `400`: malformed request or invalid operation for the current state.
|
||||
- `404`: unknown file, template, job, or resource.
|
||||
- `409`: state conflict, such as attempting to start a job of the same kind while one is running, or validating a stale edit.
|
||||
- `422`: request schema or field validation failure. Return a concise plain-text explanation.
|
||||
- `503`: required local dependency unavailable (Garage or SQLite). Return a concise plain-text message without infrastructure details.
|
||||
- `500`: unexpected server error. Return a generic plain-text message; do not expose credentials, stack traces, or secrets to the browser.
|
||||
|
||||
This convention applies to HTTP error responses from every endpoint. Successful responses retain the format specified by each endpoint (JSON or PDF bytes).
|
||||
|
||||
## 3. Endpoint overview
|
||||
|
||||
| Method | Path | Purpose |
|
||||
|---|---|---|
|
||||
| GET | `/health` | Liveness check |
|
||||
| GET | `/ready` | Dependency/readiness check |
|
||||
| GET | `/files` | Browse, filter, sort, and search documents |
|
||||
| GET | `/files/{sha256}` | Document detail and metadata |
|
||||
| GET | `/files/{sha256}/pdf` | Stream the current PDF for the embedded viewer |
|
||||
| GET | `/files/{sha256}/text` | Retrieve complete extracted text |
|
||||
| POST | `/files/{sha256}/preview` | Validate proposed fields and calculate the target key without mutation |
|
||||
| POST | `/files/{sha256}/validate` | Validate metadata and apply the final move/rename if required |
|
||||
| GET | `/consistency` | List known storage/index anomalies |
|
||||
| POST | `/consistency/scan` | Start a consistency scan |
|
||||
| POST | `/consistency/rebuild` | Rebuild SQLite from authoritative sidecars |
|
||||
| GET | `/jobs/{job_id}` | Get asynchronous job status and result (maintenance or retraining) |
|
||||
| GET | `/ingestion/status` | Inspect poller status and recent activity |
|
||||
| GET | `/config` | Retrieve the current JSONC configuration file |
|
||||
| PUT | `/config` | Validate and save a modified JSONC configuration file |
|
||||
| GET | `/models` | Post-v1: list model versions |
|
||||
| POST | `/retrain` | Post-v1: start local model training |
|
||||
| POST | `/retrain/{job_id}/activate` | Post-v1: activate a trained model |
|
||||
| POST | `/retrain/{job_id}/discard` | Post-v1: discard a candidate model |
|
||||
|
||||
The endpoints in the models/retraining section are post-v1 capabilities. They may be omitted from the first deterministic release without affecting the core API.
|
||||
|
||||
## 4. Configuration file
|
||||
|
||||
### GET `/config`
|
||||
|
||||
Returns the complete current JSONC configuration as UTF-8 text. Use `Content-Type: application/jsonc; charset=utf-8` (or `text/plain; charset=utf-8`). This endpoint retrieves the configuration file itself; it does not expose a separate template-list or template-editing API.
|
||||
|
||||
### PUT `/config`
|
||||
|
||||
Accepts the complete replacement JSONC configuration as a UTF-8 text body. Validate its syntax and schema before writing. If invalid, return `422` with a concise plain-text error and leave the active file unchanged. If valid, atomically replace the configured file, reload the in-memory template registry, and return `200` JSON:
|
||||
|
||||
```json
|
||||
{"saved": true, "reloaded": true}
|
||||
```
|
||||
|
||||
The implementation must avoid exposing secrets in the configuration file. Only one complete configuration replacement is supported; individual templates are changed by editing this file.
|
||||
|
||||
## 5. Health and ingestion workflow
|
||||
|
||||
### GET `/health`
|
||||
|
||||
Liveness only. Returns `200` when the API process is running.
|
||||
|
||||
Example response:
|
||||
|
||||
```json
|
||||
{"status": "ok"}
|
||||
```
|
||||
|
||||
### GET `/ready`
|
||||
|
||||
Checks that the service has loaded its configuration and can access the dependencies required to serve requests. Do not expose endpoint URLs or credentials.
|
||||
|
||||
Example response:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "ready",
|
||||
"checks": {
|
||||
"configuration": "ok",
|
||||
"garage": "ok",
|
||||
"sqlite": "ok",
|
||||
"templates": "ok"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Return `503` with a plain-text error message if a required dependency is unavailable.
|
||||
|
||||
### GET `/ingestion/status`
|
||||
|
||||
Returns poller state and a concise operational summary. It does not expose secrets or the contents of arbitrary S3 objects.
|
||||
|
||||
Example response:
|
||||
|
||||
```json
|
||||
{
|
||||
"poller": {
|
||||
"enabled": true,
|
||||
"running": true,
|
||||
"poll_interval_seconds": 10,
|
||||
"last_poll_started_at": "2026-10-14T07:00:00Z",
|
||||
"last_poll_completed_at": "2026-10-14T07:00:01Z"
|
||||
},
|
||||
"counts": {
|
||||
"discovered": 12,
|
||||
"processed": 8,
|
||||
"pending": 3,
|
||||
"technical_failures": 1
|
||||
},
|
||||
"last_success_at": "2026-10-14T06:59:50Z"
|
||||
}
|
||||
```
|
||||
|
||||
The values above are illustrative. The configured polling interval is implementation configuration. This endpoint is observational; the dashboard does not need to start the poller or submit files to it.
|
||||
|
||||
### Import workflow (internal, not an HTTP upload endpoint)
|
||||
|
||||
1. The poller lists `inbox/` and ignores non-PDF keys, hidden files, and known temporary upload keys.
|
||||
2. A candidate is eligible only after its size and ETag remain unchanged across two consecutive listings.
|
||||
3. The service downloads the PDF, computes its content hash, and extracts all available text from every page. OCR is out of scope for the initial release.
|
||||
4. Empty/insufficient text, ambiguous deterministic classification, or invalid required fields routes the document to `pending/` with a review reason and the business status `pending`.
|
||||
5. An unambiguous classification with valid required fields produces a target key. The service resolves collisions with an incremental suffix, copies the object, records the sidecar, synchronizes SQLite, and deletes the source only according to the safe copy-before-delete procedure.
|
||||
6. A successful automatic move receives `processed`. A user-confirmed or corrected document receives `validated`.
|
||||
7. Sidecars contain the complete extracted text and event history. SQLite indexes that text for full-text search across all pages.
|
||||
8. Interrupted work is reconciled at startup and by the consistency job. The importer must be idempotent and must not delete the only recoverable copy.
|
||||
|
||||
### Technical failures
|
||||
|
||||
Technical failures are not business statuses and must not automatically set a document to `pending`. For a transient technical failure, the service makes the initial attempt and up to three retries after 5 seconds, 30 seconds, and 2 minutes. Once these retries are exhausted, processing of that item stops; no additional retry or manual-review workflow is launched. Other objects continue to be discovered and processed. The API may expose the last technical failure in operational diagnostics, separate from the document's business status.
|
||||
|
||||
## 6. File list and search
|
||||
|
||||
### GET `/files`
|
||||
|
||||
Query parameters:
|
||||
|
||||
| Parameter | Type | Description |
|
||||
|---|---|---|
|
||||
| `page` | integer | Page number, default 1 |
|
||||
| `page_size` | integer | Page size, default 50, max 200 |
|
||||
| `q` | string | Search current/original key, common fields, optional fields, and complete extracted text from all pages |
|
||||
| `status` | string | Repeatable or comma-separated values: `pending`, `processed`, `validated` |
|
||||
| `template` | string | Exact configured template name |
|
||||
| `document_type` | string | Exact document type |
|
||||
| `emitting_company` | string | Normalized company name |
|
||||
| `period` | string | Exact period string |
|
||||
| `date_from` | date | Inclusive emission-date lower bound |
|
||||
| `date_to` | date | Inclusive emission-date upper bound |
|
||||
| `sort_by` | string | Allowlisted: `emission_date`, `period`, `current_key`, `status`, `template_name`, `document_type`, `emitting_company`, `created_at`, `updated_at` |
|
||||
| `sort_order` | string | `asc` or `desc` |
|
||||
|
||||
Example: `GET /api/v1/files?status=pending&q=EDF&page=1&page_size=25`
|
||||
|
||||
Response:
|
||||
|
||||
```json
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"sha256": "0123456789abcdef...",
|
||||
"current_key": "pending/edf/document.pdf",
|
||||
"original_key": "inbox/document.pdf",
|
||||
"status": "pending",
|
||||
"template_name": "invoice",
|
||||
"emission_date": "2026-09-20",
|
||||
"period": "September 2026",
|
||||
"emitting_company": "EDF",
|
||||
"document_type": "invoice",
|
||||
"confidence": null,
|
||||
"updated_at": "2026-10-14T07:00:00Z"
|
||||
}
|
||||
],
|
||||
"page": 1,
|
||||
"page_size": 25,
|
||||
"total_items": 1,
|
||||
"total_pages": 1
|
||||
}
|
||||
```
|
||||
|
||||
The response must not claim ML confidence for a rules-only decision. `confidence` is nullable and may be omitted if not applicable. List items use `sha256` as the canonical identifier and return the computed `period`. Search must include the full extracted text from every page, not only the first page. Matching semantics (substring/token behavior) should be documented by the implementation and tested; use SQLite indexes/FTS rather than loading every record into application memory.
|
||||
|
||||
## 7. Document detail and PDF content
|
||||
|
||||
### GET `/files/{sha256}`
|
||||
|
||||
Returns the latest sidecar-backed record, including fields needed by the detail page. SQLite may be used to locate the record, but authoritative metadata must be reconciled from the sidecar where needed.
|
||||
|
||||
Example response:
|
||||
|
||||
```json
|
||||
{
|
||||
"sha256": "0123456789abcdef...",
|
||||
"original_key": "inbox/document.pdf",
|
||||
"current_key": "pending/document.pdf",
|
||||
"status": "pending",
|
||||
"template_name": "invoice",
|
||||
"emission_date": "2026-09-20",
|
||||
"period": "September 2026",
|
||||
"emitting_company": "EDF",
|
||||
"document_type": "invoice",
|
||||
"optional_fields": {"invoice_number": null, "amount": 125.5},
|
||||
"target_key_preview": null,
|
||||
"confidence": null,
|
||||
"events": [
|
||||
{
|
||||
"type": "imported",
|
||||
"timestamp": "2026-10-14T07:00:00Z",
|
||||
"payload": {"source_key": "inbox/document.pdf"}
|
||||
}
|
||||
],
|
||||
"created_at": "2026-10-14T07:00:00Z",
|
||||
"updated_at": "2026-10-14T07:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
The event history is read-only through this endpoint. Do not expose internal credentials or unnecessary infrastructure details.
|
||||
|
||||
### GET `/files/{sha256}/pdf`
|
||||
|
||||
Streams the current PDF from Garage through the backend for the embedded viewer. It must not return a public or signed URL that bypasses the local service.
|
||||
|
||||
Requirements:
|
||||
- Return `Content-Type: application/pdf` and a safe inline `Content-Disposition`.
|
||||
- Support HTTP range requests (`206 Partial Content`) if supported by the chosen PDF viewer and storage client; otherwise document the limitation.
|
||||
- Return `404` if the object is genuinely missing and create/retain a consistency anomaly for diagnosis.
|
||||
- Never return the extracted text in place of the PDF bytes.
|
||||
|
||||
### GET `/files/{sha256}/text`
|
||||
|
||||
Returns the complete extracted text stored in the sidecar/replica, including all pages. Useful for a separate text panel or troubleshooting; the UI may instead use the detail response if the payload size is acceptable.
|
||||
|
||||
```json
|
||||
{
|
||||
"sha256": "0123456789abcdef...",
|
||||
"text": "Complete text from page 1...\n\nComplete text from page 2..."
|
||||
}
|
||||
```
|
||||
|
||||
## 8. Preview and validation
|
||||
|
||||
All mutation endpoints must validate the requested state, recompute the period from `emission_date` and the applicable template period rules, calculate the destination key from the template's key pattern, and prevent an invalid document from being renamed. Filename parsing must never be used as a source for the period.
|
||||
|
||||
### Shared request fields
|
||||
|
||||
```json
|
||||
{
|
||||
"template_name": "invoice",
|
||||
"emission_date": "2026-09-20",
|
||||
"emitting_company": "EDF",
|
||||
"document_type": "invoice",
|
||||
"optional_fields": {
|
||||
"invoice_number": "INV-2026-001",
|
||||
"amount": 125.5
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `template_name`, `emission_date`, `emitting_company`, and `document_type` must conform to the selected template's schema.
|
||||
- The server selects the applicable period rules from the configured mapping for the company and document type, calculates `period` from `emission_date`, and never accepts a client-supplied period or period format.
|
||||
- `optional_fields` may contain only fields declared by the selected template. Empty optional fields are allowed where the schema permits them.
|
||||
- The server must never trust a target key supplied by the browser; it calculates the target key itself.
|
||||
|
||||
### POST `/files/{sha256}/preview`
|
||||
|
||||
Validates a proposed set of values and returns the computed period, target key, and a simple error message if validation fails without changing the sidecar, status, or S3 objects. The UI calls this as values change (debounced by the frontend).
|
||||
|
||||
Request: shared request fields above.
|
||||
|
||||
Response when valid:
|
||||
|
||||
```json
|
||||
{
|
||||
"valid": true,
|
||||
"computed_period": "September 2026",
|
||||
"target_key": "files/edf/2026/09/invoice-INV-2026-001.pdf",
|
||||
"error": null
|
||||
}
|
||||
```
|
||||
|
||||
Response when invalid is still a normal validation result (`200`), not an infrastructure error:
|
||||
|
||||
```json
|
||||
{
|
||||
"valid": false,
|
||||
"computed_period": "September 2026",
|
||||
"target_key": null,
|
||||
"error": "Select a configured company."
|
||||
}
|
||||
```
|
||||
|
||||
The preview response contains the computed target key.
|
||||
|
||||
### POST `/files/{sha256}/validate`
|
||||
|
||||
Finalizes a document after the user confirms or corrects its metadata.
|
||||
|
||||
Request:
|
||||
|
||||
```json
|
||||
{
|
||||
"template_name": "invoice",
|
||||
"emission_date": "2026-09-20",
|
||||
"emitting_company": "EDF",
|
||||
"document_type": "invoice",
|
||||
"optional_fields": {"invoice_number": "INV-2026-001", "amount": 125.5}
|
||||
}
|
||||
```
|
||||
|
||||
Success response:
|
||||
|
||||
```json
|
||||
{
|
||||
"sha256": "0123456789abcdef...",
|
||||
"status": "validated",
|
||||
"previous_key": "pending/document.pdf",
|
||||
"current_key": "files/edf/2026/09/invoice-INV-2026-001.pdf",
|
||||
"renamed": true,
|
||||
"sidecar_updated": true,
|
||||
"sqlite_synced": true,
|
||||
"updated_at": "2026-10-14T07:05:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
Behavior:
|
||||
1. Validate the requested template and every required field.
|
||||
2. Recompute and validate the period from emission date and template rules.
|
||||
3. Compute the target key on the server and resolve collisions with an incremental suffix.
|
||||
4. If the target key differs from the current key, copy to the target before deleting the source. If the key is unchanged, do not perform an unnecessary rename.
|
||||
5. Persist the authoritative sidecar, including corrected values and a `validated` event, then synchronize SQLite.
|
||||
6. Return the final key and status.
|
||||
|
||||
If required values are invalid, return `422` with a concise plain-text error message and do not rename. If a storage operation is interrupted, leave a recoverable state and reconcile from the sidecar; never delete the only recoverable copy. The operation must be idempotent.
|
||||
|
||||
## 9. Consistency and maintenance jobs
|
||||
|
||||
### GET `/consistency`
|
||||
|
||||
Returns known anomalies, optionally filtered by `kind`, `page`, and `page_size`.
|
||||
|
||||
Anomaly kinds include:
|
||||
- `orphaned_object`: a PDF object exists without a sidecar.
|
||||
- `missing_object`: a sidecar refers to a PDF key that is not present.
|
||||
- `missing_sidecar`: an indexed/managed PDF has no authoritative sidecar.
|
||||
- `sqlite_drift`: SQLite differs from the sidecar.
|
||||
- `interrupted_operation`: state suggests a copy/delete operation was interrupted.
|
||||
|
||||
Response items should include `id`, `kind`, `sha256` when known, affected key, detection timestamp, concise details, and whether a safe repair is available. Listing anomalies must not mutate storage.
|
||||
|
||||
### POST `/consistency/scan`
|
||||
|
||||
Starts a consistency scan. Returns `202 Accepted`:
|
||||
|
||||
```json
|
||||
{"job_id": "job_abc", "kind": "consistency_scan", "status": "queued"}
|
||||
```
|
||||
|
||||
Only one scan of the same kind may run at a time; a duplicate start returns `409`.
|
||||
|
||||
### POST `/consistency/rebuild`
|
||||
|
||||
Rebuilds the SQLite index from authoritative sidecars. Returns `202 Accepted` with a job ID. This operation must not rewrite or delete PDF objects and sidecars just to match SQLite. It replaces/recreates the replica and records a job result.
|
||||
|
||||
### GET `/jobs/{job_id}`
|
||||
|
||||
Returns job status for consistency scan, SQLite rebuild, or other asynchronous maintenance tasks.
|
||||
|
||||
```json
|
||||
{
|
||||
"job_id": "job_abc",
|
||||
"kind": "sqlite_rebuild",
|
||||
"status": "running",
|
||||
"started_at": "2026-10-14T07:00:00Z",
|
||||
"finished_at": null,
|
||||
"progress": {"completed": 125, "total": 500},
|
||||
"result": null,
|
||||
"error": null
|
||||
}
|
||||
```
|
||||
|
||||
Allowed job statuses are technical job states such as `queued`, `running`, `succeeded`, and `failed`; these are not document business statuses. This generic endpoint reports maintenance and retraining jobs. If the process restarts and in-memory job state is lost, the API should report the job as interrupted/unknown and use persisted status files or database state where available.
|
||||
|
||||
## 10. Post-v1 model endpoints
|
||||
|
||||
These routes are future extensions; they are not required for the deterministic initial release.
|
||||
|
||||
- `GET /models`: list locally stored model versions and identify the active model.
|
||||
- `POST /retrain`: start training using the initial corpus plus eligible `validated` documents. Returns `202` and a job ID.
|
||||
- `GET /jobs/{job_id}`: show training state, dataset/model version, accuracy, confusion matrix, warnings, and output artifact.
|
||||
- `POST /retrain/{job_id}/activate`: atomically activate a successfully evaluated candidate.
|
||||
- `POST /retrain/{job_id}/discard`: discard a candidate without changing the active model.
|
||||
|
||||
Training and activation must remain local. Failed training must not replace the active model. ML confidence and candidate scores are absent or null when no model was used.
|
||||
|
||||
## 11. Consistency and implementation rules
|
||||
|
||||
1. Sidecar is the source of truth; SQLite is a rebuildable query replica.
|
||||
2. Every mutation writes an auditable sidecar event and updates `updated_at`; event types include `imported`, `classified`, `renamed`, `corrected`, `validated`, and `reprocessed` where applicable.
|
||||
3. Full extracted text from every PDF page is persisted in the sidecar and indexed in SQLite for full-text search.
|
||||
4. Period is derived only from `emission_date` and the selected template's configured period rules. The filename is never a period input.
|
||||
5. All final object keys are calculated server-side from the selected template and validated field values.
|
||||
6. Rename/move uses copy-before-delete, resolves collisions with an incremental suffix, and records the final key.
|
||||
7. All mutation endpoints are idempotent where feasible and protect against stale edits using `expected_revision` or an equivalent concurrency token.
|
||||
8. Technical processing failures are not assigned a business status. After the defined retry sequence is exhausted, that item stops processing without another workflow.
|
||||
9. The API does not expose an endpoint to undo or reverse a file rename.
|
||||
10. No endpoint uploads a PDF directly; the Nextcloud/S3 inbox remains the ingestion boundary.
|
||||
|
||||
## 12. Open implementation choices
|
||||
|
||||
The following are implementation details, not reasons to block the API contract:
|
||||
- Exact stable ID strategy and revision token format.
|
||||
- Exact SQLite full-text search tokenizer and phrase/prefix matching semantics.
|
||||
- Whether PDF byte-range requests are supported by the chosen S3 client/viewer.
|
||||
- Exact representation of technical processing diagnostics after retries are exhausted.
|
||||
+205
-25
@@ -30,8 +30,7 @@ flowchart TD
|
||||
worker --> db
|
||||
worker --> model
|
||||
worker --> s3
|
||||
poll --> s3
|
||||
poll --> db
|
||||
poll --> api
|
||||
nc --> s3
|
||||
user --> nc
|
||||
```
|
||||
@@ -39,7 +38,7 @@ flowchart TD
|
||||
## 2. Architecture Goals and Constraints
|
||||
|
||||
- Offline-first: no runtime downloads, no external services (NFR1, AC10).
|
||||
- Zero silent wrong renames: every ambiguous path routes to not processed (NFR3).
|
||||
- Zero silent wrong renames: every ambiguous path routes to `pending` (NFR3).
|
||||
- Crash safety: copy-then-delete renames, atomic sidecar writes, SQLite repairable from sidecars alone (NFR4, AC8, AC9).
|
||||
- Minimal operational surface: one container, one JSONC configuration file, env-var credentials (NFR6).
|
||||
- Single user, local network: performance targets are modest (NFR2) but correctness targets are strict.
|
||||
@@ -53,7 +52,7 @@ flowchart TD
|
||||
| Text extraction | PyMuPDF | Fastest option; AGPL acceptable for a personal offline tool. Wrapped behind a small extractor interface so it remains swappable (PRD open question resolved). |
|
||||
| S3 client | boto3 | Most complete API coverage (copy-object, versioning, ETag), mature Garage compatibility. Wrapped behind a thin storage interface; v1 ships one implementation. |
|
||||
| SQLite access | SQLAlchemy ORM, WAL mode | Declared replica schema, typed models, migrations via Alembic. Sidecar remains authoritative; ORM layer only reads/writes the mirror. |
|
||||
| ML classification | scikit-learn TF-IDF + Logistic Regression | PRD FR3.1; small corpus is sufficient, no deep learning needed. |
|
||||
| Initial classification | Configured deterministic rules and reference data | Initial release must work without ML; TF-IDF + Logistic Regression is a post-v1 enhancement. |
|
||||
| Fuzzy matching | rapidfuzz | PRD FR4.2, counterparty gazetteer. |
|
||||
| Dates | python-dateutil | PRD FR4.2, emission date parsing and period computation. |
|
||||
| Frontend | React + Vite, react-pdf viewer | NFR7; built to static assets served by FastAPI. |
|
||||
@@ -73,7 +72,7 @@ One Python process runs:
|
||||
- Poller task: asyncio loop, configurable interval, lists the `inbox/` prefix.
|
||||
- Job supervisor: spawns subprocesses for retrain, consistency scan, and SQLite rebuild; tracks their status in memory and in SQLite.
|
||||
|
||||
On startup, the process runs the recovery scan (FR1.3, FR10.3): objects in `inbox/` and `pending/` without sidecar records are enqueued for processing.
|
||||
On startup, the process recovers interrupted technical operations and discovers eligible objects in `inbox/`. The `pending/` prefix is reconciled for metadata and anomalies; documents awaiting manual correction are not automatically reclassified or renamed until corrected/requested.
|
||||
|
||||
### 4.2 Module layout
|
||||
|
||||
@@ -101,18 +100,21 @@ One ingestion run, fully synchronous within the poller task:
|
||||
```mermaid
|
||||
flowchart TD
|
||||
in["Stable object in inbox/"] --> dl["Download PDF"]
|
||||
dl --> tx["Extract first-page text (PyMuPDF)"]
|
||||
dl --> tx["Extract text from every PDF page (PyMuPDF)"]
|
||||
tx --> empty{"Text empty?"}
|
||||
empty -- yes --> pending["Move to pending/ (reason: empty text)"]
|
||||
empty -- no --> cls["Classify (TF-IDF + LogReg)"]
|
||||
cls --> conf{"Confidence >= threshold?"}
|
||||
conf -- no --> pending
|
||||
conf -- yes --> fill["Fill fields: regexes, companies list, additional fields from groups, period from emission date"]
|
||||
empty -- no --> cls["Classify using deterministic rules"]
|
||||
cls --> unamb{"Exactly one unambiguous rule match?"}
|
||||
unamb -- no --> pending
|
||||
unamb -- yes --> fill["Fill fields: regexes, companies list, additional fields from groups, period from emission date"]
|
||||
fill --> valid{"Fields valid?"}
|
||||
valid -- no --> pending
|
||||
valid -- yes --> rename["Compute key, copy-then-delete, write sidecar + SQLite"]
|
||||
valid -- yes --> rename["Compute key and copy object to target"]
|
||||
rename --> sidecar["Persist sidecar with target key and event"]
|
||||
sidecar --> delete["Delete source only after sidecar is durable"]
|
||||
delete --> sqlite["Synchronize SQLite replica"]
|
||||
rename --> done["Status: processed"]
|
||||
pending --> wait["Status: not processed (sidecar with failing fields)"]
|
||||
pending --> wait["Status: pending (sidecar with failing fields)"]
|
||||
```
|
||||
|
||||
Every terminal step writes a sidecar event (classified, renamed, corrected, validated) per FR5.2. Sidecar first, SQLite second (FR5.1); the consistency job repairs any drift after a crash.
|
||||
@@ -126,17 +128,193 @@ Every terminal step writes a sidecar event (classified, renamed, corrected, vali
|
||||
### 4.5 Metadata and replica
|
||||
|
||||
- Sidecar JSON is the sole source of truth; schema per PRD section 7 (now including the period field).
|
||||
- SQLAlchemy models `files` and `consistency_flags` mirror it. `extracted_text` is stored in SQLite for search; it is a replica, and the rebuild command repopulates it from sidecars.
|
||||
- SQLAlchemy models `files` and `consistency_flags` mirror it. `extracted_text` contains the complete text extracted from every PDF page and is stored in SQLite for full-text search alongside metadata/field search; it is a replica, and the rebuild command repopulates it from sidecars.
|
||||
- `POST /consistency/rebuild` and the CLI rebuild command both clear and repopulate SQLite from all sidecars (AC9).
|
||||
- Divergence resolution is always sidecar-wins (PRD 3.2); the consistency job flags drift rather than guessing.
|
||||
|
||||
### 4.6 Period computation
|
||||
### 4.5.1 Sidecar file format and schema
|
||||
|
||||
A pure function of (emission date, template period rules): granularity in {day, month, quarter, year} and offset in {current, next}. Produces the human-readable period string (e.g. "Q3 2026", "September 2026", "2026"). Recomputed whenever the emission date or template changes on the detail page; the user can override the suggested value, and validation checks consistency between period, date, and template rules (FR4.2, FR4.3, AC5c).
|
||||
The sidecar is the authoritative metadata record for one PDF. Its schema and serialization behavior are implemented in `app/sidecar.py`; this section documents that implementation. The current schema version is `"1.0"`.
|
||||
|
||||
#### File naming and location
|
||||
|
||||
A sidecar key is derived from the PDF object key by replacing a terminal, lowercase `.pdf` extension with `.meta.json`. If the key does not end in `.pdf`, `.meta.json` is appended. For example:
|
||||
|
||||
- `files/invoice.pdf` → `files/invoice.meta.json`
|
||||
- `inbox/document.pdf` → `inbox/document.meta.json`
|
||||
|
||||
|
||||
#### Canonical schema example
|
||||
|
||||
The following is a JSONC-readable example. Comments explain the fields and are not emitted by the current serializer.
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"schema_version": "1.0", // Required. Sidecar schema version.
|
||||
"sha256": "0123456789abcdef...", // Required. SHA-256 hex digest of the PDF bytes.
|
||||
"original_key": "inbox/invoice.pdf", // Required. Original S3 object key before renames.
|
||||
"current_key": "files/acme/invoice-2026-01.pdf", // Required. Current S3 key for the PDF.
|
||||
"status": "processed", // Required. One of the three business statuses below.
|
||||
"template_name": "invoice", // Required. Template/document family selected for this PDF.
|
||||
"period_format": "monthly", // Required. Format/rule identifier used for the period string.
|
||||
"confidence": null, // Null when no ML model produced a score; never fabricate a rules-only confidence.
|
||||
"emission_date": "2026-01-15", // Required. Emission date; expected ISO YYYY-MM-DD.
|
||||
"period": "2026-01", // Required. Computed or validated period string.
|
||||
"emitting_company": "ACME", // Required. Normalized name of the issuing company.
|
||||
"document_type": "invoice", // Required. Fine-grained document type.
|
||||
"optional_fields": { // Serialized by the model; template-specific extracted values.
|
||||
"invoice_number": "INV-2026-001",
|
||||
"amount": 125.5,
|
||||
"currency": "EUR"
|
||||
},
|
||||
"extracted_text": "Invoice ACME ... [text from all pages]", // Complete text extracted from every page.
|
||||
"created_at": "2026-01-15T10:20:30+00:00", // Required. Creation timestamp in ISO 8601.
|
||||
"updated_at": "2026-01-15T10:21:05+00:00", // Required. Last sidecar mutation timestamp in ISO 8601.
|
||||
"events": [ // Required array; may be empty.
|
||||
{
|
||||
"type": "classified", // Event name; see supported event names below.
|
||||
"payload": {
|
||||
"template_name": "invoice",
|
||||
"confidence": null
|
||||
}, // Required object; event-specific data, intentionally extensible.
|
||||
"timestamp": "2026-01-15T10:20:45+00:00" // Required ISO 8601 timestamp.
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
#### Field definitions
|
||||
|
||||
| Field | Type | Presence and meaning |
|
||||
|---|---|---|
|
||||
| `schema_version` | string | Required. Must equal `"1.0"` in the current implementation. |
|
||||
| `sha256` | string | Required. SHA-256 digest of the exact PDF bytes and canonical document identifier. Any byte-level content change creates a new identifier; renames/moves that preserve bytes do not. |
|
||||
| `original_key` | string | Required. S3 key where the PDF was first registered, retained across renames. |
|
||||
| `current_key` | string | Required. S3 key currently associated with the PDF. |
|
||||
| `status` | string | Required. One of `pending`, `processed`, or `validated`. |
|
||||
| `template_name` | string | Required. Name of the configured template/document family. |
|
||||
| `period_format` | string | Required. Period format/rule label, e.g. `monthly` or `yearly`. |
|
||||
| `confidence` | number or null | Nullable; absent/null when no ML model produced a score. Never default to `0.0` for deterministic rules. |
|
||||
| `emission_date` | string | Required. Document emission date; `build_sidecar` documents ISO `YYYY-MM-DD`. |
|
||||
| `period` | string | Required. Period derived from the emission date and applicable rules. |
|
||||
| `emitting_company` | string | Required. Issuing company name. |
|
||||
| `document_type` | string | Required. More specific document type within the template. |
|
||||
| `optional_fields` | object | Emitted by the dataclass serializer, default `{}`. Template-specific values; values may be strings, numbers, booleans, arrays, objects, or null as allowed by the application schema. |
|
||||
| `extracted_text` | string | Emitted by the dataclass serializer, default empty string. Complete text extracted from all PDF pages. |
|
||||
| `created_at` | string | Required. ISO 8601 timestamp set on object construction if not supplied. |
|
||||
| `updated_at` | string | Required. ISO 8601 timestamp set on construction and refreshed by `add_event` and `set_status`. |
|
||||
| `events` | array of objects | Required. Chronological event history; each event has `type`, `payload`, and `timestamp`. |
|
||||
|
||||
The dataclass serializer emits all fields listed above, including `confidence`, `optional_fields`, and `extracted_text`, even though those three are not in the implementation's required-field set. `Sidecar.from_dict` supplies defaults for omitted fields, but `validate_sidecar` still requires the core required-field set described below.
|
||||
|
||||
#### Business statuses and event types
|
||||
|
||||
The only business statuses accepted by `validate_sidecar` are:
|
||||
|
||||
- `pending` — deterministic rules could not produce a valid result; the item awaits manual correction.
|
||||
- `processed` — processed by the pipeline.
|
||||
- `validated` — reviewed/confirmed by the owner.
|
||||
|
||||
Technical progress and error details must not be invented as additional business statuses. Store retry state, attempt count, next retry time, and failure details in technical processing metadata/events, separate from the three business statuses. The retry schedule is fixed: 3 retries after the initial attempt, at 5 seconds, 30 seconds, and 2 minutes. Once these retries are exhausted, stop processing that item without launching another workflow. Use `pending` only when manual correction of document classification or fields is required.
|
||||
|
||||
The event names declared by `EventType` are:
|
||||
|
||||
- `classified`
|
||||
- `renamed`
|
||||
- `validated`
|
||||
- `corrected`
|
||||
- `imported`
|
||||
- `reprocessed`
|
||||
|
||||
Each event is an object with exactly the following required structure:
|
||||
|
||||
- `type`: string event name.
|
||||
- `payload`: object containing event-specific details; the schema intentionally allows different payloads for different event types.
|
||||
- `timestamp`: ISO 8601 timestamp generated in UTC when `Sidecar.add_event` is called.
|
||||
|
||||
The current validator checks that these three event properties have the correct broad types, but it does not enforce the event-name enum, timestamp parseability, or a per-event payload schema.
|
||||
|
||||
### 4.5.2 SQLite database schema
|
||||
|
||||
This section describes the schema and indexes used by the database. SQLite is a local, rebuildable metadata index; sidecars in object storage remain the source of truth. The default database path is `./renamer.db`, overridden by the `SQLITE_PATH` environment variable. The resolved parent directory is created automatically.
|
||||
|
||||
#### Table: `files`
|
||||
|
||||
The implementation should create one table, `files`. Columns are stored as follows:
|
||||
|
||||
| Column | SQLite type | Constraints / default | Meaning |
|
||||
|---|---|---|---|
|
||||
| `sha256` | `TEXT` | `PRIMARY KEY` | SHA-256 digest of exact PDF bytes; canonical document identifier. |
|
||||
| `original_key` | `TEXT` | Nullable | Original object key. |
|
||||
| `current_key` | `TEXT` | Nullable | Current object key. |
|
||||
| `status` | `TEXT` | Nullable | Processing/business status. |
|
||||
| `template_name` | `TEXT` | Nullable | Selected document template. |
|
||||
| `period_format` | `TEXT` | Nullable | Period-format rule used for the document. |
|
||||
| `document_type` | `TEXT` | Nullable | Document type. |
|
||||
| `emitting_company` | `TEXT` | Nullable | Issuing company. |
|
||||
| `period` | `TEXT` | Nullable | Computed or corrected accounting/document period. |
|
||||
| `emission_date` | `TEXT` | Nullable | Document emission date. |
|
||||
| `confidence` | `REAL` | Nullable | Classification confidence. |
|
||||
| `optional_fields` | `TEXT` | Nullable | JSON-serialized object of template-specific fields. |
|
||||
| `extracted_text` | `TEXT` | Nullable | Complete extracted document text from all pages, used by full-text search. |
|
||||
| `created_at` | `TEXT` | Nullable | Creation timestamp, represented as text (normally ISO 8601). |
|
||||
| `updated_at` | `TEXT` | Nullable | Last-update timestamp, represented as text (normally ISO 8601). |
|
||||
|
||||
SQLite does not declare SQL-level defaults for these columns beyond the constraints shown. Defaults and normalization are supplied by Python.
|
||||
|
||||
#### Indexes
|
||||
|
||||
The code should creates these named indexes :
|
||||
|
||||
| Index | Indexed column(s) | Purpose / usage |
|
||||
|---|---|---|
|
||||
| `idx_files_status` | `status` | Supports filtering by status and status-oriented listings. |
|
||||
|
||||
SQLite creates the primary-key index for `sha256`.
|
||||
|
||||
## 4.6 Period computation
|
||||
|
||||
#### 4.6.1 Requirement
|
||||
|
||||
The system shall generate the `period` field exclusively from the document's `emission_date` and the period rule configured by the selected template. The filename shall not be used as an input to period generation or to override the generated value. The system shall parse and validate `emission_date`, apply the template's granularity, offset, and output format, and store the resulting string in `period`. If the date or rule is invalid or insufficient to generate a period, the system shall report a clear error rather than infer a value from the filename.
|
||||
|
||||
#### 4.6.2 Source date
|
||||
|
||||
`emission_date` is the sole date input for period generation. The system shall accept a date value or supported date strings, including ISO `YYYY-MM-DD`. Invalid or ambiguous values that cannot be resolved safely shall produce a validation error.
|
||||
|
||||
#### 4.6.3 Template period rules
|
||||
|
||||
The selected template shall define the period granularity and may define an offset and a custom output format.
|
||||
|
||||
| Property | Allowed values | Meaning |
|
||||
|---|---|---|
|
||||
| `granularity` | `day`, `week`, `month`, `quarter`, `semester`, `year` | Unit represented by the generated period. Defaults to `month` if omitted. |
|
||||
| `offset` | `current`, `previous`, `following` | Period containing the emission date, the immediately preceding period, or the immediately following period. Defaults to `current`. |
|
||||
| `format` | Template format string | Defines the final representation of the period. |
|
||||
|
||||
The system shall apply the offset at the selected granularity, correctly handling month, quarter, semester, year, and ISO-week boundaries. Weekly periods shall use ISO week numbering and the associated ISO week-year. Unknown granularity or offset values shall be reported as configuration errors; they shall not trigger filename-based fallback.
|
||||
|
||||
#### 4.6.4 Period formatting
|
||||
|
||||
If a custom format is defined, the system shall generate `period` by substituting supported placeholders with values derived from the emission date after applying the template's granularity and offset. Supported placeholders are:
|
||||
|
||||
| Placeholder | Meaning |
|
||||
|---|---|
|
||||
| `{year}` | Four-digit year associated with the computed period. |
|
||||
| `{year_next}` / `{year_previous}` | Following / preceding calendar year. |
|
||||
| `{month}` | Zero-padded month number (`01`–`12`). |
|
||||
| `{month_name}` | Full French month name. |
|
||||
| `{month_abbr}` | English three-letter month abbreviation. |
|
||||
| `{day}` | Zero-padded day; use the first day as the representative day for month-, quarter-, semester-, and year-based periods. |
|
||||
| `{quarter}` | Quarter number (`1`–`4`). |
|
||||
| `{semester}` | Semester number (`1`–`2`). |
|
||||
| `{week}` | ISO week number (`1`–`53`). |
|
||||
|
||||
When no custom format is provided, the system shall use a deterministic standard representation appropriate to the granularity, with French month names by default. An unknown placeholder shall produce a clear format error.
|
||||
|
||||
### 4.7 Frontend
|
||||
|
||||
The dashboard is served as static assets by FastAPI. Views: file list (filters, counters, badges), detail page (PDF preview and editable metadata), and a Config page for editing JSONC configuration. Saving validates the content before updating the configured file and reloading the in-memory template registry. Auth: API routes use the configured dashboard credentials.
|
||||
The dashboard is served as static assets by FastAPI. Views: file list (filters, status badges, consistency indicator), detail page (PDF preview and editable metadata), and a Config page for editing the JSONC configuration file. The editor retrieves the current file through `GET /config` and saves the modified file through `PUT /config`; the server validates it before replacing the configured file and reloading the in-memory template registry. Auth: API routes use the configured dashboard credentials.
|
||||
|
||||
## 5. Data Flow: Renaming and Validation
|
||||
|
||||
@@ -146,13 +324,15 @@ sequenceDiagram
|
||||
participant API as FastAPI
|
||||
participant S3 as Garage
|
||||
participant DB as SQLite
|
||||
U->>API: POST /files/{id}/validate (template, fields)
|
||||
U->>API: POST /files/{sha256}/validate (template, fields)
|
||||
API->>API: Validate fields, compute target key
|
||||
alt key changed
|
||||
API->>S3: Copy object to target key
|
||||
API->>S3: Delete old key
|
||||
API->>S3: Atomic sidecar update with new current_key and validated event
|
||||
API->>S3: Delete old key only after sidecar is durable
|
||||
else key unchanged
|
||||
API->>S3: Atomic sidecar update (status: validated, event)
|
||||
end
|
||||
API->>S3: Atomic sidecar update (status: validated, event)
|
||||
API->>DB: Mirror update
|
||||
API-->>U: 200, new key
|
||||
```
|
||||
@@ -172,13 +352,13 @@ Subprocesses report progress and metrics by writing a status JSON consumed by th
|
||||
## 7. Configuration and Secrets
|
||||
|
||||
- Templates: single JSONC file defining: document family, document types, period rules, optional field schema, extraction regexes, validation rules, key pattern, target prefix.
|
||||
- Environment: S3 endpoint, access key, secret key, bucket name, poll interval, confidence threshold, dashboard credentials, paths (SQLite file, model registry, template YAML).
|
||||
- No secrets in the YAML; no secrets in sidecars.
|
||||
- Environment: S3 endpoint, access key, secret key, bucket name, poll interval, optional post-v1 ML confidence threshold, dashboard credentials, paths (SQLite file, model registry, template JSONC).
|
||||
- No secrets in the JSONC; no secrets in sidecars.
|
||||
|
||||
## 8. Error Handling and Crash Safety
|
||||
|
||||
- Ingestion: any unexpected exception marks the file not processed with the reason in the sidecar; the poller continues.
|
||||
- Rename: copy strictly before delete; a crash between the two leaves both keys, and the startup scan re-evaluates (the sidecar's current key is authoritative).
|
||||
- Ingestion: classify errors as transient/retryable technical failures, permanent technical failures, or document/business validation failures. Retry transient failures up to 3 times after the initial attempt, with delays of 5 seconds, 30 seconds, and 2 minutes, using idempotent recovery. After the final retry fails, stop processing that item without another retry or review workflow. Do not set business status `pending` solely because a technical operation failed. Record technical state and error details separately from business status, and continue processing other objects.
|
||||
- Rename: copy to the target key, persist the sidecar with the new current key and event, then delete the old key; synchronize SQLite after the authoritative sidecar is durable. If a crash occurs before deletion, recovery follows the sidecar's current key and safely removes or reconciles the stale copy.
|
||||
- Sidecar/SQLite: sidecar first; drift repaired by the consistency job; rebuild is always possible from sidecars alone.
|
||||
- Retrain: trained models land in a new versioned directory; activation swaps an active-model pointer; a failed training run leaves the current model untouched.
|
||||
- All object-not-found conditions during processing are treated as consistency anomalies and flagged (FR5.3).
|
||||
@@ -201,11 +381,11 @@ Subprocesses report progress and metrics by writing a status JSON consumed by th
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| PyMuPDF AGPL incompatibility if the tool is ever shared | Extractor interface keeps the swap to pdfplumber cheap. |
|
||||
| Garage quirks in copy-object/versioning | Week 1 spike per PRD; storage interface isolates the client. |
|
||||
| Garage quirks in copy-object/versioning | Storage interface isolates the client. |
|
||||
| Corpus too small for reliable classification | Low-confidence routing to pending; retrain on validated files grows the corpus. |
|
||||
| SQLite drift after crash | Sidecar-first ordering + consistency job + full rebuild path. |
|
||||
| Single-process blast radius | Long jobs are subprocesses; the poller catches per-file exceptions. |
|
||||
|
||||
## 12. Traceability to PRD
|
||||
|
||||
Every epic maps to a component: ingestion (jobs/poller), extraction and classification (pipeline), data model and validation (templates, period, gazetteer, pipeline), renaming and logging (rename, sidecar), dashboards (web + api), template management (templates.py + config), retraining (jobs/retrain + models), consistency and recovery (jobs/consistency, replica/rebuild). Acceptance criteria AC1-AC12 are covered by the components above; AC8-AC10 by section 8, AC9 by section 4.5.
|
||||
Every epic maps to a component: ingestion (jobs/poller), extraction and classification (pipeline), data model and validation (templates, period, gazetteer, pipeline), renaming and logging (rename, sidecar), dashboards (web + api), JSONC configuration editing (config), retraining (jobs/retrain + models), consistency and recovery (jobs/consistency, replica/rebuild). Acceptance criteria AC1-AC12 are covered by the components above; AC8-AC10 by section 8, AC9 by section 4.5.
|
||||
|
||||
@@ -2,7 +2,9 @@
|
||||
|
||||
## 1. Vision
|
||||
|
||||
A personal, 100% offline tool that watches an S3 bucket hosted on a self-hosted Garage storage cluster, automatically identifies the type of each incoming PDF (invoice, purchase order, contract, etc.) from its content, applies the corresponding naming template by extracting the fields (date, counterparty, reference, amount), and moves the object into an organized S3 prefix structure. Users upload, download, and share files through Nextcloud, which mounts the bucket as external storage. Low-confidence decisions go through human review in a local web dashboard. Every operation is logged in JSON sidecar files (source of truth) and replicated to SQLite, and every rename is reversible.
|
||||
**Document identity:** the SHA-256 digest of the exact PDF bytes is the unique identifier. Any byte-level change to a PDF creates a new document identity; moving or renaming an unchanged PDF does not. Two imports with identical PDF bytes resolve to the same document identity, not two independent document records.
|
||||
|
||||
A personal, 100% offline tool that watches an S3 bucket hosted on a self-hosted Garage storage cluster, automatically identifies the type of each incoming PDF (invoice, purchase order, contract, etc.) from its content, applies the corresponding naming template by extracting the fields (date, counterparty, reference, amount), and moves the object into an organized S3 prefix structure. Users upload, download, and share files through Nextcloud, which mounts the bucket as external storage. Uncertain or incomplete results go through human review in a local web dashboard. Every operation is logged in JSON sidecar files (source of truth) and replicated to SQLite.
|
||||
|
||||
## 2. Problem and Context
|
||||
|
||||
@@ -10,11 +12,11 @@ PDFs arrive continuously in an inbox S3 prefix (dropped via Nextcloud by the use
|
||||
|
||||
## 3. Objectives
|
||||
|
||||
- Classify each incoming PDF against a known naming template, based on the file content (first-page text, header/footer), not the current object name.
|
||||
- Classify each incoming PDF against a known naming template, based on the full text content of every page (including headers and footers), not the current object name.
|
||||
- Fill the fields of the selected template (date, counterparty, reference, amount) from the content.
|
||||
- Rename and move the object within the bucket into an organized prefix structure depending on the document type.
|
||||
- High confidence: automatic processing. Low confidence or incomplete extraction: queued for human review in the dashboard.
|
||||
- Write metadata as JSON sidecar objects in the bucket (source of truth) and replicate every decision into SQLite for fast queries, with rollback capability and retraining from corrections.
|
||||
- Unambiguous deterministic rules and valid fields: automatic processing. If rules cannot determine a valid result or extraction is incomplete, queue the file with status `pending` for human review in the dashboard.
|
||||
- Write metadata as JSON sidecar objects in the bucket (source of truth) and replicate every decision into SQLite for fast queries, with retraining from corrections.
|
||||
- Run fully offline on self-hosted infrastructure: no cloud APIs, no data ever leaves the server.
|
||||
|
||||
## 4. Scope
|
||||
@@ -22,13 +24,13 @@ PDFs arrive continuously in an inbox S3 prefix (dropped via Nextcloud by the use
|
||||
### In scope (v1)
|
||||
|
||||
- Ingestion by polling the `inbox/` prefix of a Garage S3 bucket for new objects only. Nextcloud (External Storage app) is the user-facing interface to upload, download, and share files.
|
||||
- PDF text extraction. All files are text-based, no OCR in v1.
|
||||
- Template classification: TF-IDF + Logistic Regression (scikit-learn) on the extracted text, with a configurable confidence threshold.
|
||||
- Extract all available text from every page of each PDF. All files are text-based; OCR for scanned/image-only documents is out of scope for v1.
|
||||
- Deterministic template classification in v1 using configured rules, extracted text, and reference data; uncertain or insufficient results are routed to `pending/` for manual correction. TF-IDF + Logistic Regression is a post-v1 enhancement, not a v1 dependency.
|
||||
- Per-template slot filling: regex + dateutil for dates, amounts, references; gazetteer + fuzzy matching (rapidfuzz) for counterparties, with spaCy/CamemBERT NER as fallback if needed.
|
||||
- Templates and extractors declared in configuration (YAML/JSON), not in code.
|
||||
- Templates and extractors declared in configuration (JSONC format), not in code.
|
||||
- Storage: Garage (self-hosted S3-compatible, https://garagehq.deuxfleurs.fr/) with a single versioned bucket; objects addressed by S3 keys, no local filesystem for documents.
|
||||
- Metadata: one JSON sidecar object per PDF in the bucket, holding template, fields, confidence, status, and history — the source of truth. SQLite acts as a replicated index for dashboard queries, rebuildable from the sidecars.
|
||||
- Local web dashboard: all processed files with status (not processed / processed / validated), file detail page with PDF preview streamed from S3, template and field correction, validation, history, statistics.
|
||||
- Local web dashboard: all processed files with status (pending / processed / validated), file detail page with PDF preview streamed from S3, template and field correction, validation, history, and full-text search across the entire extracted document in addition to metadata/field search.
|
||||
- On-demand classifier retraining, using the existing corpus based on all the files with "validated" status.
|
||||
|
||||
### Out of scope (v1)
|
||||
@@ -45,22 +47,22 @@ PDFs arrive continuously in an inbox S3 prefix (dropped via Nextcloud by the use
|
||||
Single user (personal use), technically proficient. Nominal flow:
|
||||
|
||||
1. A PDF is uploaded into the `inbox/` prefix via Nextcloud (or any S3 client).
|
||||
2. The poller detects the new object, waits for write completion (stable ETag/content-length on repeated listing), downloads it, extracts the text, classifies, extracts the fields, validates.
|
||||
3. High confidence and valid fields: the object is copied to its renamed key in the target prefix and the inbox copy deleted; metadata is written to the sidecar and replicated to SQLite; the event is logged.
|
||||
4. Otherwise: the file appears in the dashboard review queue; the user corrects or validates; the correction feeds the training corpus.
|
||||
2. The poller detects the new object, waits for write completion (stable ETag/content-length on repeated listing), downloads it, extracts text from every page, applies deterministic classification and field-extraction rules, then validates.
|
||||
3. A deterministic result with valid fields: the object is copied to its renamed key in the target prefix; the sidecar is updated, then SQLite is synchronized, and the inbox copy is deleted only after the sidecar records the new key.
|
||||
4. If deterministic rules cannot produce a valid result: the file is moved to `pending/` and appears in the dashboard review queue; the user corrects or validates it. Validated corrections may feed a future ML training corpus.
|
||||
|
||||
## 6. Functional Requirements
|
||||
|
||||
- FR1: detect new PDFs in a configurable `inbox/` S3 prefix by polling the bucket listing at a configurable interval, ignoring non-PDF keys and temporary uploads (e.g. Nextcloud `.part` files).
|
||||
- FR2: classify the document into one of the configured templates, with a confidence score and an "unknown" class below the threshold.
|
||||
- FR2: classify the document using configured deterministic rules and reference data. If the evidence is insufficient or contradictory, do not guess; set status `pending` and queue the file for manual correction. Confidence scores and an "unknown" class belong to the post-v1 ML enhancement.
|
||||
- FR3: fill each template field according to its schema (type, format, validity range).
|
||||
- FR4: validate fields before renaming (parseable date, numeric amount, recognized or flagged counterparty); any failed validation routes to review.
|
||||
- FR5: rename and move within the bucket via S3 copy + delete to the template's target prefix, resolving key collisions (incremental suffix).
|
||||
- FR6: write a JSON sidecar per file (old key → new key → decision mapping, per-file status not processed / processed / validated) and replicate to SQLite, enabling rollback and selecting training data.
|
||||
- FR6: write a JSON sidecar per file (key mapping, decisions, and business status) and replicate to SQLite, enabling auditability and selecting training data.
|
||||
- FR7: local web dashboard (FastAPI + React): file list, detail page, review, correction, validation, history, retraining trigger. PDFs are streamed from the bucket for preview.
|
||||
- FR7a: the dashboard shows every processed file with its status: "not processed" (confidence was too low, rename pending), "processed" (automatically renamed and moved), or "validated" (the user confirmed or corrected the result — such files are eligible for future model retraining).
|
||||
- FR7a: the dashboard shows each document's status: `pending` (deterministic classification or required-field validation needs manual correction), `processed` (automatically renamed and moved), or `validated` (the user confirmed or corrected the result — eligible for future model retraining).
|
||||
- FR7b: clicking a file opens a detail page displaying the PDF, where the user can change the template if misclassified, correct the extracted field values, and validate the result. Validation sets the status to "validated" and feeds the corrected data into the training corpus.
|
||||
- FR8: adding a new template is done through a configuration entry (YAML) plus example files, with no code changes.
|
||||
- FR8: adding a new template is done by editing the JSONC configuration file plus example files, with no code changes. Dedicated endpoints retrieve the current configuration file and upload a validated modified version.
|
||||
|
||||
## 7. Non-Functional Requirements
|
||||
|
||||
@@ -76,15 +78,15 @@ Single user (personal use), technically proficient. Nominal flow:
|
||||
|
||||
- ≥ 95% correct template classification on typical files after initial training on the existing corpus.
|
||||
- ≥ 80% of files processed without human intervention (the rest in review, never a silent wrong rename).
|
||||
- Zero non-reversible renames.
|
||||
- Every rename is logged and traceable; no user-facing undo or rollback of file renames is required.
|
||||
- Adding a new template in under 30 minutes without writing code.
|
||||
- SQLite can be fully rebuilt from the bucket sidecars at any time.
|
||||
|
||||
## 9. Risks and Mitigations
|
||||
|
||||
- Limited training corpus (~500 files): start simple (TF-IDF + LogReg), optionally sentence-transformers embeddings if accuracy is insufficient; the review loop continuously enriches the corpus.
|
||||
- Limited training corpus (~500 files): start with deterministic rules; evaluate TF-IDF + Logistic Regression as a post-v1 enhancement; the review loop continuously enriches the corpus.
|
||||
- Templates too weakly distinguished in the text: enrich features (header, issuer, business keywords per template).
|
||||
- Poorly extracted or empty text files: systematic detection and routing to review.
|
||||
- Poorly extracted or empty text files: systematic detection and routing to review; extraction and search must cover all pages, not only page one.
|
||||
- Partial uploads (Nextcloud chunked uploads): poll-only keys with a stable size across two listing cycles and a `.part`/`.~lock` exclusion list.
|
||||
- Sidecar/object drift (sidecar missing or out of sync with SQLite): sidecars win; a consistency job rebuilds SQLite and flags orphaned or missing sidecars.
|
||||
- Garage version of S3 API: pin a Garage release and run a compatibility spike (listing, multipart, copy-object, versioning).
|
||||
|
||||
+53
-101
@@ -5,9 +5,9 @@ version: "1.0"
|
||||
status: "draft"
|
||||
date: "2026-10-13"
|
||||
inputDocuments:
|
||||
- "automatic-pdf-renamer/brief.md"
|
||||
- "automatic-pdf-renamer/prd.md"
|
||||
- "automatic-pdf-renamer/arch.md"
|
||||
- "brief.md"
|
||||
- "prd.md"
|
||||
- "arch.md"
|
||||
---
|
||||
|
||||
# Automatic PDF Renamer — Epics and Stories
|
||||
@@ -43,12 +43,12 @@ The following inventory consolidates the requirements from the project brief and
|
||||
|
||||
- **FR-ING-01 — Polling and discovery:** poll a configurable Garage S3 `inbox/` prefix at a configurable interval; consider only PDFs and ignore temporary, hidden, and lock files.
|
||||
- **FR-ING-02 — Upload stability:** process a PDF only after its size and ETag remain stable across consecutive listings.
|
||||
- **FR-ING-03 — Restart recovery:** on startup, discover eligible inbox or pending objects that have not completed processing.
|
||||
- **FR-TXT-01 — Text extraction:** extract text from the first page of a PDF using a local library; no OCR in v1.
|
||||
- **FR-ING-03 — Restart recovery:** on startup, recover interrupted technical operations and discover eligible inbox objects that have not completed processing; reconcile `pending/` without automatically reprocessing documents awaiting manual correction.
|
||||
- **FR-TXT-01 — Text extraction:** extract all available text from every page of a PDF using a local library; no OCR in v1.
|
||||
- **FR-TXT-02 — Empty text handling:** route PDFs with empty or insufficient extracted text to human review with an explicit reason.
|
||||
- **FR-TXT-03 — Text persistence:** persist extracted text in the sidecar and SQLite replica so normal review and training workflows do not need to re-parse the PDF.
|
||||
- **FR-CLS-01 — Classification:** classify documents against configured templates. The initial light release uses deterministic rules; the ML epic adds TF-IDF and Logistic Regression classification.
|
||||
- **FR-CLS-02 — Confidence and unknown class:** ML classification exposes confidence and routes results below a configurable threshold to review.
|
||||
- **FR-CLS-02 — Confidence and unknown class (post-v1):** ML classification exposes confidence and routes results below a configurable threshold to review. The deterministic initial release uses explicit rule outcomes and does not generate artificial confidence scores.
|
||||
- **FR-FLD-01 — Common fields:** every document has template, emission date, emitting company, document type, and period.
|
||||
- **FR-FLD-02 — Optional fields:** templates define optional fields and reusable field groups with formats, allowed values, and extraction rules.
|
||||
- **FR-FLD-03 — Deterministic extraction:** the light release uses configured regular expressions, date parsing, company gazetteers, and fuzzy matching where useful.
|
||||
@@ -56,18 +56,17 @@ The following inventory consolidates the requirements from the project brief and
|
||||
- **FR-FLD-05 — Period computation:** derive period from emission date and template/company/document-type rules, including granularity, formatting, and offset.
|
||||
- **FR-FLD-06 — Validation:** validate dates, companies, document types, periods, and optional field formats before applying a rename. Invalid or missing required values must be reviewed.
|
||||
- **FR-STO-01 — Garage storage:** use a versioned Garage S3 bucket and the `inbox/`, `pending/`, and `files/` key areas.
|
||||
- **FR-STO-02 — Sidecar source of truth:** store a JSON sidecar for each PDF with identifiers, current and original keys, status, extracted fields/text, confidence where applicable, timestamps, and event history.
|
||||
- **FR-STO-03 — SQLite replica:** mirror sidecar data into SQLite for fast listing, filtering, search, and statistics.
|
||||
- **FR-STO-02 — Sidecar source of truth:** store a JSON sidecar for each PDF with SHA-256 as its unique document identifier, current and original keys, status, extracted fields/text, confidence where applicable, timestamps, and event history. Any byte-level PDF content change creates a new identity; a rename/move preserving bytes does not.
|
||||
- **FR-STO-03 — SQLite replica:** mirror sidecar data into SQLite for fast listing, filtering, search, and consistency diagnostics.
|
||||
- **FR-STO-04 — Safe moves:** move objects using copy-before-delete, resolve destination collisions with an incremental suffix, and record the actual final key.
|
||||
- **FR-STO-05 — Reversible operations:** preserve sufficient history and key information to trace and reverse a rename without silently overwriting another object.
|
||||
- **FR-STO-05 — Operation audit trail:** preserve sufficient history and key information to trace file moves and renames. A user-facing undo/rollback feature for file renames is out of scope.
|
||||
- **FR-DASH-01 — File list:** list all known files and show emission date, period, current key/name, status, template, document type, company, and confidence where available.
|
||||
- **FR-DASH-02 — Search and filters:** filter by status, template, document type, company, period, and date range; sort columns and search object names and metadata, with extracted-text search where feasible.
|
||||
- **FR-DASH-03 — Statistics:** show counts by status, automation and validation rates, and consistency indicators.
|
||||
- **FR-DASH-02 — Search and filters:** filter by status, template, document type, company, period, and date range; sort columns and search object names and metadata, and full extracted text across every PDF page.
|
||||
- **FR-DASH-04 — PDF detail and preview:** stream the PDF from Garage through the backend and show it alongside extracted metadata.
|
||||
- **FR-DASH-05 — Correction and validation:** allow the owner to change the template and field values, preview the resulting target key, and validate the final result.
|
||||
- **FR-DASH-06 — Lifecycle:** support the business statuses `not processed`, `processed`, and `validated`; validated records are eligible for training.
|
||||
- **FR-DASH-06 — Lifecycle:** support the business statuses `pending`, `processed`, and `validated`; validated records are eligible for training.
|
||||
- **FR-TPL-01 — Template configuration:** define templates, fields, extraction and validation rules, period rules, destination prefixes, and key patterns in configuration.
|
||||
- **FR-TPL-02 — Template discoverability:** show configured templates, their fields/rules, and corpus counts in the dashboard.
|
||||
- **FR-TPL-02 — Configuration file editing:** retrieve and update the JSONC configuration through dedicated endpoints; validate changes before activating them.
|
||||
- **FR-ML-01 — Training corpus:** train from the initial labeled corpus and files validated by the owner.
|
||||
- **FR-ML-02 — Evaluation:** show evaluation metrics, including accuracy and confusion matrix, before model activation.
|
||||
- **FR-ML-03 — Model lifecycle:** version model artifacts, allow activation or discard, keep one active model, and activate without restarting the service.
|
||||
@@ -82,7 +81,7 @@ The following inventory consolidates the requirements from the project brief and
|
||||
- **NFR-01 Privacy:** no network calls except the local Garage bucket, local dashboard, and required local infrastructure. No runtime downloads or telemetry.
|
||||
- **NFR-02 Performance:** process a standard PDF in under 5 seconds on the target server, including the S3 download. Dashboard list loads in under 1 second for 10,000 records; the PRD also calls out a 1,000-file acceptance check.
|
||||
- **NFR-03 Reliability:** no silent invalid renames; ambiguous or invalid results are routed to review.
|
||||
- **NFR-04 Durability:** bucket versioning is enabled; SQLite uses WAL; sidecar writes are atomic; crash recovery repairs replica drift.
|
||||
- **NFR-04 Durability:** bucket versioning is enabled; SQLite uses WAL; sidecar writes are atomic; crash recovery repairs replica drift; retries are idempotent and technical failures remain distinct from business statuses.
|
||||
- **NFR-05 Language:** French documents in v1; the UI may use one locale, English or French.
|
||||
- **NFR-06 Deployment/configuration:** containerized deployment, environment variables for secrets and endpoints, and a single documented template configuration format.
|
||||
- **NFR-07 Maintainability:** Python/FastAPI, Garage-compatible S3 client, SQLite, React dashboard, local PDF text extraction, and scikit-learn for ML.
|
||||
@@ -93,10 +92,7 @@ The following inventory consolidates the requirements from the project brief and
|
||||
|
||||
- Existing correctly named documents provide the initial labeled corpus; v1 processes new incoming files only, not the historical backlog.
|
||||
- OCR, scanned PDFs, multilingual documents, cloud services, and automatic periodic retraining are out of scope for v1.
|
||||
- The three business statuses must not be overloaded with transient technical states. Processing failures and recovery steps need separate technical flags/reasons.
|
||||
- **Configuration format must be reconciled:** the brief and PRD refer to YAML, while the architecture specifies JSONC. For this backlog, JSONC is the planning baseline because it is the architecture's stated decision. Update the brief/PRD or explicitly reverse this decision before implementation.
|
||||
- **Period rules must be reconciled:** the PRD requires day/week/month/quarter/semester/year and current/previous/following offsets; the architecture currently describes a narrower set. The implementation must support the full PRD set unless the product owner approves a reduced scope.
|
||||
- **Authentication scope must be reconciled:** the brief and PRD exclude dashboard authentication for the single-user v1, while the architecture describes a password-protected dashboard. This backlog follows the brief/PRD: no application-level authentication in v1; revisit before exposing the dashboard beyond the trusted local network.
|
||||
- The three business statuses must not be overloaded with technical states. Transient technical failures use a separate retry/error state, with attempt count, next retry time, and error details. Retry up to 3 times after the initial attempt, with delays of 5 seconds, 30 seconds, and 2 minutes. After the final failed retry, stop processing that item without another retry or review workflow; only document/business issues requiring manual correction use `pending`.
|
||||
- S3 object operations, sidecar updates, and SQLite writes do not form one distributed transaction. Each state transition must be recoverable and verifiable rather than relying on a claim of cross-system atomicity.
|
||||
|
||||
## 3. Functional Requirements Coverage Map
|
||||
@@ -107,7 +103,7 @@ The following inventory consolidates the requirements from the project brief and
|
||||
| FR-TXT-01 to FR-TXT-03: text extraction and persistence | Epic 2 |
|
||||
| FR-CLS-01 to FR-CLS-02: rules-first classification and ML confidence | Epics 2 and 3 |
|
||||
| FR-FLD-01 to FR-FLD-06: fields, deterministic/ML extraction, periods, validation | Epics 2 and 3; review overrides in Epic 4 |
|
||||
| FR-STO-01 to FR-STO-05: S3, sidecars, SQLite, safe and reversible moves | Epics 1 and 2; recovery in Epic 5 |
|
||||
| FR-STO-01 to FR-STO-05: S3, sidecars, SQLite, safe and traceable moves | Epics 1 and 2; recovery in Epic 5 |
|
||||
| FR-DASH-01 to FR-DASH-06: list, search, preview, correction, validation, lifecycle | Epic 4 |
|
||||
| FR-TPL-01 to FR-TPL-02: templates and template visibility | Epics 2 and 4 |
|
||||
| FR-ML-01 to FR-ML-03: training, evaluation, model lifecycle | Epic 3 |
|
||||
@@ -121,7 +117,7 @@ The following inventory consolidates the requirements from the project brief and
|
||||
1. **Technical Foundation, Storage, and Metadata** — establish the offline service, Garage access, sidecar schema, and rebuildable SQLite index.
|
||||
2. **Rules-Based PDF Ingestion and Classification (Light Release)** — deliver a complete deterministic pipeline using text extraction, regular expressions, configured lookup data, validation, and safe renaming; no ML.
|
||||
3. **Machine Learning for PDF Ingestion** — add local model training and inference for document classification and configured-value extraction, integrated with the existing pipeline.
|
||||
4. **Dashboard, Human Review, and Template Management** — make results searchable and correctable, support PDF preview and validation, and expose configured templates.
|
||||
4. **Dashboard, Human Review, and Configuration Editing** — make results searchable and correctable, support PDF preview and validation, and retrieve/update the JSONC configuration file.
|
||||
5. **Consistency, Recovery, and Operational Readiness** — detect and repair drift, recover safely from interruptions, verify offline behavior and performance, and document deployment/operations.
|
||||
|
||||
---
|
||||
@@ -179,7 +175,7 @@ So that processing decisions and file history survive restarts and can be audite
|
||||
|
||||
As the owner,
|
||||
I want sidecar metadata indexed in SQLite,
|
||||
So that dashboard queries and statistics remain fast without making SQLite authoritative.
|
||||
So that dashboard queries remain fast without making SQLite authoritative.
|
||||
|
||||
**Acceptance Criteria:**
|
||||
|
||||
@@ -207,7 +203,7 @@ So that a damaged or deleted local database does not require reprocessing the PD
|
||||
|
||||
## Epic 2: Rules-Based PDF Ingestion and Classification (Light Release)
|
||||
|
||||
**Goal:** Deliver a complete end-to-end workflow for new PDFs using deterministic rules only. The application polls Garage, extracts first-page text, fills fields with regexes and configured lookup data, validates the result, and safely renames or routes the file for review. This epic must not depend on a machine-learning model.
|
||||
**Goal:** Deliver a complete end-to-end workflow for new PDFs using deterministic rules only. The application polls Garage, extracts text from every PDF page, fills fields with regexes and configured lookup data, validates the result, and safely renames or routes the file for review. This epic must not depend on a machine-learning model.
|
||||
|
||||
**Requirements covered:** FR-ING-01 to FR-ING-03, FR-TXT-01 to FR-TXT-03, deterministic part of FR-CLS-01, FR-FLD-01 to FR-FLD-03 and FR-FLD-05 to FR-FLD-06, FR-STO-04 to FR-STO-05, FR-TPL-01, FR-OPS-02, NFR-02 to NFR-04.
|
||||
|
||||
@@ -223,20 +219,22 @@ So that newly uploaded documents enter the pipeline automatically.
|
||||
**When** its size and ETag are unchanged across two consecutive listings
|
||||
**Then** it becomes eligible for processing
|
||||
**And** non-PDF files, hidden files, `.part` files, and lock files are ignored
|
||||
**And** a file that is still being uploaded is not processed prematurely
|
||||
**And** a poller error for one object does not stop discovery of other objects.
|
||||
**And** a file that is still being uploaded is not eligible for processing prematurely
|
||||
**And** a poller error for one object does not stop discovery of other objects
|
||||
**And** transient technical failures are retried up to 3 times after the initial attempt, with delays of 5 seconds, 30 seconds, and 2 minutes, and recorded separately from business status
|
||||
**And** a technical failure alone never moves a document to `pending/`.
|
||||
|
||||
### Story 2.2: Extract and persist first-page PDF text
|
||||
### Story 2.2: Extract and persist full PDF text
|
||||
|
||||
As the application,
|
||||
I want to extract first-page text from each eligible PDF,
|
||||
I want to extract and persist all available text from every page of each eligible PDF,
|
||||
So that downstream classification and field extraction use the document's content.
|
||||
|
||||
**Acceptance Criteria:**
|
||||
|
||||
**Given** an eligible PDF contains extractable text
|
||||
**When** the extraction step runs
|
||||
**Then** the configured local PDF library extracts the first page's text
|
||||
**Then** the configured local PDF library extracts text from every page of the PDF
|
||||
**And** the text is saved in the sidecar and mirrored to SQLite
|
||||
**And** the application does not call an external OCR or document-processing service
|
||||
**And** if the text is empty or below the configured minimum length, the file is routed to review with reason `empty text`.
|
||||
@@ -264,7 +262,7 @@ So that the light release can classify and name common document types without ML
|
||||
|
||||
**Acceptance Criteria:**
|
||||
|
||||
**Given** first-page text and a configured set of templates are available
|
||||
**Given** full-document extracted text and a configured set of templates are available
|
||||
**When** deterministic extraction runs
|
||||
**Then** it extracts candidate values for emission date, emitting company, document type, and configured optional fields using the applicable regexes and lookup lists
|
||||
**And** date parsing uses a local date parser and company matching uses the configured gazetteer, with fuzzy matching only where configured
|
||||
@@ -302,8 +300,8 @@ So that routine files are organized without risking silent data loss.
|
||||
**And** the sidecar records the original key, current key, extracted values, decision, status, and rename events
|
||||
**And** the file receives status `processed` only after the required state updates have been persisted or recoverably recorded
|
||||
**And** when confidence is not applicable in this rules-only release, the decision is based on explicit rule outcomes and field validation rather than a fabricated ML confidence score
|
||||
**And** ambiguous, incomplete, or invalid files are moved to `pending/` and receive status `not processed` with a review reason
|
||||
**And** the operation can be audited and safely reversed without overwriting another object.
|
||||
**And** ambiguous, incomplete, or invalid files are moved to `pending/` and receive status `pending` with a review reason
|
||||
**And** the operation can be audited from its recorded history; no user-facing rename reversal is required.
|
||||
|
||||
### Story 2.7: Make ingestion idempotent and restart-safe
|
||||
|
||||
@@ -318,7 +316,10 @@ So that retries do not create duplicate processing or lose track of a file.
|
||||
**Then** objects without a completed sidecar state are discovered and resumed or flagged for review
|
||||
**And** objects already completed are not processed again in a way that creates duplicate renames or events
|
||||
**And** if a crash occurs between copy and delete, the recovery logic detects both keys and resolves the state from the sidecar and object hashes
|
||||
**And** transient technical failures are distinguishable from the three business statuses.
|
||||
**And** transient technical failures are recorded separately from the three business statuses
|
||||
**And** retryable failures are retried safely up to 3 times after the initial attempt, with delays of 5 seconds, 30 seconds, and 2 minutes
|
||||
**And** after retries are exhausted, processing of that item stops without launching another retry or manual-review workflow
|
||||
**And** a technical failure alone does not move the document to `pending/`.
|
||||
|
||||
---
|
||||
|
||||
@@ -383,7 +384,7 @@ So that the application can resolve more document-specific values without hard-c
|
||||
|
||||
**Acceptance Criteria:**
|
||||
|
||||
**Given** a template maps a company/document type to a predefined field group with allowed values
|
||||
**Given** a template maps a period format/emitting company/document type/additional fields to a predefined field group with allowed values
|
||||
**When** ML-assisted extraction is enabled for that field
|
||||
**Then** the extractor proposes a value from the configured candidate set and records the method and score where available
|
||||
**And** regex and deterministic lookup results remain available as inputs or fallback paths
|
||||
@@ -409,7 +410,7 @@ So that model improvements can be adopted safely and reversed if quality regress
|
||||
|
||||
---
|
||||
|
||||
## Epic 4: Dashboard, Human Review, and Template Management
|
||||
## Epic 4: Dashboard, Human Review, and Configuration Editing
|
||||
|
||||
**Goal:** Give the owner a local interface to find documents, understand automated decisions, correct metadata, validate files, and inspect template configuration. The dashboard is the human safety net for both the rules-only and ML-enabled ingestion paths.
|
||||
|
||||
@@ -427,25 +428,11 @@ So that I can find documents and understand their processing status.
|
||||
**When** I open the dashboard file list
|
||||
**Then** each row shows emission date, period, current object name/key, status, template, document type, emitting company, and confidence where available
|
||||
**And** I can filter by status, template, document type, company, period, and date range
|
||||
**And** I can sort by supported columns and search object names and metadata
|
||||
**And** status badges distinguish `not processed`, `processed`, and `validated`
|
||||
**And** I can sort by supported columns and search object names, metadata fields, and the full extracted text across all pages
|
||||
**And** status badges distinguish `pending`, `processed`, and `validated`
|
||||
**And** the list is served from SQLite rather than performing a full S3 scan for every request.
|
||||
|
||||
### Story 4.2: Show useful statistics and consistency indicators
|
||||
|
||||
As the owner,
|
||||
I want summary counts and automation statistics,
|
||||
So that I can monitor whether the system is working as expected.
|
||||
|
||||
**Acceptance Criteria:**
|
||||
|
||||
**Given** the indexed file records and known consistency flags are available
|
||||
**When** I view the dashboard summary
|
||||
**Then** it shows counts by business status, automation rate, validation rate, and known consistency anomalies
|
||||
**And** the displayed counts are derived from the same indexed records as the file list
|
||||
**And** unavailable or stale statistics are indicated rather than presented as current.
|
||||
|
||||
### Story 4.3: Inspect a PDF and its processing decision
|
||||
### Story 4.2: Inspect a PDF and its processing decision
|
||||
|
||||
As the owner,
|
||||
I want to open a file's detail page with its PDF and extracted metadata,
|
||||
@@ -456,10 +443,10 @@ So that I can understand and correct the system's decision.
|
||||
**Given** a file appears in the list
|
||||
**When** I open its detail page
|
||||
**Then** the page displays the PDF streamed from Garage through the backend alongside its extracted fields
|
||||
**And** it displays the current status, template, extracted text, decision/review reason, and ML candidates/scores when available
|
||||
**And** it displays the current status, metadata, extracted text, decision/review reason, and ML candidates/scores when available
|
||||
**And** a missing or unavailable PDF is reported clearly without breaking the rest of the dashboard.
|
||||
|
||||
### Story 4.4: Correct metadata and preview the target key
|
||||
### Story 4.3: Correct metadata and preview the target key
|
||||
|
||||
As the owner,
|
||||
I want to change a template and edit extracted fields before validation,
|
||||
@@ -470,11 +457,11 @@ So that the final file name and destination reflect the correct document data.
|
||||
**Given** I am viewing a document detail page
|
||||
**When** I change the template
|
||||
**Then** the page keeps the five common fields, reloads the optional fields for the selected template, and performs best-effort re-extraction without silently overwriting user edits
|
||||
**And** the period is recomputed from the emission date and the new template rules but remains editable
|
||||
**And** the period is recomputed from the emission date and the new template rules and is read-only
|
||||
**And** the target key and destination prefix update as I edit fields
|
||||
**And** invalid or missing required fields are highlighted and prevent final validation until corrected or explicitly handled by a defined rule.
|
||||
|
||||
### Story 4.5: Validate a file and preserve the correction history
|
||||
### Story 4.4: Validate a file and preserve the correction history
|
||||
|
||||
As the owner,
|
||||
I want to confirm or correct a file's metadata and validate the result,
|
||||
@@ -490,20 +477,20 @@ So that the file is moved to its final location and my correction can improve fu
|
||||
**And** the validated record becomes eligible for the ML training corpus
|
||||
**And** the operation is idempotent and its history remains auditable.
|
||||
|
||||
### Story 4.6: Inspect configured templates and their corpus coverage
|
||||
### Story 4.5: Retrieve and update the JSONC configuration file
|
||||
|
||||
As the owner,
|
||||
I want to see the configured templates, their rules, and the number of associated examples,
|
||||
So that I can understand and maintain the document taxonomy.
|
||||
I want to retrieve and edit the JSONC configuration file from the dashboard,
|
||||
So that I can change templates and extraction rules without editing application code.
|
||||
|
||||
**Acceptance Criteria:**
|
||||
|
||||
**Given** a valid template registry is loaded
|
||||
**When** I open the template view
|
||||
**Then** I can see each template's fields, period rules, target prefix, key pattern, and extraction/validation configuration
|
||||
**And** the view shows the number of corpus files associated with each template
|
||||
**And** configuration errors are surfaced with actionable messages
|
||||
**And** a new template becomes available to classification and the detail-page selector after valid configuration is loaded, without code changes.
|
||||
**Given** the current configuration file is available
|
||||
**When** I open the configuration editor
|
||||
**Then** the dashboard retrieves the complete JSONC file through `GET /config`
|
||||
**And** I can submit the complete modified file through `PUT /config`
|
||||
**And** invalid JSONC or schema errors are reported without replacing the active configuration
|
||||
**And** a valid file is saved atomically and the in-memory configuration is reloaded.
|
||||
|
||||
---
|
||||
|
||||
@@ -557,37 +544,6 @@ So that files are not stranded in an unknown state.
|
||||
**And** it repairs SQLite drift from authoritative sidecars
|
||||
**And** it never deletes the only known copy of a PDF as an unverified recovery shortcut.
|
||||
|
||||
### Story 5.4: Verify offline operation and Garage compatibility
|
||||
|
||||
As the owner,
|
||||
I want the deployment and critical storage operations tested against the actual local environment,
|
||||
So that the system meets its privacy and reliability requirements in practice.
|
||||
|
||||
**Acceptance Criteria:**
|
||||
|
||||
**Given** the application is deployed with a pinned Garage version
|
||||
**When** the integration test suite runs
|
||||
**Then** listing, HEAD/GET, PUT, copy, delete, versioning, and collision handling are tested against Garage
|
||||
**And** a network audit confirms that runtime traffic is limited to approved local services and no cloud API or telemetry endpoint is contacted
|
||||
**And** tests cover interruptions during rename, sidecar update, SQLite synchronization, and rebuild
|
||||
**And** the critical ingestion-to-review and ingestion-to-rename journeys are exercised end to end.
|
||||
|
||||
### Story 5.5: Verify performance and document operations
|
||||
|
||||
As the owner,
|
||||
I want measurable performance checks and clear operating instructions,
|
||||
So that I can deploy, monitor, and troubleshoot the application confidently.
|
||||
|
||||
**Acceptance Criteria:**
|
||||
|
||||
**Given** a representative test dataset and target server are available
|
||||
**When** performance checks run
|
||||
**Then** standard-PDF processing is measured against the under-5-second target
|
||||
**And** the dashboard list is measured against the under-1-second target for 10,000 records, with the PRD's 1,000-file check retained as an additional acceptance test
|
||||
**And** failures and slow operations produce actionable local logs without exposing credentials
|
||||
**And** deployment documentation covers configuration, Garage bucket/versioning prerequisites, Nextcloud visibility, backup/rebuild, consistency checks, model registry backup, and troubleshooting
|
||||
**And** the v1 decision to omit at-rest encryption and application-level dashboard authentication is explicitly documented, including the trusted-network assumption.
|
||||
|
||||
---
|
||||
|
||||
## 5. Cross-Epic Definition of Done
|
||||
@@ -595,7 +551,7 @@ So that I can deploy, monitor, and troubleshoot the application confidently.
|
||||
A story is complete only when all applicable items below are satisfied:
|
||||
|
||||
- Acceptance criteria are implemented and verified with automated tests where practical.
|
||||
- Errors and ambiguous outcomes are explicit; no invalid document is silently renamed.
|
||||
- Errors and ambiguous outcomes are explicit; no invalid document is silently renamed. Transient technical failures are retried separately and never mislabeled as `pending`.
|
||||
- Metadata changes follow the sidecar-first, SQLite-replica model and can be recovered.
|
||||
- No external network dependency is introduced.
|
||||
- Configuration, logs, and documentation are updated where behavior changes.
|
||||
@@ -614,12 +570,8 @@ Adds Epic 3 and its integration with the review workflow. ML is an enhancement,
|
||||
|
||||
### Release C — Operationally complete v1
|
||||
|
||||
Completes the remaining Epic 4 capabilities and Epic 5, including consistency repair, rebuild, recovery, offline verification, performance checks, and deployment documentation.
|
||||
Completes the remaining Epic 4 capabilities and Epic 5, including consistency repair, rebuild, recovery, offline verification, and deployment documentation.
|
||||
|
||||
## 7. Open Decisions to Resolve Before the Affected Stories
|
||||
|
||||
1. **Template format:** select YAML or JSONC as the single supported format and align brief, PRD, and architecture. This backlog currently assumes JSONC.
|
||||
2. **Period configuration:** confirm that all PRD granularities and offsets are required in v1 and update the architecture accordingly.
|
||||
3. **Dashboard authentication:** confirm the brief/PRD decision to omit application-level authentication for the single-user v1, despite the architecture's proposed password-based login.
|
||||
4. **Reversibility semantics:** define the user-facing undo operation, collision behavior, and exact state transitions before implementing the final rollback workflow.
|
||||
5. **ML field extraction:** specify which field groups use ML in v1 and what evaluation threshold is required before enabling each extractor. Classification and deterministic validation must remain usable independently.
|
||||
2. **ML field extraction:** specify which field groups use ML in v1 and what evaluation threshold is required before enabling each extractor. Classification and deterministic validation must remain usable independently.
|
||||
|
||||
@@ -4,7 +4,7 @@ Product Requirements Document. Derived from the project brief (BMAD). Personal t
|
||||
|
||||
## 1. Product Overview
|
||||
|
||||
A server-side service with a local web dashboard that polls an `inbox/` prefix of a Garage S3 bucket for incoming PDFs, identifies the document type from its content, applies the matching naming template, fills the template fields, moves the object to an organized prefix tree within the bucket, and tracks every file through a three-state lifecycle (not processed / processed / validated). Metadata lives in per-file JSON sidecars (source of truth), replicated to SQLite for queries. Low-confidence or invalid files wait for human correction; validated files feed future model retraining. Users upload, download, and share documents through Nextcloud, which mounts the bucket as external storage.
|
||||
A server-side service with a local web dashboard that polls an `inbox/` prefix of a Garage S3 bucket for incoming PDFs, identifies the document type from its content, applies the matching naming template, fills the template fields, moves the object to an organized prefix tree within the bucket, and tracks every file through a three-state lifecycle (`pending` / `processed` / `validated`). Metadata lives in per-file JSON sidecars (source of truth), replicated to SQLite for queries. Low-confidence or invalid files wait for human correction; validated files feed future model retraining. Users upload, download, and share documents through Nextcloud, which mounts the bucket as external storage.
|
||||
|
||||
## 2. Personas
|
||||
|
||||
@@ -15,16 +15,16 @@ Single user: the owner. Technically proficient, wants minimal manual work but ze
|
||||
### 3.1 Object storage
|
||||
|
||||
- Garage ([https://garagehq.deuxfleurs.fr/](https://garagehq.deuxfleurs.fr/)), self-hosted, S3 API. One versioned bucket, e.g. `pdf-renamer`.
|
||||
- Key layout: `inbox/` (entry point), `pending/` (not-processed files moved out of inbox to avoid re-processing), and `files/` for the processes files (validated or not).
|
||||
- Key layout: `inbox/` (entry point), `pending/` (files awaiting manual correction, moved out of inbox to avoid re-processing), and `files/` for the processes files (validated or not).
|
||||
- Sidecars: `{object-key}.meta.json` next to each PDF.
|
||||
- No encryption at rest in v1 (documents not sensitive); bucket served on the local network only.
|
||||
- Renames are copy-object + delete within the bucket; collisions resolved with an incremental suffix.
|
||||
|
||||
### 3.2 Metadata: sidecars as source of truth, SQLite as replica
|
||||
|
||||
- Each PDF has a sidecar JSON object holding: sha256, original key, current key, status, template, document type, emission date, period, emitting company, all optional fields, confidence, and the full event history (classified, renamed, validated, undone, corrected).
|
||||
- Each PDF has a sidecar JSON object holding: sha256, original key, current key, status, template, document type, emission date, period, emitting company, all optional fields, confidence, and the full event history (classified, renamed, validated, corrected).
|
||||
- Sidecar is updated atomically on every state change: write the new sidecar to a temp key, copy over the final sidecar key.
|
||||
- SQLite mirrors the sidecar content in queryable tables for the dashboard (lists, filters, statistics). It is a cache/replica, never authoritative. Any sidecar↔SQLite divergence is resolved in favor of the sidecar.
|
||||
- SQLite mirrors the sidecar content in queryable tables for the dashboard (lists, filters, search, and consistency diagnostics). It is a cache/replica, never authoritative. Any sidecar↔SQLite divergence is resolved in favor of the sidecar.
|
||||
- A rebuild command reconstructs the full SQLite database by scanning all sidecars; a consistency job detects PDFs without sidecars (orphaned) and sidecars without PDFs (missing object) and flags them in the dashboard.
|
||||
|
||||
### 3.3 Nextcloud integration
|
||||
@@ -38,16 +38,16 @@ Single user: the owner. Technically proficient, wants minimal manual work but ze
|
||||
|
||||
| Status | Meaning |
|
||||
| ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| not processed | Classification confidence below threshold, or field validation failed. Object moved to the `pending/` prefix, waiting for review. |
|
||||
| pending | Deterministic rules cannot determine a valid classification or required field, or validation fails. Object moved to the `pending/` prefix, waiting for review. |
|
||||
| processed | Automatically classified, fields validated, renamed and moved. |
|
||||
| validated | The user reviewed the result (confirmed or corrected template/fields). Eligible for model retraining. Terminal state for the review flow; the object was renamed and moved. |
|
||||
|
||||
|
||||
Transitions:
|
||||
|
||||
- new object in `inbox/` → not processed (low confidence or invalid fields; object copied to `pending/`)
|
||||
- new object in `inbox/` → processed (high confidence, valid fields)
|
||||
- not processed → validated (user corrects and validates in the detail page; rename then applied)
|
||||
- new object in `inbox/` → pending (deterministic rules cannot determine a valid result; object copied to `pending/`)
|
||||
- new object in `inbox/` → processed (deterministic rules yield an unambiguous result and all required fields are valid)
|
||||
- pending → validated (user corrects and validates in the detail page; rename then applied)
|
||||
- processed → validated (user confirms or corrects and validates in the detail page; rename only if the user maid some changes)
|
||||
|
||||
## 5. Functional Requirements
|
||||
@@ -56,23 +56,23 @@ Transitions:
|
||||
|
||||
- FR1.1 Poll the configurable `inbox/` prefix at a configurable interval; process only keys ending in `.pdf`. Ignore other extensions and temporary files (Nextcloud `.part`, `.~lock`, hidden files).
|
||||
- FR1.2 Wait for object stability: process a key only after its size and ETag are unchanged across two consecutive listings (handles chunked uploads and partial writes).
|
||||
- FR1.3 On service restart, detect objects present in `inbox/` (or `pending/`) with no sidecar record and process them.
|
||||
- FR1.3 On service restart, recover eligible objects in `inbox/` and resume interrupted technical operations. Do not automatically reprocess a `pending` document that is awaiting manual correction unless the user requests processing after correction.
|
||||
- FR1.4 Optionally reduce latency with bucket notification hooks if the Garage deployment supports them; polling is the v1 baseline.
|
||||
- FR1.5 After processing, the inbox copy is deleted (processed files) or the object is copied to `pending/` and the inbox copy deleted (not-processed files), so `inbox/` always converges to empty.
|
||||
- FR1.5 After successful processing, the inbox copy is deleted. Documents requiring manual correction are copied to `pending/` and removed from `inbox/`. Transient technical failures remain eligible for retry/recovery and must not be mislabeled as `pending`.
|
||||
|
||||
### Epic 2 — Text extraction
|
||||
|
||||
- FR2.1 Download the object from S3 and extract text with pdfplumber or PyMuPDF (choice fixed at implementation, documented in config).
|
||||
- FR2.2 Extract the full first page only.
|
||||
- FR2.3 If extracted text is empty or below a minimum length, route the file to not processed with reason "empty text".
|
||||
- FR2.4 Store the extracted text in the sidecar (and replicated to SQLite) for the file's lifetime, so the detail page and retraining never need to re-download and re-parse the object.
|
||||
- FR2.2 Extract all available text from every page of each PDF; extraction must not stop after the first page. OCR of scanned/image-only documents remains out of scope for v1.
|
||||
- FR2.3 If extracted text is empty or below a minimum length, route the file to `pending` with reason "empty text".
|
||||
- FR2.4 Store the complete extracted text from all pages in the sidecar and replicate it to SQLite for the file's lifetime, so search, the detail page, and retraining do not need to re-download and re-parse the object.
|
||||
|
||||
### Epic 3 — Template classification
|
||||
|
||||
- FR3.1 Classify the document into one of the configured templates using TF-IDF + Logistic Regression on the extracted text.
|
||||
- FR3.2 Output a confidence score per template; the top score below a configurable threshold (default 0.8) yields "unknown" and routes the file to not processed with reason "low confidence".
|
||||
- FR3.3 Display the top-N template candidates with scores on the detail page.
|
||||
- FR3.4 Retrain the classifier on demand from: the initial corpus + all files with validated status. Retraining is a manual action from the dashboard, with the training date and resulting evaluation metrics stored in the sidecar-adjacent model registry and displayed.
|
||||
- FR3.1 Classify the document into one of the configured templates using configured deterministic rules, extracted text, and reference data in the initial release. TF-IDF + Logistic Regression is a post-v1 enhancement specified in Epic 3, not a dependency of the initial release.
|
||||
- FR3.2 In the initial release, if deterministic rules do not yield one unambiguous template, route the file to `pending` with reason `insufficient deterministic evidence`; do not fabricate a confidence score. In the post-v1 ML enhancement, expose model scores and route predictions below the configured threshold to `pending`.
|
||||
- FR3.3 In the initial release, display the matching rules and ambiguity/review reason on the detail page. When ML is enabled in a future release, also display ranked candidates and scores.
|
||||
- FR3.4 Post-v1 only: retrain the classifier on demand from the initial corpus plus eligible validated files. Record the training date and evaluation metrics in the local model registry and display them in the dashboard.
|
||||
|
||||
### Epic 4 — Document data model, slot filling and validation
|
||||
|
||||
@@ -94,37 +94,37 @@ The object key pattern of each template is composed from these common fields, pl
|
||||
#### 4.2 Optional fields (template-specific)
|
||||
|
||||
- FR4.1 Each template declares in JSONC its optional fields beyond the common five. Additional fields are organized in predefined field groups (e.g., `bank_accounts`, `properties`, `contract_references`, `customer_numbers`) with their possible values. The (company, document_type) mapping specifies which field groups to extract for each document. Examples: bank statements extract from `bank_accounts` group; rent receipts extract from `properties` group.
|
||||
- FR4.2 Fill common and optional fields using per-template regexes for dates and the companies list for company matching. Additional fields are extracted from their predefined field groups using ML from the possible values. The period is computed from the emission date using the period format's granularity, format template, and offset (current/previous/following); it is recomputed whenever the emission date or period format changes.
|
||||
- FR4.3 Validate fields before renaming: emission date parses and is within a plausible range, period is non-empty and consistent with the emission date and the template's granularity/offset, emitting company recognized, document type among the template's allowed values, each optional field matches its declared format (empty optional fields are allowed). Any failure routes to not processed with the failing fields identified.
|
||||
- FR4.2 Fill common and optional fields using per-template regexes for dates and the companies list for company matching. Additional fields are extracted in the initial release using configured deterministic rules and lookup lists. ML-assisted selection from predefined possible values is a post-v1 enhancement. The period is computed server-side from the emission date using the configured granularity, format template, and offset (current/previous/following); it is recomputed whenever the emission date or applicable template rules change and cannot be edited or submitted by the user.
|
||||
- FR4.3 Validate fields before renaming: emission date parses and is within a plausible range, period is non-empty and consistent with the emission date and the template's granularity/offset, emitting company recognized, document type among the template's allowed values, each optional field matches its declared format (empty optional fields are allowed). Any failure routes to `pending` with the failing fields identified.
|
||||
- FR4.4 Collisions in the target prefix are resolved with an incremental suffix; the final key is always the one logged.
|
||||
|
||||
### Epic 5 — Renaming, moving, logging
|
||||
|
||||
- FR5.1 Rename and move the object within the bucket (S3 copy to the target key per the template pattern, then delete of the source key). Update the sidecar and SQLite in the same transaction boundary: sidecar first, then SQLite; a crash between the two is repaired by the consistency job.
|
||||
- FR5.1 Rename and move the object within the bucket using a recoverable sequence: copy to the target key, persist the sidecar with the new current key and event, delete the source key only after the sidecar is durable, then synchronize SQLite. These stores do not share a transaction. A crash or replica failure is recovered by the consistency job. If a transient technical operation fails, retry it safely for up to 3 times after the initial attempt, with delays of 5 seconds, 30 seconds, and 2 minutes. Do not assign business status `pending` solely because of technical failure.
|
||||
- FR5.2 Every event is logged in the sidecar history: file hash, old key, new key, template, extracted fields, confidence, decision, status, timestamps.
|
||||
- FR5.3 Objects missing from the bucket while not in a terminal state are detected by the consistency job and the record is flagged as orphaned.
|
||||
|
||||
### Epic 6 — Dashboard: file list
|
||||
|
||||
- FR6.1 A single list shows every file the system has seen, with columns: emission date, period, current object name, status, template (document family), document type, emitting company, confidence. Served from SQLite.
|
||||
- FR6.2 Filter by status, template, document type, emitting company, period, and date range; sort by any column; free-text search on object name, any field and optionnally first-page content.
|
||||
- FR6.3 Status is visually distinguishable (badge): not processed, processed, validated.
|
||||
- FR6.4 The list shows counters: files per status, automation rate (processed / total), validation rate, plus a consistency indicator (orphaned objects, missing sidecars).
|
||||
- FR6.1 A single list shows every file the system has seen, with columns: emission date, period, current object name, status, template (document family), document type, emitting company, and confidence where applicable. Served from SQLite.
|
||||
- FR6.2 Filter by status, template, document type, emitting company, period, and date range; sort by any column; free-text search on object name, metadata fields, and the complete extracted text from every page of the PDF.
|
||||
- FR6.3 Status is visually distinguishable (badge): `pending`, `processed`, `validated`.
|
||||
- FR6.4 The list exposes a consistency indicator for known anomalies (orphaned objects, missing sidecars), without a separate statistics dashboard or aggregate KPI counters.
|
||||
|
||||
### Epic 7 — Dashboard: file detail page
|
||||
|
||||
- FR7.1 Clicking a file in the list opens a detail page.
|
||||
- FR7.2 The detail page displays the PDF (embedded viewer), streamed from the bucket by the backend, alongside the extracted data.
|
||||
- FR7.3 The user can change the template (dropdown of configured templates, showing the classifier's top candidates and scores).
|
||||
- FR7.4 The detail page always shows the five common fields (template, emission date, emitting company, document type, period) plus the current template's optional fields; the user can correct each value inline; changing the template reloads the optional fields with the new template's schema and best-effort re-extraction, keeping the common field values, and recomputes the period from the new template's granularity and offset. The period is editable and displayed as a free-text field with a suggested value derived from the emission date.
|
||||
- FR7.4 The detail page always shows the five common fields (template, emission date, emitting company, document type, period) plus the current template's optional fields; the user can correct editable values inline. Changing the template reloads the optional fields with the new template's schema and best-effort re-extraction, keeps the other common field values, and recomputes the period from the emission date and the selected rules. The period is read-only, calculated by the server, and never accepted as user input.
|
||||
- FR7.5 A preview shows the target object key and destination prefix resulting from the current values, updating live as fields change.
|
||||
- FR7.6 A validate action applies the rename/move (if not already done), sets the status to validated, and records the corrected values as training data.
|
||||
|
||||
### Epic 8 — Template management
|
||||
### Epic 8 — Configuration file management
|
||||
|
||||
- FR8.1 Templates are defined in YAML configuration files.
|
||||
- FR8.2 Adding a template = YAML entry (including the period granularity and offset) + example files; no code change.
|
||||
- FR8.3 The dashboard lists configured templates with their slot definitions, period rules, target prefix, and the count of corpus files per template.
|
||||
- FR8.1 Templates and extraction rules are defined in a JSONC configuration file; there is no separate template-management API or template registry endpoint.
|
||||
- FR8.2 Adding or changing a template means editing the JSONC configuration file (including period granularity and offset) plus example files; no code change is required.
|
||||
- FR8.3 The dashboard configuration editor retrieves the current JSONC file and uploads a modified version through dedicated configuration endpoints. The server validates the file before replacing the active configuration and reloading the in-memory template registry.
|
||||
|
||||
### Epic 9 — Retraining
|
||||
|
||||
@@ -135,52 +135,53 @@ The object key pattern of each template is composed from these common fields, pl
|
||||
|
||||
- FR10.1 A periodic consistency job verifies, for every sidecar: object exists at current key, sidecar in sync with SQLite; and for every PDF object: sidecar exists. Anomalies are listed in the dashboard with repair actions.
|
||||
- FR10.2 Full SQLite rebuild from sidecars is available as a dashboard action and a CLI command.
|
||||
- FR10.3 The pending prefix is re-scanned at startup to resume interrupted processing.
|
||||
- FR10.3 At startup, recover interrupted technical operations from recorded state. Re-scan `pending/` to reconcile metadata and detect anomalies, but do not automatically repeat classification or rename for documents awaiting manual correction.
|
||||
|
||||
## 6. Non-Functional Requirements
|
||||
|
||||
- NFR1 Privacy: fully offline on self-hosted infrastructure. No network calls except the local Garage bucket, and serving the local dashboard. Dependencies must not require downloads at runtime.
|
||||
- NFR2 Performance: end-to-end processing of a standard PDF under 5 seconds on the target server, including the S3 download; dashboard file list loads under 1 s for 10,000 records (served from SQLite).
|
||||
- NFR3 Reliability: zero silent wrong renames. Every ambiguous case routes to not processed.
|
||||
- NFR4 Durability: bucket versioning enabled; SQLite with WAL. A crash at any point leaves no half-renamed state: the S3 copy is a non-destructive copy-then-delete, the sidecar is written atomically, and the consistency job repairs SQLite-sidecar drift.
|
||||
- NFR3 Reliability: zero silent wrong renames. Every ambiguous case routes to `pending`.
|
||||
- NFR4 Durability: bucket versioning enabled; SQLite with WAL. Operations must be recoverable after crashes using copy-then-delete, atomic sidecar writes, idempotent retries, and consistency repair for SQLite-sidecar drift. For transient technical failures, perform at most 3 retries after the initial attempt, with delays of 5 seconds, 30 seconds, and 2 minutes. After the 3 retries are exhausted, stop processing that item; do not launch another retry or manual-review workflow, and do not change its business status to `pending` solely because of the technical failure.
|
||||
- NFR5 Language: documents in French; UI in English or French (single locale acceptable in v1).
|
||||
- NFR6 Deployment: runs as a container next to the Garage deployment and the Nextcloud server; configuration via a single YAML file (templates) and environment variables (S3 endpoint, credentials, bucket).
|
||||
- NFR6 Deployment: runs as a container next to the Garage deployment and the Nextcloud server; configuration via a single JSONC file (templates) and environment variables (S3 endpoint, credentials, bucket).
|
||||
- NFR7 Stack: Python (FastAPI, boto3 or minio S3 client, scikit-learn, pdfplumber/PyMuPDF, rapidfuzz, spaCy fr model optional), React (Vite, a lightweight PDF viewer such as react-pdf), SQLite, Garage for S3 storage, Nextcloud with External Storage (S3) as the user-facing file interface.
|
||||
- NFR9 No encryption at rest in v1: documents are not sensitive; this decision is recorded explicitly and can be revisited.
|
||||
|
||||
## 7. Data Model
|
||||
|
||||
Sidecar JSON (source of truth), one per PDF at `{key}.meta.json`:
|
||||
Sidecar JSON (source of truth), one per PDF at `{key}.meta.json`. The `sha256` field is the unique document identifier and must match the digest of the exact PDF bytes. Any byte-level change creates a new document record; a rename/move without content changes preserves the identifier. Identical PDF bytes map to the same record, so ingestion must detect an existing SHA-256 and handle it as a duplicate rather than create a second row.
|
||||
|
||||
- schema_version, sha256, original_key, current_key, status, template_name, confidence, emission_date, period, emitting_company, document_type, optional_fields (dict of additional field values), extracted_text, created_at, updated_at, events[] (type: classified / renamed / validated / corrected, payload, timestamp).
|
||||
- schema_version, sha256, original_key, current_key, status, template_name, confidence, emission_date, period, emitting_company, document_type, optional_fields (dict of additional field values), extracted_text (full text from all pages), created_at, updated_at, events[] (type: classified / renamed / validated / corrected, payload, timestamp).
|
||||
|
||||
SQLite replica (rebuilt from sidecars):
|
||||
|
||||
- files: id, sha256, original_key, current_key, status, template_id, confidence, emission_date, period, emitting_company, document_type, optional_fields (JSON), created_at, updated_at, extracted_text, created_at, updated_at.
|
||||
- consistency_flags: id, file_id, kind (orphaned_object, missing_sidecar, drift).
|
||||
- files: sha256 (primary key), original_key, current_key, status, template_id, confidence, emission_date, period, emitting_company, document_type, optional_fields (JSON), created_at, updated_at, extracted_text.
|
||||
- consistency_flags: id, sha256 (nullable reference to the affected PDF digest), kind (orphaned_object, missing_sidecar, drift).
|
||||
|
||||
## 8. API Surface (indicative)
|
||||
|
||||
- GET /files?status=&template=&q=&page= — list with filters (from SQLite).
|
||||
- GET /files/{id} — detail: text, fields, candidates, current key.
|
||||
- GET /files/{id}/pdf — stream the PDF from the bucket for the viewer.
|
||||
- POST /files/{id}/validate — body: template_id, fields; applies rename if needed, sets validated.
|
||||
- GET /templates — configured templates.
|
||||
- POST /retrain — starts training; GET /retrain/status; POST /retrain/{id}/activate.
|
||||
- GET /files/{sha256} — detail: text, fields, candidates, current key.
|
||||
- GET /files/{sha256}/pdf — stream the PDF from the bucket for the viewer.
|
||||
- POST /files/{sha256}/validate — body: template_id, fields; applies rename if needed, sets validated.
|
||||
- GET /config — retrieve the current JSONC configuration file.
|
||||
- PUT /config — validate and save a replacement JSONC configuration file, then reload the configuration.
|
||||
- POST /retrain — starts training and returns a generic job ID; GET /jobs/{job_id} reports status and result.
|
||||
- GET /consistency — list anomalies; POST /consistency/rebuild — rebuild SQLite from sidecars.
|
||||
|
||||
## 9. Acceptance Criteria (v1)
|
||||
|
||||
- AC1 A new PDF dropped in `inbox/` via Nextcloud with high-confidence classification and valid fields appears as processed within the poll interval, renamed and moved in the bucket.
|
||||
- AC2 A PDF below the confidence threshold appears as not processed, is moved to `pending/`; correcting and validating it on the detail page renames it and sets validated.
|
||||
- AC1 A new PDF dropped in `inbox/` via Nextcloud whose deterministic rules yield an unambiguous template and valid fields appears as processed within the poll interval, renamed and moved in the bucket.
|
||||
- AC2 A PDF that cannot be classified unambiguously by deterministic rules, or has missing/invalid required fields, appears as `pending` and is moved to `pending/`; correcting and validating it on the detail page renames it and sets validated.
|
||||
- AC3 Opening a processed file and validating without changes sets validated without re-renaming.
|
||||
- AC4 The file list filters by status, template, document type, emitting company, period, and date; counters match the data.
|
||||
- AC5 Changing the template on the detail page keeps the common fields, reloads the optional fields per the new template schema, recomputes the period per the new template's granularity and offset, and shows the live target key preview.
|
||||
- AC5b Every file in every state carries the five common fields (template, emission date, emitting company, document type, period); a document whose fields cannot be recognized is routed to not processed rather than renamed.
|
||||
- AC4 The file list filters by status, template, document type, emitting company, period, and date; the consistency indicator matches the known anomalies.
|
||||
- AC5 Changing the template on the detail page keeps the other common fields, reloads the optional fields per the new template schema, recomputes the read-only period per the new template's granularity and offset, and shows the live target key preview.
|
||||
- AC5b Every file in every state carries the five common fields (template, emission date, emitting company, document type, period); a document whose fields cannot be recognized is routed to `pending` rather than renamed.
|
||||
- AC5c Period computation per template rules: a document dated 2026-09-20 under a monthly template with offset current yields "September 2026"; under a quarterly template with offset next it yields "Q4 2026"; under a yearly template it yields "2026".
|
||||
- AC6 Retraining uses initial corpus + validated files, shows metrics, and activation swaps the model without service restart.
|
||||
- AC7 Adding a template via YAML (with example files) makes it available for classification and in the detail page dropdown, with no code change.
|
||||
- AC8 Killing the service mid-processing leaves no half-renamed object: the copy-then-delete order and the sidecar-first write guarantee a consistent state on restart, and the consistency job repairs SQLite.
|
||||
- AC6 Retraining uses the initial corpus + validated files, runs as a generic asynchronous job queried through `GET /jobs/{job_id}`, shows metrics when complete, and activation swaps the model without service restart.
|
||||
- AC7 Editing and uploading the JSONC configuration (with example files) makes a new template available for classification and correction; the server validates the replacement before activating it, with no code change.
|
||||
- AC8 Killing the service mid-processing leaves a recoverable state: startup recovery and the consistency job reconcile object keys, sidecars, and SQLite; retries do not duplicate renames or delete the only recoverable copy.
|
||||
- AC9 Deleting the SQLite database entirely and running the rebuild command restores a fully functional dashboard from the sidecars alone.
|
||||
- AC10 No network traffic other than the local Garage bucket, local Nextcloud, and local dashboard serving (verifiable with a firewall audit).
|
||||
- AC11 1,000-file list view renders with filters in under 1 second.
|
||||
@@ -195,6 +196,6 @@ SQLite replica (rebuilt from sidecars):
|
||||
|
||||
- Choice between pdfplumber and PyMuPDF (license and performance tradeoff) — decide in Week 1 spike.
|
||||
- S3 client library: boto3 vs minio-py against Garage — covered by the Week 1 Garage compatibility spike (listing, copy-object, multipart, versioning behavior).
|
||||
- Exact confidence threshold and per-template overrides — calibrate on the initial corpus.
|
||||
- Post-v1 ML decision: exact model confidence threshold and per-template overrides, to be calibrated on the initial corpus before enabling ML.
|
||||
- Poll interval tradeoff between responsiveness and Garage listing load.
|
||||
- Exact human-readable period formats (French month names, quarter notation) and their rendering in the object key — to be fixed in the template YAML spec.
|
||||
- Exact human-readable period formats (French month names, quarter notation) and their rendering in the object key — to be fixed in the template JSONC spec.
|
||||
|
||||
Reference in New Issue
Block a user