From 0748584b22d9e675d8803e8ba31a7fb873d75eb8 Mon Sep 17 00:00:00 2001 From: napnap75 Date: Mon, 19 Oct 2026 16:20:34 +0200 Subject: [PATCH] Add local ML models and document suggestions (#13) --- automatic-pdf-renamer/API.md | 71 ++- automatic-pdf-renamer/Dockerfile | 4 +- automatic-pdf-renamer/app/api/routes.py | 371 ++++++++------ automatic-pdf-renamer/app/config.py | 56 ++- automatic-pdf-renamer/app/ml.py | 363 ++++++++++++++ automatic-pdf-renamer/app/model_manager.py | 378 +++++++++++++++ automatic-pdf-renamer/app/pipeline.py | 456 ++++++------------ automatic-pdf-renamer/app/sidecar.py | 1 + automatic-pdf-renamer/arch.md | 52 +- automatic-pdf-renamer/brief.md | 5 +- automatic-pdf-renamer/epics.md | 8 +- automatic-pdf-renamer/prd.md | 21 +- .../tests/test_document_suggestions.py | 308 ++++++++++++ .../tests/test_ml_pipeline_order.py | 152 ++++++ .../tests/test_model_manager.py | 124 +++++ .../tests/test_pipeline_date_extraction.py | 45 ++ automatic-pdf-renamer/web/app.js | 2 +- automatic-pdf-renamer/web/config.js | 4 +- automatic-pdf-renamer/web/document.html | 6 + automatic-pdf-renamer/web/document.js | 257 +++++++++- automatic-pdf-renamer/web/maintenance.html | 13 + automatic-pdf-renamer/web/maintenance.js | 61 ++- automatic-pdf-renamer/web/styles.css | 151 ++++++ 23 files changed, 2372 insertions(+), 537 deletions(-) create mode 100644 automatic-pdf-renamer/app/ml.py create mode 100644 automatic-pdf-renamer/app/model_manager.py create mode 100644 automatic-pdf-renamer/tests/test_document_suggestions.py create mode 100644 automatic-pdf-renamer/tests/test_ml_pipeline_order.py create mode 100644 automatic-pdf-renamer/tests/test_model_manager.py create mode 100644 automatic-pdf-renamer/tests/test_pipeline_date_extraction.py diff --git a/automatic-pdf-renamer/API.md b/automatic-pdf-renamer/API.md index b2f6781..b9b7e1c 100644 --- a/automatic-pdf-renamer/API.md +++ b/automatic-pdf-renamer/API.md @@ -1,7 +1,7 @@ # API — Automatic PDF Renamer Version: 1.0 (design specification) -Base path: `/api/v1` +Base path: `/` Backend: FastAPI; storage: Garage S3-compatible bucket; metadata source of truth: per-file sidecars; query index: SQLite. This document defines the HTTP API consumed by the local web dashboard and the operational API used to observe ingestion, recover the index, and run maintenance jobs. The API is a local, same-origin interface; it does not expose the S3 credentials or bypass the sidecar-as-source-of-truth rule. @@ -9,10 +9,10 @@ This document defines the HTTP API consumed by the local web dashboard and the o ## 1. API principles - **Nextcloud/S3 remains the upload interface.** A user uploads a PDF to the configured `inbox/` prefix through Nextcloud or an S3 client. The API does not accept PDF uploads in v1. -- **Ingestion is server-side and automatic.** The poller discovers stable PDF objects in `inbox/`, extracts text from every page, applies deterministic rules, validates fields, and either renames the file or moves it to `pending/`. +- **Ingestion is server-side and automatic.** The poller discovers stable PDF objects in `inbox/`, extracts text from every page, follows the ordered local-ML and configured-regex pipeline in Section 10, validates fields, and either renames the file or moves it to `pending/`. - **Sidecars are authoritative.** For a mutation, persist the sidecar first, then synchronize SQLite. SQLite is used for list/search and consistency diagnostics, and can be rebuilt from sidecars. - **Business statuses are exactly** `pending`, `processed`, and `validated`. Technical job state is separate and must not be represented as a business status. -- **The initial release is deterministic.** API responses may include a nullable confidence/model section for forward compatibility, but the rules-only release must not fabricate ML scores. +- **Classification is local and model-driven.** Company, document type, additional fields, template, and period format are predicted by local models. Emission dates are extracted only by configured regular expressions; period and storage key are calculated deterministically. Unavailable or low-confidence required predictions are routed to review. - **No rename undo endpoint exists.** Renames are recorded for audit, not reversed by a user-facing API. - The API and processing pipeline remain offline except for the configured local Garage endpoint. @@ -73,12 +73,9 @@ This convention applies to HTTP error responses from every endpoint. Successful | GET | `/ingestion/status` | Inspect poller status and recent activity | | GET | `/config` | Retrieve the current JSONC configuration file | | PUT | `/config` | Validate and save a modified JSONC configuration file | -| GET | `/models` | Post-v1: list model versions | -| POST | `/retrain` | Post-v1: start local model training | -| POST | `/retrain/{job_id}/activate` | Post-v1: activate a trained model | -| POST | `/retrain/{job_id}/discard` | Post-v1: discard a candidate model | - -The endpoints in the models/retraining section are post-v1 capabilities. They may be omitted from the first deterministic release without affecting the core API. +| GET | `/models` | Report whether the current complete model set is available | +| POST | `/models/train` | Start full model retraining and replace the active set on success | +The model-maintenance API is described in [Section 10](#10-local-model-maintenance). ## 4. Configuration file @@ -191,7 +188,7 @@ Query parameters: | `sort_by` | string | Allowlisted: `emission_date`, `period`, `current_key`, `status`, `template_name`, `document_type`, `emitting_company`, `created_at`, `updated_at` | | `sort_order` | string | `asc` or `desc` | -Example: `GET /api/v1/files?status=pending&q=EDF&page=1&page_size=25` +Example: `GET /files?status=pending&q=EDF&page=1&page_size=25` Response: @@ -288,7 +285,7 @@ All mutation endpoints must validate the requested state, recompute the period f ``` - `template_name`, `emission_date`, `emitting_company`, and `document_type` must conform to the selected template's schema. -- The server selects the applicable period rules from the configured mapping for the company and document type, calculates `period` from `emission_date`, and never accepts a client-supplied period or period format. +- The server validates the model-selected template and final `period_format` against configuration, calculates `period` from `emission_date` exactly once, and never accepts a client-supplied derived `period` or `current_key` as authoritative. - `optional_fields` may contain only fields declared by the selected template. Empty optional fields are allowed where the schema permits them. - The server must never trust a target key supplied by the browser; it calculates the target key itself. @@ -411,17 +408,53 @@ Returns job status for consistency scan, SQLite rebuild, or other asynchronous m Allowed job statuses are technical job states such as `queued`, `running`, `succeeded`, and `failed`; these are not document business statuses. This generic endpoint reports maintenance and retraining jobs. If the process restarts and in-memory job state is lost, the API should report the job as interrupted/unknown and use persisted status files or database state where available. -## 10. Post-v1 model endpoints +## 10. Local model maintenance -These routes are future extensions; they are not required for the deterministic initial release. +The Maintenance page exposes **one model action only**: retrain the complete model set. There are no per-model training actions, model activation controls, or model-history/version selection in the UI. -- `GET /models`: list locally stored model versions and identify the active model. -- `POST /retrain`: start training using the initial corpus plus eligible `validated` documents. Returns `202` and a job ID. -- `GET /jobs/{job_id}`: show training state, dataset/model version, accuracy, confusion matrix, warnings, and output artifact. -- `POST /retrain/{job_id}/activate`: atomically activate a successfully evaluated candidate. -- `POST /retrain/{job_id}/discard`: discard a candidate without changing the active model. +- `POST /models/train`: start a background job that retrains all required models in dependency order from indexed documents with business status `validated`. The request has no `model_name` parameter. Return a job identifier. +- `GET /jobs/{job_id}`: report job state, current stage, progress percentage, and a user-readable status message. The UI displays a progress bar using the same job-progress mechanism as SQLite rebuild and consistency check. +- `GET /models` may remain as a read-only diagnostic endpoint to report whether the complete active model set is present, but it must not expose a model-version history or offer activation. -Training and activation must remain local. Failed training must not replace the active model. ML confidence and candidate scores are absent or null when no model was used. +Retraining covers all ML-supported stages: `field_emitting_company`, `field_document_type`, `additional_field_groups`, `additional_field_values`, `template_name`, and `period_format`. There is **no `field_emission_date` model**: emission dates are extracted only with the ordered regexes in `config.jsonc` under `date_extraction`. + +Training labels come from validated sidecar values. Additional-field value labels are usable only when the value appears in the saved extracted text. The shared `additional_field_values` model uses one token-level classifier for all eligible additional fields. + +The replacement policy is **replace, do not version**. Train the full replacement set in a staging location; report progress for the complete job; only after all required models train successfully, replace the active set and delete the previous artifacts. If training fails, report a failed job and keep the previous complete model set usable. No historical versions are retained after a successful replacement. Model artifacts are trusted local joblib files; loading them can execute code. + +### Required inference order + +1. Predict `emitting_company` and `document_type` from the complete extracted PDF text. +2. Extract `emission_date` with regex only, using the patterns in the `date_extraction` configuration section. Choose the earliest valid match in document order; if matches begin at the same position, use the order in the configuration. Normalize to `YYYY-MM-DD`. No ML model may predict the emission date. +3. Predict only the applicable additional-field list/group from `emitting_company` and `document_type`. Do not predict `period_format` at this stage. +4. Predict each additional-field value from the complete extracted PDF text with the shared token-level extraction model. +5. Predict `template_name` and `period_format` from `emitting_company`, `document_type`, and the extracted `additional_fields`. +6. Calculate `period` once from the emission date and final period-format configuration; calculate `current_key` from the selected template's key pattern and final values. These are deterministic derived values, not model predictions. + +Any required ML result that is missing or below the configured confidence threshold is unresolved and must follow the existing pending/review behavior; do not substitute a hard-coded mapping. + + +### POST `/files/{sha256}/suggestions` + +Returns asynchronous, confidence-gated proposals for the document detail page. The browser first renders the saved document, then requests suggestions in the background and refreshes them after edits to template, period format, company, document type, emission date, or additional fields. Model-backed proposals are omitted when the model is unavailable or its confidence is below `CONFIDENCE_THRESHOLD`. Additional-field proposals are limited to complete configured values found in the document; ambiguous matches and values absent from the configured list are omitted. The response includes `template_name`, `period_format`, `emitting_company`, and `document_type` when confident, plus `optional_fields` containing canonical configured matches, and derived `period` and `current_key` values when the corresponding inputs can be calculated. Derived values are recalculated from the current form values and are not independent ML predictions. + +Example response: + +```json +{ + "suggestions": { + "template_name": {"value": "invoice", "confidence": 0.94, "source": "model"}, + "period_format": {"value": "monthly", "confidence": 0.91, "source": "model"}, + "emitting_company": {"value": "Example Energy", "confidence": 0.97, "source": "model"}, + "document_type": {"value": "invoice", "confidence": 0.96, "source": "model"}, + "period": {"value": "2026-09", "confidence": 0.91, "source": "derived"}, + "current_key": {"value": "files/example/2026-09/invoice.pdf", "confidence": 0.91, "source": "derived"} + }, + "confidence_threshold": 0.8 +} +``` + +The frontend debounces changes and cancels stale requests. A suggestion is shown only when the corresponding prediction meets the confidence threshold. Users can apply model-backed suggestions to editable controls; calculated period and key previews remain derived from the current values and are recalculated on save. ## 11. Consistency and implementation rules diff --git a/automatic-pdf-renamer/Dockerfile b/automatic-pdf-renamer/Dockerfile index 600d06e..5798d7a 100644 --- a/automatic-pdf-renamer/Dockerfile +++ b/automatic-pdf-renamer/Dockerfile @@ -17,7 +17,9 @@ RUN python -m pip install --upgrade pip && \ boto3 \ pymupdf \ python-dateutil \ - PyYAML + PyYAML \ + joblib \ + scikit-learn EXPOSE 8000 diff --git a/automatic-pdf-renamer/app/api/routes.py b/automatic-pdf-renamer/app/api/routes.py index 581389c..b5ac268 100644 --- a/automatic-pdf-renamer/app/api/routes.py +++ b/automatic-pdf-renamer/app/api/routes.py @@ -4,6 +4,8 @@ import json import logging import os import re +import threading +import uuid from datetime import datetime, timezone from pathlib import Path from typing import Any @@ -13,7 +15,10 @@ from fastapi import APIRouter, Depends, HTTPException, Request, Response, status from app.config import TemplateRegistry, load_templates from app.db import FileRecord, SQLiteFileRepository from app.period import format_period_from_emission_date +from app.pipeline import DocumentProcessor from app.storage import StorageBackend +from app.model_manager import list_models, train_all_models +from app.ml import LocalMLPipeline from .deps import get_current_user from .models import ( @@ -93,7 +98,10 @@ def _as_file_item(file_id: str, metadata: dict[str, Any]) -> FileListItem: @router.get("/health", response_model=HealthResponse) async def health() -> HealthResponse: """Service health checkpoint.""" - return HealthResponse(status="ok", app="automatic-pdf-renamer") + logger.debug("API GET /health: health check requested") + result = HealthResponse(status="ok", app="automatic-pdf-renamer") + logger.debug("API GET /health: status=%s", result.status) + return result @router.get("/files", response_model=list[FileListItem]) @@ -116,6 +124,8 @@ async def list_files( page: int = 1, ) -> list[FileListItem]: """List documents and report filtered and global counts in response headers.""" + logger.debug("API GET /files: user=%s page=%s sort_by=%s sort_order=%s", (current_user or {}).get("username", "anonymous"), page, sort_by, sort_order) + logger.debug("API GET /files: active filters=%s", {key: value for key, value in {"status": status, "template": template, "period_format": period_format, "q": q, "current_key": current_key, "emitting_company": emitting_company, "document_type": document_type, "period": period, "confidence": confidence, "updated_at": updated_at}.items() if value is not None}) storage = _storage(request) db = _db(request) filters = { @@ -149,6 +159,7 @@ async def list_files( ) response.headers["X-Total-Count"] = str(db.count_files(**filters)) response.headers["X-Status-Counts"] = json.dumps(db.get_status_counts()) + logger.debug("API GET /files: returning %s records; filtered_total=%s; status_counts=%s", len(records), response.headers["X-Total-Count"], response.headers["X-Status-Counts"]) return [ FileListItem( @@ -176,6 +187,7 @@ async def get_file( current_user: dict[str, str] | None = Depends(get_current_user), ) -> FileDetail: """Return metadata and extracted text for a single file.""" + logger.debug("API GET /files/%s: loading document metadata", file_id) storage = _storage(request) db = _db(request) record = db.get_by_id(file_id) @@ -234,7 +246,7 @@ async def get_file( "history": sidecar_metadata.get("events", []), } item = _as_file_item(record.id, metadata) - return FileDetail( + result = FileDetail( id=item.id, status=item.status, current_key=item.current_key, @@ -253,6 +265,148 @@ async def get_file( optional_fields=record.optional_fields, history=metadata["history"] if isinstance(metadata["history"], list) else [], ) + logger.debug("API GET /files/%s: loaded status=%s current_key=%s optional_field_count=%s", file_id, result.status, result.current_key, len(result.optional_fields or {})) + return result + + +@router.post("/files/{file_id}/suggestions") +async def get_document_model_suggestions( + request: Request, + file_id: str, + payload: dict[str, Any], + current_user: dict[str, str] | None = Depends(get_current_user), +) -> dict[str, Any]: + """Return confidence-gated model suggestions without delaying document rendering.""" + db = _db(request) + record = db.get_by_id(file_id) + if record is None: + raise HTTPException(status_code=404, detail=f"File '{file_id}' not found") + + registry = _registry(request) + threshold = float(os.environ.get("CONFIDENCE_THRESHOLD", "0.8")) + ml = LocalMLPipeline(confidence_threshold=threshold) + text = record.extracted_text or "" + company = str(payload.get("emitting_company") or record.emitting_company or "").strip() + document_type = str(payload.get("document_type") or record.document_type or "").strip() + current_fields = dict(record.optional_fields or {}) + submitted_fields = payload.get("optional_fields") + if isinstance(submitted_fields, dict): + current_fields.update({ + str(name): str(value) + for name, value in submitted_fields.items() + if value is not None and str(value).strip() + }) + suggestions, field_decisions = ml.suggest_document_fields( + text, + company, + document_type, + current_fields, + registry.get_additional_fields(), + ) + field_decisions = { + **field_decisions, + "emission_date": "unavailable: no valid configured date or stored date", + "period": "skipped: missing date or period format", + "current_key": "skipped: missing template or key pattern", + } + + # Recommend the first valid date matched in document order by configured regexes. + # This is deterministic content extraction, not an ML prediction. + try: + recommended_date, date_error = DocumentProcessor( + registry, ml_pipeline=ml + )._extract_emission_date(text) + if not recommended_date: + # The stored date was originally produced by the same configured + # content-extraction pipeline. Keep it available as a recommendation + # when the current extraction configuration cannot reproduce it. + stored_date = str(getattr(record, "emission_date", "") or "").strip() + if stored_date: + recommended_date = stored_date + field_decisions["emission_date"] = "content: using stored date" + if recommended_date: + suggestions["emission_date"] = { + "value": recommended_date, + "source": "content", + } + field_decisions["emission_date"] = f"content: {recommended_date}" + elif date_error: + field_decisions["emission_date"] = f"unavailable: {date_error}" + except Exception as exc: + stored_date = str(getattr(record, "emission_date", "") or "").strip() + if stored_date: + suggestions["emission_date"] = { + "value": stored_date, + "source": "content", + } + field_decisions["emission_date"] = f"content: using stored date {stored_date}" + else: + field_decisions["emission_date"] = f"unavailable: extraction failed ({exc})" + + derived_template = str(payload.get("template_name") or record.template_name or "") + derived_format = str(payload.get("period_format") or record.period_format or "") + + try: + raw_date = str(payload.get("emission_date") or record.emission_date or "") + emission_date = datetime.strptime(raw_date, "%Y-%m-%d").date() + period_config = registry.get_period_format_config(derived_format) + if period_config is not None: + calculated_period = format_period_from_emission_date(emission_date, period_config.to_dict()) + suggestions["period"] = { + "value": calculated_period, + "source": "derived", + } + field_decisions["period"] = f"derived: {calculated_period}" + key_pattern = registry.get_key_pattern(derived_template) + if key_pattern: + optional_fields = payload.get("optional_fields") + if not isinstance(optional_fields, dict): + optional_fields = record.optional_fields or {} + pattern_values = dict(optional_fields) + pattern_values.update({ + "company": company, + "document_type": document_type, + "emission_date": emission_date.isoformat(), + "period": calculated_period, + "period_format": derived_format, + "template_name": derived_template, + "emitting_company": company, + "additional_fields": ", ".join(str(value) for value in optional_fields.values()), + "original_filename": (record.original_key or "").rsplit("/", 1)[-1], + }) + for _ in range(20): + try: + calculated_key = key_pattern.format(**pattern_values).strip("/") + break + except KeyError as error: + pattern_values[str(error.args[0])] = "unknown" + else: + calculated_key = "" + if calculated_key: + if not calculated_key.startswith("files/"): + calculated_key = f"files/{calculated_key}" + if not calculated_key.lower().endswith(".pdf"): + calculated_key = f"{calculated_key}.pdf" + suggestions["current_key"] = { + "value": calculated_key, + "source": "derived", + } + field_decisions["current_key"] = f"derived: {calculated_key}" + else: + field_decisions["current_key"] = "unavailable: key pattern could not be formatted" + else: + field_decisions["current_key"] = "skipped: template has no key pattern" + else: + field_decisions["period"] = f"unavailable: unknown period format {derived_format!r}" + except (TypeError, ValueError) as exc: + field_decisions["period"] = f"unavailable: {exc}" + + logger.debug( + "API POST /files/%s/suggestions decisions=%s", + file_id, + field_decisions, + ) + return {"suggestions": suggestions, "confidence_threshold": threshold} @router.get("/files/{file_id}/pdf") @@ -262,6 +416,7 @@ async def get_file_pdf( current_user: dict[str, str] | None = Depends(get_current_user), ) -> Response: """Stream a PDF by key from storage when present.""" + logger.debug("API GET /files/%s/pdf: resolving PDF storage key", file_id) storage = _storage(request) db = _db(request) record = db.get_by_id(file_id) @@ -301,6 +456,7 @@ async def edit_file( current_user: dict[str, str] | None = Depends(get_current_user), ) -> FileDetail: """Recalculate and persist user-edited document metadata and its storage key.""" + logger.debug("API POST /files/%s/edit: template=%s period_format=%s company=%s document_type=%s optional_fields=%s", file_id, payload.template_name, payload.period_format, payload.emitting_company, payload.document_type, sorted((payload.optional_fields or {}).keys())) storage = _storage(request) db = _db(request) registry = _registry(request) @@ -545,12 +701,13 @@ async def validate_file( ) -@router.get("/api/document-options") +@router.get("/document-options") async def document_edit_options( request: Request, current_user: dict[str, str] | None = Depends(get_current_user), ) -> dict[str, Any]: """Return configured templates and existing document values for edit suggestions.""" + logger.debug("API GET /document-options: loading template and field options") registry = _registry(request) db = _db(request) records = db.list_files(page_size=10000) @@ -606,7 +763,7 @@ async def document_edit_options( } -@router.get("/api/config") +@router.get("/config") async def get_config( current_user: dict[str, str] | None = Depends(get_current_user), ) -> dict[str, str]: @@ -618,7 +775,7 @@ async def get_config( raise HTTPException(status_code=500, detail="Unable to read the configured template file.") from error -@router.put("/api/config") +@router.put("/config") async def update_config( payload: ConfigUpdateRequest, request: Request, @@ -663,37 +820,81 @@ async def list_templates( return items -@router.post("/retrain") -async def start_retrain( +@router.get("/models") +async def get_models( + current_user: dict[str, str] | None = Depends(get_current_user), +) -> dict[str, Any]: + """Report whether each model in the current complete set is available.""" + result = list_models() + logger.debug("API GET /models: complete=%s", result.get("complete")) + return result + + +@router.post("/models/train", status_code=202) +async def train_models( request: Request, current_user: dict[str, str] | None = Depends(get_current_user), ) -> dict[str, Any]: - """Start a retraining job. The current implementation records a job stub and returns its status.""" - request.app.state.last_retrain_job = { - "job_id": "retrain-1", - "status": "queued", - "message": "Retrain job accepted. Replace with a subprocess-backed job in production.", + """Start one background job that replaces the complete model set.""" + jobs = getattr(request.app.state, "model_training_jobs", None) + if jobs is None: + jobs = {} + request.app.state.model_training_jobs = jobs + if any(job.get("status") == "running" for job in jobs.values()): + raise HTTPException(status_code=409, detail="A full model retraining job is already running.") + + records = _db(request).list_files(status="validated", page=1, page_size=100000) + job_id = uuid.uuid4().hex + jobs[job_id] = { + "job_id": job_id, + "status": "running", + "progress": 0, + "stage": "Starting model training", + "message": "Preparing validated documents", } - return request.app.state.last_retrain_job + + def update_progress(progress: int, message: str) -> None: + jobs[job_id].update({ + "progress": max(0, min(100, int(progress))), + "stage": message, + "message": message, + }) + + def run_training() -> None: + try: + result = train_all_models(records, progress_callback=update_progress) + jobs[job_id].update({ + "status": "completed", + "progress": 100, + "stage": "Completed", + "message": "All models have been retrained and replaced.", + "result": result, + }) + except Exception as error: + logger.exception("Full model retraining job %s failed", job_id) + jobs[job_id].update({ + "status": "failed", + "stage": "Failed", + "message": str(error), + }) + + threading.Thread(target=run_training, name=f"model-training-{job_id[:8]}", daemon=True).start() + logger.debug("API POST /models/train: started full retraining job %s", job_id) + return {"job_id": job_id, "status": "running"} -@router.get("/retrain/status") -async def get_retrain_status( +@router.get("/jobs/{job_id}") +async def get_job_status( request: Request, - current_user: dict[str, str] | None = Depends(get_current_user), -) -> dict[str, Any]: - """Read the active retraining status.""" - job = getattr(request.app.state, "last_retrain_job", {"job_id": "retrain-1", "status": "idle"}) - return job - - -@router.post("/retrain/{job_id}/activate") -async def activate_model( job_id: str, current_user: dict[str, str] | None = Depends(get_current_user), ) -> dict[str, Any]: - """Activate a completed training model.""" - return {"job_id": job_id, "status": "activated", "message": "Model activation accepted."} + """Return current progress for a background model-training job.""" + jobs = getattr(request.app.state, "model_training_jobs", {}) + job = jobs.get(job_id) + if job is None: + raise HTTPException(status_code=404, detail="Job not found.") + return dict(job) @router.get("/consistency") @@ -702,6 +903,7 @@ async def list_consistency_flags( current_user: dict[str, str] | None = Depends(get_current_user), ) -> list[dict[str, Any]]: """Return consistency anomalies. In the current implementation, this is a lightweight placeholder.""" + logger.debug("API GET /consistency: scanning storage and database consistency") storage = _storage(request) db = _db(request) anomalies: list[dict[str, Any]] = [] @@ -741,120 +943,3 @@ async def rebuild_consistency( "count": len(rebuilt), } return request.app.state.last_rebuild - - -@router.get("/api/health") -async def api_health() -> HealthResponse: - """Alias for /health.""" - return await health() - - -@router.get("/api/files", response_model=list[FileListItem]) -async def list_files_api( - request: Request, - response: Response, - current_user: dict[str, str] | None = Depends(get_current_user), - status: str | None = None, - template: str | None = None, - period_format: str | None = None, - q: str | None = None, - current_key: str | None = None, - emitting_company: str | None = None, - document_type: str | None = None, - period: str | None = None, - confidence: str | None = None, - updated_at: str | None = None, - sort_by: str = "updated_at", - sort_order: str = "desc", - page: int = 1, -) -> list[FileListItem]: - """Compatibility alias for /files.""" - return await list_files( - request=request, - response=response, - current_user=current_user, - status=status, - template=template, - period_format=period_format, - q=q, - current_key=current_key, - emitting_company=emitting_company, - document_type=document_type, - period=period, - confidence=confidence, - updated_at=updated_at, - sort_by=sort_by, - sort_order=sort_order, - page=page, - ) - - -@router.get("/api/files/{file_id}", response_model=FileDetail) -async def get_file_api( - request: Request, - file_id: str, - current_user: dict[str, str] | None = Depends(get_current_user), -) -> FileDetail: - """Compatibility alias for /files/{file_id}.""" - return await get_file(request=request, file_id=file_id, current_user=current_user) - - -@router.post("/api/files/{file_id}/edit", response_model=FileDetail) -async def edit_file_api( - request: Request, - file_id: str, - payload: DocumentEditRequest, - current_user: dict[str, str] | None = Depends(get_current_user), -) -> FileDetail: - """Compatibility alias for POST /files/{file_id}/edit.""" - return await edit_file( - request=request, - file_id=file_id, - payload=payload, - current_user=current_user, - ) - - -@router.get("/api/templates") -async def list_templates_api( - request: Request, - current_user: dict[str, str] | None = Depends(get_current_user), -) -> list[dict[str, Any]]: - """Compatibility alias for /templates.""" - return await list_templates(request=request, current_user=current_user) - - -@router.post("/api/retrain") -async def start_retrain_api( - request: Request, - current_user: dict[str, str] | None = Depends(get_current_user), -) -> dict[str, Any]: - """Compatibility alias for /retrain.""" - return await start_retrain(request=request, current_user=current_user) - - -@router.get("/api/retrain/status") -async def get_retrain_status_api( - request: Request, - current_user: dict[str, str] | None = Depends(get_current_user), -) -> dict[str, Any]: - """Compatibility alias for /retrain/status.""" - return await get_retrain_status(request=request, current_user=current_user) - - -@router.get("/api/consistency") -async def list_consistency_flags_api( - request: Request, - current_user: dict[str, str] | None = Depends(get_current_user), -) -> list[dict[str, Any]]: - """Compatibility alias for /consistency.""" - return await list_consistency_flags(request=request, current_user=current_user) - - -@router.post("/api/consistency/rebuild") -async def rebuild_consistency_api( - request: Request, - current_user: dict[str, str] | None = Depends(get_current_user), -) -> dict[str, Any]: - """Compatibility alias for /consistency/rebuild.""" - return await rebuild_consistency(request=request, current_user=current_user) diff --git a/automatic-pdf-renamer/app/config.py b/automatic-pdf-renamer/app/config.py index fee4b89..0f65a29 100644 --- a/automatic-pdf-renamer/app/config.py +++ b/automatic-pdf-renamer/app/config.py @@ -130,6 +130,7 @@ class MappingMatch: company: str | None = None document_type: str | None = None + additional_fields: dict[str, Any] = field(default_factory=dict) @classmethod def from_dict(cls, data: dict[str, Any] | None) -> "MappingMatch": @@ -139,19 +140,40 @@ class MappingMatch: return cls( company=data.get("company"), document_type=data.get("document_type"), + additional_fields=data.get("additional_fields", {}), ) - def matches(self, company: str, document_type: str) -> bool: - """Check if this match criteria matches the given company and document type. - - Empty match (both None) matches everything. - company=None in match means match any company. - document_type=None in match means match any document type. - """ + def to_dict(self) -> dict[str, Any]: + """Serialize configured match criteria, including optional-field predicates.""" + result: dict[str, Any] = {} + if self.company is not None: + result["company"] = self.company + if self.document_type is not None: + result["document_type"] = self.document_type + if self.additional_fields: + result["additional_fields"] = self.additional_fields.copy() + return result + + def matches_identity(self, company: str, document_type: str) -> bool: + """Match company/document type independently of extracted optional fields.""" company_match = self.company is None or (isinstance(self.company, list) and company in self.company) or self.company == company doc_type_match = self.document_type is None or (isinstance(self.document_type, list) and document_type in self.document_type) or self.document_type == document_type return company_match and doc_type_match + def matches( + self, + company: str, + document_type: str, + optional_fields: dict[str, Any] | None = None, + ) -> bool: + """Match document identity and, when configured, extracted field values.""" + if not self.matches_identity(company, document_type): + return False + # Additional-field criteria are optional for backward compatibility. + # When configured, every named field must equal its expected value. + extracted = optional_fields or {} + return all(extracted.get(name) == expected for name, expected in self.additional_fields.items()) + @dataclass class MappingSet: @@ -215,9 +237,14 @@ class MappingRule: result["set"] = set_dict return result - def matches(self, company: str, document_type: str) -> bool: - """Check if this rule matches the given company and document type.""" - return self.match.matches(company, document_type) + def matches( + self, + company: str, + document_type: str, + optional_fields: dict[str, Any] | None = None, + ) -> bool: + """Check if this rule matches document identity and extracted values.""" + return self.match.matches(company, document_type, optional_fields) @dataclass @@ -507,7 +534,12 @@ class TemplateRegistry: """ return self._mappings - def resolve_mapping(self, company: str, document_type: str) -> dict[str, Any]: + def resolve_mapping( + self, + company: str, + document_type: str, + optional_fields: dict[str, Any] | None = None, + ) -> dict[str, Any]: """Resolve the mapping for a specific (company, document_type) tuple. Processes all rules in order. For each matching rule, applies the set fields. @@ -524,7 +556,7 @@ class TemplateRegistry: result: dict[str, Any] = {} for rule in self._mappings: - if rule.matches(company, document_type): + if rule.matches(company, document_type, optional_fields): # Apply set fields, overriding previous values if rule.set.template is not None: result["template"] = rule.set.template diff --git a/automatic-pdf-renamer/app/ml.py b/automatic-pdf-renamer/app/ml.py new file mode 100644 index 0000000..ad6f34e --- /dev/null +++ b/automatic-pdf-renamer/app/ml.py @@ -0,0 +1,363 @@ +"""Offline ML inference for document field extraction and routing decisions. + +Model artifacts are trusted, locally trained joblib objects. No rule-based or +configuration-based classifier is used when an artifact is unavailable. +Each single-label artifact exposes predict([sample]) and, optionally, +predict_proba([sample]). The group selector returns an iterable of group labels +for a (company, document_type) sample. +""" + +from __future__ import annotations + +import logging +import os +import re +from pathlib import Path +from typing import Any + +logger = logging.getLogger(__name__) + + +class ModelUnavailableError(RuntimeError): + """Raised when a required local ML model is missing or invalid.""" + + +class AmbiguousPredictionError(RuntimeError): + """Raised when an ML prediction does not meet the configured confidence.""" + + +class LocalMLPipeline: + """Load local model artifacts and provide confidence-gated predictions.""" + + def __init__(self, model_dir: str | None = None, confidence_threshold: float | None = None): + self.model_dir = Path(model_dir or os.environ.get("MODEL_DIR", "./models")) + self.confidence_threshold = ( + float(confidence_threshold) + if confidence_threshold is not None + else float(os.environ.get("CONFIDENCE_THRESHOLD", "0.8")) + ) + self._models: dict[str, Any] = {} + self._model_versions: dict[str, str] = {} + + def _load(self, name: str) -> Any: + active_path = self.model_dir / "active_models.json" + try: + active = __import__("json").loads(active_path.read_text(encoding="utf-8")) + except (OSError, ValueError): + active = {} + version = active.get(name) + if not version: + raise ModelUnavailableError( + f"No active model artifact for {name}; retrain all models from Maintenance." + ) + if name in self._models and self._model_versions.get(name) == version: + return self._models[name] + path = self.model_dir / "sets" / version / f"{name}.joblib" + if not path.is_file(): + # Keep the pre-retraining model layout usable until the first full replacement. + path = self.model_dir / "versions" / name / f"{version}.joblib" + if not path.is_file(): + raise ModelUnavailableError( + f"Active model artifact is missing for {name} version {version}." + ) + try: + import joblib + model = joblib.load(path) + except Exception as exc: + raise ModelUnavailableError(f"Could not load ML model {path.name}: {exc}") from exc + self._models[name] = model + self._model_versions[name] = version + return model + + def _predict_label(self, name: str, sample: Any) -> tuple[str, float]: + model = self._load(name) + try: + predictions = model.predict([sample]) + logger.debug("ML model %s predicted %s for sample=%r", name, predictions, sample) + if len(predictions) != 1: + raise ValueError("Expected exactly one prediction") + label = predictions[0] + if not isinstance(label, str) or not label.strip(): + raise ValueError("The model returned an empty or non-string label") + confidence = 0.0 + if hasattr(model, "predict_proba"): + probabilities = model.predict_proba([sample]) + confidence = float(max(probabilities[0])) + elif hasattr(model, "decision_function"): + # Decision scores are not calibrated probabilities; do not + # misrepresent them as confidence. + raise ModelUnavailableError( + f"Model {name} must expose calibrated predict_proba() scores" + ) + else: + raise ModelUnavailableError( + f"Model {name} must expose predict_proba() for confidence gating" + ) + if confidence < self.confidence_threshold: + raise AmbiguousPredictionError( + f"Model {name} confidence {confidence:.3f} is below " + f"threshold {self.confidence_threshold:.3f}" + ) + return label.strip(), confidence + except (ModelUnavailableError, AmbiguousPredictionError) as exc: + logger.debug("ML model %s failed for sample=%r: %s", name, sample, exc) + raise + except Exception as exc: + raise ModelUnavailableError(f"ML inference failed for {name}: {exc}") from exc + + def predict_labels( + self, + fields: dict[str, tuple[str, Any]], + ) -> tuple[dict[str, tuple[str, float]], dict[str, Exception]]: + """Predict independent fields without letting one failure block the others.""" + predictions: dict[str, tuple[str, float]] = {} + errors: dict[str, Exception] = {} + for field, (model_name, sample) in fields.items(): + try: + predictions[field] = self._predict_label(model_name, sample) + except (ModelUnavailableError, AmbiguousPredictionError) as exc: + errors[field] = exc + return predictions, errors + + def extract_initial_fields(self, text: str) -> dict[str, tuple[str, float]]: + """Predict common fields directly from full document text.""" + fields, errors = self.predict_labels({ + field: (f"field_{field}", text) + for field in ("emitting_company", "document_type") + }) + if errors: + raise next(iter(errors.values())) + logger.debug("Extracted initial fields %r from text=%r", fields, text) + return fields + + def predict_additional_groups(self, company: str, document_type: str) -> tuple[list[str], float]: + """Predict applicable additional-field groups from company and type.""" + model = self._load("additional_field_groups") + sample = f"{company}\n{document_type}" + try: + predictions = model.predict([sample]) + if len(predictions) != 1: + raise ValueError("Expected exactly one group prediction") + predicted = predictions[0] + if isinstance(predicted, str): + groups = [predicted] if predicted.strip() else [] + else: + groups = [str(value).strip() for value in predicted if str(value).strip()] + if not groups: + raise AmbiguousPredictionError("ML model predicted no additional-field groups") + if not hasattr(model, "predict_proba"): + raise ModelUnavailableError( + "Model additional_field_groups must expose calibrated predict_proba() scores" + ) + probabilities = model.predict_proba([sample]) + first = probabilities[0] + if hasattr(first, "toarray"): + first = first.toarray().ravel() + try: + confidence = float(max(first)) + except (TypeError, ValueError): + confidence = float(getattr(model, "confidence_threshold", 0.0)) + if confidence < self.confidence_threshold: + raise AmbiguousPredictionError( + f"Additional-field group confidence {confidence:.3f} is below " + f"threshold {self.confidence_threshold:.3f}" + ) + logger.debug('Predicted additional-field group %r (confidence=%.3f)', groups[0], confidence) + return groups, confidence + except (ModelUnavailableError, AmbiguousPredictionError): + raise + except Exception as exc: + raise ModelUnavailableError(f"ML inference failed for additional_field_groups: {exc}") from exc + + def predict_additional_values( + self, + field_names: list[str], + text: str, + *, + allow_partial: bool = False, + ) -> dict[str, tuple[str, float]]: + """Extract requested fields, optionally omitting individual low-confidence results.""" + model = self._load("additional_field_values") + if not hasattr(model, "predict_values"): + raise ModelUnavailableError( + "The active additional_field_values artifact does not support shared extraction; retrain it." + ) + try: + predictions = model.predict_values(text, field_names) + results: dict[str, tuple[str, float]] = {} + for field_name in field_names: + value, confidence = predictions[field_name] + if not value or confidence < self.confidence_threshold: + error = AmbiguousPredictionError( + f"Additional field '{field_name}' confidence {confidence:.3f} is below " + f"threshold {self.confidence_threshold:.3f}" + ) + if allow_partial: + logger.debug("Omitting low-confidence additional field %s: %s", field_name, error) + continue + raise error + results[field_name] = (str(value), float(confidence)) + logger.debug('Predicted additional field %s=%r (confidence=%.3f)', field_name, value, confidence) + return results + except (ModelUnavailableError, AmbiguousPredictionError): + raise + except ValueError as exc: + raise ModelUnavailableError(str(exc)) from exc + except Exception as exc: + raise ModelUnavailableError(f"ML inference failed for additional_field_values: {exc}") from exc + + def predict_template_and_period_format( + self, + emitting_company: str, + document_type: str, + additional_fields: dict[str, str], + ) -> tuple[str, float, str, float]: + """Predict template and period format from company, type, and extracted fields.""" + fields_text = "\n".join(f"{key}: {value}" for key, value in sorted(additional_fields.items())) + sample = f"{emitting_company}\n{document_type}\n{fields_text}" + predictions, errors = self.predict_labels({ + "template_name": ("template_name", sample), + "period_format": ("period_format", sample), + }) + if errors: + raise next(iter(errors.values())) + template, template_confidence = predictions["template_name"] + period_format, period_confidence = predictions["period_format"] + logger.debug( + "Predicted template=%s (confidence=%.3f), period_format=%s (confidence=%.3f) " + "for company=%r, document_type=%r, additional_fields=%r", + template, + template_confidence, + period_format, + period_confidence, + emitting_company, + document_type, + additional_fields, + ) + return template, template_confidence, period_format, period_confidence + + @staticmethod + def match_configured_additional_value( + text: str, + predicted_value: str, + configured_values: list[str], + ) -> str | None: + """Resolve an extracted span to a complete, unambiguous configured value.""" + matched_values = [] + for raw_value in configured_values: + value = str(raw_value).strip() + words = value.split() + if not words: + continue + pattern = r"(? tuple[dict[str, dict[str, Any]], dict[str, str]]: + """Collect independent model and configured-value suggestions for one document.""" + suggestions: dict[str, dict[str, Any]] = {} + decisions = { + "emitting_company": "skipped: model not evaluated", + "document_type": "skipped: model not evaluated", + "template_name": "skipped: prerequisites not met", + "period_format": "skipped: prerequisites not met", + "additional_fields": "skipped: groups not evaluated", + } + initial, initial_errors = self.predict_labels({ + "emitting_company": ("field_emitting_company", text), + "document_type": ("field_document_type", text), + }) + for field, (value, confidence) in initial.items(): + suggestions[field] = {"value": value, "confidence": confidence, "source": "model"} + decisions[field] = f"suggested: {value!r} ({confidence:.3f})" + for field, error in initial_errors.items(): + decisions[field] = f"unavailable: {error}" if isinstance(error, ModelUnavailableError) else f"ambiguous: {error}" + + suggested_company = initial.get("emitting_company", (company, 0.0))[0] + suggested_type = initial.get("document_type", (document_type, 0.0))[0] + if not suggested_company or not suggested_type: + missing = [ + name for name, value in (("emitting_company", suggested_company), ("document_type", suggested_type)) + if not value + ] + reason = f"skipped: missing {', '.join(missing)}" + decisions["template_name"] = reason + decisions["period_format"] = reason + return suggestions, decisions + + try: + groups, _ = self.predict_additional_groups(suggested_company, suggested_type) + value_model = self._load("additional_field_values") + trained_fields = getattr(value_model, "trained_fields", None) + if trained_fields is None: + extractable_groups = groups + unsupported_groups: list[str] = [] + else: + supported_fields = set(trained_fields) + extractable_groups = [name for name in groups if name in supported_fields] + unsupported_groups = sorted(set(groups) - supported_fields) + decisions["additional_fields"] = ( + f"predicted={groups}; extracting={extractable_groups}; unsupported={unsupported_groups}" + ) + extracted = self.predict_additional_values( + extractable_groups, text, allow_partial=True + ) if extractable_groups else {} + matched = { + name: value + for name, (span, _confidence) in extracted.items() + if (value := self.match_configured_additional_value( + text, span, configured_values.get(name, []) + )) is not None + } + if matched: + suggestions["optional_fields"] = {"value": matched, "source": "configured"} + model_spans = { + name: {"value": value, "confidence": confidence} + for name, (value, confidence) in extracted.items() + } + decisions["additional_fields"] += ( + f"; model_spans={model_spans}; configured_matches={matched}; " + f"omitted_low_confidence={sorted(set(extractable_groups) - set(extracted))}; " + f"omitted_without_configured_match={sorted(set(extracted) - set(matched))}" + ) + inference_fields = {**matched, **current_fields} + fields_text = "\n".join( + f"{key}: {value}" for key, value in sorted(inference_fields.items()) + ) + sample = f"{suggested_company}\n{suggested_type}\n{fields_text}" + final, final_errors = self.predict_labels({ + "template_name": ("template_name", sample), + "period_format": ("period_format", sample), + }) + for field, (value, confidence) in final.items(): + suggestions[field] = {"value": value, "confidence": confidence, "source": "model"} + decisions[field] = f"suggested: {value!r} ({confidence:.3f})" + for field, error in final_errors.items(): + decisions[field] = f"unavailable: {error}" if isinstance(error, ModelUnavailableError) else f"ambiguous: {error}" + except (ModelUnavailableError, AmbiguousPredictionError) as exc: + reason = f"skipped: additional-field inference failed ({exc})" + decisions["additional_fields"] = f"unavailable: {exc}" + decisions["template_name"] = reason + decisions["period_format"] = reason + return suggestions, decisions diff --git a/automatic-pdf-renamer/app/model_manager.py b/automatic-pdf-renamer/app/model_manager.py new file mode 100644 index 0000000..e381cf0 --- /dev/null +++ b/automatic-pdf-renamer/app/model_manager.py @@ -0,0 +1,378 @@ +"""Train and replace the complete local document model set.""" + +from __future__ import annotations + +import json +import os +import re +from datetime import datetime, timezone +import shutil +import uuid +from pathlib import Path +from typing import Any + +from joblib import dump +from sklearn.feature_extraction import DictVectorizer +from sklearn.feature_extraction.text import TfidfVectorizer +from sklearn.linear_model import LogisticRegression +from sklearn.multiclass import OneVsRestClassifier +from sklearn.pipeline import Pipeline +from sklearn.preprocessing import MultiLabelBinarizer + + +SINGLE_LABEL_MODELS = { + "field_emitting_company": ("extracted_text", "emitting_company"), + "field_document_type": ("extracted_text", "document_type"), + "template_name": ("company_type_fields", "template_name"), + "period_format": ("company_type_fields", "period_format"), +} +ADDITIONAL_VALUES_MODEL = "additional_field_values" +BASE_MODELS = ( + "field_emitting_company", + "field_document_type", + "additional_field_groups", + ADDITIONAL_VALUES_MODEL, + "template_name", + "period_format", +) + + +class AdditionalGroupsModel: + """Predict a multi-label set of additional-field groups.""" + + def __init__(self, vectorizer: TfidfVectorizer, classifier: OneVsRestClassifier, labels: list[str]): + self.vectorizer = vectorizer + self.classifier = classifier + self.labels = labels + + def predict(self, samples): + probabilities = self.predict_proba(samples) + return [[label for label, probability in zip(self.labels, row) if probability >= 0.5] + for row in probabilities] + + def predict_proba(self, samples): + return self.classifier.predict_proba(self.vectorizer.transform(samples)) + + +class AdditionalValuesModel: + """One shared token-level model that extracts values as spans from document text.""" + + def __init__(self, vectorizer: DictVectorizer, classifier: Any, trained_fields: list[str]): + self.vectorizer = vectorizer + self.classifier = classifier + self.trained_fields = trained_fields + + @staticmethod + def _tokens(text: str) -> list[tuple[str, int, int]]: + return [(match.group(), match.start(), match.end()) for match in re.finditer(r"\S+", text)] + + @staticmethod + def _features(tokens: list[tuple[str, int, int]], index: int) -> dict[str, Any]: + token = tokens[index][0] + normalized = token.casefold() + features: dict[str, Any] = { + "token": normalized, "prefix2": normalized[:2], "prefix4": normalized[:4], + "suffix2": normalized[-2:], "suffix4": normalized[-4:], + "shape_has_digit": any(char.isdigit() for char in token), + "shape_has_alpha": any(char.isalpha() for char in token), + "shape_upper": token.isupper(), "shape_title": token.istitle(), + "shape_length": min(len(token), 30), "position_bucket": min(index // 10, 20), + } + for offset in (-2, -1, 1, 2): + neighbor_index = index + offset + features[f"token_{offset}"] = ( + tokens[neighbor_index][0].casefold() + if 0 <= neighbor_index < len(tokens) else "" + ) + return features + + def predict_values(self, text: str, fields: list[str]) -> dict[str, tuple[str, float]]: + tokens = self._tokens(text) + if not tokens: + raise ValueError("The document contains no tokens for additional-field extraction.") + unknown = sorted(set(fields) - set(self.trained_fields)) + if unknown: + raise ValueError( + "The shared extraction model has no training spans for: " + ", ".join(unknown) + + ". Validate documents where these values appear in the extracted text, then retrain." + ) + features = self.vectorizer.transform([ + self._features(tokens, index) for index in range(len(tokens)) + ]) + probabilities = self.classifier.predict_proba(features) + classes = list(self.classifier.classes_) + predicted = self.classifier.predict(features) + output: dict[str, tuple[str, float]] = {} + for field_name in fields: + spans: list[tuple[int, int, float]] = [] + current_start: int | None = None + current_end: int | None = None + scores: list[float] = [] + for index, tag in enumerate(predicted): + token_tag = str(tag) + if token_tag == f"B::{field_name}": + if current_start is not None and current_end is not None: + spans.append((current_start, current_end, sum(scores) / len(scores))) + current_start, current_end = tokens[index][1], tokens[index][2] + scores = [float(probabilities[index][classes.index(token_tag)])] + elif token_tag == f"I::{field_name}" and current_start is not None: + current_end = tokens[index][2] + scores.append(float(probabilities[index][classes.index(token_tag)])) + elif current_start is not None and current_end is not None: + spans.append((current_start, current_end, sum(scores) / len(scores))) + current_start = current_end = None + scores = [] + if current_start is not None and current_end is not None: + spans.append((current_start, current_end, sum(scores) / len(scores))) + if not spans: + output[field_name] = ("", 0.0) + continue + span_start, span_end, confidence = max(spans, key=lambda span: span[2]) + output[field_name] = (text[span_start:span_end].strip(" :;,. \t\r\n"), confidence) + return output + + +def model_root() -> Path: + return Path(os.environ.get("MODEL_DIR", "./models")).resolve() + + +def _active_file() -> Path: + return model_root() / "active_models.json" + + +def _read_active() -> dict[str, str]: + path = _active_file() + if not path.is_file(): + return {} + try: + value = json.loads(path.read_text(encoding="utf-8")) + return value if isinstance(value, dict) else {} + except (OSError, json.JSONDecodeError): + return {} + + +def list_models() -> dict[str, Any]: + """Return the current model set without exposing model history.""" + root = model_root() + active = _read_active() + models = [] + for name in BASE_MODELS: + version = active.get(name) + artifact = root / "sets" / version / f"{name}.joblib" if version else None + if version and not (artifact and artifact.is_file()): + artifact = root / "versions" / name / f"{version}.joblib" + models.append({"name": name, "has_active_model": bool(artifact and artifact.is_file())}) + return {"models": models, "model_dir": str(root), "training_source": "validated documents", + "complete": all(model["has_active_model"] for model in models)} + + +def _new_classifier() -> Pipeline: + return Pipeline([ + ("vectorizer", TfidfVectorizer(ngram_range=(1, 2), sublinear_tf=True)), + ("classifier", LogisticRegression(max_iter=1000, class_weight="balanced")), + ]) + + +def train_model(model_name: str, records: list[Any], output_root: Path) -> dict[str, Any]: + """Train one component of the full model set into a staging directory.""" + if not re.fullmatch(r"[A-Za-z0-9_-]+", model_name): + raise ValueError("Model names may contain only letters, numbers, underscores, and hyphens.") + if model_name not in SINGLE_LABEL_MODELS and model_name not in { + "additional_field_groups", ADDITIONAL_VALUES_MODEL + }: + if model_name.startswith("additional_value_"): + raise ValueError( + "Per-field value models have been replaced by the shared " + "'additional_field_values' model." + ) + raise ValueError(f"Unsupported model type: {model_name}") + + usable = [record for record in records if (record.extracted_text or "").strip()] + if model_name == ADDITIONAL_VALUES_MODEL: + token_samples: list[dict[str, Any]] = [] + token_labels: list[str] = [] + field_counts: dict[str, int] = {} + for record in usable: + text = str(record.extracted_text or "") + tokens = AdditionalValuesModel._tokens(text) + if not tokens: + continue + labels_for_tokens = ["O"] * len(tokens) + occupied: set[int] = set() + for raw_name, raw_value in (record.optional_fields or {}).items(): + field_name = str(raw_name).strip() + value = str(raw_value).strip() if raw_value is not None else "" + if not field_name or not value: + continue + match = re.search(re.escape(value), text, flags=re.IGNORECASE) + if not match: + continue + covered = [ + index for index, (_, token_start, token_end) in enumerate(tokens) + if token_start < match.end() and token_end > match.start() + ] + if not covered or any(index in occupied for index in covered): + continue + labels_for_tokens[covered[0]] = f"B::{field_name}" + for index in covered[1:]: + labels_for_tokens[index] = f"I::{field_name}" + occupied.update(covered) + field_counts[field_name] = field_counts.get(field_name, 0) + 1 + token_samples.extend( + AdditionalValuesModel._features(tokens, index) for index in range(len(tokens)) + ) + token_labels.extend(labels_for_tokens) + + trained_fields = sorted(name for name, count in field_counts.items() if count >= 2) + if not trained_fields: + raise ValueError( + "At least one additional field must have two validated examples whose labeled values " + "appear verbatim in the extracted document text." + ) + valid_tags = {"O"} | { + tag for field_name in trained_fields for tag in (f"B::{field_name}", f"I::{field_name}") + } + filtered_samples = [sample for sample, label in zip(token_samples, token_labels) if label in valid_tags] + filtered_labels = [label for label in token_labels if label in valid_tags] + vectorizer = DictVectorizer(sparse=True) + features = vectorizer.fit_transform(filtered_samples) + classifier = LogisticRegression(max_iter=1000, class_weight="balanced") + classifier.fit(features, filtered_labels) + model = AdditionalValuesModel(vectorizer, classifier, trained_fields) + sample_count = len(usable) + elif model_name == "additional_field_groups": + samples, labels = [], [] + for record in usable: + if not record.emitting_company or not record.document_type: + continue + groups = sorted(str(key) for key in (record.optional_fields or {}).keys() if str(key).strip()) + samples.append(f"{record.emitting_company}\n{record.document_type}") + labels.append(groups) + unique_labels = sorted({label for row in labels for label in row}) + if len(samples) < 2 or not unique_labels: + raise ValueError("At least two validated documents and one labeled additional-field group are required.") + eligible_labels = [ + label for label in unique_labels + if 0 < sum(label in row for row in labels) < len(labels) + ] + if not eligible_labels: + raise ValueError("No additional-field group has both positive and negative validated examples.") + unique_labels = eligible_labels + labels = [[label for label in row if label in unique_labels] for row in labels] + mlb = MultiLabelBinarizer(classes=unique_labels) + target = mlb.fit_transform(labels) + vectorizer = TfidfVectorizer(ngram_range=(1, 2), sublinear_tf=True) + features = vectorizer.fit_transform(samples) + classifier = OneVsRestClassifier(LogisticRegression(max_iter=1000, class_weight="balanced")) + classifier.fit(features, target) + model = AdditionalGroupsModel(vectorizer, classifier, unique_labels) + sample_count = len(samples) + trained_fields = [] + else: + samples, labels = [], [] + source, target_name = SINGLE_LABEL_MODELS[model_name] + for record in usable: + if source in {"company_type", "company_type_fields"}: + if not record.emitting_company or not record.document_type: + continue + sample = f"{record.emitting_company}\\n{record.document_type}" + if source == "company_type_fields": + fields = record.optional_fields or {} + sample += "\\n" + "\\n".join( + f"{key}: {value}" for key, value in sorted(fields.items()) + ) + label = getattr(record, target_name, None) + else: + sample = record.extracted_text + label = getattr(record, target_name, None) + if label is None or not str(label).strip(): + continue + samples.append(str(sample)) + labels.append(str(label).strip()) + if len(samples) < 2 or len(set(labels)) < 2: + raise ValueError("At least two labeled validated documents and two distinct target values are required.") + model = _new_classifier() + model.fit(samples, labels) + sample_count = len(samples) + trained_fields = [] + + version = "current" + destination = output_root / "versions" / model_name + destination.mkdir(parents=True, exist_ok=True) + artifact = destination / f"{version}.joblib" + dump(model, artifact) + result = { + "name": model_name, + "version": version, + "samples": sample_count, + "status": "trained", + "active": False, + "message": "Model trained in staging; it is activated only when the full model set is replaced.", + } + if model_name == ADDITIONAL_VALUES_MODEL: + result["trained_fields"] = trained_fields + return result + + +def train_all_models(records: list[Any], progress_callback=None) -> dict[str, Any]: + """Train all models in staging and atomically switch the active-set pointer.""" + root = model_root() + root.mkdir(parents=True, exist_ok=True) + staging = root / f".staging-{uuid.uuid4().hex}" + model_names = list(BASE_MODELS) + generation = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S%fZ") + set_published = False + pointer_switched = False + try: + staging.mkdir() + results = [] + for index, model_name in enumerate(model_names, start=1): + if progress_callback: + progress_callback( + round((index - 1) * 90 / len(model_names)), + f"Training {model_name} ({index}/{len(model_names)})", + ) + results.append(train_model(model_name, records, output_root=staging)) + + staged_set = staging / "sets" / generation + staged_set.mkdir(parents=True) + for model_name, result in zip(model_names, results): + artifact = staging / "versions" / model_name / "current.joblib" + artifact.rename(staged_set / f"{model_name}.joblib") + result["version"] = generation + shutil.rmtree(staging / "versions", ignore_errors=True) + + active = {name: generation for name in model_names} + staged_active = staging / "active_models.json" + staged_active.write_text(json.dumps(active, indent=2), encoding="utf-8") + if progress_callback: + progress_callback(95, "Switching to the new model set") + + sets_root = root / "sets" + sets_root.mkdir(parents=True, exist_ok=True) + published_set = sets_root / generation + staged_set.rename(published_set) + set_published = True + + # A single atomic pointer replacement switches every model at once. + os.replace(staged_active, _active_file()) + pointer_switched = True + + # The new set is active and complete; remove all old artifacts and history. + for old_set in sets_root.iterdir(): + if old_set.is_dir() and old_set.name != generation: + shutil.rmtree(old_set, ignore_errors=True) + shutil.rmtree(root / "versions", ignore_errors=True) + if progress_callback: + progress_callback(100, "All models trained and replaced successfully") + return {"status": "completed", "models": results, "count": len(results)} + except Exception: + import logging + logging.getLogger(__name__).exception( + "Full model retraining failed; the previous active model set was retained" + ) + if set_published and not pointer_switched: + shutil.rmtree(root / "sets" / generation, ignore_errors=True) + raise + finally: + shutil.rmtree(staging, ignore_errors=True) diff --git a/automatic-pdf-renamer/app/pipeline.py b/automatic-pdf-renamer/app/pipeline.py index da2ae0d..fbe9fe3 100644 --- a/automatic-pdf-renamer/app/pipeline.py +++ b/automatic-pdf-renamer/app/pipeline.py @@ -1,16 +1,12 @@ """Processing pipeline module for Automatic PDF Renamer. -Implements the document processing workflow: -1. Load the first page and extract text using PyMuPDF -2. Find the emission date using regex patterns from config (ordered list, first match wins) -3. Find company and document_type using ML, validated against predefined lists in config - - Unknown values must route to review -4. Look up the (company, document_type) tuple in mappings to get template, period_format, and additional_fields - - Mappings use match/set structure: match criteria (company, document_type, both, or neither) - - Rules are processed in order, later matches override earlier ones - - Empty match {} matches all documents (default rule) -5. Apply period_format (with granularity, format template, and offset) to calculate the human-readable period string -6. Apply template key_pattern to generate the final filename +Implements the ordered ML and deterministic processing workflow: +1. Predict emitting_company and document_type from extracted PDF text +2. Extract emission_date using configured date_extraction regexes only +3. Predict applicable additional-field groups from emitting_company and document_type +4. Extract additional-field values from the full text +5. Predict template_name and period_format from emitting_company, document_type, and additional fields +6. Calculate period and current_key from the final predictions and configuration """ import logging @@ -29,6 +25,7 @@ from .period import ( format_period_from_emission_date, ) from .sidecar import Sidecar, SidecarStatus, build_sidecar +from .ml import LocalMLPipeline, ModelUnavailableError, AmbiguousPredictionError logger = logging.getLogger(__name__) @@ -66,6 +63,7 @@ class DocumentProcessor: self, registry: TemplateRegistry, min_text_length: int = 10, + ml_pipeline: LocalMLPipeline | None = None, ): """Initialize the document processor. @@ -75,6 +73,7 @@ class DocumentProcessor: """ self.registry = registry self.min_text_length = min_text_length + self.ml_pipeline = ml_pipeline or LocalMLPipeline() def process( self, @@ -99,109 +98,116 @@ class DocumentProcessor: result = ProcessingResult(success=False) try: - # Step 1: Extract document text + # Preparation: Extract document text if extracted_text is None: extracted_text = self._extract_text(pdf_bytes) if extracted_text is None: result.errors.append("Failed to extract text from PDF") - result.status = SidecarStatus.NOT_PROCESSED.value + result.status = SidecarStatus.PENDING.value return result - # Step 2: Find emission date using regex patterns from config + # Step 1: Predict emitting_company and document_type from the full text. + try: + initial_fields = self.ml_pipeline.extract_initial_fields(extracted_text) + emitting_company = initial_fields["emitting_company"][0] + doc_type = initial_fields["document_type"][0] + initial_confidence = min(value[1] for value in initial_fields.values()) + except (ModelUnavailableError, AmbiguousPredictionError) as exc: + result.errors.append(str(exc)) + result.status = SidecarStatus.PENDING.value + return result + + # Step 2: Extract the emission date using configured regexes only. emission_date, date_error = self._extract_emission_date(extracted_text) if date_error: result.errors.append(date_error) - result.status = SidecarStatus.NOT_PROCESSED.value + result.status = SidecarStatus.PENDING.value return result - result.emission_date = emission_date - # Step 3: Find company and document_type using ML - company, doc_type, ml_errors = self._extract_company_and_type(extracted_text) - - if ml_errors: - result.errors.extend(ml_errors) - result.status = SidecarStatus.NOT_PROCESSED.value - return result - - result.company = company + result.company = emitting_company result.document_type = doc_type - - # Validate company and document_type against config lists - # Strict validation: unknown values route to review - if not self.registry.is_valid_company(company): + company = emitting_company # Compatibility alias for existing key-pattern placeholders. + if not self.registry.is_valid_company(emitting_company): result.errors.append( - f"Unknown company '{company}'. " - f"Valid companies: {', '.join(self.registry.get_companies()[:10])}..." + f"Unknown emitting company '{emitting_company}'. " + f"Valid companies: {', '.join(self.registry.get_companies()[:10])}" ) - result.status = SidecarStatus.NOT_PROCESSED.value + result.status = SidecarStatus.PENDING.value return result - if not self.registry.is_valid_document_type(doc_type): result.errors.append( f"Unknown document type '{doc_type}'. " - f"Valid types: {', '.join(self.registry.get_document_types()[:10])}..." + f"Valid types: {', '.join(self.registry.get_document_types()[:10])}" ) - result.status = SidecarStatus.NOT_PROCESSED.value + result.status = SidecarStatus.PENDING.value return result - # Step 4: Resolve mapping using match/set rules - # Rules are processed in order, later matches override earlier ones - resolved = self.registry.resolve_mapping(company, doc_type) - - if not resolved: - result.errors.append( - f"No mapping found for company='{company}', " - f"document_type='{doc_type}'. Routing to review." - ) - result.status = SidecarStatus.NOT_PROCESSED.value + # Step 3: Determine the additional-field groups from emitting_company + # and document_type; the period format is not predicted at this stage. + # Template and period format are deliberately not consumed until this + # first-stage field extraction has completed. + try: + additional_field_groups, groups_confidence = self.ml_pipeline.predict_additional_groups(emitting_company, doc_type) + except (ModelUnavailableError, AmbiguousPredictionError) as exc: + result.errors.append(str(exc)) + result.status = SidecarStatus.PENDING.value return result - - # Extract resolved fields - template_name = resolved.get("template") - period_format = resolved.get("period_format") - additional_field_groups = resolved.get("additional_fields", []) - + + # Step 4: Extract each selected additional-field value from the full text. + optional_fields: dict[str, str] = {} + field_confidences = [initial_confidence, groups_confidence] + if additional_field_groups: + try: + extracted_additional_fields = self.ml_pipeline.predict_additional_values( + additional_field_groups, extracted_text + ) + except (ModelUnavailableError, AmbiguousPredictionError) as exc: + result.errors.append(str(exc)) + result.status = SidecarStatus.PENDING.value + return result + for field_name, (field_value, field_confidence) in extracted_additional_fields.items(): + optional_fields[field_name] = field_value + field_confidences.append(field_confidence) + + # Step 5: Predict template and final period format using all extracted fields. + # Configuration validates that predicted identifiers can be executed downstream. + try: + template_name, template_confidence, period_format, period_confidence = ( + self.ml_pipeline.predict_template_and_period_format(emitting_company, doc_type, optional_fields) + ) + except (ModelUnavailableError, AmbiguousPredictionError) as exc: + result.errors.append(str(exc)) + result.status = SidecarStatus.PENDING.value + return result + field_confidences.extend([template_confidence, period_confidence]) if not template_name: result.errors.append( f"No template resolved for company='{company}', " f"document_type='{doc_type}'. Routing to review." ) - result.status = SidecarStatus.NOT_PROCESSED.value + result.status = SidecarStatus.PENDING.value return result - - result.template_name = template_name - result.period_format = period_format - - # Step 4b: Extract additional fields from additional_fields lists - optional_fields = {} - for group_name in additional_field_groups: - field_value, field_error = self._extract_field_value( - extracted_text, group_name - ) - if field_error: - result.errors.append(field_error) - result.status = SidecarStatus.NOT_PROCESSED.value - return result - if field_value: - optional_fields[group_name] = field_value - - # Step 5: Calculate period using period_format configuration if not period_format: result.errors.append( f"No period_format resolved for company='{company}', " f"document_type='{doc_type}'. Routing to review." ) - result.status = SidecarStatus.NOT_PROCESSED.value + result.status = SidecarStatus.PENDING.value return result + + result.template_name = template_name + result.period_format = period_format + + # Step 6: Calculate period once using the final period-format configuration. period_format_config = self.registry.get_period_format_config(period_format) if period_format_config is None: result.errors.append( f"Period format '{period_format}' not found in configuration" ) - result.status = SidecarStatus.NOT_PROCESSED.value + result.status = SidecarStatus.PENDING.value return result try: @@ -212,10 +218,10 @@ class DocumentProcessor: result.period = period except Exception as e: result.errors.append(f"Failed to compute period: {e}") - result.status = SidecarStatus.NOT_PROCESSED.value + result.status = SidecarStatus.PENDING.value return result - # Step 6: Generate the final key using template's key_pattern + # Step 6: Generate current_key from the template and final field values. key_pattern = self.registry.get_key_pattern(template_name) if key_pattern: try: @@ -233,14 +239,14 @@ class DocumentProcessor: ) except Exception as e: result.errors.append(f"Failed to format key pattern: {e}") - result.status = SidecarStatus.NOT_PROCESSED.value + result.status = SidecarStatus.PENDING.value return result else: # Fallback: use a simple pattern final_key = f"{template_name}/{period}/{company}/{doc_type}/{emission_date}_{original_key.split('/')[-1]}" # Step 7: Build the sidecar - confidence = 1.0 # Will be set by ML classifier + confidence = min(field_confidences) sidecar = build_sidecar( sha256=sha256, original_key=original_key, @@ -284,7 +290,7 @@ class DocumentProcessor: return result def _extract_text(self, pdf_bytes: bytes) -> str | None: - """Extract first page text from PDF. + """Extract text from every page of the PDF. Args: pdf_bytes: The PDF content as bytes. @@ -302,39 +308,55 @@ class DocumentProcessor: return None def _extract_emission_date(self, text: str) -> tuple[str, str | None]: - """Extract emission date from text using regex patterns from config. - - Uses ordered list of regex patterns. Tries them in order and stops - at first match. - - Args: - text: The extracted text from the PDF. - - Returns: - Tuple of (date_string, error_message). - If successful, error_message is None. - """ + """Return the earliest valid date matched by configured regexes only.""" patterns = self.registry.get_date_extraction_compiled() - - if not patterns: - # Fallback: try to find any date-like pattern - return self._extract_emission_date_fallback(text) - - # Try each pattern in order, first match wins - for idx, pattern in enumerate(patterns): - matches = pattern.finditer(text) - for match in matches: - # Get the first date-like match - groups = match.groups() - date_str = self._reconstruct_date(groups) - if date_str: - # Validate it looks like a date - if self._is_valid_date_string(date_str): - return (date_str, None) - - # If no patterns matched, try fallback - return self._extract_emission_date_fallback(text) + matches: list[tuple[int, int, str]] = [] + for pattern_index, pattern in enumerate(patterns): + for match in pattern.finditer(text): + # Try captured groups first, but do not let an unrelated/context + # capture prevent a valid full-match date from being considered. + candidates = [self._reconstruct_date(match.groups()), match.group(0)] + normalized = next( + ( + parsed + for candidate in candidates + if candidate + for parsed in [self._normalize_emission_date(candidate)] + if parsed + ), + None, + ) + if normalized: + matches.append((match.start(), pattern_index, normalized)) + if not matches: + return "", "Could not find a valid emission date matching configured date_extraction patterns" + matches.sort(key=lambda item: (item[0], item[1])) + return matches[0][2], None + def _normalize_emission_date(self, value: str) -> str | None: + """Normalize a regex-captured date to YYYY-MM-DD.""" + value = " ".join(value.strip().split()).casefold() + french_months = { + "janvier": "01", "fevrier": "02", "mars": "03", "avril": "04", + "mai": "05", "juin": "06", "juillet": "07", "aout": "08", + "septembre": "09", "octobre": "10", "novembre": "11", "decembre": "12", + } + for month_name, month_number in french_months.items(): + value = re.sub(rf"\b{month_name}\b", month_number, value) + formats = ( + "%Y-%m-%d", "%d/%m/%Y", "%d-%m-%Y", "%Y/%m/%d", + "%Y/%m", "%Y-%m", "%m/%Y", "%m-%Y", + "%d %m %Y", "%Y %m %d", "%m %Y", "%Y%m%d", "%Y", + ) + for date_format in formats: + try: + parsed = datetime.strptime(value, date_format) + if date_format == "%Y" or date_format in {"%Y/%m", "%Y-%m", "%m/%Y", "%m-%Y"}: + parsed = parsed.replace(month=1 if date_format == "%Y" else parsed.month, day=1) + return parsed.strftime("%Y-%m-%d") + except ValueError: + continue + return None def _reconstruct_date(self, groups: tuple[str, ...]) -> str | None: """Reconstruct a date string from regex match groups. @@ -350,214 +372,18 @@ class DocumentProcessor: return " ".join(non_empty) if len(non_empty) > 1 else non_empty[0] return None - def _is_valid_date_string(self, date_str: str) -> bool: - """Check if a string looks like a valid date. - - Args: - date_str: The string to check. - - Returns: - True if it looks like a valid date. - """ - # Simple heuristic: contains digits and possibly separators - if not date_str: - return False - - # Check for common date patterns - date_patterns = [ - r"\d{4}-\d{2}-\d{2}", # YYYY-MM-DD - r"\d{2}/\d{2}/\d{4}", # DD/MM/YYYY - r"\d{2}-\d{2}-\d{4}", # DD-MM-YYYY - r"\d{4}", # YYYY - r"\d{4}/\d{2}", # YYYY/MM - r"\d{2}\s+[a-zA-Z]+\s+\d{4}", # DD Month YYYY - ] - - for p in date_patterns: - if re.match(p, date_str.strip()): - return True - - return False - - def _extract_emission_date_fallback(self, text: str) -> tuple[str, str | None]: - """Fallback method for extracting emission date. - - Args: - text: The extracted text. - - Returns: - Tuple of (date_string, error_message). - """ - # Try common date patterns - fallback_patterns = [ - r"(\d{4}-\d{2}-\d{2})", # YYYY-MM-DD - r"(\d{2}/\d{2}/\d{4})", # DD/MM/YYYY - r"(\d{2}-\d{2}-\d{4})", # DD-MM-YYYY - r"(\d{4})", # YYYY - ] - - for pattern in fallback_patterns: - matches = re.finditer(pattern, text) - for match in matches: - date_str = match.group(1) - if self._is_valid_date_string(date_str): - return (date_str, None) - - return ("", "Could not find emission date in text") - def _extract_company_and_type(self, text: str) -> tuple[str, str, list[str]]: - """Extract company and document type from text using ML. - - This is a placeholder for the actual ML implementation. - For now, it tries to match company and document type names from config. - - Args: - text: The extracted text. - - Returns: - Tuple of (company, document_type, errors). - """ - errors = [] - - # Try to find company names from the config - companies = self.registry.get_companies() - for company in companies: - if company.lower() in text.lower(): - # Try to find document type - doc_types = self.registry.get_document_types() - for doc_type in doc_types: - if doc_type.lower() in text.lower(): - return (company, doc_type, errors) - - # If no document type found, use first one as default - if doc_types: - return (company, doc_types[0], [f"Could not determine document type, using default: {doc_types[0]}"]) - - # If no company found, return unknown - return ("", "", ["Could not determine company and document type"]) + """Predict emitting company and document type with local ML models.""" + try: + fields = self.ml_pipeline.extract_initial_fields(text) + return fields["emitting_company"][0], fields["document_type"][0], [] + except (ModelUnavailableError, AmbiguousPredictionError) as exc: + return "", "", [str(exc)] def _extract_field_value(self, text: str, group_name: str) -> tuple[str, str | None]: - """Extract a field value from text using ML from predefined list. - - For each additional field group in the mapping, the value should be found - using ML from the predefined list of possible values for that group. - - Strict validation: if value not found in predefined list, route to review. - - Args: - text: The extracted text. - group_name: The name of the additional field group. - - Returns: - Tuple of (field_value, error_message). - If successful, error_message is None. - If no value can be determined, error_message describes the issue. - """ - # Get the predefined list of values for this additional field group - possible_values = self.registry.get_additional_field_values(group_name) - - if not possible_values: - return ("", f"No predefined values for additional field group '{group_name}'") - - # Try to find a value from the predefined list in the text - # This is a placeholder for ML-based extraction - # For now, we'll do simple string matching - text_lower = text.lower() - for value in possible_values: - if value.lower() in text_lower: - return (value, None) - - # Strict validation: if no value found in predefined list, route to review - return ("", f"Could not determine value for additional field '{group_name}' from predefined list") - - def _format_key_pattern( - self, - pattern: str, - **kwargs: Any, - ) -> str: - """Format a key pattern with the provided values. - - Args: - pattern: The key pattern with placeholders. - **kwargs: Values to substitute into the pattern. - - Returns: - Formatted key string. - """ + """Predict an additional-field value from text using a local ML model.""" try: - return pattern.format(**kwargs) - except KeyError as e: - # If a placeholder is missing, use a default value - missing = str(e).strip("'") - kwargs[missing] = "unknown" - return pattern.format(**kwargs) - - -class DateExtractor: - """Specialized class for extracting dates from text using config patterns.""" - - def __init__(self, registry: TemplateRegistry): - """Initialize the date extractor. - - Args: - registry: TemplateRegistry with date extraction patterns. - """ - self.registry = registry - self._patterns = registry.get_date_extraction_compiled() - - def extract_first_date(self, text: str) -> str | None: - """Extract the first date found in text using configured patterns. - - Args: - text: The text to search. - - Returns: - The first date string found, or None. - """ - if not self._patterns: - return None - - for pattern in self._patterns: - match = pattern.search(text) - if match: - groups = match.groups() - date_str = " ".join([g for g in groups if g]) - if date_str.strip(): - return date_str.strip() - - return None - - def extract_all_dates(self, text: str) -> list[str]: - """Extract all dates found in text using configured patterns. - - Args: - text: The text to search. - - Returns: - List of date strings found. - """ - dates = [] - if not self._patterns: - return dates - - for pattern in self._patterns: - matches = pattern.finditer(text) - for match in matches: - groups = match.groups() - date_str = " ".join([g for g in groups if g]) - if date_str.strip() and date_str.strip() not in dates: - dates.append(date_str.strip()) - - return dates - - -def create_processor(registry: TemplateRegistry) -> DocumentProcessor: - """Factory function to create a DocumentProcessor. - - Args: - registry: TemplateRegistry with configuration. - - Returns: - DocumentProcessor instance. - """ - return DocumentProcessor(registry) + value, _confidence = self.ml_pipeline.predict_additional_value(group_name, text) + return value, None + except (ModelUnavailableError, AmbiguousPredictionError) as exc: + return "", str(exc) diff --git a/automatic-pdf-renamer/app/sidecar.py b/automatic-pdf-renamer/app/sidecar.py index 3d96305..229abde 100644 --- a/automatic-pdf-renamer/app/sidecar.py +++ b/automatic-pdf-renamer/app/sidecar.py @@ -20,6 +20,7 @@ class SidecarStatus(Enum): """File processing status (PRD section 4).""" NOT_PROCESSED = "not processed" + PENDING = "pending" PROCESSED = "processed" VALIDATED = "validated" diff --git a/automatic-pdf-renamer/arch.md b/automatic-pdf-renamer/arch.md index baeb9dc..7c5f1e6 100644 --- a/automatic-pdf-renamer/arch.md +++ b/automatic-pdf-renamer/arch.md @@ -19,7 +19,7 @@ flowchart TD poll["Inbox poller (asyncio task)"] s3["Garage S3 bucket"] db[("SQLite replica (WAL)")] - worker["Subprocess jobs: retrain / consistency / rebuild"] + worker["Background jobs: retrain / consistency / rebuild"] model[("Model registry on disk")] user --> dash @@ -52,13 +52,13 @@ flowchart TD | Text extraction | PyMuPDF | Fastest option; AGPL acceptable for a personal offline tool. Wrapped behind a small extractor interface so it remains swappable (PRD open question resolved). | | S3 client | boto3 | Most complete API coverage (copy-object, versioning, ETag), mature Garage compatibility. Wrapped behind a thin storage interface; v1 ships one implementation. | | SQLite access | SQLAlchemy ORM, WAL mode | Declared replica schema, typed models, migrations via Alembic. Sidecar remains authoritative; ORM layer only reads/writes the mirror. | -| Initial classification | Configured deterministic rules and reference data | Initial release must work without ML; TF-IDF + Logistic Regression is a post-v1 enhancement. | +| ML inference | Local model artifacts for field extraction, additional-field group selection, template prediction, and period-format prediction | No predefined mappings decide classification outcomes; missing or low-confidence predictions route to `pending`. | | Fuzzy matching | rapidfuzz | PRD FR4.2, counterparty gazetteer. | | Dates | python-dateutil | PRD FR4.2, emission date parsing and period computation. | | Frontend | React + Vite, react-pdf viewer | NFR7; built to static assets served by FastAPI. | | Process model | API, poller, and background jobs in one container | Single-user tool; a process manager is unnecessary complexity. | | Inbox poller | asyncio background task inside the FastAPI process | Lightweight, no queue infrastructure; restart scan (FR1.3) covers crash recovery. | -| Long jobs (retrain, consistency, rebuild) | Subprocesses launched and monitored by the API | Keeps the event loop responsive; status exposed via the existing endpoints (FR9, FR10). | +| Long jobs (retrain, consistency, rebuild) | Background worker or subprocess, depending on operation | Keeps the event loop responsive; status and progress are exposed through job endpoints (FR9, FR10). | | Dashboard auth | Username + password (single account, env-var credentials, session cookie) | Local network but not fully trusted; cheap to implement. | | Testing | Unit tests only, pytest | Single-user offline tool; the storage layer is behind interfaces so mocks are cheap. Integration/CI coverage deferred. | @@ -70,7 +70,7 @@ One Python process runs: - FastAPI app: serves the API, the static dashboard, and the PDF streaming proxy. - Poller task: asyncio loop, configurable interval, lists the `inbox/` prefix. -- Job supervisor: spawns subprocesses for retrain, consistency scan, and SQLite rebuild; tracks their status in memory and in SQLite. +- Job supervisor: runs full model retraining in a background worker and launches subprocesses for consistency scan and SQLite rebuild; tracks current job state for the API. On startup, the process recovers interrupted technical operations and discovers eligible objects in `inbox/`. The `pending/` prefix is reconciled for metadata and anomalies; documents awaiting manual correction are not automatically reclassified or renamed until corrected/requested. @@ -87,14 +87,23 @@ app/ period.py # period computation from emission date + period format config templates.py # JSONC template registry (slots, period rules, key patterns) rename.py # copy-then-delete move + event logging - jobs/ # poller, consistency scan, retrain subprocess, rebuild - api/ # routes: files, templates, retrain, consistency, auth + jobs/ # poller, consistency scan, rebuild + api/ # routes: files, templates, model retraining, consistency, auth web/ # React + Vite app (built to static assets) -models/ # versioned model registry (initial + trained) +models/ # current model set; prior artifacts removed after successful replacement ``` ### 4.3 Processing pipeline +The ingestion pipeline follows this strict analysis order. Model predictions that are unavailable or below the configured confidence threshold are treated as unresolved; the document is routed to `pending/` rather than silently substituting a deterministic mapping. + +1. **Company and document type:** ML models predict `emitting_company` and `document_type` from the complete extracted PDF text. +2. **Emission date:** no ML model is used. Scan the extracted text for dates matching the regular expressions listed in the `date_extraction` section of `config.jsonc`. Select the earliest matching date by its position in the document; use configuration order to resolve matches starting at the same position. Normalize the captured date to `YYYY-MM-DD`. If no valid match exists, the date is unresolved. +3. **Additional-field list:** an ML model predicts the applicable list/group of additional fields using `emitting_company` and `document_type`. The period format is not predicted at this stage. +4. **Additional-field values:** the shared token-level ML extractor predicts each configured additional-field value from the complete extracted PDF text, conditioned on the field name/group as represented by the model. +5. **Template and final period format:** ML models predict `template_name` and `period_format` using `emitting_company`, `document_type`, and the extracted `additional_fields`. The period format is predicted only at this stage. +6. **Derived values:** calculate `period` once from the emission date and final `period_format` definition; calculate `current_key` from the selected template's key pattern and final field values. Neither `period` nor `current_key` is an ML prediction or an independently editable input. + One ingestion run, fully synchronous within the poller task: ```mermaid @@ -103,13 +112,15 @@ flowchart TD dl --> tx["Extract text from every PDF page (PyMuPDF)"] tx --> empty{"Text empty?"} empty -- yes --> pending["Move to pending/ (reason: empty text)"] - empty -- no --> cls["Classify using deterministic rules"] - cls --> unamb{"Exactly one unambiguous rule match?"} - unamb -- no --> pending - unamb -- yes --> fill["Fill fields: regexes, companies list, additional fields from groups, period from emission date"] - fill --> valid{"Fields valid?"} + empty -- no --> company["1. ML: emitting_company + document_type from text"] + company --> date["2. Regex: first matching emission date from config date_extraction"] + date --> groups["3. ML: additional-field list from emitting_company/document_type"] + groups --> values["4. ML: additional-field values from full document text"] + values --> final["5. ML: template + period_format from emitting_company/document_type/fields"] + final --> derived["6. Calculate period + current_key from config"] + derived --> valid{"Required predictions and derived values valid?"} valid -- no --> pending - valid -- yes --> rename["Compute key and copy object to target"] + valid -- yes --> rename["Copy object to target key"] rename --> sidecar["Persist sidecar with target key and event"] sidecar --> delete["Delete source only after sidecar is durable"] delete --> sqlite["Synchronize SQLite replica"] @@ -344,10 +355,10 @@ sequenceDiagram | Inbox poll | Interval (configurable, default from PRD) | asyncio task, in-process | | Stability check | Two consecutive listings, same size + ETag | Poller state kept in memory | | Consistency scan | Periodic (configurable) + on demand | Subprocess | -| Retrain | Dashboard action | Subprocess; writes a new versioned model dir; activation is an atomic pointer swap, no restart (AC6) | +| Retrain all models | Single Maintenance dashboard action | One background worker trains every required model in dependency order; the API exposes job state and percentage progress. On success, an atomic active-set pointer switch replaces the previous set and old artifacts are deleted. | | SQLite rebuild | Dashboard action or CLI | Subprocess | -Subprocesses report progress and metrics by writing a status JSON consumed by the API's status endpoints. Only one job of each kind runs at a time; the API refuses a duplicate start. +Subprocess-based jobs report progress and metrics by writing a status JSON consumed by the API's status endpoints. Model retraining reports in-memory job state and progress through the API. Only one retraining job runs at a time; the API refuses a duplicate start. Model retraining is one atomic maintenance operation: there are no per-model train/activate controls and no model-version history. The existing model set must remain usable if training fails before the replacement set is complete; once the replacement succeeds, remove the previous artifacts. ## 7. Configuration and Secrets @@ -360,7 +371,7 @@ Subprocesses report progress and metrics by writing a status JSON consumed by th - Ingestion: classify errors as transient/retryable technical failures, permanent technical failures, or document/business validation failures. Retry transient failures up to 3 times after the initial attempt, with delays of 5 seconds, 30 seconds, and 2 minutes, using idempotent recovery. After the final retry fails, stop processing that item without another retry or review workflow. Do not set business status `pending` solely because a technical operation failed. Record technical state and error details separately from business status, and continue processing other objects. - Rename: copy to the target key, persist the sidecar with the new current key and event, then delete the old key; synchronize SQLite after the authoritative sidecar is durable. If a crash occurs before deletion, recovery follows the sidecar's current key and safely removes or reconciles the stale copy. - Sidecar/SQLite: sidecar first; drift repaired by the consistency job; rebuild is always possible from sidecars alone. -- Retrain: trained models land in a new versioned directory; activation swaps an active-model pointer; a failed training run leaves the current model untouched. +- Retrain: train the complete model set in staging, then replace all active artifacts as one maintenance operation. A failed training run leaves the current model set usable; a successful run deletes all previous artifacts and retains no history. - All object-not-found conditions during processing are treated as consistency anomalies and flagged (FR5.3). ## 9. Deployment @@ -384,8 +395,13 @@ Subprocesses report progress and metrics by writing a status JSON consumed by th | Garage quirks in copy-object/versioning | Storage interface isolates the client. | | Corpus too small for reliable classification | Low-confidence routing to pending; retrain on validated files grows the corpus. | | SQLite drift after crash | Sidecar-first ordering + consistency job + full rebuild path. | -| Single-process blast radius | Long jobs are subprocesses; the poller catches per-file exceptions. | +| Single-process blast radius | Model retraining runs in a background worker; consistency/rebuild use subprocesses; the poller catches per-file exceptions. | ## 12. Traceability to PRD -Every epic maps to a component: ingestion (jobs/poller), extraction and classification (pipeline), data model and validation (templates, period, gazetteer, pipeline), renaming and logging (rename, sidecar), dashboards (web + api), JSONC configuration editing (config), retraining (jobs/retrain + models), consistency and recovery (jobs/consistency, replica/rebuild). Acceptance criteria AC1-AC12 are covered by the components above; AC8-AC10 by section 8, AC9 by section 4.5. +Every epic maps to a component: ingestion (jobs/poller), extraction and classification (pipeline), data model and validation (templates, period, gazetteer, pipeline), renaming and logging (rename, sidecar), dashboards (web + api), JSONC configuration editing (config), retraining (API background job + model_manager), consistency and recovery (jobs/consistency, replica/rebuild). Acceptance criteria AC1-AC12 are covered by the components above; AC8-AC10 by section 8, AC9 by section 4.5. + + +### Local ML inference + +The runtime loads trusted, locally stored model artifacts from `MODEL_DIR`. The analysis order is mandatory: (1) predict `emitting_company` and `document_type` from extracted text; (2) extract `emission_date` using the configured `date_extraction` regexes only, selecting the earliest valid match in document order (no date ML model); (3) predict only the applicable additional-field groups from `emitting_company` and `document_type`; (4) extract each additional-field value from the full text using one shared token-level model; (5) predict `template_name` and `period_format` from `emitting_company`, `document_type`, and additional fields; and (6) calculate `period` once from the emission date and final period format, and calculate `current_key` from the template key pattern and final values. Training labels come from validated sidecar values, and additional-field values are trained only when they appear in the saved extracted text. Predictions are confidence-gated; missing or uncertain required predictions route to `pending`. Templates and period-format definitions provide calculation/formatting rules, not hard-coded classification mappings. diff --git a/automatic-pdf-renamer/brief.md b/automatic-pdf-renamer/brief.md index 0ddffa4..ba63339 100644 --- a/automatic-pdf-renamer/brief.md +++ b/automatic-pdf-renamer/brief.md @@ -12,8 +12,9 @@ PDFs arrive continuously in an inbox S3 prefix (dropped via Nextcloud by the use ## 3. Objectives -- Classify each incoming PDF against a known naming template, based on the full text content of every page (including headers and footers), not the current object name. -- Fill the fields of the selected template (date, counterparty, reference, amount) from the content. +- Extract the emitting company, document type, additional-field values, and emission date from the full text content of every page (including headers and footers), not from the current object name. +- Use ML to select additional-field groups from emitting company and document type, then use one shared token-level extraction model to identify all requested field values in the text. Independently use ML to predict template and period format; do not use predefined mappings for these decisions. +- Calculate the period and storage key only after the required extracted fields and configured mapping have been validated. - Rename and move the object within the bucket into an organized prefix structure depending on the document type. - Unambiguous deterministic rules and valid fields: automatic processing. If rules cannot determine a valid result or extraction is incomplete, queue the file with status `pending` for human review in the dashboard. - Write metadata as JSON sidecar objects in the bucket (source of truth) and replicate every decision into SQLite for fast queries, with retraining from corrections. diff --git a/automatic-pdf-renamer/epics.md b/automatic-pdf-renamer/epics.md index aa5e48c..5fcf189 100644 --- a/automatic-pdf-renamer/epics.md +++ b/automatic-pdf-renamer/epics.md @@ -30,8 +30,8 @@ Build a personal, fully offline application that watches a self-hosted Garage S3 ### Delivery strategy 1. Deliver the technical foundation and durable metadata model. -2. Deliver a complete, useful ingestion flow using deterministic rules only. -3. Add ML classification and ML-assisted extraction, training and model lifecycle management. +2. Deliver the ingestion and storage foundation, with classification outcomes delegated to local ML models. +3. Add local ML inference for all extraction and classification decisions, then training and model lifecycle management. 4. Deliver the dashboard for search, review, correction, validation, and template visibility. 5. Complete consistency checks, recovery, deployment hardening, and operational verification. @@ -47,7 +47,7 @@ The following inventory consolidates the requirements from the project brief and - **FR-TXT-01 — Text extraction:** extract all available text from every page of a PDF using a local library; no OCR in v1. - **FR-TXT-02 — Empty text handling:** route PDFs with empty or insufficient extracted text to human review with an explicit reason. - **FR-TXT-03 — Text persistence:** persist extracted text in the sidecar and SQLite replica so normal review and training workflows do not need to re-parse the PDF. -- **FR-CLS-01 — Classification:** classify documents against configured templates. The initial light release uses deterministic rules; the ML epic adds TF-IDF and Logistic Regression classification. +- **FR-CLS-01 — Ordered ML-assisted extraction:** use local ML to extract emitting company, document type, and emission date; use another ML model to select additional-field groups from company and document type, then one shared token-level ML model to extract their values; independently predict template and period format from company and document type; finally calculate period and storage key deterministically. No predefined mapping selects these outcomes. - **FR-CLS-02 — Confidence and unknown class (post-v1):** ML classification exposes confidence and routes results below a configurable threshold to review. The deterministic initial release uses explicit rule outcomes and does not generate artificial confidence scores. - **FR-FLD-01 — Common fields:** every document has template, emission date, emitting company, document type, and period. - **FR-FLD-02 — Optional fields:** templates define optional fields and reusable field groups with formats, allowed values, and extraction rules. @@ -574,4 +574,4 @@ Completes the remaining Epic 4 capabilities and Epic 5, including consistency re ## 7. Open Decisions to Resolve Before the Affected Stories -2. **ML field extraction:** specify which field groups use ML in v1 and what evaluation threshold is required before enabling each extractor. Classification and deterministic validation must remain usable independently. +2. **ML field extraction:** specify which field groups use ML in v1 and what evaluation threshold is required before enabling the shared extractor. Classification and deterministic validation must remain usable independently. diff --git a/automatic-pdf-renamer/prd.md b/automatic-pdf-renamer/prd.md index 40fe8cd..0053e58 100644 --- a/automatic-pdf-renamer/prd.md +++ b/automatic-pdf-renamer/prd.md @@ -67,12 +67,14 @@ Transitions: - FR2.3 If extracted text is empty or below a minimum length, route the file to `pending` with reason "empty text". - FR2.4 Store the complete extracted text from all pages in the sidecar and replicate it to SQLite for the file's lifetime, so search, the detail page, and retraining do not need to re-download and re-parse the object. -### Epic 3 — Template classification +### Epic 3 — ML-assisted field extraction and classification -- FR3.1 Classify the document into one of the configured templates using configured deterministic rules, extracted text, and reference data in the initial release. TF-IDF + Logistic Regression is a post-v1 enhancement specified in Epic 3, not a dependency of the initial release. -- FR3.2 In the initial release, if deterministic rules do not yield one unambiguous template, route the file to `pending` with reason `insufficient deterministic evidence`; do not fabricate a confidence score. In the post-v1 ML enhancement, expose model scores and route predictions below the configured threshold to `pending`. -- FR3.3 In the initial release, display the matching rules and ambiguity/review reason on the detail page. When ML is enabled in a future release, also display ranked candidates and scores. -- FR3.4 Post-v1 only: retrain the classifier on demand from the initial corpus plus eligible validated files. Record the training date and evaluation metrics in the local model registry and display them in the dashboard. +- FR3.1 Process the full extracted document text in ordered stages. Stage 1 extracts `emitting_company`, `document_type`, `additional_fields`, and `emission_date`; emission date extraction runs in the same stage as the other text-derived fields. +- FR3.2 Use local ML to predict applicable additional-field groups from `(emitting_company, document_type)`, then use one shared local token-level extraction model to extract all selected values from document text. No predefined mapping selects these groups. Do not select or consume the template and period format before this extraction stage completes. +- FR3.3 Stage 2 resolves `template_name` and `period_format` from the configured mapping using the first-stage fields. Stage 3 calculates `period` from `emission_date` and `period_format`, then generates and validates the target storage key. +- FR3.4 Ambiguous or missing indispensable values, unknown values, unresolved mappings, and invalid calculations must route to `pending`; the pipeline must never substitute a default company or document type or fabricate a candidate. +- FR3.5 ML is an assistive candidate-proposal/ranking layer, not an authority that bypasses configuration. Deterministic rules and configured allow-lists remain available as fallback and validation boundaries. ML must be disabled or unavailable without breaking the rules-only path. Exact model strategy and confidence thresholds remain an explicit calibration decision. +- FR3.6 Retrain on demand from the initial labeled corpus plus eligible validated files. Record the training date and evaluation metrics in the local model registry; require evaluation before activation. ### Epic 4 — Document data model, slot filling and validation @@ -91,10 +93,10 @@ Every document, regardless of template, carries at least these five common field The object key pattern of each template is composed from these common fields, plus the template's optional fields. -#### 4.2 Optional fields (template-specific) +#### 4.2 Optional fields (template-specific)\n\n**Required processing order:** (1) extract emitting company, document type, additional-field values, and emission date from text; (2) resolve template and period format using the first-stage fields and configured mapping; (3) calculate period and storage key. The first stage must not select a default class. If an indispensable field is missing or ambiguous, route the document to `pending`. - FR4.1 Each template declares in JSONC its optional fields beyond the common five. Additional fields are organized in predefined field groups (e.g., `bank_accounts`, `properties`, `contract_references`, `customer_numbers`) with their possible values. The (company, document_type) mapping specifies which field groups to extract for each document. Examples: bank statements extract from `bank_accounts` group; rent receipts extract from `properties` group. -- FR4.2 Fill common and optional fields using per-template regexes for dates and the companies list for company matching. Additional fields are extracted in the initial release using configured deterministic rules and lookup lists. ML-assisted selection from predefined possible values is a post-v1 enhancement. The period is computed server-side from the emission date using the configured granularity, format template, and offset (current/previous/following); it is recomputed whenever the emission date or applicable template rules change and cannot be edited or submitted by the user. +- FR4.2 Extract `emitting_company`, `document_type`, `additional_fields`, and `emission_date` from the document text in the first processing stage. Predict additional-field groups with a local ML model from `(emitting_company, document_type)`, then extract their values using one shared token-level model trained from validated field spans in extracted text. No predefined mapping or allow-list determines these decisions. Only after this stage resolves required fields may the system select `template_name` and `period_format`. Compute `period` server-side from `emission_date` and the selected period format, then generate the storage key from the validated fields and template pattern. Recompute period and key whenever a relevant source field or mapping changes; period is not client-editable. - FR4.3 Validate fields before renaming: emission date parses and is within a plausible range, period is non-empty and consistent with the emission date and the template's granularity/offset, emitting company recognized, document type among the template's allowed values, each optional field matches its declared format (empty optional fields are allowed). Any failure routes to `pending` with the failing fields identified. - FR4.4 Collisions in the target prefix are resolved with an incremental suffix; the final key is always the one logged. @@ -199,3 +201,8 @@ SQLite replica (rebuilt from sidecars): - Post-v1 ML decision: exact model confidence threshold and per-template overrides, to be calibrated on the initial corpus before enabling ML. - Poll interval tradeoff between responsiveness and Garage listing load. - Exact human-readable period formats (French month names, quarter notation) and their rendering in the object key — to be fixed in the template JSONC spec. + + +### ML-only inference contract + +The runtime uses local model artifacts for emitting-company, document-type and emission-date extraction; additional-field-group selection from emitting company and document type; shared token-level additional-field span extraction; and independent `template_name` and `period_format` prediction from emitting company and document type. Configuration may define template key patterns and period arithmetic parameters, but it must not select classification outcomes. Missing models, invalid predictions, or confidence below `CONFIDENCE_THRESHOLD` route the document to `pending`. The `period` and final storage key remain deterministic calculations after model inference. diff --git a/automatic-pdf-renamer/tests/test_document_suggestions.py b/automatic-pdf-renamer/tests/test_document_suggestions.py new file mode 100644 index 0000000..dda7337 --- /dev/null +++ b/automatic-pdf-renamer/tests/test_document_suggestions.py @@ -0,0 +1,308 @@ +"""Tests for asynchronous, confidence-gated document suggestions.""" + +import asyncio +import re +import unittest +from types import SimpleNamespace +from unittest.mock import patch + +from app.api import routes +from app.ml import AmbiguousPredictionError, LocalMLPipeline + + +class FakeDatabase: + def __init__(self, record): + self.record = record + + def get_by_id(self, file_id): + return self.record if file_id == self.record.id else None + + +class FakeRegistry: + def get_additional_fields(self): + return { + "invoice_number": ["INV-123"], + "amount": ["42"], + } + + def get_date_extraction_compiled(self): + return [re.compile(r"([0-9]{1,2})/([0-9]{1,2})/([0-9]{4})")] + + def get_period_format_config(self, name): + return SimpleNamespace(to_dict=lambda: {}) + + def get_key_pattern(self, name): + return "example/{company}/{period}/{document_type}" + + +class FakeMLPipeline(LocalMLPipeline): + def __init__(self, confidence_threshold=None): + super().__init__(confidence_threshold=confidence_threshold) + + def _load(self, model_name): + if model_name == "additional_field_values": + return SimpleNamespace(trained_fields=["invoice_number", "amount"]) + raise AssertionError(f"Unexpected model load: {model_name}") + + def _predict_label(self, model_name, sample): + values = { + "field_emitting_company": ("Suggested Company", 0.96), + "field_document_type": ("invoice", 0.95), + "template_name": ("invoice-template", 0.94), + "period_format": ("monthly", 0.93), + } + if model_name == "template_name" and "invoice_number: INV-123" not in sample: + raise AmbiguousPredictionError("Expected extracted additional field in sample") + if model_name == "template_name" and "LowConfidence" in sample: + raise AmbiguousPredictionError("Below threshold") + return values[model_name] + + def predict_additional_groups(self, emitting_company, document_type): + return ["invoice_number", "tags"], 0.91 + + def predict_additional_values(self, field_names, text, *, allow_partial=False): + if "tags" in field_names: + raise ValueError("The shared extraction model has no training spans for: tags") + return { + field_name: ("INV-123" if field_name == "invoice_number" else "42", 0.92) + for field_name in field_names + } + +class DocumentSuggestionTests(unittest.TestCase): + def setUp(self): + self.record = SimpleNamespace( + id="document-1", + extracted_text="Suggested Company invoice\nNumber: INV-123\nAmount: 42\nDate: 20/09/2026", + emitting_company="Old Company", + document_type="receipt", + template_name="old-template", + period_format="monthly", + emission_date="2026-09-20", + optional_fields={"invoice_number": "INV-123"}, + original_key="inbox/original.pdf", + ) + self.request = SimpleNamespace(app=SimpleNamespace(state=SimpleNamespace())) + + def test_partial_additional_value_extraction_omits_only_low_confidence_fields(self): + pipeline = LocalMLPipeline(confidence_threshold=0.3) + pipeline._load = lambda _name: SimpleNamespace( + predict_values=lambda _text, _fields: { + "appartement": ("", 0.0), + "impots": ("Impots sur le revenu", 0.84), + } + ) + + result = pipeline.predict_additional_values( + ["appartement", "impots"], "document text", allow_partial=True + ) + + self.assertEqual(result, {"impots": ("Impots sur le revenu", 0.84)}) + with self.assertRaises(AmbiguousPredictionError): + pipeline.predict_additional_values(["appartement", "impots"], "document text") + + def test_returns_confident_model_suggestions_and_derived_values(self): + with ( + patch.object(routes, "_db", return_value=FakeDatabase(self.record)), + patch.object(routes, "_registry", return_value=FakeRegistry()), + patch.object(routes, "LocalMLPipeline", FakeMLPipeline), + patch.object(routes, "format_period_from_emission_date", return_value="2026-09"), + patch.object(routes.logger, "debug") as debug_log, + ): + result = asyncio.run(routes.get_document_model_suggestions( + self.request, + "document-1", + { + "emitting_company": "Edited Company", + "document_type": "invoice", + "template_name": "edited-template", + "period_format": "monthly", + "emission_date": "2026-09-20", + "optional_fields": {}, + }, + current_user=None, + )) + + suggestions = result["suggestions"] + self.assertEqual(suggestions["emission_date"]["value"], "2026-09-20") + self.assertEqual(suggestions["emission_date"]["source"], "content") + self.assertEqual(suggestions["emitting_company"]["value"], "Suggested Company") + self.assertEqual( + suggestions["optional_fields"]["value"], + {"invoice_number": "INV-123"}, + ) + self.assertEqual(suggestions["optional_fields"]["source"], "configured") + decisions = next( + call.args[2] + for call in debug_log.call_args_list + if call.args[0] == "API POST /files/%s/suggestions decisions=%s" + ) + self.assertIn("configured_matches={'invoice_number': 'INV-123'}", decisions["additional_fields"]) + self.assertEqual(suggestions["template_name"]["value"], "invoice-template") + self.assertEqual(suggestions["period_format"]["value"], "monthly") + self.assertEqual(suggestions["period"]["value"], "2026-09") + self.assertEqual(suggestions["current_key"]["value"], "files/example/Edited Company/2026-09/invoice.pdf") + + def test_derived_values_use_form_values_not_model_predictions(self): + class ConflictingPipeline(FakeMLPipeline): + def _predict_label(self, model_name, sample): + predictions = { + "field_emitting_company": ("Model Company", 0.96), + "field_document_type": ("model-document", 0.95), + "template_name": ("model-template", 0.94), + "period_format": ("model-period", 0.93), + } + return predictions[model_name] + + class TrackingRegistry(FakeRegistry): + def __init__(self): + self.period_format_used = None + self.template_used = None + + def get_period_format_config(self, name): + self.period_format_used = name + return SimpleNamespace(to_dict=lambda: {}) + + def get_key_pattern(self, name): + self.template_used = name + return "{template_name}/{company}/{document_type}/{period_format}/{period}" + + registry = TrackingRegistry() + with ( + patch.object(routes, "_db", return_value=FakeDatabase(self.record)), + patch.object(routes, "_registry", return_value=registry), + patch.object(routes, "LocalMLPipeline", ConflictingPipeline), + patch.object(routes, "format_period_from_emission_date", return_value="user-date-period"), + ): + result = asyncio.run(routes.get_document_model_suggestions( + self.request, + "document-1", + { + "emitting_company": "User Company", + "document_type": "user-document", + "template_name": "user-template", + "period_format": "user-period-format", + "emission_date": "2026-09-20", + }, + current_user=None, + )) + + suggestions = result["suggestions"] + self.assertEqual(suggestions["emitting_company"]["value"], "Model Company") + self.assertEqual(suggestions["document_type"]["value"], "model-document") + self.assertEqual(suggestions["template_name"]["value"], "model-template") + self.assertEqual(suggestions["period_format"]["value"], "model-period") + self.assertEqual(suggestions["period"]["value"], "user-date-period") + self.assertEqual(registry.period_format_used, "user-period-format") + self.assertEqual(registry.template_used, "user-template") + self.assertEqual( + suggestions["current_key"]["value"], + "files/user-template/User Company/user-document/user-period-format/user-date-period.pdf", + ) + + def test_suggestion_orchestrator_collects_independent_model_results(self): + pipeline = FakeMLPipeline(confidence_threshold=0.3) + + suggestions, decisions = pipeline.suggest_document_fields( + self.record.extracted_text, + "Edited Company", + "invoice", + {"invoice_number": "INV-123"}, + FakeRegistry().get_additional_fields(), + ) + + self.assertEqual(suggestions["emitting_company"]["value"], "Suggested Company") + self.assertEqual(suggestions["document_type"]["value"], "invoice") + self.assertEqual(suggestions["optional_fields"]["value"], {"invoice_number": "INV-123"}) + self.assertEqual(suggestions["template_name"]["value"], "invoice-template") + self.assertEqual(suggestions["period_format"]["value"], "monthly") + self.assertIn("configured_matches={'invoice_number': 'INV-123'}", decisions["additional_fields"]) + + def test_configured_value_matching_corrects_partial_case_insensitive_spans(self): + text = "65 RUE RAYMOND LOSSERAND\nTaxe foncière" + + self.assertEqual( + LocalMLPipeline.match_configured_additional_value( + text, + "65 RUE RAYMOND", + ["65 rue Raymond Losserand", "20 rue Paul Fort"], + ), + "65 rue Raymond Losserand", + ) + self.assertEqual( + LocalMLPipeline.match_configured_additional_value( + text, + "Taxe", + ["Taxe foncière", "Taxe d'habitation"], + ), + "Taxe foncière", + ) + self.assertIsNone( + LocalMLPipeline.match_configured_additional_value( + "Some unrelated text", + "unrelated", + ["Taxe foncière"], + ) + ) + self.assertIsNone( + LocalMLPipeline.match_configured_additional_value( + "Taxe foncière et Taxe d'habitation", + "Taxe", + ["Taxe foncière", "Taxe d'habitation"], + ) + ) + + def test_unavailable_or_low_confidence_predictions_are_omitted(self): + class LowConfidencePipeline(FakeMLPipeline): + def _predict_label(self, model_name, sample): + if model_name == "template_name": + raise AmbiguousPredictionError("Below threshold") + return super()._predict_label(model_name, sample) + + with ( + patch.object(routes, "_db", return_value=FakeDatabase(self.record)), + patch.object(routes, "_registry", return_value=FakeRegistry()), + patch.object(routes, "LocalMLPipeline", LowConfidencePipeline), + patch.object(routes, "format_period_from_emission_date", return_value="2026-09"), + ): + result = asyncio.run(routes.get_document_model_suggestions( + self.request, + "document-1", + {"emission_date": "2026-09-20"}, + current_user=None, + )) + + self.assertNotIn("template_name", result["suggestions"]) + self.assertEqual(result["suggestions"]["period_format"]["value"], "monthly") + + def test_low_confidence_additional_field_does_not_block_other_suggestions(self): + class PartialConfidencePipeline(FakeMLPipeline): + def predict_additional_groups(self, emitting_company, document_type): + return ["invoice_number", "amount"], 0.91 + + def predict_additional_values(self, field_names, text, *, allow_partial=False): + if field_names == ["invoice_number"]: + raise AmbiguousPredictionError("invoice number below threshold") + return {"amount": ("42", 0.87)} + + with ( + patch.object(routes, "_db", return_value=FakeDatabase(self.record)), + patch.object(routes, "_registry", return_value=FakeRegistry()), + patch.object(routes, "LocalMLPipeline", PartialConfidencePipeline), + patch.object(routes, "format_period_from_emission_date", return_value="2026-09"), + ): + result = asyncio.run(routes.get_document_model_suggestions( + self.request, + "document-1", + {"emission_date": "2026-09-20"}, + current_user=None, + )) + + suggestions = result["suggestions"] + self.assertEqual(suggestions["optional_fields"]["value"], {"amount": "42"}) + self.assertEqual(suggestions["optional_fields"]["source"], "configured") + self.assertEqual(suggestions["template_name"]["value"], "invoice-template") + self.assertEqual(suggestions["period_format"]["value"], "monthly") + + +if __name__ == "__main__": + unittest.main() diff --git a/automatic-pdf-renamer/tests/test_ml_pipeline_order.py b/automatic-pdf-renamer/tests/test_ml_pipeline_order.py new file mode 100644 index 0000000..593fe7e --- /dev/null +++ b/automatic-pdf-renamer/tests/test_ml_pipeline_order.py @@ -0,0 +1,152 @@ +"""Tests for local ML inference and confidence-gated routing decisions.""" + +import json +import tempfile +import unittest +from pathlib import Path +from unittest.mock import patch + +from app.ml import LocalMLPipeline, AmbiguousPredictionError, ModelUnavailableError + + +class FakeModel: + def __init__(self, label, confidence=0.95): + self.label = label + self.confidence = confidence + + def predict(self, samples): + return [self.label] + + def predict_proba(self, samples): + return [[self.confidence, 1.0 - self.confidence]] + + +class FakeSharedValuesModel: + def predict_values(self, text, fields): + return {field: (f"value-{field}", 0.95) for field in fields} + + +class FakeGroupsModel: + def __init__(self, groups, confidence=0.95): + self.groups = groups + self.confidence = confidence + + def predict(self, samples): + return [self.groups] + + def predict_proba(self, samples): + return [[self.confidence, 1.0 - self.confidence]] + + +class LocalMLPipelineTests(unittest.TestCase): + def setUp(self): + self.temp_dir = tempfile.TemporaryDirectory() + self.model_dir = Path(self.temp_dir.name) + + def tearDown(self): + self.temp_dir.cleanup() + + def pipeline_with(self, models, threshold=0.8): + active = {} + for name in models: + version = "v1" + artifact = self.model_dir / "versions" / name / f"{version}.joblib" + artifact.parent.mkdir(parents=True, exist_ok=True) + artifact.touch() + active[name] = version + (self.model_dir / "active_models.json").write_text(json.dumps(active)) + pipeline = LocalMLPipeline(str(self.model_dir), threshold) + loader = patch("joblib.load", side_effect=lambda path: models[Path(path).parent.name]) + loader.start() + self.addCleanup(loader.stop) + return pipeline + + def test_common_fields_are_predicted_by_models(self): + models = { + "field_emitting_company": FakeModel("Example Energy"), + "field_document_type": FakeModel("invoice"), + } + pipeline = self.pipeline_with(models) + fields = pipeline.extract_initial_fields("document text") + self.assertEqual(fields["emitting_company"][0], "Example Energy") + self.assertEqual(fields["document_type"][0], "invoice") + self.assertEqual(set(fields), {"emitting_company", "document_type"}) + + def test_predict_labels_keeps_successful_fields_when_another_is_ambiguous(self): + pipeline = self.pipeline_with({ + "field_emitting_company": FakeModel("Example Energy", 0.4), + "field_document_type": FakeModel("invoice", 0.95), + }) + + predictions, errors = pipeline.predict_labels({ + "emitting_company": ("field_emitting_company", "document text"), + "document_type": ("field_document_type", "document text"), + }) + + self.assertEqual(predictions, {"document_type": ("invoice", 0.95)}) + self.assertIsInstance(errors["emitting_company"], AmbiguousPredictionError) + + def test_extract_initial_fields_remains_strict(self): + pipeline = self.pipeline_with({ + "field_emitting_company": FakeModel("Example Energy", 0.4), + "field_document_type": FakeModel("invoice", 0.95), + }) + + with self.assertRaises(AmbiguousPredictionError): + pipeline.extract_initial_fields("document text") + + def test_additional_groups_are_predicted_with_confidence(self): + pipeline = self.pipeline_with({ + "additional_field_groups": FakeGroupsModel(["appartement", "impots"]), + }) + + groups, confidence = pipeline.predict_additional_groups("Impots", "Avis") + + self.assertEqual(groups, ["appartement", "impots"]) + self.assertEqual(confidence, 0.95) + + def test_low_confidence_prediction_is_rejected(self): + pipeline = self.pipeline_with({"field_document_type": FakeModel("invoice", 0.4)}) + with self.assertRaises(AmbiguousPredictionError): + pipeline._predict_label("field_document_type", "document text") + + def test_shared_model_extracts_multiple_fields(self): + pipeline = self.pipeline_with({"additional_field_values": FakeSharedValuesModel()}, threshold=0.8) + values = pipeline.predict_additional_values(["invoice_number", "account_number"], "document text") + self.assertEqual(values["invoice_number"][0], "value-invoice_number") + self.assertEqual(values["account_number"][0], "value-account_number") + + def test_missing_model_is_not_replaced_by_rules(self): + pipeline = LocalMLPipeline(str(self.model_dir), 0.8) + with self.assertRaises(ModelUnavailableError): + pipeline._predict_label("field_document_type", "invoice text") + + def test_low_confidence_additional_field_is_rejected(self): + pipeline = self.pipeline_with({"additional_field_values": FakeSharedValuesModel()}, threshold=0.99) + with self.assertRaises(AmbiguousPredictionError): + pipeline.predict_additional_values(["invoice_number"], "document text") + + def test_template_and_period_format_are_predicted_together(self): + pipeline = self.pipeline_with({ + "template_name": FakeModel("invoice-template", 0.94), + "period_format": FakeModel("monthly", 0.91), + }) + + result = pipeline.predict_template_and_period_format( + "Example Energy", "invoice", {"invoice_number": "INV-123"} + ) + + self.assertEqual(result, ("invoice-template", 0.94, "monthly", 0.91)) + + def test_template_and_period_format_remain_strict(self): + pipeline = self.pipeline_with({ + "template_name": FakeModel("invoice-template", 0.4), + "period_format": FakeModel("monthly", 0.91), + }) + + with self.assertRaises(AmbiguousPredictionError): + pipeline.predict_template_and_period_format("Example Energy", "invoice", {}) + + +if __name__ == "__main__": + unittest.main() diff --git a/automatic-pdf-renamer/tests/test_model_manager.py b/automatic-pdf-renamer/tests/test_model_manager.py new file mode 100644 index 0000000..b4a653f --- /dev/null +++ b/automatic-pdf-renamer/tests/test_model_manager.py @@ -0,0 +1,124 @@ +"""Tests for full-set model retraining and shared additional-field extraction.""" + +import json +import os +import tempfile +import unittest +from pathlib import Path +from types import SimpleNamespace +from unittest.mock import patch + +from app.model_manager import BASE_MODELS, list_models, train_all_models +from app.ml import LocalMLPipeline, ModelUnavailableError + + +def validated_records(): + return [ + SimpleNamespace( + extracted_text="Invoice number INV-100 for account AC-10", + optional_fields={"invoice_number": "INV-100", "account_number": "AC-10"}, + emitting_company="Company A", + document_type="invoice", + template_name="invoice_template", + period_format="monthly", + ), + SimpleNamespace( + extracted_text="Invoice number INV-200", + optional_fields={"invoice_number": "INV-200"}, + emitting_company="Company A", + document_type="invoice", + template_name="invoice_template", + period_format="quarterly", + ), + SimpleNamespace( + extracted_text="Account statement for AC-30", + optional_fields={"account_number": "AC-30"}, + emitting_company="Company B", + document_type="statement", + template_name="statement_template", + period_format="yearly", + ), + ] + + +class FullModelRetrainingTests(unittest.TestCase): + def setUp(self): + self.temp_dir = tempfile.TemporaryDirectory() + self.root = Path(self.temp_dir.name) + self.env = patch.dict(os.environ, {"MODEL_DIR": str(self.root)}) + self.env.start() + self.addCleanup(self.env.stop) + + def tearDown(self): + self.temp_dir.cleanup() + + def test_retraining_replaces_all_models_without_history(self): + old_artifact = self.root / "sets" / "old-version" / "field_document_type.joblib" + old_artifact.parent.mkdir(parents=True) + old_artifact.write_bytes(b"old artifact") + (self.root / "active_models.json").write_text( + json.dumps({"field_document_type": "old-version"}), encoding="utf-8" + ) + + progress = [] + result = train_all_models(validated_records(), lambda value, message: progress.append((value, message))) + + self.assertEqual(result["status"], "completed") + self.assertEqual(result["count"], len(BASE_MODELS)) + active = json.loads((self.root / "active_models.json").read_text()) + self.assertEqual(set(active), set(BASE_MODELS)) + self.assertEqual(len(set(active.values())), 1) + for name, version in active.items(): + self.assertTrue((self.root / "sets" / version / f"{name}.joblib").is_file()) + self.assertFalse(old_artifact.exists()) + self.assertEqual( + {path.name.removesuffix(".joblib") for path in (self.root / "sets" / next(iter(active.values()))).glob("*.joblib")}, + set(BASE_MODELS), + ) + self.assertEqual(len(list((self.root / "sets").glob("*"))), 1) + self.assertFalse((self.root / "versions").exists()) + self.assertEqual(progress[-1][0], 100) + + def test_failed_retraining_keeps_the_previous_active_set(self): + old_generation = "existing-set" + old_artifact = self.root / "sets" / old_generation / "field_document_type.joblib" + old_artifact.parent.mkdir(parents=True) + old_artifact.write_bytes(b"previous active artifact") + active_before = {"field_document_type": old_generation} + (self.root / "active_models.json").write_text(json.dumps(active_before), encoding="utf-8") + + with self.assertRaises(ValueError): + train_all_models([]) + + self.assertEqual( + json.loads((self.root / "active_models.json").read_text()), + active_before, + ) + self.assertTrue(old_artifact.is_file()) + self.assertFalse(any(path.name.startswith(".staging-") for path in self.root.iterdir())) + + def test_inventory_has_no_version_history_or_emission_date_model(self): + result = list_models() + names = {model["name"] for model in result["models"]} + self.assertNotIn("field_emission_date", names) + self.assertTrue(all("versions" not in model for model in result["models"])) + + def test_inference_requires_a_complete_active_model_set(self): + pipeline = LocalMLPipeline(str(self.root), 0.8) + with self.assertRaises(ModelUnavailableError): + pipeline._load("field_document_type") + + def test_shared_additional_value_model_extracts_multiple_fields(self): + train_all_models(validated_records()) + pipeline = LocalMLPipeline(str(self.root), 0.0) + predictions = pipeline.predict_additional_values( + ["invoice_number", "account_number"], + "Invoice number INV-200 for account AC-20", + ) + self.assertEqual(set(predictions), {"invoice_number", "account_number"}) + self.assertTrue(predictions["invoice_number"][0]) + self.assertTrue(predictions["account_number"][0]) + + +if __name__ == "__main__": + unittest.main() diff --git a/automatic-pdf-renamer/tests/test_pipeline_date_extraction.py b/automatic-pdf-renamer/tests/test_pipeline_date_extraction.py new file mode 100644 index 0000000..b18158f --- /dev/null +++ b/automatic-pdf-renamer/tests/test_pipeline_date_extraction.py @@ -0,0 +1,45 @@ +"""Tests for configured-regex emission-date extraction.""" + +import re +import unittest +from types import SimpleNamespace + +from app.pipeline import DocumentProcessor + + +class EmissionDateExtractionTests(unittest.TestCase): + def processor(self, patterns): + processor = object.__new__(DocumentProcessor) + processor.registry = SimpleNamespace( + get_date_extraction_compiled=lambda: [re.compile(pattern) for pattern in patterns] + ) + return processor + + def test_uses_first_date_in_document_order_not_earliest_calendar_date(self): + processor = self.processor([ + r"([0-9]{1,2})/([0-9]{1,2})/([0-9]{4})", + r"([0-9]{4})-([0-9]{2})-([0-9]{2})", + ]) + date, error = processor._extract_emission_date( + "Issued 2025-02-03; reference date 01/01/2024" + ) + self.assertIsNone(error) + self.assertEqual(date, "2025-02-03") + + def test_normalizes_french_month_names(self): + processor = self.processor([ + r"([0-9]{1,2}) (janvier|fevrier|mars|avril|mai|juin|juillet|aout|septembre|octobre|novembre|decembre) ([0-9]{4})" + ]) + date, error = processor._extract_emission_date("Émis le 1 janvier 2025") + self.assertIsNone(error) + self.assertEqual(date, "2025-01-01") + + def test_does_not_fall_back_when_no_configured_regex_matches(self): + processor = self.processor([r"([0-9]{4})-([0-9]{2})-([0-9]{2})"]) + date, error = processor._extract_emission_date("Date: 01/02/2025") + self.assertEqual(date, "") + self.assertIsNotNone(error) + + +if __name__ == "__main__": + unittest.main() diff --git a/automatic-pdf-renamer/web/app.js b/automatic-pdf-renamer/web/app.js index 758eef8..ca85119 100644 --- a/automatic-pdf-renamer/web/app.js +++ b/automatic-pdf-renamer/web/app.js @@ -131,7 +131,7 @@ const loadDocuments = async ({ append = false } = {}) => { params.set('page', String(page)); try { - const response = await fetch(`/api/files?${params.toString()}`); + const response = await fetch(`/files?${params.toString()}`); if (!response.ok) { throw new Error(`HTTP ${response.status}`); } diff --git a/automatic-pdf-renamer/web/config.js b/automatic-pdf-renamer/web/config.js index ac2e6b4..68a5bcb 100644 --- a/automatic-pdf-renamer/web/config.js +++ b/automatic-pdf-renamer/web/config.js @@ -5,7 +5,7 @@ const status = document.getElementById('config-status'); const loadConfig = async () => { try { - const response = await fetch('/api/config'); + const response = await fetch('/config'); if (!response.ok) { throw new Error(`HTTP ${response.status}`); } @@ -28,7 +28,7 @@ form.addEventListener('submit', async (event) => { status.textContent = 'Validating and saving configuration...'; try { - const response = await fetch('/api/config', { + const response = await fetch('/config', { method: 'PUT', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ content: editor.value }), diff --git a/automatic-pdf-renamer/web/document.html b/automatic-pdf-renamer/web/document.html index bc4f090..ecd17b0 100644 --- a/automatic-pdf-renamer/web/document.html +++ b/automatic-pdf-renamer/web/document.html @@ -54,23 +54,28 @@
+
+
+
+
Use dd/mm/yyyy +
@@ -93,6 +98,7 @@

Additional fields

+
diff --git a/automatic-pdf-renamer/web/document.js b/automatic-pdf-renamer/web/document.js index 362d45d..6bf2713 100644 --- a/automatic-pdf-renamer/web/document.js +++ b/automatic-pdf-renamer/web/document.js @@ -4,6 +4,7 @@ const content = document.getElementById('document-content'); const technicalDataList = document.getElementById('technical-data'); const additionalSection = document.getElementById('additional-section'); const additionalFields = document.getElementById('additional-fields'); +const additionalFieldSuggestions = document.getElementById('additional-field-suggestions'); const addAdditionalFieldButton = document.getElementById('add-additional-field'); const historyList = document.getElementById('document-history'); const extractedText = document.getElementById('extracted-text'); @@ -18,6 +19,7 @@ const periodFormatSelect = document.getElementById('edit-period-format'); const companyInput = document.getElementById('edit-company'); const documentTypeInput = document.getElementById('edit-document-type'); const emissionDateInput = document.getElementById('edit-emission-date'); +const emissionDateSuggestion = document.getElementById('suggestion-emission-date'); const periodPreview = document.getElementById('period-preview'); const keyPreview = document.getElementById('current-key-preview'); const saveButton = document.getElementById('save-document-button'); @@ -83,7 +85,7 @@ const loadDocumentNavigation = async (fileId) => { while (true) { const params = new URLSearchParams(filters); params.set('page', String(page)); - const response = await fetch(`/api/files?${params.toString()}`); + const response = await fetch(`/files?${params.toString()}`); if (!response.ok) throw new Error(`HTTP ${response.status}`); const items = await response.json(); total = Number(response.headers.get('X-Total-Count')) || total; @@ -96,7 +98,7 @@ const loadDocumentNavigation = async (fileId) => { if (!following && position < total) { const nextParams = new URLSearchParams(filters); nextParams.set('page', String(page + 1)); - const nextResponse = await fetch(`/api/files?${nextParams.toString()}`); + const nextResponse = await fetch(`/files?${nextParams.toString()}`); if (!nextResponse.ok) throw new Error(`HTTP ${nextResponse.status}`); following = (await nextResponse.json())[0] || null; } @@ -190,6 +192,7 @@ const buildAdditionalFieldRow = (label = '', value = '', suggestions = {}) => { removeButton.textContent = 'Remove'; removeButton.addEventListener('click', () => { row.remove(); + scheduleModelSuggestions(); if (!additionalFields.querySelector('.additional-field-row')) { const empty = document.createElement('p'); empty.className = 'additional-field-empty'; @@ -335,7 +338,6 @@ const renderDocument = (documentData, fileId) => { const technicalMetadata = [ ['Status', documentData.status], - ['Confidence', documentData.confidence], ['Original key', documentData.original_key], ['Current key', documentData.current_key], ['SHA-256', documentData.id], @@ -354,6 +356,219 @@ const renderDocument = (documentData, fileId) => { statusMessage.textContent = ''; }; +const suggestionElements = { + template_name: document.getElementById('suggestion-template_name'), + period_format: document.getElementById('suggestion-period_format'), + emitting_company: document.getElementById('suggestion-emitting_company'), + document_type: document.getElementById('suggestion-document_type'), +}; + +let suggestionRequestSequence = 0; +let suggestionTimer = null; +let suggestionAbortController = null; + +const hideModelSuggestions = () => { + Object.values(suggestionElements).forEach((element) => { + if (element) { + element.setAttribute('aria-hidden', 'true'); + element.replaceChildren(); + } + }); + if (additionalFieldSuggestions) { + additionalFieldSuggestions.setAttribute('aria-hidden', 'true'); + additionalFieldSuggestions.replaceChildren(); + } +}; + +const renderAdditionalFieldSuggestions = (suggestion) => { + if (!additionalFieldSuggestions) return; + additionalFieldSuggestions.replaceChildren(); + const predictedFields = suggestion?.value; + if (!predictedFields || typeof predictedFields !== 'object' || Array.isArray(predictedFields)) { + additionalFieldSuggestions.setAttribute('aria-hidden', 'true'); + return; + } + + const currentFields = getAdditionalFieldEntries(); + Object.entries(predictedFields).forEach(([fieldName, rawValue]) => { + const value = String(rawValue ?? '').trim(); + if (!fieldName.trim() || !value || currentFields[fieldName] === value) return; + + const proposal = document.createElement('div'); + proposal.className = 'additional-field-proposal'; + const confidence = suggestion.confidence?.[fieldName]; + const description = document.createElement('span'); + const confidenceLabel = Number.isFinite(confidence) + ? ` (${Math.round(confidence * 100)}% confidence)` + : ''; + const sourceLabel = suggestion.source === 'configured' ? 'Configured value found' : 'Model suggestion'; + description.textContent = `${sourceLabel}: ${fieldName}: ${value}${confidenceLabel}`; + description.title = description.textContent; + + const applyButton = document.createElement('button'); + applyButton.type = 'button'; + applyButton.textContent = Object.hasOwn(currentFields, fieldName) ? 'Use suggestion' : 'Add field'; + applyButton.setAttribute('aria-label', `Apply suggested value for ${fieldName}`); + applyButton.addEventListener('click', () => { + const existingRow = [...additionalFields.querySelectorAll('.additional-field-row')] + .find((row) => row.querySelector('.additional-field-label')?.value === fieldName); + if (existingRow) { + const valueSelect = existingRow.querySelector('.additional-field-value'); + if (![...valueSelect.options].some((option) => option.value === value)) { + const option = document.createElement('option'); + option.value = value; + option.textContent = value; + valueSelect.append(option); + } + valueSelect.value = value; + } else { + additionalFields.querySelector('.additional-field-empty')?.remove(); + additionalFields.append(buildAdditionalFieldRow(fieldName, value, fieldSuggestionState)); + } + updateSaveButtonState(); + scheduleModelSuggestions(); + }); + proposal.append(description, ' ', applyButton); + additionalFieldSuggestions.append(proposal); + }); + additionalFieldSuggestions.setAttribute( + 'aria-hidden', + String(!additionalFieldSuggestions.childElementCount), + ); +}; + +const renderModelSuggestion = (fieldName, suggestion) => { + const container = suggestionElements[fieldName]; + if (!container) return; + container.replaceChildren(); + if (!suggestion || typeof suggestion.value !== 'string' || !suggestion.value.trim()) { + container.setAttribute('aria-hidden', 'true'); + return; + } + + const prefix = document.createElement('span'); + prefix.textContent = `Model suggestion: ${suggestion.value}`; + prefix.title = prefix.textContent; + container.append(prefix); + + const controls = { + template_name: templateSelect, + period_format: periodFormatSelect, + emitting_company: companyInput, + document_type: documentTypeInput, + }; + const control = controls[fieldName]; + if (control) { + const applyButton = document.createElement('button'); + applyButton.type = 'button'; + applyButton.textContent = 'Use suggestion'; + applyButton.setAttribute('aria-label', `Use model suggestion for ${fieldName.replaceAll('_', ' ')}`); + applyButton.addEventListener('click', () => { + const options = [...control.options].map((option) => option.value); + if (control.tagName === 'SELECT' && !options.includes(suggestion.value)) { + const option = document.createElement('option'); + option.value = suggestion.value; + option.textContent = suggestion.value; + control.append(option); + } + control.value = suggestion.value; + control.dispatchEvent(new Event('change', { bubbles: true })); + updateSaveButtonState(); + scheduleModelSuggestions(); + }); + container.append(' ', applyButton); + } + container.setAttribute('aria-hidden', 'false'); +}; + +const renderEmissionDateSuggestion = (suggestion) => { + if (!emissionDateSuggestion) return; + emissionDateSuggestion.replaceChildren(); + if (!suggestion || typeof suggestion.value !== 'string' || !suggestion.value.trim()) { + emissionDateSuggestion.setAttribute('aria-hidden', 'true'); + return; + } + + const recommendedDate = formatEmissionDateInput(suggestion.value); + const label = document.createElement('span'); + label.textContent = `Content recommendation: ${recommendedDate}`; + label.title = label.textContent; + emissionDateSuggestion.append(label); + + const applyButton = document.createElement('button'); + applyButton.type = 'button'; + applyButton.textContent = 'Use recommendation'; + applyButton.setAttribute('aria-label', 'Use recommended emission date extracted from document content'); + applyButton.addEventListener('click', () => { + emissionDateInput.value = recommendedDate; + emissionDateInput.dispatchEvent(new Event('input', { bubbles: true })); + emissionDateInput.dispatchEvent(new Event('change', { bubbles: true })); + updateSaveButtonState(); + scheduleModelSuggestions(); + }); + emissionDateSuggestion.append(' ', applyButton); + emissionDateSuggestion.setAttribute('aria-hidden', 'false'); +}; + +const scheduleModelSuggestions = () => { + if (!currentFileId) return; + hideModelSuggestions(); + suggestionRequestSequence += 1; + if (suggestionAbortController) suggestionAbortController.abort(); + if (suggestionTimer) clearTimeout(suggestionTimer); + suggestionTimer = setTimeout(() => { + fetchModelSuggestions(); + }, 250); +}; + +const fetchModelSuggestions = async () => { + if (!currentFileId) return; + const sequence = ++suggestionRequestSequence; + if (suggestionAbortController) suggestionAbortController.abort(); + suggestionAbortController = new AbortController(); + + const payload = { + template_name: templateSelect.value, + period_format: periodFormatSelect.value, + emitting_company: companyInput.value.trim(), + document_type: documentTypeInput.value.trim(), + emission_date: parseEmissionDateInput(emissionDateInput.value), + optional_fields: getAdditionalFieldEntries(), + }; + try { + const response = await fetch(`/files/${encodeURIComponent(currentFileId)}/suggestions`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(payload), + signal: suggestionAbortController.signal, + }); + if (!response.ok) { + console.debug( + `Document suggestions request failed (HTTP ${response.status}) for file ${currentFileId}.`, + ); + return; + } + const result = await response.json(); + if (sequence !== suggestionRequestSequence || !result || !result.suggestions) return; + if (!result.suggestions.emission_date) { + console.debug( + `No emission date recommendation returned for file ${currentFileId}; keeping any stored content-extracted date visible.`, + ); + } + Object.keys(suggestionElements).forEach((fieldName) => { + renderModelSuggestion(fieldName, result.suggestions[fieldName]); + }); + renderAdditionalFieldSuggestions(result.suggestions.optional_fields); + renderEmissionDateSuggestion(result.suggestions.emission_date); + const calculatedPeriod = result.suggestions.period?.value; + const calculatedKey = result.suggestions.current_key?.value; + periodPreview.textContent = calculatedPeriod || '—'; + keyPreview.textContent = calculatedKey || '—'; + } catch (error) { + if (error.name !== 'AbortError') console.debug('Document model suggestions are unavailable.', error); + } +}; + const saveDocument = async (event) => { event.preventDefault(); if (!currentFileId || saveButton.disabled) return; @@ -372,7 +587,7 @@ const saveDocument = async (event) => { optional_fields: getAdditionalFieldEntries(), }; try { - const response = await fetch(`/api/files/${encodeURIComponent(currentFileId)}/edit`, { + const response = await fetch(`/files/${encodeURIComponent(currentFileId)}/edit`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(payload), @@ -380,6 +595,7 @@ const saveDocument = async (event) => { const result = await response.json(); if (!response.ok) throw new Error(result.detail || `HTTP ${response.status}`); renderDocument(result, currentFileId); + scheduleModelSuggestions(); editStatus.textContent = 'Changes saved. The document is validated.'; updateSaveButtonState(); await loadDocumentNavigation(currentFileId); @@ -397,13 +613,26 @@ if (addAdditionalFieldButton) { if (emptyState) emptyState.remove(); additionalFields.append(buildAdditionalFieldRow('', '', fieldSuggestionState)); updateSaveButtonState(); + scheduleModelSuggestions(); }); } -editForm.addEventListener('input', updateSaveButtonState); -editForm.addEventListener('change', updateSaveButtonState); -additionalFields.addEventListener('input', updateSaveButtonState); -additionalFields.addEventListener('change', updateSaveButtonState); +editForm.addEventListener('input', () => { + updateSaveButtonState(); + scheduleModelSuggestions(); +}); +editForm.addEventListener('change', () => { + updateSaveButtonState(); + scheduleModelSuggestions(); +}); +additionalFields.addEventListener('input', () => { + updateSaveButtonState(); + scheduleModelSuggestions(); +}); +additionalFields.addEventListener('change', () => { + updateSaveButtonState(); + scheduleModelSuggestions(); +}); const loadDocument = async () => { const fileId = new URLSearchParams(window.location.search).get('id'); @@ -415,8 +644,8 @@ const loadDocument = async () => { try { const [documentResponse, optionsResponse] = await Promise.all([ - fetch(`/api/files/${encodeURIComponent(fileId)}`), - fetch('/api/document-options'), + fetch(`/files/${encodeURIComponent(fileId)}`), + fetch('/document-options'), ]); if (!documentResponse.ok) throw new Error(`HTTP ${documentResponse.status}`); if (!optionsResponse.ok) throw new Error(`Unable to load edit options (HTTP ${optionsResponse.status})`); @@ -434,7 +663,15 @@ const loadDocument = async () => { setSelectOptions(documentTypeInput, options.document_types || [], documentData.document_type); renderDocument(documentData, fileId); renderAdditionalFields(documentData.optional_fields || {}, fieldSuggestionState); + // Show the date already extracted during ingestion immediately; the async + // suggestions response will replace it if a fresh regex extraction succeeds. + renderEmissionDateSuggestion( + documentData.emission_date + ? { value: documentData.emission_date, source: 'content' } + : null, + ); updateSaveButtonState(); + scheduleModelSuggestions(); await loadDocumentNavigation(fileId); } catch (error) { title.textContent = 'Unable to load document'; diff --git a/automatic-pdf-renamer/web/maintenance.html b/automatic-pdf-renamer/web/maintenance.html index 6fd084f..b3b4eb2 100644 --- a/automatic-pdf-renamer/web/maintenance.html +++ b/automatic-pdf-renamer/web/maintenance.html @@ -28,6 +28,19 @@
+
+
+

Machine learning models

+

Retrain the complete model set from validated documents.

+

+
+ + +
+

Rebuild database

diff --git a/automatic-pdf-renamer/web/maintenance.js b/automatic-pdf-renamer/web/maintenance.js index c71a555..88e0946 100644 --- a/automatic-pdf-renamer/web/maintenance.js +++ b/automatic-pdf-renamer/web/maintenance.js @@ -30,7 +30,7 @@ rebuildButton.addEventListener('click', () => runAction({ operation: document.getElementById('rebuild-operation'), progress: document.getElementById('rebuild-progress'), status: document.getElementById('rebuild-status'), - endpoint: '/api/consistency/rebuild', + endpoint: '/consistency/rebuild', method: 'POST', confirmation: 'Rebuild the database from sidecar files? Existing database records will be replaced.', getResult: (result) => `Database rebuilt from ${result.count} sidecar file${result.count === 1 ? '' : 's'}.`, @@ -40,10 +40,65 @@ checkButton.addEventListener('click', () => runAction({ operation: document.getElementById('check-operation'), progress: document.getElementById('check-progress'), status: document.getElementById('check-status'), - endpoint: '/api/consistency', + endpoint: '/consistency', method: 'GET', confirmation: 'Run a consistency check? This scans metadata and database records against storage.', getResult: (anomalies) => anomalies.length ? `Consistency check complete: ${anomalies.length} issue${anomalies.length === 1 ? '' : 's'} found.` : 'Consistency check complete: no issues found.', -})); \ No newline at end of file +})); + +const trainModelButton = document.getElementById('train-model-button'); +const modelTrainingOperation = document.getElementById('model-training-operation'); +const modelTrainingProgress = document.getElementById('model-training-progress'); +const modelsStatus = document.getElementById('models-status'); +const modelsInventoryStatus = document.getElementById('models-inventory-status'); + +const sleep = (milliseconds) => new Promise((resolve) => setTimeout(resolve, milliseconds)); + +const pollModelTraining = async (jobId) => { + while (true) { + const response = await fetch(`/jobs/${encodeURIComponent(jobId)}`); + const result = await response.json(); + if (!response.ok) throw new Error(result.detail || `HTTP ${response.status}`); + + modelTrainingProgress.value = Number(result.progress || 0); + modelsStatus.textContent = result.message || result.stage || 'Training models…'; + + if (result.status === 'completed') { + modelTrainingProgress.value = 100; + modelsInventoryStatus.textContent = 'All models were replaced successfully. Previous model artifacts have been removed.'; + return; + } + if (result.status === 'failed') { + throw new Error(result.message || 'Model retraining failed.'); + } + await sleep(1000); + } +}; + +trainModelButton.addEventListener('click', async () => { + if (!window.confirm('Retrain all models from validated documents? The current models will be replaced after the complete training succeeds.')) return; + + trainModelButton.disabled = true; + actionButtons.forEach((button) => { button.disabled = true; }); + modelTrainingOperation.hidden = false; + modelTrainingProgress.value = 0; + modelsInventoryStatus.textContent = ''; + modelsStatus.textContent = 'Starting full model retraining…'; + + try { + const response = await fetch('/models/train', { method: 'POST' }); + const result = await response.json(); + if (!response.ok) throw new Error(result.detail || `HTTP ${response.status}`); + await pollModelTraining(result.job_id); + modelsStatus.textContent = 'Model retraining completed.'; + } catch (error) { + modelsStatus.textContent = `Model retraining failed: ${error.message}`; + modelsInventoryStatus.textContent = 'The previous model set should remain active if training failed before replacement.'; + console.error(error); + } finally { + trainModelButton.disabled = false; + actionButtons.forEach((button) => { button.disabled = false; }); + } +}); diff --git a/automatic-pdf-renamer/web/styles.css b/automatic-pdf-renamer/web/styles.css index 6012808..8e7b734 100644 --- a/automatic-pdf-renamer/web/styles.css +++ b/automatic-pdf-renamer/web/styles.css @@ -770,3 +770,154 @@ body { justify-content: flex-start; } } + + +.model-card-heading { + display: flex; + align-items: flex-start; + justify-content: space-between; + gap: 16px; +} + +.model-card-heading h4 { + margin: 0 0 6px; +} + +.model-card-heading p, +.model-help, +.model-empty { + color: var(--muted); + line-height: 1.5; + margin: 0; +} + +.model-help { + font-size: 0.85rem; +} + +.models-list { + display: grid; + grid-template-columns: repeat(auto-fit, minmax(280px, 1fr)); + gap: 12px; + margin-top: 14px; +} + +.model-card { + border: 1px solid var(--border); + border-radius: 10px; + padding: 14px; + min-width: 0; +} + +.model-card-heading { + margin-bottom: 12px; +} + +.model-card-heading h4 { + overflow-wrap: anywhere; +} + +.model-card-heading p { + font-size: 0.84rem; +} + +.model-count { + flex-shrink: 0; + color: var(--muted); + font-size: 0.8rem; +} + +.model-versions { + border-top: 1px solid var(--border); +} + +.model-version { + display: flex; + justify-content: space-between; + align-items: center; + gap: 10px; + padding: 10px 0; + border-bottom: 1px solid var(--border); +} + +.model-version:last-child { + border-bottom: 0; +} + +.model-version code { + overflow-wrap: anywhere; + font-size: 0.8rem; +} + +.model-active-tag { + display: inline-block; + color: var(--success); + background: #e9f7ef; + border-radius: 999px; + padding: 3px 7px; + font-size: 0.72rem; + font-weight: 700; +} + +@media (max-width: 820px) { + .model-card-heading { + flex-direction: column; + } +} + + +/* Compact, asynchronously loaded model proposals under editable document fields. */ +.model-suggestion { + display: flex; + align-items: center; + gap: 0.35rem; + box-sizing: border-box; + height: 1.6rem; + min-width: 0; + margin-top: 0.25rem; + color: var(--muted, #667085); + font-size: 0.75rem; + line-height: 1.35; + white-space: nowrap; + overflow: hidden; +} + +.model-suggestion[aria-hidden="true"] { + visibility: hidden; +} + +.model-suggestion > span, +.additional-field-proposal > span { + min-width: 0; + overflow: hidden; + text-overflow: ellipsis; +} + +.model-suggestion button { + flex: 0 0 auto; + margin: 0; + padding: 0; + border: 0; + background: transparent; + color: var(--accent, #3157a4); + font: inherit; + text-decoration: underline; + cursor: pointer; +} + +#additional-field-suggestions { + overflow-x: auto; +} + +.additional-field-proposal { + display: flex; + flex: 0 0 auto; + align-items: center; + gap: 0.35rem; + min-width: 0; +} + +.model-suggestion button:focus-visible { + outline: 2px solid currentColor; + outline-offset: 2px; +}