Files
multiarch-docker-images/automatic-pdf-renamer/app/storage/__init__.py
T
72f8254e10 US0: Paperless-ngx Initial Import - Reusable Components + Import Script (#9)
* Add sidecar.py module for User Story 0 (Paperless-ngx import)

Implements the sidecar data model, schema validation, atomic writes,
and I/O helpers per PRD section 7.

- Sidecar and SidecarEvent dataclasses with full PRD schema
- SidecarStatus and EventType enums
- build_sidecar(), serialize(), deserialize() functions
- validate_sidecar() with type and required field checking
- get_sidecar_key() for .meta.json suffix generation
- write_atomic() for crash-safe temp-key + copy + delete pattern
- read_sidecar(), sidecar_exists(), matching_sha256() for storage I/O

This is a reusable component for both the import script and main service.

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Add storage/ module with S3 and File backends

Implements the storage layer abstraction for User Story 0.

- StorageBackend ABC: defines interface for all storage implementations
- S3Storage: boto3-based backend for Garage S3 (production)
- FileStorage: filesystem-based backend for testing
- get_storage_backend() factory function
- Atomic operations: put, get, delete, copy, exists, list, head

Both backends implement the same interface, allowing seamless
switching between production S3 and local testing.

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Add period.py, extractor.py, and config.py reusable modules

- period.py: Period parsing from filenames and computation from dates
  - parse_period_from_filename(): Extracts month-year, quarter, year, date patterns
  - compute_period_from_date(): Computes period per template rules (granularity + offset)
  - get_period(): Main entry point with filename fallback to rule-based computation
  - validate_period_consistency(): Checks period vs emission_date + rule
  - Supports French month names (août, janvier, etc.)

- extractor.py: PDF text extraction using PyMuPDF
  - extract_first_page_text(): Extracts text from first PDF page
  - extract_and_validate(): Extracts and checks minimum text length
  - TextExtractor class for dependency injection and mocking

- config.py: Configuration management
  - S3Config, PaperlessConfig, PollerConfig, ClassifierConfig dataclasses
  - AppConfig: Main application configuration from env vars
  - TemplateRegistry: Loads and manages template YAML configurations
  - load_config(), load_templates() helper functions

All modules are reusable by both import script and main service.

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Add import_paperless.py script for User Story 0

Implements the standalone import script that:
- Connects to Paperless-ngx REST API (PAPERLESS_URL + token from env)
- Connects to Garage S3 bucket via boto3 (S3_ENDPOINT_URL, credentials, bucket)
- Lists all documents from Paperless via paginated /api/documents/
- Fetches document metadata and maps to sidecar fields:
  - template: first path segment of storage_path
  - emission_date: document created field
  - emitting_company: correspondent name
  - document_type: document type name
  - period: parsed from filename tail, fallback to month-year from emission_date
  - extracted_text: first page text via PyMuPDF
  - optional_fields: from Paperless labels
- Downloads PDF via /api/documents/{id}/download/
- Stores PDF at files/{template}/{original filename}
- Writes sidecar {key}.meta.json with atomic temp-key + copy pattern
- Supports resumability (skip if PDF + sidecar exist with matching SHA256)
- Dry-run flag for testing
- Limit flag for trial runs
- Summary reporting at end
- Exit code: 0 if all succeed, 1 if any failures

Dependencies: requests, boto3, pymupdf, python-dateutil

Usage: python scripts/import_paperless.py [--dry-run] [--limit N] [--verbose]
Environment: PAPERLESS_URL, PAPERLESS_TOKEN, S3_ENDPOINT_URL, S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY, S3_BUCKET_NAME

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Add requirements.txt for import script dependencies

Lists all Python dependencies needed to run scripts/import_paperless.py:
- requests: Paperless-ngx API client
- boto3: Garage S3 client
- pymupdf: PDF text extraction
- python-dateutil: Date parsing
- PyYAML: YAML config parsing (optional)

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Update import script: add storage backend selection (s3/file)

- Added --storage-backend argument (choices: s3, file, default: s3)
- Added --file-base-dir argument for file storage
- Storage backend validation based on selection
- Updated help text and epilog with backend-specific env vars
- Importer class now accepts storage_backend and file_base_dir params
- _init_storage() method handles backend initialization
- Summary now shows which storage backend was used

Usage examples:
  # S3 storage (default)
  python scripts/import_paperless.py

  # File storage for testing
  python scripts/import_paperless.py --storage-backend file --file-base-dir ./test_storage

  # Explicit S3
  python scripts/import_paperless.py --storage-backend s3 --s3-endpoint ... --s3-bucket ...

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Add README.md for scripts directory

Comprehensive documentation for import_paperless.py script:
- Overview and features
- Requirements and installation
- Usage examples (env vars + CLI args)
- Storage backends (S3 vs file)
- Metadata mapping table
- Output structure
- Resumability notes
- Exit codes
- Example output
- Troubleshooting guide
- Development/testing notes
- Architecture overview

Helps users understand how to use the import script for User Story 0.

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Fix Paperless API field mapping

The Paperless-ngx API returns IDs for correspondent, document_type, and tags,
not the actual string values. This commit fixes the import script to:

- Use 'archived_file_name' instead of 'filename' for the actual filename
- Fetch correspondent name from /api/correspondents/{id}/ endpoint
- Fetch document type name from /api/document_types/{id}/ endpoint
- Fetch tag names from /api/tags/{id}/ endpoint
- Add caching for fetched entities to avoid duplicate API requests

This ensures the sidecar metadata contains actual human-readable strings
instead of numeric IDs.

Changes:
- PaperlessClient: Added get_correspondent_name(), get_document_type_name(), get_tag_name()
- PaperlessClient: Added caches for fetched entities
- get_all_documents(): Now resolves IDs to names for all fields
- Updated module docstring to document this behavior

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Refactor PaperlessClient: change labels to list, add storage path caching, and improve document fetching logic

---------

Co-authored-by: Vibe Nuage Agent <vibe@mistral.ai>
Co-authored-by: napnap75 <napnap75@users.noreply.github.com>
2026-10-03 10:05:49 +02:00

120 lines
3.0 KiB
Python

"""Storage layer for Automatic PDF Renamer.
Provides a unified interface for object storage operations.
Implementations:
- S3Storage: For Garage S3 (production)
- FileStorage: For local filesystem (testing)
"""
from abc import ABC, abstractmethod
from typing import Any
class StorageBackend(ABC):
"""Abstract base class for storage backends.
All storage implementations must provide these methods.
"""
@abstractmethod
def put_object(self, key: str, body: str | bytes) -> None:
"""Upload an object to the storage.
Args:
key: The object key/path.
body: The object content as string or bytes.
"""
pass
@abstractmethod
def get_object(self, key: str) -> str | bytes | None:
"""Retrieve an object from storage.
Args:
key: The object key/path.
Returns:
The object content as string or bytes, or None if not found.
"""
pass
@abstractmethod
def delete_object(self, key: str) -> None:
"""Delete an object from storage.
Args:
key: The object key/path.
"""
pass
@abstractmethod
def copy_object(self, src_key: str, dst_key: str) -> None:
"""Copy an object within the storage.
Args:
src_key: Source object key/path.
dst_key: Destination object key/path.
"""
pass
@abstractmethod
def object_exists(self, key: str) -> bool:
"""Check if an object exists in storage.
Args:
key: The object key/path.
Returns:
True if the object exists, False otherwise.
"""
pass
@abstractmethod
def list_objects(self, prefix: str = "") -> list[str]:
"""List all object keys under a prefix.
Args:
prefix: The prefix to filter by.
Returns:
List of object keys (full paths).
"""
pass
@abstractmethod
def head_object(self, key: str) -> dict[str, Any] | None:
"""Get metadata for an object.
Args:
key: The object key/path.
Returns:
Dict with metadata (e.g., size, etag, content_type), or None if not found.
"""
pass
# Convenience function to get the appropriate storage backend
# This will be configured by the application
def get_storage_backend(backend_type: str, **kwargs) -> StorageBackend:
"""Factory function to get a storage backend instance.
Args:
backend_type: Either 's3' or 'file'.
**kwargs: Backend-specific configuration.
Returns:
A StorageBackend instance.
Raises:
ValueError: If backend_type is not recognized.
"""
if backend_type == "s3":
from .s3_storage import S3Storage
return S3Storage(**kwargs)
elif backend_type == "file":
from .file_storage import FileStorage
return FileStorage(**kwargs)
else:
raise ValueError(f"Unknown storage backend type: {backend_type}")