Files
multiarch-docker-images/automatic-pdf-renamer/app/extractor.py
T
72f8254e10 US0: Paperless-ngx Initial Import - Reusable Components + Import Script (#9)
* Add sidecar.py module for User Story 0 (Paperless-ngx import)

Implements the sidecar data model, schema validation, atomic writes,
and I/O helpers per PRD section 7.

- Sidecar and SidecarEvent dataclasses with full PRD schema
- SidecarStatus and EventType enums
- build_sidecar(), serialize(), deserialize() functions
- validate_sidecar() with type and required field checking
- get_sidecar_key() for .meta.json suffix generation
- write_atomic() for crash-safe temp-key + copy + delete pattern
- read_sidecar(), sidecar_exists(), matching_sha256() for storage I/O

This is a reusable component for both the import script and main service.

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Add storage/ module with S3 and File backends

Implements the storage layer abstraction for User Story 0.

- StorageBackend ABC: defines interface for all storage implementations
- S3Storage: boto3-based backend for Garage S3 (production)
- FileStorage: filesystem-based backend for testing
- get_storage_backend() factory function
- Atomic operations: put, get, delete, copy, exists, list, head

Both backends implement the same interface, allowing seamless
switching between production S3 and local testing.

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Add period.py, extractor.py, and config.py reusable modules

- period.py: Period parsing from filenames and computation from dates
  - parse_period_from_filename(): Extracts month-year, quarter, year, date patterns
  - compute_period_from_date(): Computes period per template rules (granularity + offset)
  - get_period(): Main entry point with filename fallback to rule-based computation
  - validate_period_consistency(): Checks period vs emission_date + rule
  - Supports French month names (août, janvier, etc.)

- extractor.py: PDF text extraction using PyMuPDF
  - extract_first_page_text(): Extracts text from first PDF page
  - extract_and_validate(): Extracts and checks minimum text length
  - TextExtractor class for dependency injection and mocking

- config.py: Configuration management
  - S3Config, PaperlessConfig, PollerConfig, ClassifierConfig dataclasses
  - AppConfig: Main application configuration from env vars
  - TemplateRegistry: Loads and manages template YAML configurations
  - load_config(), load_templates() helper functions

All modules are reusable by both import script and main service.

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Add import_paperless.py script for User Story 0

Implements the standalone import script that:
- Connects to Paperless-ngx REST API (PAPERLESS_URL + token from env)
- Connects to Garage S3 bucket via boto3 (S3_ENDPOINT_URL, credentials, bucket)
- Lists all documents from Paperless via paginated /api/documents/
- Fetches document metadata and maps to sidecar fields:
  - template: first path segment of storage_path
  - emission_date: document created field
  - emitting_company: correspondent name
  - document_type: document type name
  - period: parsed from filename tail, fallback to month-year from emission_date
  - extracted_text: first page text via PyMuPDF
  - optional_fields: from Paperless labels
- Downloads PDF via /api/documents/{id}/download/
- Stores PDF at files/{template}/{original filename}
- Writes sidecar {key}.meta.json with atomic temp-key + copy pattern
- Supports resumability (skip if PDF + sidecar exist with matching SHA256)
- Dry-run flag for testing
- Limit flag for trial runs
- Summary reporting at end
- Exit code: 0 if all succeed, 1 if any failures

Dependencies: requests, boto3, pymupdf, python-dateutil

Usage: python scripts/import_paperless.py [--dry-run] [--limit N] [--verbose]
Environment: PAPERLESS_URL, PAPERLESS_TOKEN, S3_ENDPOINT_URL, S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY, S3_BUCKET_NAME

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Add requirements.txt for import script dependencies

Lists all Python dependencies needed to run scripts/import_paperless.py:
- requests: Paperless-ngx API client
- boto3: Garage S3 client
- pymupdf: PDF text extraction
- python-dateutil: Date parsing
- PyYAML: YAML config parsing (optional)

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Update import script: add storage backend selection (s3/file)

- Added --storage-backend argument (choices: s3, file, default: s3)
- Added --file-base-dir argument for file storage
- Storage backend validation based on selection
- Updated help text and epilog with backend-specific env vars
- Importer class now accepts storage_backend and file_base_dir params
- _init_storage() method handles backend initialization
- Summary now shows which storage backend was used

Usage examples:
  # S3 storage (default)
  python scripts/import_paperless.py

  # File storage for testing
  python scripts/import_paperless.py --storage-backend file --file-base-dir ./test_storage

  # Explicit S3
  python scripts/import_paperless.py --storage-backend s3 --s3-endpoint ... --s3-bucket ...

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Add README.md for scripts directory

Comprehensive documentation for import_paperless.py script:
- Overview and features
- Requirements and installation
- Usage examples (env vars + CLI args)
- Storage backends (S3 vs file)
- Metadata mapping table
- Output structure
- Resumability notes
- Exit codes
- Example output
- Troubleshooting guide
- Development/testing notes
- Architecture overview

Helps users understand how to use the import script for User Story 0.

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Fix Paperless API field mapping

The Paperless-ngx API returns IDs for correspondent, document_type, and tags,
not the actual string values. This commit fixes the import script to:

- Use 'archived_file_name' instead of 'filename' for the actual filename
- Fetch correspondent name from /api/correspondents/{id}/ endpoint
- Fetch document type name from /api/document_types/{id}/ endpoint
- Fetch tag names from /api/tags/{id}/ endpoint
- Add caching for fetched entities to avoid duplicate API requests

This ensures the sidecar metadata contains actual human-readable strings
instead of numeric IDs.

Changes:
- PaperlessClient: Added get_correspondent_name(), get_document_type_name(), get_tag_name()
- PaperlessClient: Added caches for fetched entities
- get_all_documents(): Now resolves IDs to names for all fields
- Updated module docstring to document this behavior

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Refactor PaperlessClient: change labels to list, add storage path caching, and improve document fetching logic

---------

Co-authored-by: Vibe Nuage Agent <vibe@mistral.ai>
Co-authored-by: napnap75 <napnap75@users.noreply.github.com>
2026-10-03 10:05:49 +02:00

176 lines
4.5 KiB
Python

"""Text extraction module for Automatic PDF Renamer.
Provides PDF text extraction using PyMuPDF (PRD FR2.1, FR2.2).
This is the same extraction used by both the import script and the service.
"""
import io
import logging
from typing import Any
logger = logging.getLogger(__name__)
# Minimum text length threshold (PRD FR2.3)
MIN_TEXT_LENGTH = 10
class TextExtractionError(Exception):
"""Error raised when text extraction fails."""
pass
def extract_first_page_text(pdf_bytes: bytes) -> str:
"""Extract text from the first page of a PDF.
Uses PyMuPDF for fast text extraction (PRD stack decision).
Extracts only the first page as per FR2.2.
Args:
pdf_bytes: The PDF file content as bytes.
Returns:
The extracted text from the first page.
Raises:
TextExtractionError: If extraction fails or PDF is invalid.
"""
try:
import fitz # PyMuPDF
except ImportError:
raise TextExtractionError(
"PyMuPDF is required for text extraction. Install with: pip install pymupdf"
)
try:
# Open PDF from bytes
doc = fitz.open(stream=pdf_bytes)
if len(doc) == 0:
logger.warning("PDF has no pages")
return ""
# Extract text from first page only (FR2.2)
first_page = doc[0]
text = first_page.get_text()
doc.close()
return text
except Exception as e:
logger.error(f"Failed to extract text from PDF: {e}")
raise TextExtractionError(f"Text extraction failed: {e}") from e
def extract_first_page_text_from_file(filepath: str) -> str:
"""Extract text from the first page of a PDF file.
Args:
filepath: Path to the PDF file.
Returns:
The extracted text from the first page.
Raises:
TextExtractionError: If extraction fails or PDF is invalid.
FileNotFoundError: If the file does not exist.
"""
try:
with open(filepath, "rb") as f:
pdf_bytes = f.read()
except FileNotFoundError:
raise FileNotFoundError(f"PDF file not found: {filepath}")
return extract_first_page_text(pdf_bytes)
def is_text_empty(text: str, min_length: int = MIN_TEXT_LENGTH) -> bool:
"""Check if extracted text is effectively empty.
Considers text empty if:
- It's an empty string
- It's only whitespace
- It's shorter than min_length characters
Args:
text: The text to check.
min_length: Minimum length threshold (default: 10).
Returns:
True if text is considered empty, False otherwise.
"""
if not text:
return True
if not text.strip():
return True
if len(text.strip()) < min_length:
return True
return False
def extract_and_validate(pdf_bytes: bytes, min_length: int = MIN_TEXT_LENGTH) -> tuple[str, bool]:
"""Extract first page text and check if it's valid.
Args:
pdf_bytes: The PDF file content as bytes.
min_length: Minimum text length threshold.
Returns:
Tuple of (extracted_text, is_valid).
is_valid is False if text is empty or below min_length.
Raises:
TextExtractionError: If extraction fails.
"""
text = extract_first_page_text(pdf_bytes)
is_valid = not is_text_empty(text, min_length)
return (text, is_valid)
class TextExtractor:
"""Text extractor class for dependency injection.
Wraps the extraction functions to allow for mocking in tests
and potential future swapping of extraction libraries.
"""
def __init__(self, min_length: int = MIN_TEXT_LENGTH):
"""Initialize the text extractor.
Args:
min_length: Minimum text length threshold.
"""
self.min_length = min_length
def extract(self, pdf_bytes: bytes) -> str:
"""Extract first page text from PDF bytes.
Args:
pdf_bytes: The PDF file content as bytes.
Returns:
The extracted text.
"""
return extract_first_page_text(pdf_bytes)
def extract_and_validate(self, pdf_bytes: bytes) -> tuple[str, bool]:
"""Extract and validate text.
Args:
pdf_bytes: The PDF file content as bytes.
Returns:
Tuple of (text, is_valid).
"""
return extract_and_validate(pdf_bytes, self.min_length)
def is_empty(self, text: str) -> bool:
"""Check if text is empty.
Args:
text: The text to check.
Returns:
True if text is considered empty.
"""
return is_text_empty(text, self.min_length)