mirror of
https://github.com/napnap75/multiarch-docker-images.git
synced 2026-11-11 00:22:12 +01:00
* Add sidecar.py module for User Story 0 (Paperless-ngx import) Implements the sidecar data model, schema validation, atomic writes, and I/O helpers per PRD section 7. - Sidecar and SidecarEvent dataclasses with full PRD schema - SidecarStatus and EventType enums - build_sidecar(), serialize(), deserialize() functions - validate_sidecar() with type and required field checking - get_sidecar_key() for .meta.json suffix generation - write_atomic() for crash-safe temp-key + copy + delete pattern - read_sidecar(), sidecar_exists(), matching_sha256() for storage I/O This is a reusable component for both the import script and main service. Co-authored-by: napnap75 <napnap75@users.noreply.github.com> * Add storage/ module with S3 and File backends Implements the storage layer abstraction for User Story 0. - StorageBackend ABC: defines interface for all storage implementations - S3Storage: boto3-based backend for Garage S3 (production) - FileStorage: filesystem-based backend for testing - get_storage_backend() factory function - Atomic operations: put, get, delete, copy, exists, list, head Both backends implement the same interface, allowing seamless switching between production S3 and local testing. Co-authored-by: napnap75 <napnap75@users.noreply.github.com> * Add period.py, extractor.py, and config.py reusable modules - period.py: Period parsing from filenames and computation from dates - parse_period_from_filename(): Extracts month-year, quarter, year, date patterns - compute_period_from_date(): Computes period per template rules (granularity + offset) - get_period(): Main entry point with filename fallback to rule-based computation - validate_period_consistency(): Checks period vs emission_date + rule - Supports French month names (août, janvier, etc.) - extractor.py: PDF text extraction using PyMuPDF - extract_first_page_text(): Extracts text from first PDF page - extract_and_validate(): Extracts and checks minimum text length - TextExtractor class for dependency injection and mocking - config.py: Configuration management - S3Config, PaperlessConfig, PollerConfig, ClassifierConfig dataclasses - AppConfig: Main application configuration from env vars - TemplateRegistry: Loads and manages template YAML configurations - load_config(), load_templates() helper functions All modules are reusable by both import script and main service. Co-authored-by: napnap75 <napnap75@users.noreply.github.com> * Add import_paperless.py script for User Story 0 Implements the standalone import script that: - Connects to Paperless-ngx REST API (PAPERLESS_URL + token from env) - Connects to Garage S3 bucket via boto3 (S3_ENDPOINT_URL, credentials, bucket) - Lists all documents from Paperless via paginated /api/documents/ - Fetches document metadata and maps to sidecar fields: - template: first path segment of storage_path - emission_date: document created field - emitting_company: correspondent name - document_type: document type name - period: parsed from filename tail, fallback to month-year from emission_date - extracted_text: first page text via PyMuPDF - optional_fields: from Paperless labels - Downloads PDF via /api/documents/{id}/download/ - Stores PDF at files/{template}/{original filename} - Writes sidecar {key}.meta.json with atomic temp-key + copy pattern - Supports resumability (skip if PDF + sidecar exist with matching SHA256) - Dry-run flag for testing - Limit flag for trial runs - Summary reporting at end - Exit code: 0 if all succeed, 1 if any failures Dependencies: requests, boto3, pymupdf, python-dateutil Usage: python scripts/import_paperless.py [--dry-run] [--limit N] [--verbose] Environment: PAPERLESS_URL, PAPERLESS_TOKEN, S3_ENDPOINT_URL, S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY, S3_BUCKET_NAME Co-authored-by: napnap75 <napnap75@users.noreply.github.com> * Add requirements.txt for import script dependencies Lists all Python dependencies needed to run scripts/import_paperless.py: - requests: Paperless-ngx API client - boto3: Garage S3 client - pymupdf: PDF text extraction - python-dateutil: Date parsing - PyYAML: YAML config parsing (optional) Co-authored-by: napnap75 <napnap75@users.noreply.github.com> * Update import script: add storage backend selection (s3/file) - Added --storage-backend argument (choices: s3, file, default: s3) - Added --file-base-dir argument for file storage - Storage backend validation based on selection - Updated help text and epilog with backend-specific env vars - Importer class now accepts storage_backend and file_base_dir params - _init_storage() method handles backend initialization - Summary now shows which storage backend was used Usage examples: # S3 storage (default) python scripts/import_paperless.py # File storage for testing python scripts/import_paperless.py --storage-backend file --file-base-dir ./test_storage # Explicit S3 python scripts/import_paperless.py --storage-backend s3 --s3-endpoint ... --s3-bucket ... Co-authored-by: napnap75 <napnap75@users.noreply.github.com> * Add README.md for scripts directory Comprehensive documentation for import_paperless.py script: - Overview and features - Requirements and installation - Usage examples (env vars + CLI args) - Storage backends (S3 vs file) - Metadata mapping table - Output structure - Resumability notes - Exit codes - Example output - Troubleshooting guide - Development/testing notes - Architecture overview Helps users understand how to use the import script for User Story 0. Co-authored-by: napnap75 <napnap75@users.noreply.github.com> * Fix Paperless API field mapping The Paperless-ngx API returns IDs for correspondent, document_type, and tags, not the actual string values. This commit fixes the import script to: - Use 'archived_file_name' instead of 'filename' for the actual filename - Fetch correspondent name from /api/correspondents/{id}/ endpoint - Fetch document type name from /api/document_types/{id}/ endpoint - Fetch tag names from /api/tags/{id}/ endpoint - Add caching for fetched entities to avoid duplicate API requests This ensures the sidecar metadata contains actual human-readable strings instead of numeric IDs. Changes: - PaperlessClient: Added get_correspondent_name(), get_document_type_name(), get_tag_name() - PaperlessClient: Added caches for fetched entities - get_all_documents(): Now resolves IDs to names for all fields - Updated module docstring to document this behavior Co-authored-by: napnap75 <napnap75@users.noreply.github.com> * Refactor PaperlessClient: change labels to list, add storage path caching, and improve document fetching logic --------- Co-authored-by: Vibe Nuage Agent <vibe@mistral.ai> Co-authored-by: napnap75 <napnap75@users.noreply.github.com>
176 lines
4.5 KiB
Python
176 lines
4.5 KiB
Python
"""Text extraction module for Automatic PDF Renamer.
|
|
|
|
Provides PDF text extraction using PyMuPDF (PRD FR2.1, FR2.2).
|
|
This is the same extraction used by both the import script and the service.
|
|
"""
|
|
|
|
import io
|
|
import logging
|
|
from typing import Any
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
# Minimum text length threshold (PRD FR2.3)
|
|
MIN_TEXT_LENGTH = 10
|
|
|
|
|
|
class TextExtractionError(Exception):
|
|
"""Error raised when text extraction fails."""
|
|
pass
|
|
|
|
|
|
def extract_first_page_text(pdf_bytes: bytes) -> str:
|
|
"""Extract text from the first page of a PDF.
|
|
|
|
Uses PyMuPDF for fast text extraction (PRD stack decision).
|
|
Extracts only the first page as per FR2.2.
|
|
|
|
Args:
|
|
pdf_bytes: The PDF file content as bytes.
|
|
|
|
Returns:
|
|
The extracted text from the first page.
|
|
|
|
Raises:
|
|
TextExtractionError: If extraction fails or PDF is invalid.
|
|
"""
|
|
try:
|
|
import fitz # PyMuPDF
|
|
except ImportError:
|
|
raise TextExtractionError(
|
|
"PyMuPDF is required for text extraction. Install with: pip install pymupdf"
|
|
)
|
|
|
|
try:
|
|
# Open PDF from bytes
|
|
doc = fitz.open(stream=pdf_bytes)
|
|
|
|
if len(doc) == 0:
|
|
logger.warning("PDF has no pages")
|
|
return ""
|
|
|
|
# Extract text from first page only (FR2.2)
|
|
first_page = doc[0]
|
|
text = first_page.get_text()
|
|
|
|
doc.close()
|
|
return text
|
|
|
|
except Exception as e:
|
|
logger.error(f"Failed to extract text from PDF: {e}")
|
|
raise TextExtractionError(f"Text extraction failed: {e}") from e
|
|
|
|
|
|
def extract_first_page_text_from_file(filepath: str) -> str:
|
|
"""Extract text from the first page of a PDF file.
|
|
|
|
Args:
|
|
filepath: Path to the PDF file.
|
|
|
|
Returns:
|
|
The extracted text from the first page.
|
|
|
|
Raises:
|
|
TextExtractionError: If extraction fails or PDF is invalid.
|
|
FileNotFoundError: If the file does not exist.
|
|
"""
|
|
try:
|
|
with open(filepath, "rb") as f:
|
|
pdf_bytes = f.read()
|
|
except FileNotFoundError:
|
|
raise FileNotFoundError(f"PDF file not found: {filepath}")
|
|
|
|
return extract_first_page_text(pdf_bytes)
|
|
|
|
|
|
def is_text_empty(text: str, min_length: int = MIN_TEXT_LENGTH) -> bool:
|
|
"""Check if extracted text is effectively empty.
|
|
|
|
Considers text empty if:
|
|
- It's an empty string
|
|
- It's only whitespace
|
|
- It's shorter than min_length characters
|
|
|
|
Args:
|
|
text: The text to check.
|
|
min_length: Minimum length threshold (default: 10).
|
|
|
|
Returns:
|
|
True if text is considered empty, False otherwise.
|
|
"""
|
|
if not text:
|
|
return True
|
|
if not text.strip():
|
|
return True
|
|
if len(text.strip()) < min_length:
|
|
return True
|
|
return False
|
|
|
|
|
|
def extract_and_validate(pdf_bytes: bytes, min_length: int = MIN_TEXT_LENGTH) -> tuple[str, bool]:
|
|
"""Extract first page text and check if it's valid.
|
|
|
|
Args:
|
|
pdf_bytes: The PDF file content as bytes.
|
|
min_length: Minimum text length threshold.
|
|
|
|
Returns:
|
|
Tuple of (extracted_text, is_valid).
|
|
is_valid is False if text is empty or below min_length.
|
|
|
|
Raises:
|
|
TextExtractionError: If extraction fails.
|
|
"""
|
|
text = extract_first_page_text(pdf_bytes)
|
|
is_valid = not is_text_empty(text, min_length)
|
|
return (text, is_valid)
|
|
|
|
|
|
class TextExtractor:
|
|
"""Text extractor class for dependency injection.
|
|
|
|
Wraps the extraction functions to allow for mocking in tests
|
|
and potential future swapping of extraction libraries.
|
|
"""
|
|
|
|
def __init__(self, min_length: int = MIN_TEXT_LENGTH):
|
|
"""Initialize the text extractor.
|
|
|
|
Args:
|
|
min_length: Minimum text length threshold.
|
|
"""
|
|
self.min_length = min_length
|
|
|
|
def extract(self, pdf_bytes: bytes) -> str:
|
|
"""Extract first page text from PDF bytes.
|
|
|
|
Args:
|
|
pdf_bytes: The PDF file content as bytes.
|
|
|
|
Returns:
|
|
The extracted text.
|
|
"""
|
|
return extract_first_page_text(pdf_bytes)
|
|
|
|
def extract_and_validate(self, pdf_bytes: bytes) -> tuple[str, bool]:
|
|
"""Extract and validate text.
|
|
|
|
Args:
|
|
pdf_bytes: The PDF file content as bytes.
|
|
|
|
Returns:
|
|
Tuple of (text, is_valid).
|
|
"""
|
|
return extract_and_validate(pdf_bytes, self.min_length)
|
|
|
|
def is_empty(self, text: str) -> bool:
|
|
"""Check if text is empty.
|
|
|
|
Args:
|
|
text: The text to check.
|
|
|
|
Returns:
|
|
True if text is considered empty.
|
|
"""
|
|
return is_text_empty(text, self.min_length)
|