Files
multiarch-docker-images/automatic-pdf-renamer/app/period.py
T
72f8254e10 US0: Paperless-ngx Initial Import - Reusable Components + Import Script (#9)
* Add sidecar.py module for User Story 0 (Paperless-ngx import)

Implements the sidecar data model, schema validation, atomic writes,
and I/O helpers per PRD section 7.

- Sidecar and SidecarEvent dataclasses with full PRD schema
- SidecarStatus and EventType enums
- build_sidecar(), serialize(), deserialize() functions
- validate_sidecar() with type and required field checking
- get_sidecar_key() for .meta.json suffix generation
- write_atomic() for crash-safe temp-key + copy + delete pattern
- read_sidecar(), sidecar_exists(), matching_sha256() for storage I/O

This is a reusable component for both the import script and main service.

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Add storage/ module with S3 and File backends

Implements the storage layer abstraction for User Story 0.

- StorageBackend ABC: defines interface for all storage implementations
- S3Storage: boto3-based backend for Garage S3 (production)
- FileStorage: filesystem-based backend for testing
- get_storage_backend() factory function
- Atomic operations: put, get, delete, copy, exists, list, head

Both backends implement the same interface, allowing seamless
switching between production S3 and local testing.

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Add period.py, extractor.py, and config.py reusable modules

- period.py: Period parsing from filenames and computation from dates
  - parse_period_from_filename(): Extracts month-year, quarter, year, date patterns
  - compute_period_from_date(): Computes period per template rules (granularity + offset)
  - get_period(): Main entry point with filename fallback to rule-based computation
  - validate_period_consistency(): Checks period vs emission_date + rule
  - Supports French month names (août, janvier, etc.)

- extractor.py: PDF text extraction using PyMuPDF
  - extract_first_page_text(): Extracts text from first PDF page
  - extract_and_validate(): Extracts and checks minimum text length
  - TextExtractor class for dependency injection and mocking

- config.py: Configuration management
  - S3Config, PaperlessConfig, PollerConfig, ClassifierConfig dataclasses
  - AppConfig: Main application configuration from env vars
  - TemplateRegistry: Loads and manages template YAML configurations
  - load_config(), load_templates() helper functions

All modules are reusable by both import script and main service.

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Add import_paperless.py script for User Story 0

Implements the standalone import script that:
- Connects to Paperless-ngx REST API (PAPERLESS_URL + token from env)
- Connects to Garage S3 bucket via boto3 (S3_ENDPOINT_URL, credentials, bucket)
- Lists all documents from Paperless via paginated /api/documents/
- Fetches document metadata and maps to sidecar fields:
  - template: first path segment of storage_path
  - emission_date: document created field
  - emitting_company: correspondent name
  - document_type: document type name
  - period: parsed from filename tail, fallback to month-year from emission_date
  - extracted_text: first page text via PyMuPDF
  - optional_fields: from Paperless labels
- Downloads PDF via /api/documents/{id}/download/
- Stores PDF at files/{template}/{original filename}
- Writes sidecar {key}.meta.json with atomic temp-key + copy pattern
- Supports resumability (skip if PDF + sidecar exist with matching SHA256)
- Dry-run flag for testing
- Limit flag for trial runs
- Summary reporting at end
- Exit code: 0 if all succeed, 1 if any failures

Dependencies: requests, boto3, pymupdf, python-dateutil

Usage: python scripts/import_paperless.py [--dry-run] [--limit N] [--verbose]
Environment: PAPERLESS_URL, PAPERLESS_TOKEN, S3_ENDPOINT_URL, S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY, S3_BUCKET_NAME

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Add requirements.txt for import script dependencies

Lists all Python dependencies needed to run scripts/import_paperless.py:
- requests: Paperless-ngx API client
- boto3: Garage S3 client
- pymupdf: PDF text extraction
- python-dateutil: Date parsing
- PyYAML: YAML config parsing (optional)

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Update import script: add storage backend selection (s3/file)

- Added --storage-backend argument (choices: s3, file, default: s3)
- Added --file-base-dir argument for file storage
- Storage backend validation based on selection
- Updated help text and epilog with backend-specific env vars
- Importer class now accepts storage_backend and file_base_dir params
- _init_storage() method handles backend initialization
- Summary now shows which storage backend was used

Usage examples:
  # S3 storage (default)
  python scripts/import_paperless.py

  # File storage for testing
  python scripts/import_paperless.py --storage-backend file --file-base-dir ./test_storage

  # Explicit S3
  python scripts/import_paperless.py --storage-backend s3 --s3-endpoint ... --s3-bucket ...

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Add README.md for scripts directory

Comprehensive documentation for import_paperless.py script:
- Overview and features
- Requirements and installation
- Usage examples (env vars + CLI args)
- Storage backends (S3 vs file)
- Metadata mapping table
- Output structure
- Resumability notes
- Exit codes
- Example output
- Troubleshooting guide
- Development/testing notes
- Architecture overview

Helps users understand how to use the import script for User Story 0.

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Fix Paperless API field mapping

The Paperless-ngx API returns IDs for correspondent, document_type, and tags,
not the actual string values. This commit fixes the import script to:

- Use 'archived_file_name' instead of 'filename' for the actual filename
- Fetch correspondent name from /api/correspondents/{id}/ endpoint
- Fetch document type name from /api/document_types/{id}/ endpoint
- Fetch tag names from /api/tags/{id}/ endpoint
- Add caching for fetched entities to avoid duplicate API requests

This ensures the sidecar metadata contains actual human-readable strings
instead of numeric IDs.

Changes:
- PaperlessClient: Added get_correspondent_name(), get_document_type_name(), get_tag_name()
- PaperlessClient: Added caches for fetched entities
- get_all_documents(): Now resolves IDs to names for all fields
- Updated module docstring to document this behavior

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Refactor PaperlessClient: change labels to list, add storage path caching, and improve document fetching logic

---------

Co-authored-by: Vibe Nuage Agent <vibe@mistral.ai>
Co-authored-by: napnap75 <napnap75@users.noreply.github.com>
2026-10-03 10:05:49 +02:00

485 lines
15 KiB
Python

"""Period computation module for Automatic PDF Renamer.
Provides period parsing from filenames and computation from emission dates
per template rules (PRD section 4.1, FR4.2, FR4.3).
"""
import re
from dataclasses import dataclass
from datetime import date, datetime
from enum import Enum
from typing import Any
from dateutil.relativedelta import relativedelta
class PeriodGranularity(Enum):
"""Granularity of period computation (PRD section 4.1)."""
DAY = "day"
MONTH = "month"
QUARTER = "quarter"
YEAR = "year"
class PeriodOffset(Enum):
"""Offset for period computation (PRD section 4.1)."""
CURRENT = "current"
NEXT = "next"
@dataclass
class PeriodRule:
"""Template period rule for period computation."""
granularity: PeriodGranularity
offset: PeriodOffset = PeriodOffset.CURRENT
@classmethod
def from_dict(cls, data: dict[str, Any]) -> "PeriodRule":
"""Create a PeriodRule from a dict."""
return cls(
granularity=PeriodGranularity(data.get("granularity", "month")),
offset=PeriodOffset(data.get("offset", "current")),
)
# French month names mapping
_FRENCH_MONTHS = {
"janvier": 1,
"février": 2,
"mars": 3,
"avril": 4,
"mai": 5,
"juin": 6,
"juillet": 7,
"août": 8,
"septembre": 9,
"octobre": 10,
"novembre": 11,
"décembre": 12,
# English fallbacks
"january": 1,
"february": 2,
"march": 3,
"april": 4,
"may": 5,
"june": 6,
"july": 7,
"august": 8,
"september": 9,
"october": 10,
"november": 11,
"december": 12,
}
# Month name to full French name for output
_FRENCH_MONTH_NAMES = {
1: "janvier", 2: "février", 3: "mars", 4: "avril", 5: "mai", 6: "juin",
7: "juillet", 8: "août", 9: "septembre", 10: "octobre", 11: "novembre", 12: "décembre",
}
# Month abbreviation mapping
_MONTH_ABBR = {
1: "Jan", 2: "Feb", 3: "Mar", 4: "Apr", 5: "May", 6: "Jun",
7: "Jul", 8: "Aug", 9: "Sep", 10: "Oct", 11: "Nov", 12: "Dec",
}
# Quarter pattern - matches Q3 2025, T3 2025, 3 2025, Q3-2025, trimestre 3 2025, 3T2025, etc.
_QUARTER_PATTERN = re.compile(
r"(?:Q|T|trimestre|quarter)?[\s\-]?(\d{1,2})[\s\-]?(?:Q|T|trimestre|quarter)?[\s\-]?(\d{4})\b|\b(\d{1,2})[TQ]?(\d{4})\b",
re.IGNORECASE,
)
# Year pattern
_YEAR_PATTERN = re.compile(r"\b(\d{4})\b")
# Month-year patterns (various formats)
_MONTH_YEAR_PATTERNS = [
# "août 2025", "August 2025" - French/English month name + year
re.compile(
rf"\b([{''.join(_FRENCH_MONTHS.keys())}]+)\s+(\d{{4}})\b",
re.IGNORECASE,
),
# "2025-08", "2025/08", "08-2025"
re.compile(r"(\d{4})[-/](\d{1,2})\b"),
re.compile(r"\b(\d{1,2})[-/](\d{4})\b"),
# "Aug 2025", "Aug-2025"
re.compile(r"\b([A-Za-z]{3})[-\s]?(\d{4})\b"),
# "2025 08", "08 2025"
re.compile(r"\b(\d{4})\s+(\d{1,2})\b"),
re.compile(r"\b(\d{1,2})\s+(\d{4})\b"),
]
# Date pattern (YYYY-MM-DD, DD/MM/YYYY, etc.)
_DATE_PATTERN = re.compile(
r"(\d{4})[-/](\d{1,2})[-/](\d{1,2})|(\d{1,2})[-/](\d{1,2})[-/](\d{4})"
)
def _parse_french_month(month_str: str) -> int | None:
"""Parse a French or English month name to month number."""
month_lower = month_str.lower().strip()
return _FRENCH_MONTHS.get(month_lower)
def _parse_month_abbr(abbr: str) -> int | None:
"""Parse a 3-letter month abbreviation to month number."""
abbr_upper = abbr.upper()
for num, name in _MONTH_ABBR.items():
if name.upper() == abbr_upper:
return num
return None
def parse_period_from_filename(filename: str) -> str | None:
"""Parse a period string from a filename.
Tries to extract period from the filename, looking at various positions.
Supports formats:
- Month-year: "août 2025", "August 2025", "Aug 2025", "2025-08"
- Quarter: "Q3 2025", "T3 2025", "3 2025", "Q3-2025"
- Year: "2025"
- Date: "2025-08-31", "31/08/2025"
Args:
filename: The filename to parse (without path).
Returns:
Parsed period string (e.g., "août 2025", "Q3 2025", "2025"), or None.
"""
# Get the filename without path
basename = filename.split("/")[-1].split("\\")[-1]
# Remove extension
name = basename.rsplit(".", 1)[0]
# First, try to find quarter patterns anywhere in the filename
quarter_match = _QUARTER_PATTERN.search(name)
if quarter_match:
# Handle different group patterns
groups = quarter_match.groups()
quarter_num = None
year = None
for g in groups:
if g and g.isdigit() and len(g) <= 2:
q = int(g)
if 1 <= q <= 4 and quarter_num is None:
quarter_num = q
elif g and g.isdigit() and len(g) == 4:
year = int(g)
if quarter_num and year:
return format_quarter(quarter_num, year)
# Try to find month-year patterns anywhere in the filename
for pattern in _MONTH_YEAR_PATTERNS:
match = pattern.search(name)
if match:
groups = match.groups()
year = None
month = None
for g in groups:
if g and g.isdigit() and len(g) == 4:
year = int(g)
elif g and g.isdigit() and len(g) <= 2:
if month is None:
month = int(g)
elif g:
# Try as month name (French or English)
parsed = _parse_french_month(g)
if parsed:
month = parsed
else:
parsed = _parse_month_abbr(g)
if parsed:
month = parsed
# If we found a year and a month, return it
if year is not None and month is not None:
return format_month_year(month, year)
# Try to find date patterns (YYYY-MM-DD, DD/MM/YYYY, etc.)
date_match = _DATE_PATTERN.search(name)
if date_match:
groups = date_match.groups()
year = None
month = None
day = None
for g in groups:
if g and len(g) == 4:
year = int(g)
elif g and len(g) <= 2:
if day is None:
day = int(g)
else:
month = int(g)
if year and month:
return format_month_year(month, year)
elif year:
return str(year)
# Try simple year pattern anywhere
year_match = _YEAR_PATTERN.search(name)
if year_match:
return year_match.group(1)
return None
def compute_period_from_date(
emission_date: str | date,
rule: PeriodRule,
) -> str:
"""Compute period from emission date and template rule.
Args:
emission_date: The document emission date (ISO format YYYY-MM-DD or date object).
rule: PeriodRule with granularity and offset.
Returns:
Human-readable period string.
"""
if isinstance(emission_date, str):
# Parse ISO date
try:
dt = datetime.fromisoformat(emission_date).date()
except ValueError:
# Try other formats
for fmt in ["%Y-%m-%d", "%d/%m/%Y", "%m/%d/%Y", "%d-%m-%Y"]:
try:
dt = datetime.strptime(emission_date, fmt).date()
break
except ValueError:
continue
else:
raise ValueError(f"Cannot parse emission_date: {emission_date}")
else:
dt = emission_date
if rule.granularity == PeriodGranularity.DAY:
return f"{dt.day} {_FRENCH_MONTH_NAMES[dt.month]} {dt.year}"
elif rule.granularity == PeriodGranularity.MONTH:
if rule.offset == PeriodOffset.CURRENT:
return format_month_year(dt.month, dt.year)
else: # NEXT
next_month = dt + relativedelta(months=1)
return format_month_year(next_month.month, next_month.year)
elif rule.granularity == PeriodGranularity.QUARTER:
quarter = (dt.month - 1) // 3 + 1
if rule.offset == PeriodOffset.CURRENT:
return format_quarter(quarter, dt.year)
else: # NEXT
next_quarter = quarter + 1
next_year = dt.year
if next_quarter > 4:
next_quarter = 1
next_year += 1
return format_quarter(next_quarter, next_year)
elif rule.granularity == PeriodGranularity.YEAR:
if rule.offset == PeriodOffset.CURRENT:
return str(dt.year)
else: # NEXT
return str(dt.year + 1)
else:
raise ValueError(f"Unknown granularity: {rule.granularity}")
def format_month_year(month: int, year: int, use_french: bool = True) -> str:
"""Format month and year as a human-readable string.
Uses French month names as per PRD by default.
Args:
month: Month number (1-12).
year: Year number.
use_french: If True, use French month names; otherwise use English.
Returns:
Formatted string (e.g., "août 2025" or "August 2025").
"""
if month < 1 or month > 12:
raise ValueError(f"Invalid month: {month}")
if use_french:
month_name = _FRENCH_MONTH_NAMES[month]
else:
month_name = _MONTH_ABBR[month]
return f"{month_name} {year}"
def format_quarter(quarter: int, year: int) -> str:
"""Format quarter and year as a human-readable string.
Args:
quarter: Quarter number (1-4).
year: Year number.
Returns:
Formatted string (e.g., "Q3 2025").
"""
if quarter < 1 or quarter > 4:
raise ValueError(f"Invalid quarter: {quarter}")
return f"Q{quarter} {year}"
def get_period(
filename: str | None = None,
emission_date: str | date | None = None,
rule: PeriodRule | None = None,
) -> tuple[str, bool]:
"""Get the period for a document, trying filename first, then fallback to rule.
This is the main entry point for period computation.
Args:
filename: The document filename (optional).
emission_date: The document emission date (optional).
rule: The template period rule (optional, required if emission_date provided).
Returns:
Tuple of (period_string, is_from_filename).
If period is parsed from filename, is_from_filename is True.
If computed from rule, is_from_filename is False.
Raises:
ValueError: If no period can be determined and no fallback is possible.
"""
# Try to parse from filename first
if filename:
period = parse_period_from_filename(filename)
if period:
return (period, True)
# Fallback to rule-based computation
if emission_date and rule:
period = compute_period_from_date(emission_date, rule)
return (period, False)
# If we have emission_date but no rule, use month-year as fallback
if emission_date:
if isinstance(emission_date, str):
try:
dt = datetime.fromisoformat(emission_date).date()
except ValueError:
for fmt in ["%Y-%m-%d", "%d/%m/%Y", "%m/%d/%Y", "%d-%m-%Y"]:
try:
dt = datetime.strptime(emission_date, fmt).date()
break
except ValueError:
continue
else:
raise ValueError(f"Cannot parse emission_date: {emission_date}")
else:
dt = emission_date
return (format_month_year(dt.month, dt.year), False)
raise ValueError("Cannot determine period: no filename, emission_date, or rule provided")
def validate_period_consistency(
period: str,
emission_date: str | date,
rule: PeriodRule,
) -> tuple[bool, str | None]:
"""Validate that a period is consistent with emission date and template rule.
Args:
period: The period string to validate.
emission_date: The document emission date.
rule: The template period rule.
Returns:
Tuple of (is_consistent, error_message).
If consistent, error_message is None.
"""
if isinstance(emission_date, str):
try:
dt = datetime.fromisoformat(emission_date).date()
except ValueError:
for fmt in ["%Y-%m-%d", "%d/%m/%Y", "%m/%d/%Y", "%d-%m-%Y"]:
try:
dt = datetime.strptime(emission_date, fmt).date()
break
except ValueError:
continue
else:
return (False, f"Cannot parse emission_date: {emission_date}")
else:
dt = emission_date
# Compute expected period from rule
expected_period = compute_period_from_date(dt, rule)
# Normalize both periods for comparison
# This is a simple comparison; may need enhancement for different formats
if period.lower() == expected_period.lower():
return (True, None)
# Try to parse both and compare
try:
parsed = _parse_period_string(period)
expected_parsed = _parse_period_string(expected_period)
if parsed == expected_parsed:
return (True, None)
except ValueError:
pass
return (
False,
f"Period '{period}' is inconsistent with emission_date '{emission_date}' "
f"and rule {rule.granularity.value}/{rule.offset.value}. "
f"Expected: {expected_period}",
)
def _parse_period_string(period: str) -> tuple[int, int, int | None]:
"""Parse a period string into (year, month, quarter) tuple.
Returns:
(year, month, quarter) where month and quarter may be None.
"""
period_lower = period.lower()
# Try year only
year_match = _YEAR_PATTERN.search(period_lower)
if year_match:
year = int(year_match.group(1))
return (year, None, None)
# Try quarter
quarter_match = _QUARTER_PATTERN.search(period_lower)
if quarter_match:
quarter = int(quarter_match.group(1))
year = int(quarter_match.group(2))
return (year, None, quarter)
# Try month-year
for pattern in _MONTH_YEAR_PATTERNS:
match = pattern.search(period_lower)
if match:
groups = match.groups()
year = None
month = None
for g in groups:
if g and g.isdigit() and len(g) == 4:
year = int(g)
elif g and g.isdigit() and len(g) <= 2:
month = int(g)
elif g:
parsed_month = _parse_french_month(g)
if parsed_month:
month = parsed_month
else:
parsed_abbr = _parse_month_abbr(g)
if parsed_abbr:
month = parsed_abbr
if year and month:
return (year, month, None)
raise ValueError(f"Cannot parse period string: {period}")