thomas/multiarch-docker-images
1
0
Fork 0
mirror of https://github.com/napnap75/multiarch-docker-images.git synced 2026-11-11 00:22:12 +01:00
Code Issues Projects Releases Wiki Activity
136 Commits 2 Branches 0 Tags
master
Commit Graph
6 Commits
Author SHA1 Message Date
thomas 6a47e6ad56 Add configurable document columns and allow bulk metadata editing (#14) 2026-10-19 18:15:03 +02:00
thomas 34271f3bfa feat: enhance PDF text extraction and add maintenance features 
- Refactor text extraction functions to extract text from all pages instead of just the first page.
- Implement logging for better debugging during file operations.
- Add a new maintenance page with options to rebuild the database and run consistency checks.
- Update styles for maintenance actions and improve UI elements.
- Remove obsolete rebuild database functionality from the main interface.
 2026-10-13 22:01:36 +02:00
thomas 6ba115059b Multiple corrections 2026-10-10 22:35:32 +02:00
thomas b8a8efcfd6 First version of the web UI 2026-10-07 22:14:47 +02:00
thomasnapnap75Vibe Nuage Agent
 		911ed1c027 feat: implement paperless import with config auto-update and new mapping structure (#10) 
* Add mapping-based configuration for document processing

- Create new config structure with company-type mappings
- Add date extraction regex patterns to config
- Update config.py to support new mapping-based configuration
- Create pipeline.py with full document processing workflow
- Enforce strict validation: unknown companies/types route to review
- Period rules now come from (company, type) mappings, not templates
- Remove document_family field from templates as requested

Implements workflow:
1. Extract first page text
2. Find emission date using regex from config
3. Find company/type using ML, validated against config lists
4. Look up (company, type) in mappings for template + period rule
5. If no mapping found -> route to review
6. Apply period rule to calculate period
7. Apply template key_pattern to generate filename

Closes #config-review

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Add field_groups support for additional fields in configuration

- Add field_groups section to templates.yaml with predefined lists:
  - bank_accounts: for bank statements
  - properties: for building-related documents
  - contract_references: for contracts and invoices
  - customer_numbers: for customer identification
- Add field_groups to CompanyTypeMapping dataclass
- Add get_field_groups(), get_field_group_values(), get_field_groups_for_mapping() methods
- Update pipeline to extract additional field values from predefined lists using ML
- Additional fields are extracted for each field_group in the mapping
- If field values cannot be determined -> route to review

Implements workflow:
1. Extract first page text
2. Find emission date using regex patterns from config
3. Find company and document_type using ML, validated against config lists
4. Look up (company, document_type) in mappings for template, period rule, and field groups
5. For each field group, find value using ML from predefined list
6. If no mapping found or field values cannot be determined -> route to review
7. Apply period rule to calculate period from emission date
8. Apply template key_pattern to generate filename with all fields

Closes #field-groups-review

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Update documentation and configuration for new structure

- Move configuration from config/templates.yaml to config.jsonc at root
- Rename field_groups to additional_fields throughout
- Update PRD section 4.1 to reference new period format configuration
- Update PRD FR4.1 and FR4.2 to reference JSONC and additional_fields
- Update Arch.md to reference JSONC configuration file
- Update module layout comments to reference JSONC
- Update processing pipeline diagram to reference companies list and additional fields
- Update configuration section to describe new JSONC structure

The new config.jsonc includes:
- companies: predefined list of valid companies
- document_types: predefined list of valid document types
- additional_fields: predefined value groups (bank_accounts, properties, etc.)
- date_extraction: regex patterns for finding emission date
- period_formats: customizable period formats with granularity, format template, and offset
- mappings: (company, document_type) -> template + period_format + additional_fields
- templates: template definitions with key patterns

Closes #config-review

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* feat: implement JSONC config with wildcard mappings and period formats

- Add JSONC configuration file at root (config.jsonc)
- Support simplified date_extraction as ordered list of regex strings
- Implement wildcard mappings (company OR document_type can be None)
- Add period_formats with granularity, custom format templates, and offsets
- Rename field_groups to additional_fields
- Update config.py: JSONC loading, PeriodFormatConfig, CompanyTypeMapping with optional fields
- Update period.py: format_period_from_emission_date with custom template support
- Update pipeline.py: use new config structure with strict validation

Closes #10

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* feat: implement new mapping structure with match/set rules

- New mapping structure: each rule has 'match' (criteria) and 'set' (actions)
- match: can contain company, document_type, both, or neither (empty = matches all)
- set: can contain template, period_format, additional_fields
- Rules processed in order, later matches override earlier ones
- Added default rule that applies to all documents
- Added MappingRule, MappingMatch, MappingSet dataclasses
- Updated resolve_mapping() to process all rules and apply overrides
- Added 'default' template for fallback

Closes #10

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* feat: implement paperless import with config auto-update and new mapping structure

- New mapping structure with match/set sections
- Auto-update config.jsonc with missing companies, document_types, additional_fields
- Tag-to-field matching with strict validation (unmatched/ambiguous tags route to pending)
- Route documents to pending/{paperless_id} when validation fails
- Save config at end of run if changes detected
- Use created_date from Paperless as emission_date
- Write sidecars with errors for pending documents
- Clear files/ and pending/ at start of each run

Closes #10

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Various corrections following real life tests

* Various corrections following real life tests

---------

Co-authored-by: Vibe Nuage Agent <vibe@mistral.ai>
Co-authored-by: napnap75 <napnap75@users.noreply.github.com>
 2026-10-04 22:06:01 +02:00
thomasnapnap75Vibe Nuage Agent
 		72f8254e10 US0: Paperless-ngx Initial Import - Reusable Components + Import Script (#9) 
* Add sidecar.py module for User Story 0 (Paperless-ngx import)

Implements the sidecar data model, schema validation, atomic writes,
and I/O helpers per PRD section 7.

- Sidecar and SidecarEvent dataclasses with full PRD schema
- SidecarStatus and EventType enums
- build_sidecar(), serialize(), deserialize() functions
- validate_sidecar() with type and required field checking
- get_sidecar_key() for .meta.json suffix generation
- write_atomic() for crash-safe temp-key + copy + delete pattern
- read_sidecar(), sidecar_exists(), matching_sha256() for storage I/O

This is a reusable component for both the import script and main service.

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Add storage/ module with S3 and File backends

Implements the storage layer abstraction for User Story 0.

- StorageBackend ABC: defines interface for all storage implementations
- S3Storage: boto3-based backend for Garage S3 (production)
- FileStorage: filesystem-based backend for testing
- get_storage_backend() factory function
- Atomic operations: put, get, delete, copy, exists, list, head

Both backends implement the same interface, allowing seamless
switching between production S3 and local testing.

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Add period.py, extractor.py, and config.py reusable modules

- period.py: Period parsing from filenames and computation from dates
  - parse_period_from_filename(): Extracts month-year, quarter, year, date patterns
  - compute_period_from_date(): Computes period per template rules (granularity + offset)
  - get_period(): Main entry point with filename fallback to rule-based computation
  - validate_period_consistency(): Checks period vs emission_date + rule
  - Supports French month names (août, janvier, etc.)

- extractor.py: PDF text extraction using PyMuPDF
  - extract_first_page_text(): Extracts text from first PDF page
  - extract_and_validate(): Extracts and checks minimum text length
  - TextExtractor class for dependency injection and mocking

- config.py: Configuration management
  - S3Config, PaperlessConfig, PollerConfig, ClassifierConfig dataclasses
  - AppConfig: Main application configuration from env vars
  - TemplateRegistry: Loads and manages template YAML configurations
  - load_config(), load_templates() helper functions

All modules are reusable by both import script and main service.

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Add import_paperless.py script for User Story 0

Implements the standalone import script that:
- Connects to Paperless-ngx REST API (PAPERLESS_URL + token from env)
- Connects to Garage S3 bucket via boto3 (S3_ENDPOINT_URL, credentials, bucket)
- Lists all documents from Paperless via paginated /api/documents/
- Fetches document metadata and maps to sidecar fields:
  - template: first path segment of storage_path
  - emission_date: document created field
  - emitting_company: correspondent name
  - document_type: document type name
  - period: parsed from filename tail, fallback to month-year from emission_date
  - extracted_text: first page text via PyMuPDF
  - optional_fields: from Paperless labels
- Downloads PDF via /api/documents/{id}/download/
- Stores PDF at files/{template}/{original filename}
- Writes sidecar {key}.meta.json with atomic temp-key + copy pattern
- Supports resumability (skip if PDF + sidecar exist with matching SHA256)
- Dry-run flag for testing
- Limit flag for trial runs
- Summary reporting at end
- Exit code: 0 if all succeed, 1 if any failures

Dependencies: requests, boto3, pymupdf, python-dateutil

Usage: python scripts/import_paperless.py [--dry-run] [--limit N] [--verbose]
Environment: PAPERLESS_URL, PAPERLESS_TOKEN, S3_ENDPOINT_URL, S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY, S3_BUCKET_NAME

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Add requirements.txt for import script dependencies

Lists all Python dependencies needed to run scripts/import_paperless.py:
- requests: Paperless-ngx API client
- boto3: Garage S3 client
- pymupdf: PDF text extraction
- python-dateutil: Date parsing
- PyYAML: YAML config parsing (optional)

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Update import script: add storage backend selection (s3/file)

- Added --storage-backend argument (choices: s3, file, default: s3)
- Added --file-base-dir argument for file storage
- Storage backend validation based on selection
- Updated help text and epilog with backend-specific env vars
- Importer class now accepts storage_backend and file_base_dir params
- _init_storage() method handles backend initialization
- Summary now shows which storage backend was used

Usage examples:
  # S3 storage (default)
  python scripts/import_paperless.py

  # File storage for testing
  python scripts/import_paperless.py --storage-backend file --file-base-dir ./test_storage

  # Explicit S3
  python scripts/import_paperless.py --storage-backend s3 --s3-endpoint ... --s3-bucket ...

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Add README.md for scripts directory

Comprehensive documentation for import_paperless.py script:
- Overview and features
- Requirements and installation
- Usage examples (env vars + CLI args)
- Storage backends (S3 vs file)
- Metadata mapping table
- Output structure
- Resumability notes
- Exit codes
- Example output
- Troubleshooting guide
- Development/testing notes
- Architecture overview

Helps users understand how to use the import script for User Story 0.

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Fix Paperless API field mapping

The Paperless-ngx API returns IDs for correspondent, document_type, and tags,
not the actual string values. This commit fixes the import script to:

- Use 'archived_file_name' instead of 'filename' for the actual filename
- Fetch correspondent name from /api/correspondents/{id}/ endpoint
- Fetch document type name from /api/document_types/{id}/ endpoint
- Fetch tag names from /api/tags/{id}/ endpoint
- Add caching for fetched entities to avoid duplicate API requests

This ensures the sidecar metadata contains actual human-readable strings
instead of numeric IDs.

Changes:
- PaperlessClient: Added get_correspondent_name(), get_document_type_name(), get_tag_name()
- PaperlessClient: Added caches for fetched entities
- get_all_documents(): Now resolves IDs to names for all fields
- Updated module docstring to document this behavior

Co-authored-by: napnap75 <napnap75@users.noreply.github.com>

* Refactor PaperlessClient: change labels to list, add storage path caching, and improve document fetching logic

---------

Co-authored-by: Vibe Nuage Agent <vibe@mistral.ai>
Co-authored-by: napnap75 <napnap75@users.noreply.github.com>
 2026-10-03 10:05:49 +02:00