multiarch-docker-images/automatic-pdf-renamer/scripts/README.md

Scripts

Standalone Python scripts for the Automatic PDF Renamer project.

Available Scripts

Script	Purpose	Dependencies
import_paperless.py	Import documents from Paperless-ngx to storage	requests, boto3, pymupdf, python-dateutil

---

import_paperless.py - Paperless-ngx Initial Import

User Story 0: Standalone script that imports existing paperless-ngx documents into a Garage S3 bucket (or local filesystem) with their metadata as sidecars.

Features

• ✅ Connects to Paperless-ngx REST API
• ✅ Connects to Garage S3 or local filesystem
• ✅ Lists all documents via paginated API
• ✅ Maps Paperless metadata to sidecar fields
• ✅ Downloads PDFs and extracts first-page text
• ✅ Atomic sidecar writes (temp-key + copy pattern)
• ✅ Resumability (skip if already imported with matching SHA256)
• ✅ Dry-run mode for testing
• ✅ Summary reporting
• ✅ Proper exit codes

Requirements

Install dependencies:

pip install -r scripts/requirements.txt

Or individually:

pip install requests boto3 pymupdf python-dateutil PyYAML

Usage

Environment Variables (Recommended)

Set these before running:

# Paperless-ngx
 export PAPERLESS_URL="http://paperless:8000"
 export PAPERLESS_TOKEN="your-api-token"

# S3 Storage (Garage)
 export S3_ENDPOINT_URL="http://garage:7777"
 export S3_ACCESS_KEY_ID="your-access-key"
 export S3_SECRET_ACCESS_KEY="your-secret-key"
 export S3_BUCKET_NAME="pdf-renamer"

# File Storage (for testing)
 export FILE_BASE_DIR="./import_storage"

Command Line

# S3 storage (default)
python scripts/import_paperless.py

# File storage for testing
python scripts/import_paperless.py \
    --storage-backend file \
    --file-base-dir ./test_storage

# Dry run (list what would be imported)
python scripts/import_paperless.py --dry-run

# Limited import (test with 5 documents)
python scripts/import_paperless.py --limit 5

# Verbose mode
python scripts/import_paperless.py --verbose

# Full options
python scripts/import_paperless.py \
    --paperless-url http://paperless:8000 \
    --paperless-token your-token \
    --storage-backend s3 \
    --s3-endpoint http://garage:7777 \
    --s3-access-key your-key \
    --s3-secret-key your-secret \
    --s3-bucket pdf-renamer \
    --dry-run \
    --limit 10 \
    --verbose

Storage Backends

The import script supports two storage backends:

Backend	Description	Use Case
s3	Garage S3 via boto3	Production
file	Local filesystem	Testing, development

S3 Backend

Requires:

• --s3-endpoint or S3_ENDPOINT_URL
• --s3-access-key or S3_ACCESS_KEY_ID
• --s3-secret-key or S3_SECRET_ACCESS_KEY
• --s3-bucket or S3_BUCKET_NAME

The script will refuse to run if the files/ prefix is not empty (to prevent accidental overwrites).

File Backend

Requires:

• --file-base-dir or FILE_BASE_DIR (default: ./import_storage)

Creates a directory structure matching the S3 layout:

file_base_dir/
  files/
    Template1/
      document1.pdf
      document1.pdf.meta.json
    Template2/
      document2.pdf
      document2.pdf.meta.json
  tmp/
    (temporary files for atomic writes)

Metadata Mapping

Paperless-ngx fields are mapped to sidecar fields as follows:

Sidecar Field	Paperless Source	Notes
template_name	storage_path (first segment)	Document family
emission_date	created	Document date
emitting_company	correspondent.name	Correspondent
document_type	document_type.name	Document type
period	Filename (parsed) or created (fallback)	Month-year, quarter, year
extracted_text	PDF first page	Via PyMuPDF
optional_fields	labels	Each label becomes an optional field
sha256	PDF binary	Computed from downloaded PDF
status	Hardcoded	validated
confidence	Hardcoded	1.0

Output Structure

Documents are stored at:

files/{template}/{original_filename}
files/{template}/{original_filename}.meta.json

Example:

files/Banque/Relevé de compte août 2025.pdf
files/Banque/Relevé de compte août 2025.pdf.meta.json

Resumability

The script can be interrupted and rerun safely:

• Documents already imported with matching SHA256 are skipped
• Atomic sidecar writes ensure no partial/corrupted sidecars
• Summary shows imported, skipped, and failed counts

Exit Codes

Code	Meaning
0	Success (all documents imported)
1	Failure (one or more documents failed)
130	Interrupted by user (SIGINT)

Example Output

2026-10-02 12:00:00,000 - INFO - Starting Paperless-ngx import...
2026-10-02 12:00:00,001 - INFO - Using S3 storage backend: http://garage:7777/pdf-renamer
2026-10-02 12:00:00,002 - INFO - Fetching documents from Paperless...
2026-10-02 12:00:05,123 - INFO - Retrieved 150 documents from Paperless
2026-10-02 12:00:05,124 - INFO - Processing document 1: Relevé août 2025.pdf
2026-10-02 12:00:05,456 - INFO - Successfully imported document 1
2026-10-02 12:00:05,457 - INFO - Processing document 2: Facture Q3 2025.pdf
...

============================================================
IMPORT SUMMARY
============================================================
Storage backend: s3
Imported:   150
Skipped:    0
Failed:     0
Period fallbacks: 0

Per-template counts:
  Banque: 75
  Factures: 50
  Contrats: 25
============================================================

Troubleshooting

"Missing required values" error

Make sure all required environment variables are set or passed as CLI arguments:

# Check what's set
echo $PAPERLESS_URL
echo $S3_ENDPOINT_URL

"Bucket is not empty" error

The script refuses to run if files/ prefix already has objects. Options:

1. Use a different bucket
2. Use file storage for testing: --storage-backend file --file-base-dir ./test
3. Empty the bucket first (if you're sure it's safe)

"PyMuPDF not available" warning

Text extraction will be skipped. Install it for full functionality:

pip install pymupdf

Connection errors

Check that:

• Paperless-ngx is running and accessible
• S3/Garage endpoint is correct
• Credentials are valid
• Network connectivity is available

Development / Testing

For testing without Paperless-ngx or S3:

# Use file storage
python scripts/import_paperless.py \
    --storage-backend file \
    --file-base-dir ./test_storage \
    --dry-run

# Or create a mock Paperless client (see tests/ directory)

---

Architecture

The import script uses the same reusable components as the main service:

• app/sidecar.py - Sidecar data model and I/O
• app/storage/ - Storage backend abstraction
• app/period.py - Period parsing and computation
• app/extractor.py - PDF text extraction
• app/config.py - Configuration utilities

This ensures consistency between the import script and the main service.

---

License

Same as the main project. See ../LICENSE.

Dashboard list and bulk editing

The dashboard's configurable columns and bulk metadata editing are API/UI features, not standalone scripts. Script authors must preserve the existing sidecar-source-of-truth model: any script that changes company, document type, or additional fields must use the application's validation and safe rename flow, recompute derived period/key values, append event history, and synchronize SQLite only after the sidecar is durable. Do not treat user-visible column preferences as document metadata.