User Story 0: Standalone script that imports existing paperless-ngx documents into a Garage S3 bucket (or local filesystem) with their metadata as sidecars.

Features

✅ Connects to Paperless-ngx REST API
✅ Connects to Garage S3 or local filesystem
✅ Lists all documents via paginated API
✅ Maps Paperless metadata to sidecar fields
✅ Downloads PDFs and extracts first-page text
✅ Atomic sidecar writes (temp-key + copy pattern)
✅ Resumability (skip if already imported with matching SHA256)
✅ Dry-run mode for testing
✅ Summary reporting
✅ Proper exit codes

Requirements

Install dependencies:

pip install -r scripts/requirements.txt

Or individually:

pip install requests boto3 pymupdf python-dateutil PyYAML

Usage

Environment Variables (Recommended)

Set these before running:

# Paperless-ngx
 export PAPERLESS_URL="http://paperless:8000"
 export PAPERLESS_TOKEN="your-api-token"

# S3 Storage (Garage)
 export S3_ENDPOINT_URL="http://garage:7777"
 export S3_ACCESS_KEY_ID="your-access-key"
 export S3_SECRET_ACCESS_KEY="your-secret-key"
 export S3_BUCKET_NAME="pdf-renamer"

# File Storage (for testing)
 export FILE_BASE_DIR="./import_storage"

Command Line

# S3 storage (default)
python scripts/import_paperless.py

# File storage for testing
python scripts/import_paperless.py \
    --storage-backend file \
    --file-base-dir ./test_storage

# Dry run (list what would be imported)
python scripts/import_paperless.py --dry-run

# Limited import (test with 5 documents)
python scripts/import_paperless.py --limit 5

# Verbose mode
python scripts/import_paperless.py --verbose

# Full options
python scripts/import_paperless.py \
    --paperless-url http://paperless:8000 \
    --paperless-token your-token \
    --storage-backend s3 \
    --s3-endpoint http://garage:7777 \
    --s3-access-key your-key \
    --s3-secret-key your-secret \
    --s3-bucket pdf-renamer \
    --dry-run \
    --limit 10 \
    --verbose

Storage Backends

The import script supports two storage backends:

Backend	Description	Use Case
`s3`	Garage S3 via boto3	Production
`file`	Local filesystem	Testing, development

S3 Backend

Requires:

--s3-endpoint or S3_ENDPOINT_URL
--s3-access-key or S3_ACCESS_KEY_ID
--s3-secret-key or S3_SECRET_ACCESS_KEY
--s3-bucket or S3_BUCKET_NAME

The script will refuse to run if the files/ prefix is not empty (to prevent accidental overwrites).

File Backend

Requires:

--file-base-dir or FILE_BASE_DIR (default: ./import_storage)

Creates a directory structure matching the S3 layout:

file_base_dir/
  files/
    Template1/
      document1.pdf
      document1.pdf.meta.json
    Template2/
      document2.pdf
      document2.pdf.meta.json
  tmp/
    (temporary files for atomic writes)

Metadata Mapping

Paperless-ngx fields are mapped to sidecar fields as follows:

Sidecar Field	Paperless Source	Notes
`template_name`	`storage_path` (first segment)	Document family
`emission_date`	`created`	Document date
`emitting_company`	`correspondent.name`	Correspondent
`document_type`	`document_type.name`	Document type
`period`	Filename (parsed) or `created` (fallback)	Month-year, quarter, year
`extracted_text`	PDF first page	Via PyMuPDF
`optional_fields`	`labels`	Each label becomes an optional field
`sha256`	PDF binary	Computed from downloaded PDF
`status`	Hardcoded	`validated`
`confidence`	Hardcoded	`1.0`

Output Structure

Documents are stored at:

files/{template}/{original_filename}
files/{template}/{original_filename}.meta.json

Example:

files/Banque/Relevé de compte août 2025.pdf
files/Banque/Relevé de compte août 2025.pdf.meta.json

Resumability

The script can be interrupted and rerun safely:

Documents already imported with matching SHA256 are skipped
Atomic sidecar writes ensure no partial/corrupted sidecars
Summary shows imported, skipped, and failed counts

Exit Codes

Code	Meaning
0	Success (all documents imported)
1	Failure (one or more documents failed)
130	Interrupted by user (SIGINT)

Example Output

2026-10-02 12:00:00,000 - INFO - Starting Paperless-ngx import...
2026-10-02 12:00:00,001 - INFO - Using S3 storage backend: http://garage:7777/pdf-renamer
2026-10-02 12:00:00,002 - INFO - Fetching documents from Paperless...
2026-10-02 12:00:05,123 - INFO - Retrieved 150 documents from Paperless
2026-10-02 12:00:05,124 - INFO - Processing document 1: Relevé août 2025.pdf
2026-10-02 12:00:05,456 - INFO - Successfully imported document 1
2026-10-02 12:00:05,457 - INFO - Processing document 2: Facture Q3 2025.pdf
...

============================================================
IMPORT SUMMARY
============================================================
Storage backend: s3
Imported:   150
Skipped:    0
Failed:     0
Period fallbacks: 0

Per-template counts:
  Banque: 75
  Factures: 50
  Contrats: 25
============================================================

Troubleshooting

"Missing required values" error

Make sure all required environment variables are set or passed as CLI arguments:

# Check what's set
echo $PAPERLESS_URL
echo $S3_ENDPOINT_URL

"Bucket is not empty" error

The script refuses to run if files/ prefix already has objects. Options:

Use a different bucket
Use file storage for testing: --storage-backend file --file-base-dir ./test
Empty the bucket first (if you're sure it's safe)

"PyMuPDF not available" warning

Text extraction will be skipped. Install it for full functionality:

pip install pymupdf

Connection errors

Check that:

Paperless-ngx is running and accessible
S3/Garage endpoint is correct
Credentials are valid
Network connectivity is available

Development / Testing

For testing without Paperless-ngx or S3:

# Use file storage
python scripts/import_paperless.py \
    --storage-backend file \
    --file-base-dir ./test_storage \
    --dry-run

# Or create a mock Paperless client (see tests/ directory)

Architecture

The import script uses the same reusable components as the main service:

app/sidecar.py - Sidecar data model and I/O
app/storage/ - Storage backend abstraction
app/period.py - Period parsing and computation
app/extractor.py - PDF text extraction
app/config.py - Configuration utilities

This ensures consistency between the import script and the main service.

License

Same as the main project. See ../LICENSE.

Dashboard list and bulk editing

The dashboard's configurable columns and bulk metadata editing are API/UI features, not standalone scripts. Script authors must preserve the existing sidecar-source-of-truth model: any script that changes company, document type, or additional fields must use the application's validation and safe rename flow, recompute derived period/key values, append event history, and synchronize SQLite only after the sidecar is durable. Do not treat user-visible column preferences as document metadata.

README.md

Scripts

Available Scripts

import_paperless.py - Paperless-ngx Initial Import