mirror of
https://github.com/napnap75/multiarch-docker-images.git
synced 2026-11-12 00:52:11 +01:00
US0: Paperless-ngx Initial Import - Reusable Components + Import Script (#9)
* Add sidecar.py module for User Story 0 (Paperless-ngx import) Implements the sidecar data model, schema validation, atomic writes, and I/O helpers per PRD section 7. - Sidecar and SidecarEvent dataclasses with full PRD schema - SidecarStatus and EventType enums - build_sidecar(), serialize(), deserialize() functions - validate_sidecar() with type and required field checking - get_sidecar_key() for .meta.json suffix generation - write_atomic() for crash-safe temp-key + copy + delete pattern - read_sidecar(), sidecar_exists(), matching_sha256() for storage I/O This is a reusable component for both the import script and main service. Co-authored-by: napnap75 <napnap75@users.noreply.github.com> * Add storage/ module with S3 and File backends Implements the storage layer abstraction for User Story 0. - StorageBackend ABC: defines interface for all storage implementations - S3Storage: boto3-based backend for Garage S3 (production) - FileStorage: filesystem-based backend for testing - get_storage_backend() factory function - Atomic operations: put, get, delete, copy, exists, list, head Both backends implement the same interface, allowing seamless switching between production S3 and local testing. Co-authored-by: napnap75 <napnap75@users.noreply.github.com> * Add period.py, extractor.py, and config.py reusable modules - period.py: Period parsing from filenames and computation from dates - parse_period_from_filename(): Extracts month-year, quarter, year, date patterns - compute_period_from_date(): Computes period per template rules (granularity + offset) - get_period(): Main entry point with filename fallback to rule-based computation - validate_period_consistency(): Checks period vs emission_date + rule - Supports French month names (août, janvier, etc.) - extractor.py: PDF text extraction using PyMuPDF - extract_first_page_text(): Extracts text from first PDF page - extract_and_validate(): Extracts and checks minimum text length - TextExtractor class for dependency injection and mocking - config.py: Configuration management - S3Config, PaperlessConfig, PollerConfig, ClassifierConfig dataclasses - AppConfig: Main application configuration from env vars - TemplateRegistry: Loads and manages template YAML configurations - load_config(), load_templates() helper functions All modules are reusable by both import script and main service. Co-authored-by: napnap75 <napnap75@users.noreply.github.com> * Add import_paperless.py script for User Story 0 Implements the standalone import script that: - Connects to Paperless-ngx REST API (PAPERLESS_URL + token from env) - Connects to Garage S3 bucket via boto3 (S3_ENDPOINT_URL, credentials, bucket) - Lists all documents from Paperless via paginated /api/documents/ - Fetches document metadata and maps to sidecar fields: - template: first path segment of storage_path - emission_date: document created field - emitting_company: correspondent name - document_type: document type name - period: parsed from filename tail, fallback to month-year from emission_date - extracted_text: first page text via PyMuPDF - optional_fields: from Paperless labels - Downloads PDF via /api/documents/{id}/download/ - Stores PDF at files/{template}/{original filename} - Writes sidecar {key}.meta.json with atomic temp-key + copy pattern - Supports resumability (skip if PDF + sidecar exist with matching SHA256) - Dry-run flag for testing - Limit flag for trial runs - Summary reporting at end - Exit code: 0 if all succeed, 1 if any failures Dependencies: requests, boto3, pymupdf, python-dateutil Usage: python scripts/import_paperless.py [--dry-run] [--limit N] [--verbose] Environment: PAPERLESS_URL, PAPERLESS_TOKEN, S3_ENDPOINT_URL, S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY, S3_BUCKET_NAME Co-authored-by: napnap75 <napnap75@users.noreply.github.com> * Add requirements.txt for import script dependencies Lists all Python dependencies needed to run scripts/import_paperless.py: - requests: Paperless-ngx API client - boto3: Garage S3 client - pymupdf: PDF text extraction - python-dateutil: Date parsing - PyYAML: YAML config parsing (optional) Co-authored-by: napnap75 <napnap75@users.noreply.github.com> * Update import script: add storage backend selection (s3/file) - Added --storage-backend argument (choices: s3, file, default: s3) - Added --file-base-dir argument for file storage - Storage backend validation based on selection - Updated help text and epilog with backend-specific env vars - Importer class now accepts storage_backend and file_base_dir params - _init_storage() method handles backend initialization - Summary now shows which storage backend was used Usage examples: # S3 storage (default) python scripts/import_paperless.py # File storage for testing python scripts/import_paperless.py --storage-backend file --file-base-dir ./test_storage # Explicit S3 python scripts/import_paperless.py --storage-backend s3 --s3-endpoint ... --s3-bucket ... Co-authored-by: napnap75 <napnap75@users.noreply.github.com> * Add README.md for scripts directory Comprehensive documentation for import_paperless.py script: - Overview and features - Requirements and installation - Usage examples (env vars + CLI args) - Storage backends (S3 vs file) - Metadata mapping table - Output structure - Resumability notes - Exit codes - Example output - Troubleshooting guide - Development/testing notes - Architecture overview Helps users understand how to use the import script for User Story 0. Co-authored-by: napnap75 <napnap75@users.noreply.github.com> * Fix Paperless API field mapping The Paperless-ngx API returns IDs for correspondent, document_type, and tags, not the actual string values. This commit fixes the import script to: - Use 'archived_file_name' instead of 'filename' for the actual filename - Fetch correspondent name from /api/correspondents/{id}/ endpoint - Fetch document type name from /api/document_types/{id}/ endpoint - Fetch tag names from /api/tags/{id}/ endpoint - Add caching for fetched entities to avoid duplicate API requests This ensures the sidecar metadata contains actual human-readable strings instead of numeric IDs. Changes: - PaperlessClient: Added get_correspondent_name(), get_document_type_name(), get_tag_name() - PaperlessClient: Added caches for fetched entities - get_all_documents(): Now resolves IDs to names for all fields - Updated module docstring to document this behavior Co-authored-by: napnap75 <napnap75@users.noreply.github.com> * Refactor PaperlessClient: change labels to list, add storage path caching, and improve document fetching logic --------- Co-authored-by: Vibe Nuage Agent <vibe@mistral.ai> Co-authored-by: napnap75 <napnap75@users.noreply.github.com>
This commit is contained in:
co-authored by
napnap75
Vibe Nuage Agent
napnap75
Vibe Nuage Agent
parent
2d26876bf5
commit
72f8254e10
@@ -0,0 +1,119 @@
|
||||
"""Storage layer for Automatic PDF Renamer.
|
||||
|
||||
Provides a unified interface for object storage operations.
|
||||
Implementations:
|
||||
- S3Storage: For Garage S3 (production)
|
||||
- FileStorage: For local filesystem (testing)
|
||||
"""
|
||||
|
||||
from abc import ABC, abstractmethod
|
||||
from typing import Any
|
||||
|
||||
|
||||
class StorageBackend(ABC):
|
||||
"""Abstract base class for storage backends.
|
||||
|
||||
All storage implementations must provide these methods.
|
||||
"""
|
||||
|
||||
@abstractmethod
|
||||
def put_object(self, key: str, body: str | bytes) -> None:
|
||||
"""Upload an object to the storage.
|
||||
|
||||
Args:
|
||||
key: The object key/path.
|
||||
body: The object content as string or bytes.
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def get_object(self, key: str) -> str | bytes | None:
|
||||
"""Retrieve an object from storage.
|
||||
|
||||
Args:
|
||||
key: The object key/path.
|
||||
|
||||
Returns:
|
||||
The object content as string or bytes, or None if not found.
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def delete_object(self, key: str) -> None:
|
||||
"""Delete an object from storage.
|
||||
|
||||
Args:
|
||||
key: The object key/path.
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def copy_object(self, src_key: str, dst_key: str) -> None:
|
||||
"""Copy an object within the storage.
|
||||
|
||||
Args:
|
||||
src_key: Source object key/path.
|
||||
dst_key: Destination object key/path.
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def object_exists(self, key: str) -> bool:
|
||||
"""Check if an object exists in storage.
|
||||
|
||||
Args:
|
||||
key: The object key/path.
|
||||
|
||||
Returns:
|
||||
True if the object exists, False otherwise.
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def list_objects(self, prefix: str = "") -> list[str]:
|
||||
"""List all object keys under a prefix.
|
||||
|
||||
Args:
|
||||
prefix: The prefix to filter by.
|
||||
|
||||
Returns:
|
||||
List of object keys (full paths).
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
def head_object(self, key: str) -> dict[str, Any] | None:
|
||||
"""Get metadata for an object.
|
||||
|
||||
Args:
|
||||
key: The object key/path.
|
||||
|
||||
Returns:
|
||||
Dict with metadata (e.g., size, etag, content_type), or None if not found.
|
||||
"""
|
||||
pass
|
||||
|
||||
|
||||
# Convenience function to get the appropriate storage backend
|
||||
# This will be configured by the application
|
||||
def get_storage_backend(backend_type: str, **kwargs) -> StorageBackend:
|
||||
"""Factory function to get a storage backend instance.
|
||||
|
||||
Args:
|
||||
backend_type: Either 's3' or 'file'.
|
||||
**kwargs: Backend-specific configuration.
|
||||
|
||||
Returns:
|
||||
A StorageBackend instance.
|
||||
|
||||
Raises:
|
||||
ValueError: If backend_type is not recognized.
|
||||
"""
|
||||
if backend_type == "s3":
|
||||
from .s3_storage import S3Storage
|
||||
return S3Storage(**kwargs)
|
||||
elif backend_type == "file":
|
||||
from .file_storage import FileStorage
|
||||
return FileStorage(**kwargs)
|
||||
else:
|
||||
raise ValueError(f"Unknown storage backend type: {backend_type}")
|
||||
@@ -0,0 +1,197 @@
|
||||
"""File Storage Backend for testing.
|
||||
|
||||
Local filesystem implementation for development and testing purposes.
|
||||
Mimics S3 behavior using a base directory.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from . import StorageBackend
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class FileStorage(StorageBackend):
|
||||
"""Filesystem-based storage backend for testing.
|
||||
|
||||
Uses a base directory to store objects as files.
|
||||
Keys are mapped to filesystem paths relative to the base directory.
|
||||
"""
|
||||
|
||||
def __init__(self, base_dir: str = "./storage"):
|
||||
"""Initialize file storage backend.
|
||||
|
||||
Args:
|
||||
base_dir: Base directory for storing objects.
|
||||
"""
|
||||
self.base_dir = Path(base_dir).resolve()
|
||||
self.base_dir.mkdir(parents=True, exist_ok=True)
|
||||
logger.info(f"FileStorage initialized at {self.base_dir}")
|
||||
|
||||
def _make_path(self, key: str) -> Path:
|
||||
"""Convert a storage key to a filesystem path.
|
||||
|
||||
Args:
|
||||
key: The object key/path (may contain forward slashes).
|
||||
|
||||
Returns:
|
||||
Absolute Path for the object file.
|
||||
"""
|
||||
# Normalize key: remove leading slash, replace slashes with OS separator
|
||||
normalized = key.lstrip("/")
|
||||
if os.sep != "/":
|
||||
normalized = normalized.replace("/", os.sep)
|
||||
return self.base_dir / normalized
|
||||
|
||||
def _ensure_parent_dir(self, path: Path) -> None:
|
||||
"""Ensure the parent directory of a path exists."""
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def put_object(self, key: str, body: str | bytes) -> None:
|
||||
"""Upload an object to the filesystem.
|
||||
|
||||
Args:
|
||||
key: The object key/path.
|
||||
body: The object content as string or bytes.
|
||||
"""
|
||||
path = self._make_path(key)
|
||||
self._ensure_parent_dir(path)
|
||||
|
||||
if isinstance(body, str):
|
||||
body = body.encode("utf-8")
|
||||
|
||||
with open(path, "wb") as f:
|
||||
f.write(body)
|
||||
logger.debug(f"Uploaded {key} to {path}")
|
||||
|
||||
def get_object(self, key: str) -> str | bytes | None:
|
||||
"""Retrieve an object from the filesystem.
|
||||
|
||||
Args:
|
||||
key: The object key/path.
|
||||
|
||||
Returns:
|
||||
The object content as bytes, or None if not found.
|
||||
"""
|
||||
path = self._make_path(key)
|
||||
|
||||
if not path.exists():
|
||||
logger.debug(f"Object {key} not found at {path}")
|
||||
return None
|
||||
|
||||
with open(path, "rb") as f:
|
||||
return f.read()
|
||||
|
||||
def delete_object(self, key: str) -> None:
|
||||
"""Delete an object from the filesystem.
|
||||
|
||||
Args:
|
||||
key: The object key/path.
|
||||
"""
|
||||
path = self._make_path(key)
|
||||
|
||||
if path.exists():
|
||||
path.unlink()
|
||||
logger.debug(f"Deleted {key} from {path}")
|
||||
else:
|
||||
logger.debug(f"Object {key} not found at {path}, nothing to delete")
|
||||
|
||||
def copy_object(self, src_key: str, dst_key: str) -> None:
|
||||
"""Copy an object within the filesystem.
|
||||
|
||||
Args:
|
||||
src_key: Source object key/path.
|
||||
dst_key: Destination object key/path.
|
||||
"""
|
||||
src_path = self._make_path(src_key)
|
||||
dst_path = self._make_path(dst_key)
|
||||
|
||||
self._ensure_parent_dir(dst_path)
|
||||
|
||||
if not src_path.exists():
|
||||
raise FileNotFoundError(f"Source object {src_key} not found at {src_path}")
|
||||
|
||||
# Use shutil.copy2 to preserve metadata
|
||||
import shutil
|
||||
shutil.copy2(str(src_path), str(dst_path))
|
||||
logger.debug(f"Copied {src_key} to {dst_key}")
|
||||
|
||||
def object_exists(self, key: str) -> bool:
|
||||
"""Check if an object exists in the filesystem.
|
||||
|
||||
Args:
|
||||
key: The object key/path.
|
||||
|
||||
Returns:
|
||||
True if the object exists, False otherwise.
|
||||
"""
|
||||
path = self._make_path(key)
|
||||
return path.exists()
|
||||
|
||||
def list_objects(self, prefix: str = "") -> list[str]:
|
||||
"""List all object keys under a prefix.
|
||||
|
||||
Args:
|
||||
prefix: The prefix to filter by.
|
||||
|
||||
Returns:
|
||||
List of object keys (full paths, with forward slashes).
|
||||
"""
|
||||
# Convert prefix to path
|
||||
prefix_path = self._make_path(prefix)
|
||||
|
||||
# Ensure prefix_path is relative to base_dir
|
||||
if not str(prefix_path).startswith(str(self.base_dir)):
|
||||
raise ValueError(f"Prefix {prefix} escapes base directory")
|
||||
|
||||
objects: list[str] = []
|
||||
|
||||
# Walk the directory tree starting from prefix_path
|
||||
if prefix_path.exists() and prefix_path.is_dir():
|
||||
for root, _, files in os.walk(prefix_path):
|
||||
for file in files:
|
||||
full_path = Path(root) / file
|
||||
# Convert back to storage key format (forward slashes, relative to base_dir)
|
||||
rel_path = full_path.relative_to(self.base_dir)
|
||||
key = str(rel_path).replace(os.sep, "/")
|
||||
objects.append(key)
|
||||
elif prefix_path.exists() and prefix_path.is_file():
|
||||
# If prefix is a file, return it
|
||||
rel_path = prefix_path.relative_to(self.base_dir)
|
||||
key = str(rel_path).replace(os.sep, "/")
|
||||
objects.append(key)
|
||||
|
||||
return objects
|
||||
|
||||
def head_object(self, key: str) -> dict[str, Any] | None:
|
||||
"""Get metadata for an object.
|
||||
|
||||
Args:
|
||||
key: The object key/path.
|
||||
|
||||
Returns:
|
||||
Dict with metadata (size, etag, content_type, last_modified), or None if not found.
|
||||
"""
|
||||
path = self._make_path(key)
|
||||
|
||||
if not path.exists():
|
||||
return None
|
||||
|
||||
stat = path.stat()
|
||||
|
||||
# Simple etag: use inode + size (not cryptographic, but sufficient for testing)
|
||||
etag = f"{stat.st_ino}-{stat.st_size}"
|
||||
|
||||
return {
|
||||
"size": stat.st_size,
|
||||
"etag": etag,
|
||||
"content_type": "application/octet-stream", # Default for files
|
||||
"last_modified": datetime.fromtimestamp(stat.st_mtime, tz=timezone.utc).isoformat(),
|
||||
}
|
||||
|
||||
|
||||
# Import datetime for head_object
|
||||
from datetime import datetime, timezone
|
||||
@@ -0,0 +1,258 @@
|
||||
"""S3 Storage Backend for Garage S3.
|
||||
|
||||
Production storage implementation using boto3 for Garage S3 compatibility.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
try:
|
||||
import boto3
|
||||
from botocore.client import BaseClient
|
||||
from botocore.exceptions import ClientError
|
||||
HAS_BOTO3 = True
|
||||
except ImportError:
|
||||
HAS_BOTO3 = False
|
||||
BaseClient = None
|
||||
ClientError = Exception
|
||||
|
||||
from . import StorageBackend
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class S3Storage(StorageBackend):
|
||||
"""S3-compatible storage backend for Garage.
|
||||
|
||||
Uses boto3 to interact with Garage S3 API.
|
||||
Configuration via environment variables or constructor kwargs.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
endpoint_url: str | None = None,
|
||||
aws_access_key_id: str | None = None,
|
||||
aws_secret_access_key: str | None = None,
|
||||
bucket_name: str | None = None,
|
||||
region_name: str | None = None,
|
||||
):
|
||||
"""Initialize S3 storage backend.
|
||||
|
||||
Args:
|
||||
endpoint_url: Garage S3 endpoint URL.
|
||||
aws_access_key_id: S3 access key.
|
||||
aws_secret_access_key: S3 secret key.
|
||||
bucket_name: Name of the S3 bucket.
|
||||
region_name: AWS region (default: us-east-1).
|
||||
"""
|
||||
self.endpoint_url = endpoint_url
|
||||
self.aws_access_key_id = aws_access_key_id
|
||||
self.aws_secret_access_key = aws_secret_access_key
|
||||
self.bucket_name = bucket_name
|
||||
self.region_name = region_name or "us-east-1"
|
||||
|
||||
# Initialize boto3 client
|
||||
self._client: BaseClient | None = None
|
||||
self._connect()
|
||||
|
||||
def _connect(self) -> None:
|
||||
"""Initialize the boto3 S3 client."""
|
||||
if not HAS_BOTO3:
|
||||
raise ImportError("boto3 is required for S3Storage. Install with: pip install boto3")
|
||||
self._client = boto3.client(
|
||||
"s3",
|
||||
endpoint_url=self.endpoint_url,
|
||||
aws_access_key_id=self.aws_access_key_id,
|
||||
aws_secret_access_key=self.aws_secret_access_key,
|
||||
region_name=self.region_name,
|
||||
)
|
||||
logger.info(
|
||||
f"Connected to S3 at {self.endpoint_url} with bucket {self.bucket_name}"
|
||||
)
|
||||
|
||||
def _ensure_bucket(self) -> None:
|
||||
"""Ensure the bucket is accessible."""
|
||||
if self.bucket_name is None:
|
||||
raise ValueError("bucket_name is required for S3Storage")
|
||||
|
||||
def _make_key(self, key: str) -> str:
|
||||
"""Normalize a key by removing leading slashes."""
|
||||
return key.lstrip("/")
|
||||
|
||||
def put_object(self, key: str, body: str | bytes) -> None:
|
||||
"""Upload an object to S3.
|
||||
|
||||
Args:
|
||||
key: The object key/path.
|
||||
body: The object content as string or bytes.
|
||||
"""
|
||||
self._ensure_bucket()
|
||||
normalized_key = self._make_key(key)
|
||||
|
||||
if isinstance(body, str):
|
||||
body = body.encode("utf-8")
|
||||
|
||||
try:
|
||||
self._client.put_object(
|
||||
Bucket=self.bucket_name,
|
||||
Key=normalized_key,
|
||||
Body=body,
|
||||
)
|
||||
logger.debug(f"Uploaded {normalized_key} to bucket {self.bucket_name}")
|
||||
except ClientError as e:
|
||||
logger.error(f"Failed to upload {normalized_key}: {e}")
|
||||
raise
|
||||
|
||||
def get_object(self, key: str) -> str | bytes | None:
|
||||
"""Retrieve an object from S3.
|
||||
|
||||
Args:
|
||||
key: The object key/path.
|
||||
|
||||
Returns:
|
||||
The object content as bytes, or None if not found.
|
||||
"""
|
||||
self._ensure_bucket()
|
||||
normalized_key = self._make_key(key)
|
||||
|
||||
try:
|
||||
response = self._client.get_object(
|
||||
Bucket=self.bucket_name,
|
||||
Key=normalized_key,
|
||||
)
|
||||
return response["Body"].read()
|
||||
except ClientError as e:
|
||||
if e.response["Error"]["Code"] == "NoSuchKey":
|
||||
logger.debug(f"Object {normalized_key} not found")
|
||||
return None
|
||||
logger.error(f"Failed to get {normalized_key}: {e}")
|
||||
raise
|
||||
|
||||
def delete_object(self, key: str) -> None:
|
||||
"""Delete an object from S3.
|
||||
|
||||
Args:
|
||||
key: The object key/path.
|
||||
"""
|
||||
self._ensure_bucket()
|
||||
normalized_key = self._make_key(key)
|
||||
|
||||
try:
|
||||
self._client.delete_object(
|
||||
Bucket=self.bucket_name,
|
||||
Key=normalized_key,
|
||||
)
|
||||
logger.debug(f"Deleted {normalized_key} from bucket {self.bucket_name}")
|
||||
except ClientError as e:
|
||||
logger.error(f"Failed to delete {normalized_key}: {e}")
|
||||
raise
|
||||
|
||||
def copy_object(self, src_key: str, dst_key: str) -> None:
|
||||
"""Copy an object within S3.
|
||||
|
||||
Uses S3's copy_object operation which is atomic.
|
||||
|
||||
Args:
|
||||
src_key: Source object key/path.
|
||||
dst_key: Destination object key/path.
|
||||
"""
|
||||
self._ensure_bucket()
|
||||
src = self._make_key(src_key)
|
||||
dst = self._make_key(dst_key)
|
||||
|
||||
# Get the source bucket (for cross-bucket copy support)
|
||||
# For now, assume same bucket
|
||||
copy_source = {"Bucket": self.bucket_name, "Key": src}
|
||||
|
||||
try:
|
||||
self._client.copy_object(
|
||||
Bucket=self.bucket_name,
|
||||
Key=dst,
|
||||
CopySource=copy_source,
|
||||
)
|
||||
logger.debug(f"Copied {src} to {dst} in bucket {self.bucket_name}")
|
||||
except ClientError as e:
|
||||
logger.error(f"Failed to copy {src} to {dst}: {e}")
|
||||
raise
|
||||
|
||||
def object_exists(self, key: str) -> bool:
|
||||
"""Check if an object exists in S3.
|
||||
|
||||
Args:
|
||||
key: The object key/path.
|
||||
|
||||
Returns:
|
||||
True if the object exists, False otherwise.
|
||||
"""
|
||||
self._ensure_bucket()
|
||||
normalized_key = self._make_key(key)
|
||||
|
||||
try:
|
||||
self._client.head_object(
|
||||
Bucket=self.bucket_name,
|
||||
Key=normalized_key,
|
||||
)
|
||||
return True
|
||||
except ClientError as e:
|
||||
if e.response["Error"]["Code"] == "404":
|
||||
return False
|
||||
logger.error(f"Failed to check existence of {normalized_key}: {e}")
|
||||
raise
|
||||
|
||||
def list_objects(self, prefix: str = "") -> list[str]:
|
||||
"""List all object keys under a prefix.
|
||||
|
||||
Args:
|
||||
prefix: The prefix to filter by.
|
||||
|
||||
Returns:
|
||||
List of object keys (full paths, without leading slash).
|
||||
"""
|
||||
self._ensure_bucket()
|
||||
normalized_prefix = self._make_key(prefix)
|
||||
|
||||
objects: list[str] = []
|
||||
paginator = self._client.get_paginator("list_objects_v2")
|
||||
|
||||
try:
|
||||
for page in paginator.paginate(
|
||||
Bucket=self.bucket_name,
|
||||
Prefix=normalized_prefix,
|
||||
):
|
||||
if "Contents" in page:
|
||||
for obj in page["Contents"]:
|
||||
# Return keys without leading slash
|
||||
objects.append(obj["Key"].lstrip("/"))
|
||||
return objects
|
||||
except ClientError as e:
|
||||
logger.error(f"Failed to list objects with prefix {normalized_prefix}: {e}")
|
||||
raise
|
||||
|
||||
def head_object(self, key: str) -> dict[str, Any] | None:
|
||||
"""Get metadata for an object.
|
||||
|
||||
Args:
|
||||
key: The object key/path.
|
||||
|
||||
Returns:
|
||||
Dict with metadata (size, etag, content_type, last_modified), or None if not found.
|
||||
"""
|
||||
self._ensure_bucket()
|
||||
normalized_key = self._make_key(key)
|
||||
|
||||
try:
|
||||
response = self._client.head_object(
|
||||
Bucket=self.bucket_name,
|
||||
Key=normalized_key,
|
||||
)
|
||||
return {
|
||||
"size": response.get("ContentLength"),
|
||||
"etag": response.get("ETag"),
|
||||
"content_type": response.get("ContentType"),
|
||||
"last_modified": response.get("LastModified"),
|
||||
}
|
||||
except ClientError as e:
|
||||
if e.response["Error"]["Code"] == "404":
|
||||
return None
|
||||
logger.error(f"Failed to get metadata for {normalized_key}: {e}")
|
||||
raise
|
||||
Reference in New Issue
Block a user