mirror of
https://github.com/napnap75/multiarch-docker-images.git
synced 2026-11-11 00:22:12 +01:00
archivist: simplify brief, remove redundancies
This commit is contained in:
+13
-18
@@ -2,35 +2,30 @@
|
||||
|
||||
## 1. Vision
|
||||
|
||||
A personal, 100% offline tool that watches an S3 bucket hosted on a self-hosted storage cluster, automatically identifies the type of each incoming PDF (invoice, purchase order, contract, etc.) from its content, applies the corresponding naming template by extracting the fields (date, type of document, references), and moves the object into an organized S3 prefix structure. Users upload, download, and share files through Nextcloud, which mounts the bucket as external storage. Uncertain or incomplete results go through human review in a local web dashboard.
|
||||
A personal, 100% offline tool that watches the `inbox/` prefix of a self-hosted S3 bucket, identifies each incoming PDF (invoice, purchase order, contract, etc.) from the text content of its first page, applies the corresponding naming template by extracting fields, and files the object into an organized prefix structure. Nextcloud, which mounts the bucket as external storage, is the user-facing interface for uploading, downloading, and sharing. Files that cannot be classified with certainty go to `pending/` for review in a local web dashboard.
|
||||
|
||||
## 2. Problem and Context
|
||||
|
||||
PDFs arrive continuously in an inbox S3 prefix (dropped via Nextcloud by the user or by upload automations). Current naming is manual or inconsistent, making documents hard to find and archive. The user already owns around 2000 files already correctly named.
|
||||
PDFs arrive continuously in the inbox prefix via Nextcloud or upload automations. Current naming is manual or inconsistent, making documents hard to find and archive. Around 2000 existing files are already correctly named.
|
||||
|
||||
## 3. Objectives
|
||||
|
||||
- Extract the emitting company, document type, additional fields, and emission date from the text content of the first page of the document (including headers and footers), not from the current object name.
|
||||
- Rename and move the object within the bucket into an organized prefix structure depending on the extracted data.
|
||||
- Unambiguous deterministic rules and valid fields: automatic processing. If rules cannot determine a valid result or extraction is incomplete, queue the file with status `pending` for human review in the dashboard.
|
||||
- Run fully offline on self-hosted infrastructure: no cloud APIs, no data ever leaves the server.
|
||||
- Extract the emitting company, document type, additional fields, and emission date from the text content of the first page (including headers and footers), never from the current object name.
|
||||
- Deterministic processing: unambiguous rules and valid fields → automatic filing. Anything uncertain or incomplete → status `pending` for human review in the dashboard.
|
||||
- Fully offline on self-hosted infrastructure: no cloud APIs, no data ever leaves the server.
|
||||
|
||||
## 4. Scope
|
||||
|
||||
- Ingestion by polling the `inbox/` prefix of a S3 bucket for new objects only. Nextcloud (External Storage app) is the user-facing interface to upload, download, and share files.
|
||||
- Extract the text from the first page of each PDF. All files are text-based; OCR for scanned/image-only documents is out of scope.
|
||||
- Deterministic template classification using configured rules, extracted text, and reference data (inline constant lists within rule expressions, e.g. a list of known banks); uncertain or insufficient results are routed to `pending/` for manual correction.
|
||||
- Fields to extract, rule-based extraction, and templates for renaming the file declared in configuration (JSONC format), not in code.
|
||||
- Storage: self-hosted S3-compatible with a single versioned bucket; objects addressed by S3 keys, no local filesystem for documents.
|
||||
- Local web dashboard: all processed files with status (pending / processed / validated), file detail page with PDF preview streamed from S3, template and field correction, validation, history, and full-text search across the entire extracted document in addition to metadata/field search.
|
||||
- Pre-existing files (the ~2000 already-named documents present in the bucket) are out of scope: no initial import or indexing.
|
||||
- Ingestion: polling the `inbox/` prefix for new objects. OCR for scanned/image-only documents is out of scope.
|
||||
- Fields, rule-based extraction, and naming templates declared in configuration (JSONC), not in code; constant reference lists live inline in rule expressions.
|
||||
- Storage: single versioned self-hosted S3 bucket; objects addressed by S3 keys, no local filesystem for documents.
|
||||
- Dashboard: file list with status, PDF preview streamed from S3, field and template correction, validation, history, and full-text search across the entire document in addition to metadata/field search.
|
||||
- The ~2000 pre-existing files are out of scope: no initial import or indexing.
|
||||
|
||||
## 5. User and Usage
|
||||
|
||||
Single user (personal use), technically proficient. Nominal flow:
|
||||
|
||||
1. A PDF is uploaded into the `inbox/` prefix via Nextcloud (or any S3 client).
|
||||
2. The poller detects the new object, waits for write completion, downloads it, extracts text from document, applies deterministic classification and field-extraction rules, then validates.
|
||||
3. A deterministic result with valid fields: the object is copied to its renamed key in the target prefix and the inbox copy is deleted only after the file is properly processed. The status `processed` is applied to this file.
|
||||
4. If deterministic rules cannot produce a valid result: the file is moved to `pending/`, the status `pending` is applied and the file appears in the dashboard review queue; the user will have to correct and may validate it.
|
||||
5. In the dashboard, the user may review the already `processed` or `validated` files to correct the fields and validate them.
|
||||
1. A PDF is dropped into `inbox/` via Nextcloud (or any S3 client).
|
||||
2. The tool classifies it: a valid result is filed automatically (`processed`); otherwise the file lands in `pending/` in the review queue.
|
||||
3. From the dashboard, the user corrects fields (a correctable file is re-filed and becomes `validated`) and validates `processed` files.
|
||||
|
||||
Reference in New Issue
Block a user