Technical Architecture

1. Overview

Single docker container Go application built on PocketBase as an embedded framework (https://pocketbase.io/). The binary contains:

PocketBase's HTTP/API server and admin dashboard;
the custom review dashboard;
the S3 poller and the PDF processing pipeline, executed through PocketBase's standard job scheduler (app.Cron());
the declarative rule engine (JSONC config + expr-lang/expr expressions);
access to the self-hosted S3 storage via the MinIO Go SDK.

Everything runs offline on the server infrastructure: no cloud APIs, no outbound calls other than to the S3 cluster.

2. Stack

Component	Choice
Framework / runtime	PocketBase (Go framework, single binary)
Language	Go
Scheduling	`app.Cron()` (S3 poller, periodic tasks)
S3	MinIO Go SDK (`github.com/minio/minio-go/v7`)
PDF text extraction	`github.com/ledongthuc/pdf` (native text, no OCR)
Rule expressions	`github.com/expr-lang/expr`
Regex	stdlib `regexp` (RE2); `regexp2` only if a lookaround proves indispensable
Config validation	`santhosh-tekuri/jsonschema` (schema generated from structs via `invopop/jsonschema`)
Database	SQLite (embedded in PocketBase)
Full-text search	SQLite FTS5 index synchronized via hooks

3. S3 Layout

A single versioned bucket:

inbox/       # drop zone for new PDFs (via Nextcloud / S3 client)
pending/     # files awaiting human review
files/       # renamed and filed files (configured prefix structure)

Nextcloud mounts the bucket as external storage: it is the user-facing interface for uploading, downloading, and sharing. The application only manipulates S3 keys, never a local filesystem for documents.

4. Data Model (PocketBase Collections)

`documents`

One row per processed PDF file:

sha256: the sha256 is a primary key as the system will reject as duplicate any file with the same hash
original_filename: original file name in inbox/
file_key: current S3 key of the file
extracted_text: full text extracted from all pages (base of the FTS5 index)
first_page_text: first-page text (extraction base)
size: file size
status: pending | processed | validated

all the fields declared by the user in the configuration file

The append-only document_events collection is the single source of truth for a document's history; no history is stored on the documents record itself.

`document_events`

Append-only history: document, event_type (classified, moved, corrected, validated), payload as JSON, timestamp.

5. Configuration File (JSONC)

A config.jsonc file kept alongside the binary, reloaded on restart. Three sections:

fields: declaration of fields to extract from the files:
- name: name of the field
- type: string | number | boolean
- mandatory: true | false
rules: an ordered tree of rules. Each rule contains four keys:
- when (required): an expr-lang/expr expression evaluated against the first-page text and the current field values; the rest of the rule runs only if it returns true (e.g. firstPageText contains "Relevé de compte", or emittingCompany in ["BRED", "Fortunéo"] && documentType == "Relevé de compte")
- set: list of assignments, evaluated in order — { "field": ..., "value": ... } for a constant, { "field": ..., "eval": ... } for an expression evaluated in the same context
- delete: list of field names to unset (they become undefined again for later rules and template rendering)
- rules: child rules, evaluated in order only if the parent when matched; nesting is recursive
The application registers a built-in helper in the expression environment: firstMatch(regex) returns the first match of the regex in the first-page text (first capture group when present), enabling dynamic extraction (dates, invoice numbers).
templates: named naming templates (e.g. bank_statement: {sender}/{date}_{type}_{reference}.pdf), with required-field validation; the {date} placeholder is normalized to yyyy-mm-dd ("/" is not usable in a S3 key) — a template with a missing mandatory field renders the result invalid and routes the file to pending/. The template is chosen by the rules themselves: template is a field declared in the fields section like any other and set with set (e.g. set template = "bank_statement" when emittingCompany in ["BRED", "Fortunéo"] and documentType == "Relevé de compte"). If no rule sets template, the special default template is used. A template value with no matching entry renders the result invalid and routes the file to pending/.

Validation on load: JSON schema, regex compilation, expr expression compilation (when and eval) against a typed context (Document{FirstPageText, Fields}) with the firstMatch helper, regex precompilation and caching. Full rejection if invalid, no partial processing.

6. Processing Pipeline

Executed by the PocketBase cron job (configurable interval, e.g. every minute):

Poll: list objects in inbox/; an object is a candidate only if no documents record references its key yet, and its size and ETag are stable across two consecutive polls (write-completion wait).
Download: fetch the object into memory (streamed), no disk writes. The sha256 is computed during download; if a documents record with the same hash already exists, the object is a duplicate: the inbox/ key is deleted (delete marker, data recoverable), a message is logged in the application logs, and processing stops there.
Extraction: first-page text (for rule evaluation) + full text (for FTS) via ledongthuc/pdf, including headers and footers
Rule evaluation: rules evaluated in order against the first-page text and the fields set so far; full evaluation log written to the application logs (built-in dry-run mode for debugging from the dashboard).
Validation: a result is valid if and only if: all rules have been processed with no error, all mandatory fields are identified, the selected template exists in the configuration, and the naming template evaluated with no error. Otherwise: route directly to pending/.
Record: create the documents record (sha256, original filename, current inbox key, extracted texts, fields, status) and add a classified event including the original filename. The record exists before any physical move, so an interrupted filing leaves a recoverable state.
Filing:
- valid → resolve the target name first: if a file with the same name already exists in the target prefix, append a number to the filename; then copy to the new key (files/ + rendered template — the template renders the full object path, there is no separate prefix), verify the target object exists, and delete the inbox/ key only after the copy is confirmed; finally update the documents record with the final, conflict-resolved filename (file_key, status processed, moved event).
- invalid → move to pending/ (incremental suffix on name conflict); update file_key; status pending.

The bucket is versioned: any inbox/ object deletion creates a delete marker, the data remains recoverable.

7. Review Dashboard

Custom pages on the PocketBase instance:

File list: all files with status, filters, combined search over full text (FTS5) + metadata/fields.
Detail: PDF preview streamed from S3 through a dedicated route, along with extracted fields; debug mode adds detailed log lines explaining every rule evaluation and action performed.
Correction: forms to edit the extracted fields. A save is only possible if the selected template can be evaluated without error; otherwise the form refuses the save. A successfully corrected file is re-filed (copy to the new key, delete the old one, incremental suffix on conflict) and takes the status validated, whatever its previous status; a corrected pending file is thereby removed from pending/. History is preserved.
Validation: transition from processed to validated. A pending file can only leave the review queue through a correction that can be saved, which itself sets the status to validated.
Rule management: config display and dry-run on an existing document. The configuration is reloaded only on restart; there is no runtime reload.

Authentication: single account via PocketBase's standard auth.

8. Error Handling

Transient S3 error: retry with backoff, the object stays in inbox/ and will be polled again.
Unreadable PDF: pending status, reason recorded in the payload of the classified event.
Target key conflict (two files renamed identically, whether filing to files/ or moving to pending/): automatic incremental suffix, event logged.
Invalid config: error message displayed and the application does not start.

8.8 KiB Raw Blame History