thomas/multiarch-docker-images
1
0
Fork 0
mirror of https://github.com/napnap75/multiarch-docker-images.git synced 2026-11-11 00:22:12 +01:00
Code Issues Projects Releases Wiki Activity

feat(routes) config, pdf proxy, dry-run and correct routes + e2e coverage (US-31, US-34..36)

Browse Source
This commit is contained in:
thomas
2026-11-07 19:18:34 +01:00
parent 2ba78724cc
commit b7bfe1152a
4 changed files with 760 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files
archivist/arch.md
+37 -17
View File
@@ -31,7 +31,8 @@ Everything runs offline on the server infrastructure: no cloud APIs, no outbound
## 3. S3 Layout
A single bucket. Versioning is not required, but it is strongly recommended: without it, any deletion performed by the application (or by a bug) is final; with it, delete markers keep deleted objects recoverable.
A single bucket. Versioning is not required, but it is strongly recommended: without it, any deletion performed by the application (or by a bug) is final; with it, delete markers keep deleted objects rec
overable.
```
inbox/ # drop zone for new PDFs (via Nextcloud / S3 client)
@@ -63,7 +64,8 @@ The payload is self-contained: each event carries its own type, its own timestam
Common payload keys:
- `event_type`: `imported` | `corrected` | `validated`
- `timestamp`: RFC 3339 date of the event
- `timestamp`: RFC 3339 date of the
event
- `before` / `after`: the event's diff, never full snapshots — a key appears on a side only when its value changes in the event: same value on both sides, it is omitted; absent from `before` and present in `after`, the field was added (every `after` key at import, since nothing existed before); present in `before` and absent from `after`, it was deleted. `file_key` (the S3 key, unified with the `documents` record field) follows the same rule and appears only when the object actually moves; at import the status does not exist before, so `before` carries no `status`. A missing key means the value is unknown, there is no `null`
Per event type:
@@ -105,7 +107,8 @@ On classification failure, the document is routed to `pending/` and the payload
"timestamp": "2026-01-15T14:23:07Z",
"errors": ["mandatory field missing: emittingCompany"],
"before": {
"file_key": "inbox/scan_2026-01-15.pdf"
"file_key": "inbox/scan
_2026-01-15.pdf"
},
"after": {
"file_key": "pending/scan_2026-01-15.pdf",
@@ -163,7 +166,8 @@ The diff is derived by comparing the two snapshots: a field present in `before`
A `config.jsonc` file kept alongside the binary, reloaded on restart. Three sections:
1. `fields`: declaration of fields to extract from the files:
- `name`: name of the field
- `na
me`: name of the field
- `type`: `text` | `number` | `date` | `boolean` | `enum`
- `mandatory`: true | false
@@ -182,7 +186,8 @@ A `config.jsonc` file kept alongside the binary, reloaded on restart. Three sect
The expression environment provides expr's standard `date(value, layout)` builtin plus three built-in helpers registered by the application:
- `date(value, layout)`: expr's standard builtin. Parses a string into a native date using a Go date layout (e.g. `date(`15/01/2026`, `02/01/2006`)`). The format is carried by the parsing rule, so it can differ per document type. It only knows English month names: see `frDate` below for French text.
- `date(value, layout)`: expr's standard builtin. Parses a string into a native date using a Go date layout (e.g. `date(`15/01/2026`, `02/01/2006`)`). The format is carried by the parsing rule, so it can differ per document type. It only knows English month names: see
`frDate` below for French text.
- `firstMatch(regex, texts?)`: returns the first match of the regex in the document text (first capture group when present), or an empty string when no match is found — e.g. `firstMatch(`[0-9]{2}/[0-9]{2}/[0-9]{4}`, pages[0])`. The optional second argument is an expression evaluated to the text(s) to search:
- a single page: `pages[0]` (the first page)
- a slice: `pages[0:3]` (pages 1 to 3, expr's standard slice syntax, end exclusive)
@@ -197,7 +202,8 @@ A `config.jsonc` file kept alongside the binary, reloaded on restart. Three sect
- `value`: a native date (e.g. `emissionDate`); a missing or invalid date is a rule error and routes the document to `pending/`.
- `unit`: `month` | `quarter` | `half` | `year`.
- `shift`: `-1` (previous period), `0` (period containing the date) or `+1` (next period) — e.g. a rent receipt issued mid-month uses the containing month, a service-charge call issued at the end of a quarter uses the next quarter, a bank statement uses the previous month.
- Returns a French string: `Janvier 2026`, `T2 2026`, `S1 2026` or `2026`.
- Returns a French string: `Janvier 2026`, `T2 2026`, `S1 2026` or `202
6`.
- The optional `layout` argument is a standard Go reference-date layout applied to the period's first day, with month names rendered in French by the application (default layouts: `January 2006` for `month`, `2006` for `year`); it is ignored for `quarter` and `half`, whose formats (`T<quarter> <year>`, `S<half> <year>`) are fixed.
3. `templates`: named naming templates (e.g. `bank_statement: {emittingCompany}/{emissionDate.Format(`2006-01-02`)}_{documentType}_{bankAccount}.pdf`). A template is a literal string containing `{...}` placeholders:
@@ -214,14 +220,16 @@ Validation on load: JSON schema, regex compilation, `expr` expression compilatio
## 6. Processing Pipeline
Executed by the PocketBase cron job (configurable interval, e.g. every minute):
Executed by the PocketBase cron job (
configurable interval, e.g. every minute):
1. **Poll**: list objects in `inbox/`; an object is a candidate only if no `documents` record references its key yet, and its size and ETag are stable across two consecutive polls (write-completion wait). The same pass also picks up interrupted filings: any `documents` record whose `file_key` is still under `inbox/` (a filing that was interrupted between record creation and the physical move) is completed as described below.
2. **Download**: fetch the object into memory (streamed), no disk writes. The sha256 is computed during download; if a `documents` record with the same hash already exists, the object is a duplicate: the `inbox/` key is deleted (a no-data-loss case — the original with the same content stays filed), a message is logged via the PocketBase logger (section 9), and processing stops there. By design, no `document_events` entry is recorded: the duplicate is not a document of its own, and the trace lives in the PocketBase logs only.
3. **Extraction**: page-by-page text (one entry per page, for rule evaluation and the FTS index) via `ledongthuc/pdf`, including headers and footers
4. **Rule evaluation**: rules evaluated in order against the document pages and the fields set so far; full evaluation log written to the PocketBase logger at Debug level (section 9; the same trace feeds the built-in dry-run mode for debugging from the dashboard).
5. **Validation**: a result is valid if and only if: all rules have been processed with no error, all mandatory fields are identified, the selected template exists in the configuration, and the naming template evaluated with no error. Otherwise: route directly to `pending/`.
6. **Record**: create the `documents` record (sha256, current inbox key, pages, fields, status) and add an `imported` event including the original key (`before.file_key`). The record exists before any physical move, so an interrupted filing leaves a state the poller recovers automatically on its next pass.
6. **Record**: create the `documents` record (sha256, current inbox key, pages, fields, status) and add an `imported` event including the original key (`before.file_key`). The record exists before any physical move, so an interrupted filing leaves a state the poller recovers automatically on its ne
xt pass.
7. **Filing**:
- valid → resolve the target name first: if a file with the same name already exists in the target prefix, append a number to the filename; then copy to the new key (`files/` + rendered template — the template renders the full object path, there is no separate prefix), verify the target object exists, and delete the `inbox/` key only after the copy is confirmed; finally update the `documents` record with the final, conflict-resolved filename (`file_key`, status `processed`); the whole import is recorded as the single `imported` event of section 4.
- invalid → move to `pending/` (incremental suffix on name conflict); update `file_key`; status `pending`.
@@ -235,7 +243,8 @@ The review dashboard is a static frontend — plain HTML/CSS/JS files, no build
### 7.1 Document list (US-32 / US-33)
- Table: one row per document, paginated (50 by default), default columns: all mandatory fields plus `status` and `file_key`; a column chooser exposes every declared field plus `created`/`updated`, and the choice persists in the browser (`localStorage`).
- Sorting: click on a column header, ascending/descending, executed server-side through the standard PocketBase `sort` parameter.
- Sorting: click on
a column header, ascending/descending, executed server-side through the standard PocketBase `sort` parameter.
- Per-column filters, typed from the field declaration: contains for `text`, dropdown for `enum` and `status` (vocabulary from the config metadata route), date range for `date`, tri-state for `boolean`; translated client-side into a PocketBase filter expression.
- Quick tabs with the per-status counts pre-filter the review queue (`pending` first).
- Combined search over full text (FTS5) + metadata/fields: the search runs server-side through a dedicated search route querying the FTS5 index via `app.DB()`, no search logic in the browser (US-4x).
@@ -246,7 +255,8 @@ The review dashboard is a static frontend — plain HTML/CSS/JS files, no build
- Layout: two columns — the PDF preview embedded via the proxy route of section 7.3, and the correction form; below them, the document's event history, read-only, sorted by timestamp, payloads rendered as before/after diffs.
- Form: one typed control per declared field, pre-filled with the current values; mandatory fields marked; the classification errors of the `imported` event, if any, displayed at the top.
- Pipeline proposals (US-35): under each field whose value the pipeline would extract differently today, a discreet banner offers the proposed value with one-click adoption. The proposals come from the classification dry-run (section 7.3), re-run against the stored pages.
- Live name (US-35): as the form changes, the rendered target name under `files/` is recomputed by the render dry-run and displayed with its errors; the draft save remains available, but the full correction refuses to run while errors remain.
- Live name (US-35): as the form changes, the rendered target name under `files/` is recomputed by the render dry-run and displayed with its errors; the draft save remains available, but the full correction refuses to run while errors remai
n.
- Exits, three and distinct: draft save (standard PATCH, US-22: values persist, no re-filing, no status change); full correction (US-36: the `/correct` route, re-filing and validation in one confirmed step); simple validation of a `processed` document (standard PATCH, US-23). A `pending` document only leaves the review queue through the full correction.
### 7.3 Application routes
@@ -256,7 +266,8 @@ Beyond the standard PocketBase API, authenticated only:
- `GET /api/archivist/config` — the configuration metadata the UI needs: declared fields (name, type, mandatory), the enum vocabularies computed from the constants the rules assign, the template names. Read-only; the dashboard is entirely driven by it (columns, filters, controls).
- `GET /api/archivist/documents/{id}/pdf` — proxy route streaming the document from S3; the browser never reaches the S3 cluster.
- `POST /api/archivist/documents/{id}/dry-run` — two exclusive modes, one response shape (`fields`, `template`, `rendered`, `errors`), persisting nothing. Without a body: a classification re-run — the rule tree evaluated against the stored pages, returning the proposed fields (the source of the pipeline proposals). With field values: a template render — the values converted to their native types (per-field error on type mismatch), merged with the record values, rendered against the stored pages (placeholders may call `firstMatch`/`frDate` on the document text), returning the target name and its errors. The modes are never hybrid: user values never enter the rule-evaluation context, so the proposal always stays what the pipeline would do alone.
- `POST /api/archivist/documents/{id}/correct` — the full correction of US-36: transactional orchestration (conflict-suffix resolution, S3 copy, verification, old key deletion, record update with `file_key` and status `validated`, `corrected` event carrying the full diff, S3 rollback on mid-failure with the save refused). It reuses the engine internals (`FieldsFromRecord`, `ValidateValue`, `Render`, `ResolveConflict`, `DiffSnapshots`, `AppendEvent`), no parallel business logic. A route rather than a standard PATCH because the intent (re-file + validate) is not expressible in record data, the standard PATCH must stay side-effect-free for every client and the bulk edit (US-22 / US-33), and the operation is a multi-step transaction rather than a record save — the same line drawn by `POST /api/archivist/poll`.
- `POST /api/archivist/documents/{id}/correct` — the full correction of US-36: transactional orchestration (conflict-suffix resolution, S3 copy, verification, old key deletion, record update with `file_key` and status `validated`, `corrected` event carrying the full diff, S3 rollback on mid-failure with the save refused). It reuses the engine internals (`FieldsFromRecord`, `Va
lidateValue`, `Render`, `ResolveConflict`, `DiffSnapshots`, `AppendEvent`), no parallel business logic. A route rather than a standard PATCH because the intent (re-file + validate) is not expressible in record data, the standard PATCH must stay side-effect-free for every client and the bulk edit (US-22 / US-33), and the operation is a multi-step transaction rather than a record save — the same line drawn by `POST /api/archivist/poll`.
### 7.4 Standard collections UI and hooks
@@ -264,14 +275,16 @@ Beyond the standard PocketBase API, authenticated only:
- Validation: performed from the same interface by changing the status to `validated`; the hook appends the `validated` event, and any other status change is refused. A `pending` file can only leave the review queue through a full correction from the dashboard (US-36).
- Deletion: deleting a `documents` record from the same interface first deletes the S3 object at `file_key` (US-24); on S3 failure the deletion is refused and the record stays. The cascade delete of the `document_events` relation removes the history with the record; with versioning the object stays recoverable via its delete marker.
The static pages themselves carry no data and are public by nature; every fetch is authenticated against the PocketBase API and an unauthenticated visitor is redirected to the admin UI (`/_/`). Authentication: single account via PocketBase's standard auth. The configuration is reloaded only on restart; there is no runtime reload.
The static pages themselves carry no data and are public by nature; every fetch is authenticated against the PocketBase API and an unauthenticated visitor is redirected to the admin UI (`/_/`). Authentication: single ac
count via PocketBase's standard auth. The configuration is reloaded only on restart; there is no runtime reload.
### 7.5 Design (Beszel reference)
The dashboard's visual design takes Beszel (https://github.com/henrygd/beszel) as its reference. Beszel is built on shadcn/ui + Tailwind with semantic HSL CSS variables and a light/dark pair; archivist reproduces its palette, token names and layout feel in a single hand-written stylesheet (`pb_public/styles.css`) — no Tailwind, no CSS framework, consistent with the no-build-toolchain constraint of section 7.
- Semantic tokens, same names and values as Beszel's own stylesheet. Light theme: `--background: hsl(30 8% 98%)`, `--foreground: hsl(30 0% 10%)`, `--card: hsl(30 0% 100%)`, `--card-foreground: hsl(240 6% 12%)`, `--primary: hsl(240 5.88% 10%)`, `--primary-foreground: hsl(30 0% 100%)`, `--secondary: hsl(240 4.76% 95.88%)`, `--muted: hsl(26 6% 94%)`, `--muted-foreground: hsl(24 2.79% 35.1%)`, `--accent: hsl(20 23.08% 94%)`, `--destructive: hsl(0 66% 53%)`, `--border: hsl(30 8.11% 85.49%)`, `--input: hsl(30 4.29% 72.55%)`, `--ring: hsl(30 3.97% 49.41%)`, `--table-header: hsl(225 6% 97%)`, `--radius: 0.8rem`.
- Dark theme: `--background: hsl(220 5.5% 9%)`, `--foreground: hsl(220 2% 97%)`, `--card: hsl(220 5.5% 10.5%)`, `--card-foreground: hsl(220 2% 97%)`, `--primary: hsl(220 2% 96%)`, `--primary-foreground: hsl(220 4% 10%)`, `--secondary: hsl(220 4% 16%)`, `--muted: hsl(220 6% 16%)`, `--muted-foreground: hsl(220 4% 67%)`, `--accent: hsl(220 5% 15.5%)`, `--destructive: hsl(0 62% 46%)`, `--border: hsl(220 3% 17%)`, `--input: hsl(220 4% 22%)`, `--ring: hsl(220 4% 80%)`, `--table-header: hsl(220 6% 13%)`, plus `color-scheme: dark` so native controls follow.
- Theme switching, same scheme as Beszel: a `dark` class on `<html>`, defaulted from the OS `prefers-color-scheme`, with a manual toggle persisted in `localStorage`; the class is set by an inline script in the page head to avoid a flash of the wrong theme on first paint.
- Theme switching, same scheme as Beszel: a `dark` class on `<html>`, defaulted from the OS `prefers-color-scheme`, with a manual toggle persisted in `localStorage`; the class is set by an inline script in the page head to avoid a flash of the wrong theme on first
paint.
- Component language: cards on `--card` over `--background`, hairline `--border`, `--radius` rounding, sticky table headers on `--table-header` (the document list), secondary actions on `--secondary`/`--muted`, `--destructive` reserved for destructive actions and errors, focus rings on `--ring`.
- Status colors, the only domain additions to Beszel's tokens: `pending` maps to a warm badge on `--accent`, `validated` to a green from Beszel's chart scale (`hsl(140 50% 48%)` light / `hsl(140 49% 60%)` dark), `processed` to a neutral `--muted` badge.
- Typography: Beszel ships InterVariable; archivist does not ship a font file — it uses the system font stack with Inter as the preferred family when installed, to keep the image lean. A vendored Inter subset remains an accepted option if rendering proves unsatisfactory.
@@ -291,7 +304,8 @@ Two channels, split by lifecycle:
Levels used:
- `Error`: a failed operation (S3 error, event append failure).
- `Erro
r`: a failed operation (S3 error, event append failure).
- `Warn`: a degraded but non-fatal state (versioning disabled, schema column removed, stuck record whose inbox object disappeared).
- `Info`: pipeline outcomes (document imported, duplicate removed, interrupted filing recovered).
- `Debug`: the per-rule evaluation trace of section 6, step 4.
@@ -316,22 +330,28 @@ The scenarios form one narrative; each subtest names the US it proves:
2. Import — a valid simulated statement is classified and filed to `files/` with its `imported` event (US-10 to US-17).
3. Duplicate — the same content re-uploaded is deleted from `inbox/`, no record, no event (US-12).
4. Pending — a missing mandatory date routes the document to `pending/` with `errors` in the imported event and the recognized fields kept (US-13 to US-15).
5. Correction — a field PATCH through the API persists the values with no side effect (no re-filing, no status change), `corrected` event carrying the field diff (US-22).
5. Correction — a field PATCH through the API persists the values with no side effect (no re-fil
ing, no status change), `corrected` event carrying the field diff (US-22).
6. Validation — a status-only PATCH appends the `validated` event, `file_key` untouched (US-23).
7. Append-only — PATCH and DELETE on `document_events` are refused (US-04).
8. Deletion — a deleted record removes the S3 object; the history follows via the cascade delete (US-24).
9. Recovery — a stuck record (`file_key` still under `inbox/`, created directly through the API) is completed by the next pass (US-11).
10. Restart — the container restarts with all state intact.
11. Config metadata — the configuration route serves the declared fields, the enum vocabularies and the template names (US-31).
12. PDF proxy — the route streams the S3 object, refused without superuser auth (US-34).
13. Dry-run — both modes return the shared response shape and persist nothing (US-35).
14. Full correction — a pending document is re-filed and validated; value errors leave everything unchanged; an unchanged name skips the S3 move and the diff drops file_key (US-36).
`POST /api/archivist/poll` (superuser only) runs one poll pass immediately. It exists so the gate and the operator trigger passes on demand instead of waiting for the cron; a fresh upload needs two calls because the poller waits for size/ETag stability across two passes (US-10).
When the dashboard routes land (US-30 to US-36), the gate grows matching assertions: the config metadata route, both dry-run modes, the `/correct` orchestration and the PDF proxy. The gate is a regression detector for dependency upgrades (PocketBase, minio-go, ledongthuc/pdf, expr, Garage), not an acceptance proof: acceptance criteria are reviewed on the stories themselves. `E2E_KEEP=1` leaves the containers running after the run for debugging.
The dashboard application routes are covered by the gate: the config metadata route, both dry-run modes, the `/correct` orchestration and the PDF proxy (US-30 to US-36). The gate is a regression detector for dependency upgrades (PocketBase, minio-go, ledongthuc/pdf, expr, Garage), not an acceptance proof: acceptance criteria are reviewed on the stories themselves. `E2E_KEEP=1` leaves the containers running after the run for debugging.
## 11. Packaging
`Dockerfile`, two-stage build:
- Builder: `golang:1.27-alpine` (matches the `go 1.27` directive of PocketBase v0.41.1), CGO disabled, static binary, `TARGETARCH` honored for multi-platform builds under buildx. `go mod tidy` runs there because `go.sum` is not committed; once it is committed, the line becomes `go mod download` for fully reproducible builds.
- Runtime: `scratch` with the `/archivist` binary, the CA bundle (the S3 endpoint is HTTPS by default), the timezone database (the cron and date rendering honor `TZ`), the static review-dashboard frontend copied to `/app/pb_public/` (read-only at runtime), and `/app/pb_data` + `/tmp`, the only writable directories, owned by the unprivileged runtime user 65532. `config.jsonc` is never baked in: it is mounted at runtime, `ARCHIVIST_CONFIG` defaulting to `config.jsonc` relative to `/app`.
- Runtime: `scratch` with the `/archivist` binary, the CA bundle (the S3 endpoint is HTTPS by default), the timezone database (the cron and date rendering honor `TZ`), the static review-dashboard frontend copied to `/app/pb_public/` (read-only at runtime), and `/app/pb_data` + `/tmp`, the only writable directories, owned by
the unprivileged runtime user 65532. `config.jsonc` is never baked in: it is mounted at runtime, `ARCHIVIST_CONFIG` defaulting to `config.jsonc` relative to `/app`.
The container runs as 65532:65532 (non-root), exposes 8090, declares `/app/pb_data` as a volume, and has no shell: `docker exec` is not available. `.dockerignore` keeps the build context to the Go sources and the dashboard frontend; the E2E harness is not part of the image.
archivist/e2e/e2e_test.go
+291 -15
View File
@@ -3,7 +3,8 @@
// The suite is one narrative: it starts a disposable Garage and the image
// under test, then follows documents through the whole pipeline - import,
// duplicate, pending, correction, validation, append-only, deletion, recovery,
// restart. It is a regression gate for dependency upgrades (PocketBase,
// restart, dashboard routes (config metadata, PDF proxy, dry-run, full
// correction). It is a regression gate for dependency upgrades (PocketBase,
// minio-go, ledongthuc/pdf, expr, Garage), not an acceptance proof.
//
// Run it from the archivist/ directory against an already built image:
@@ -80,7 +81,8 @@ func TestMain(m *testing.M) {
}
}
if os.Getenv("E2E_KEEP") == "1" {
fmt.Println("E2E: E2E_KEEP=1, containers left running:", appC, garageC)
fmt.
Println("E2E: E2E_KEEP=1, containers left running:", appC, garageC)
return
}
teardown()
@@ -147,6 +149,7 @@ func startGarage() error {
if _, err := garage("bucket", "create", bucket); err != nil {
return err
}
keyOut, err := garage("key", "create", bucket)
if err != nil {
return err
@@ -210,7 +213,8 @@ func startApp(image string) error {
if _, err := run("docker", args...); err != nil {
return fmt.Errorf("app container: %w", err)
}
if err := retry(60*time.Second, checkHealth); err != nil {
if err := retry(60*time.Second, checkHealth);
err != nil {
if out, _ := run("docker", "logs", appC); out != "" {
return fmt.Errorf("app never became healthy; logs:\n%s", out)
}
@@ -296,7 +300,8 @@ func firstHexToken(s string) string {
return ""
}
func lineValue(s, prefix string) string {
func lineValue(s, prefix string) strin
g {
for _, line := range strings.Split(s, "\n") {
t := strings.TrimSpace(line)
if strings.HasPrefix(t, prefix) {
@@ -373,7 +378,8 @@ func listRecords(t *testing.T, collection, filter, sort string) []map[string]any
q.Set("perPage", "200")
status, out := call(t, http.MethodGet, "/api/collections/"+collection+"/records?"+q.Encode(), nil)
if status != 200 {
t.Fatalf("list %s: status %d: %v", collection, status, out)
t.Fatalf("list %s: status %d: %v
", collection, status, out)
}
items, _ := out["items"].([]any)
recs := make([]map[string]any, 0, len(items))
@@ -450,7 +456,8 @@ func s3Exists(t *testing.T, key string) bool {
return err == nil
}
// --- in-memory test PDFs (no fixtures, no generator dependency) ----------
// --- in-memory test PDFs (no fixtures, no ge
nerator dependency) ----------
// makePDF builds a minimal valid single-page PDF whose page text is the
// given lines. The classification outcome is the proof that the
@@ -500,7 +507,8 @@ func putPDF(t *testing.T, key string, lines ...string) string {
t.Fatalf("PutObject %s: %v", key, err)
}
sum := sha256.Sum256(data)
return hex.EncodeToString(sum[:])
return hex.EncodeToString(su
m[:])
}
// --- the config.jsonc mounted into the image under test -----------------
@@ -513,7 +521,8 @@ var testConfigRaw = `
{ "name": "emittingCompany", "type": "text", "mandatory": true },
{ "name": "documentType", "type": "text", "mandatory": true },
{ "name": "emissionDate", "type": "date", "mandatory": true },
{ "name": "bankAccount", "type": "text" }
{ "name": "bankAccount", "type": "text" },
{ "name": "statementType", "type": "enum" }
],
"rules": [
{
@@ -523,7 +532,8 @@ var testConfigRaw = `
{ "field": "documentType", "value": "Releve de compte" },
{ "field": "emittingCompany", "eval": "firstMatch(~BRED|Fortuneo~, pages[0])" },
{ "field": "emissionDate", "eval": "date(firstMatch(~Date : ([0-9]{2}/[0-9]{2}/[0-9]{4})~, pages[0]), ~02/01/2006~)" },
{ "field": "bankAccount", "eval": "firstMatch(~Compte ([A-Za-z]+)~, pages[0])" }
{ "field": "bankAccount", "eval": "firstMatch(~Compte ([A-Za-z]+)~, pages[0])" },
{ "field": "statementType", "value": "bank" }
]
}
],
@@ -535,6 +545,19 @@ var testConfigRaw = `
var testConfig = strings.ReplaceAll(testConfigRaw, "~", "`")
// unauth makes one API call with the superuser token temporarily removed.
func unauth(t *testing.T, method, path string, body any) (int, map[string]any) {
t.Helper()
saved := token
token = ""
status, out, err := api(method, path, body)
token = saved
if err != nil {
t.Fatalf("%s %s: %v", method, path, err)
}
return status, out
}
func TestE2E(t *testing.T) {
// 1. Startup (US-01, US-03, US-04): the app started although Garage
@@ -549,7 +572,8 @@ func TestE2E(t *testing.T) {
}
}
// PocketBase writes log batches with a small delay.
var found bool
var fou
nd bool
deadline := time.Now().Add(15 * time.Second)
for !found && time.Now().Before(deadline) {
for _, l := range pbLogs(t) {
@@ -607,7 +631,8 @@ func TestE2E(t *testing.T) {
t.Fatalf("file_key: %v, want %v", doc["file_key"], want)
}
if !s3Exists(t, want) {
t.Fatalf("object %s missing from S3", want)
t.Fatalf("object %s missing from S3", wan
t)
}
if s3Exists(t, "inbox/statement-1.pdf") {
t.Fatal("source object still in inbox/")
@@ -662,7 +687,8 @@ func TestE2E(t *testing.T) {
p := eventPayloadOfType(t, eventsOf(t, doc["id"].(string)), "imported")
errs, ok := p["errors"].([]any)
if !ok || len(errs) == 0 {
t.Fatalf("want errors in the imported event, got %v", p)
t.Fatalf("
want errors in the imported event, got %v", p)
}
after, _ := p["after"].(map[string]any)
if after["documentType"] != "Releve de compte" {
@@ -711,7 +737,8 @@ func TestE2E(t *testing.T) {
t.Fatalf("%s must not appear in a field-edit diff: %v", s, before)
}
if _, has := after[s]; has {
t.Fatalf("%s must not appear in a field-edit diff: %v", s, after)
t.Fatalf("%s must not appear in a field-edi
t diff: %v", s, after)
}
}
})
@@ -761,7 +788,8 @@ func TestE2E(t *testing.T) {
map[string]any{"event_type": "corrected"}); status < 400 {
t.Fatalf("PATCH on document_events must be refused, got %d", status)
}
if status, _ := call(t, http.MethodDelete, "/api/collections/document_events/records/"+id, nil); status < 400 {
if status, _ := call(t, http.MethodDelete, "/api/collections/docum
ent_events/records/"+id, nil); status < 400 {
t.Fatalf("DELETE on document_events must be refused, got %d", status)
}
if n := len(listRecords(t, "document_events", "", "")); n != len(evs) {
@@ -812,7 +840,8 @@ func TestE2E(t *testing.T) {
t.Fatalf("stuck record creation: %d %v", status, out)
}
poll(t) // the recovery runs at the start of the pass
want := "files/BRED/2026-02_Releve de compte_Thomas.pdf"
want := "files/BRED/202
6-02_Releve de compte_Thomas.pdf"
doc := oneRecord(t, "documents", "(sha256='"+hash+"')")
if doc["file_key"] != want {
t.Fatalf("file_key: %v, want %v", doc["file_key"], want)
@@ -839,4 +868,251 @@ func TestE2E(t *testing.T) {
t.Fatalf("documents lost across restart: %d -> %d", len(before), len(after))
}
})
// 11. Dashboard config (US-31): the metadata route serves the declared
// fields, the enum vocabularies computed from the rule constants and
// the template names; unauthenticated requests are refused.
t.Run("config: metadata route serves fields, enums and templates", func(t *testing.T) {
status, out := call(t, http.MethodGet, "/api/archivist/config", nil)
if status != 200 {
t.Fatalf("config route: status %d: %v", status, out)
}
fields, _ := out["fields"].([]any)
if len(fields) != 6 {
t.Fatalf("want 6 declared fields, got %d", len(fields))
}
byName := map[string]map[string]any{}
for _, f := range fields {
m, _ := f.(map[string]any)
byName[m["name"].(string)] = m
}
if byName["emissionDate"]["type"] != "date" || byName["emissionDate"]["mandatory"] != true {
t.Fatalf("emissionDate metadata: %v", byName["emissionDate"])
}
if byName["statementType"]["type"] != "enum" {
t.Fatalf("statementType metadata: %v", byName["statementType"])
}
if byName["bankAccount"]["mandatory"] != false {
t.Fatalf("bankAccount metadata: %v", byName["bankAccount"])
}
enums, _ := out["enums"].(map[string]any)
vocab, _ := enums["statementType"].([]any)
if len(vocab) != 1 || vocab[0] != "bank" {
t.Fatalf("statementType vocabulary: %v", enums)
}
templates, _ := out["templates"].([]any)
wantTpl := map[string]bool{"bank_statement": false, "default": false}
for _, name := range templates {
if _, ok := wantTpl[name.(string)]; ok {
wantTpl[name.(string)] = true
}
}
for name, seen := range wantTpl {
if !seen {
t.Fatalf("template %s missing from %v", name, templates)
}
}
blob, _ := json.Marshal(out)
if strings.Contains(string(blob), s3Secret) {
t.Fatal("config route leaks the S3 secret")
}
if status, _ := unauth(t, http.MethodGet, "/api/archivist/config", nil); status != 403 {
t.Fatalf("config without superuser auth: want 403, got %d", status)
}
})
// 12. PDF proxy (US-34): the route streams the S3 object; the browser
// never reaches the S3 cluster.
t.Run("pdf: the proxy streams the S3 object", func(t *testing.T) {
doc := oneRecord(t, "documents", "(file_key='files/BRED/2025-12_Releve de compte_Thomas.pdf')")
id := doc["id"].(string)
req, _ := http.NewRequest(http.MethodGet, baseURL+"/api/archivist/documents/"+id+"/pdf", nil)
req.Header.Set("Authorization", token)
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatal(err)
}
defer resp.Body.Close()
data, _ := io.ReadAll(resp.Body)
if resp.StatusCode != 200 {
t.Fatalf("pdf proxy: status %d", resp.StatusCode)
}
if ct := resp.Header.Get("Content-Type"); !strings.HasPrefix(ct, "application/pdf") {
t.Fatalf("content type: %q", ct)
}
sum := sha256.Sum256(data)
if hex.EncodeToString(sum[:]) != doc["sha256"] {
t.Fatal("the proxied bytes differ from the stored object")
}
if status, _ := unauth(t, http.MethodGet, "/api/archivist/documents/"+id+"/pdf", nil); status != 403 {
t.Fatalf("pdf without superuser auth: want 403, got %d", status)
}
})
// 13. Dry-run (US-35): the two exclusive modes share one response
// shape and persist nothing - no record, no event, no S3 object.
t.Run("dry-run: proposals and live render, nothing persisted", func(t *testing.T) {
doc := oneRecord(t, "documents", "(file_key='files/BRED/2026-01_Releve de compte_Thomas.pdf')")
id := doc["id"].(string)
path := "/api/archivist/documents/" + id + "/dry-run"
evCount := len(eventsOf(t, id))
// Mode A - no body: the pipeline proposal.
status, out := call(t, http.MethodPost, path, nil)
if status != 200 {
t.Fatalf("dry-run proposal: %d %v", status, out)
}
fields, _ := out["fields"].(map[string]any)
if fields["emittingCompany"] != "BRED" || fields["bankAccount"] != "Thomas" {
t.Fatalf("proposed fields: %v", fields)
}
if fields["emissionDate"] != "2026-01-15T00:00:00Z" {
t.Fatalf("proposed emissionDate: %v", fields["emissionDate"])
}
if out["template"] != "bank_statement" {
t.Fatalf("template: %v", out["template"])
}
if out["rendered"] != "BRED/2026-01_Releve de compte_Thomas.pdf" {
t.Fatalf("rendered: %v", out["rendered"])
}
if errs, _ := out["errors"].([]any); len(errs) != 0 {
t.Fatalf("unexpected errors: %v", errs)
}
// Mode B - values: the merged render, same response shape.
status, out = call(t, http.MethodPost, path, map[string]any{"bankAccount": "Nappez"})
if status != 200 {
t.Fatalf("dry-run with values: %d %v", status, out)
}
fields, _ = out["fields"].(map[string]any)
if fields["bankAccount"] != "Nappez" {
t.Fatalf("merged fields: %v", fields)
}
if out["rendered"] != "BRED/2026-01_Releve de compte_Nappez.pdf" {
t.Fatalf("rendered: %v", out["rendered"])
}
// Mode B - type error: per-field error, empty rendered name.
status, out = call(t, http.MethodPost, path, map[string]any{"emissionDate": "not-a-date"})
if status != 200 {
t.Fatalf("dry-run type error: %d %v", status, out)
}
if errs, _ := out["errors"].([]any); len(errs) == 0 {
t.Fatalf("want a per-field error, got %v", out)
}
if out["rendered"] != "" {
t.Fatalf("rendered must be empty on error: %v", out["rendered"])
}
// Unknown fields are refused outright.
if status, _ = call(t, http.MethodPost, path, map[string]any{"nope": "x"}); status != 400 {
t.Fatalf("unknown field must be refused with 400, got %d", status)
}
// Nothing persisted.
doc2 := oneRecord(t, "documents", "(id='" + id + "')")
if doc2["file_key"] != doc["file_key"] || doc2["status"] != doc["status"] {
t.Fatal("dry-run must not touch the record")
}
if n := len(eventsOf(t, id)); n != evCount {
t.Fatalf("dry-run created %d event(s)", n-evCount)
}
if status, _ := unauth(t, http.MethodPost, path, nil); status != 403 {
t.Fatalf("dry-run without superuser auth: want 403, got %d", status)
}
})
// 14. Full correction (US-36).
t.Run("correction: a pending document is re-filed and validated", func(t *testing.T) {
hash := putPDF(t, "inbox/undated-2.pdf", "BRED", "Releve de compte", "Compte Thomas")
poll(t)
poll(t)
doc := oneRecord(t, "documents", "(sha256='" + hash + "')")
id := doc["id"].(string)
if doc["status"] != "pending" {
t.Fatalf("status: %v", doc["status"])
}
want := "files/BRED/2026-03_Releve de compte_Thomas.pdf"
status, out := call(t, http.MethodPost, "/api/archivist/documents/"+id+"/correct",
map[string]any{"emissionDate": "2026-03-10"})
if status != 200 {
t.Fatalf("correct refused: %d %v", status, out)
}
doc = oneRecord(t, "documents", "(id='" + id + "')")
if doc["file_key"] != want {
t.Fatalf("file_key: %v, want %v", doc["file_key"], want)
}
if doc["status"] != "validated" {
t.Fatalf("status: %v", doc["status"])
}
if !s3Exists(t, want) {
t.Fatalf("object %s missing from S3", want)
}
if s3Exists(t, "pending/undated-2.pdf") {
t.Fatal("the pending/ object was not removed")
}
p := eventPayloadOfType(t, eventsOf(t, id), "corrected")
before, _ := p["before"].(map[string]any)
after, _ := p["after"].(map[string]any)
if before["file_key"] != "pending/undated-2.pdf" {
t.Fatalf("before.file_key: %v", before)
}
if after["file_key"] != want || after["status"] != "validated" {
t.Fatalf("after: %v", after)
}
if _, has := before["emissionDate"]; has {
t.Fatalf("emissionDate was unknown before the correction: %v", before)
}
if after["emissionDate"] != "2026-03-10T00:00:00Z" {
t.Fatalf("after.emissionDate: %v", after)
}
if status, _ := unauth(t, http.MethodPost, "/api/archivist/documents/"+id+"/correct",
map[string]any{"emissionDate": "2026-03-10"}); status != 403 {
t.Fatalf("correct without superuser auth: want 403, got %d", status)
}
})
t.Run("correction: a value error leaves everything unchanged", func(t *testing.T) {
doc := oneRecord(t, "documents", "(file_key='files/BRED/2026-01_Releve de compte_Thomas.pdf')")
id := doc["id"].(string)
evCount := len(eventsOf(t, id))
status, out := call(t, http.MethodPost, "/api/archivist/documents/"+id+"/correct",
map[string]any{"emissionDate": "not-a-date"})
if status != 400 {
t.Fatalf("want 400, got %d %v", status, out)
}
doc2 := oneRecord(t, "documents", "(id='" + id + "')")
if doc2["file_key"] != doc["file_key"] || doc2["status"] != doc["status"] {
t.Fatal("a refused correction must leave the record unchanged")
}
if n := len(eventsOf(t, id)); n != evCount {
t.Fatalf("a refused correction created %d event(s)", n-evCount)
}
})
t.Run("correction: an unchanged name skips S3 and the diff drops file_key", func(t *testing.T) {
doc := oneRecord(t, "documents", "(file_key='files/BRED/2026-01_Releve de compte_Thomas.pdf')")
id := doc["id"].(string)
status, out := call(t, http.MethodPost, "/api/archivist/documents/"+id+"/correct", map[string]any{})
if status != 200 {
t.Fatalf("correct refused: %d %v", status, out)
}
doc2 := oneRecord(t, "documents", "(id='" + id + "')")
if doc2["file_key"] != doc["file_key"] {
t.Fatalf("file_key must not change: %v -> %v", doc["file_key"], doc2["file_key"])
}
if strings.Contains(doc2["file_key"].(string), " (") {
t.Fatalf("a conflict suffix was added: %v", doc2["file_key"])
}
p := eventPayloadOfType(t, eventsOf(t, id), "corrected")
before, _ := p["before"].(map[string]any)
after, _ := p["after"].(map[string]any)
if _, has := before["file_key"]; has {
t.Fatalf("file_key must be absent from the diff: %v", before)
}
if _, has := after["file_key"]; has {
t.Fatalf("file_key must be absent from the diff: %v", after)
}
})
}
archivist/internal/pbapp/routes.go
+426
View File
@@ -0,0 +1,426 @@
// Routes.go implements the review-dashboard application routes of
// arch.md section 7.3: configuration metadata (US-31), PDF proxy (US-34),
// dry-run in its two exclusive modes (US-35) and the full correction (US-36).
package pbapp
import (
"bytes"
"encoding/json"
"fmt"
"io"
"sort"
"strconv"
"strings"
"time"
"ganigeorgiev.com/pocketbase/pocketbase/core"
"github.com/napnap75/archivist/internal/config"
"github.com/napnap75/archivist/internal/storage"
)
// BindRoutes registers the dashboard application routes. They are
// superuser-only: the dashboard authenticates through the standard
// PocketBase auth, but these routes bypass the record API and its hooks.
func BindRoutes(app core.App, cfg *config.Config, s3 *storage.Client) {
app.OnServe().BindFunc(func(e *core.ServeEvent) error {
e.Router.GET("/api/archivist/config", func(re *core.RequestEvent) error {
if !re.HasSuperuserAuth() {
return re.ForbiddenError("superusers only", nil)
}
return re.JSON(200, configMetadata(cfg))
})
e.Router.GET("/api/archivist/documents/{id}/pdf", func(re *core.RequestEvent) error {
return handlePdf(re, s3)
})
e.Router.POST("/api/archivist/documents/{id}/dry-run", func(re *core.RequestEvent) error {
return handleDryRun(re, cfg)
})
e.Router.POST("/api/archivist/documents/{id}/correct", func(re *core.RequestEvent) error {
return handleCorrect(re, cfg, s3)
})
return e.Next()
})
}
// configMetadata serves the read-only configuration the UI is driven by:
// declared fields, enum vocabularies, the fixed statuses, template names.
// Nothing technical leaks (no S3 credentials, no rules).
func configMetadata(cfg *config.Config) map[string]any {
return map[string]any{
"fields": cfg.FieldList,
"enums": enumVocabularies(cfg),
"statuses": []string{"pending", "processed", "validated"},
"templates": sortedTemplateNames(cfg),
}
}
func sortedTemplateNames(cfg *config.Config) []string {
out := make([]string, 0, len(cfg.Templates))
for name := range cfg.Templates {
out = append(out, name)
}
sort.Strings(out)
return out
}
// enumVocabularies computes each enum field vocabulary from the constants
// the rules assign (US-31): the UI dropdowns offer exactly what the rule
// engine can set.
func enumVocabularies(cfg *config.Config) map[string][]string {
enums := map[string][]string{}
seen := map[string]map[string]bool{}
var walk func(rules []*config.Rule)
walk = func(rules []*config.Rule) {
for _, r := range rules {
for _, s := range r.Set {
if s.Eval != "" {
continue
}
f, ok := cfg.Fields[s.Field]
if !ok || f.Type != "enum" {
continue
}
v, _ := s.Value.(string)
if v == "" {
continue
}
if seen[s.Field] == nil {
seen[s.Field] = map[string]bool{}
}
if !seen[s.Field][v] {
seen[s.Field][v] = true
enums[s.Field] = append(enums[s.Field], v)
}
}
walk(r.Rules)
}
}
walk(cfg.Rules)
for _, values := range enums {
sort.Strings(values)
}
return enums
}
// handlePdf streams the document from S3; the browser never reaches the
// S3 cluster (US-34).
func handlePdf(re *core.RequestEvent, s3 *storage.Client) error {
if !re.HasSuperuserAuth() {
return re.ForbiddenError("superusers only", nil)
}
id := re.Request.PathValue("id")
record, err := re.App.FindRecordById("documents", id)
if err != nil {
return re.JSON(404, map[string]any{"message": "document not found"})
}
key := record.GetString("file_key")
if key == "" {
return re.JSON(404, map[string]any{"message": "document has no file"})
}
ctx := re.Request.Context()
if !s3.Exists(ctx, key) {
return re.JSON(404, map[string]any{"message": "object not found in S3", "key": key})
}
data, _, err := s3.Get(ctx, key)
if err != nil {
return re.JSON(500, map[string]any{"message": "S3 download failed", "error": err.Error()})
}
re.Response.Header().Set("Content-Disposition", "inline; filename=\""+fileName(key)+"\"")
return re.Blob(200, "application/pdf", data)
}
func fileName(key string) string {
if i := strings.LastIndex(key, "/"); i >= 0 {
return key[i+1:]
}
return key
}
// dryRunResponse is the single response shape of the two exclusive
// dry-run modes (US-35).
type dryRunResponse struct {
Fields map[string]any "json:"fields""
Template string "json:"template""
Rendered string "json:"rendered""
Errors []string "json:"errors""
}
func handleDryRun(re *core.RequestEvent, cfg *config.Config) error {
if !re.HasSuperuserAuth() {
return re.ForbiddenError("superusers only", nil)
}
id := re.Request.PathValue("id")
record, err := re.App.FindRecordById("documents", id)
if err != nil {
return re.JSON(404, map[string]any{"message": "document not found"})
}
pages := recordPages(record)
body, err := io.ReadAll(re.Request.Body)
if err != nil {
return re.JSON(400, map[string]any{"message": "unreadable body", "error": err.Error()})
}
// Mode A - no body: the pipeline proposal, the rule tree re-evaluated
// against the stored pages. User values never take part.
if len(bytes.TrimSpace(body)) == 0 {
fields, rendered, errs := cfg.Evaluate(pages)
return re.JSON(200, dryRunResponse{
Fields: fields,
Template: templateName(fields),
Rendered: rendered,
Errors: nonNil(errs),
})
}
// Mode B - field values: the template render of the merged record and
// user values, against the stored pages.
var submitted map[string]any
if err := json.Unmarshal(body, &submitted); err != nil {
return re.JSON(400, map[string]any{"message": "invalid JSON body", "error": err.Error()})
}
merged, convErrs, unknown, _ := mergeSubmitted(cfg, FieldsFromRecord(cfg, record), submitted)
if len(unknown) > 0 {
return re.JSON(400, map[string]any{"message": "unknown field(s)", "errors": unknown})
}
tplName, rendered, evalErrs := evaluateMerged(cfg, merged, pages)
if len(convErrs) > 0 {
rendered = ""
}
return re.JSON(200, dryRunResponse{
Fields: merged,
Template: tplName,
Rendered: rendered,
Errors: nonNil(append(convErrs, evalErrs...)),
})
}
// handleCorrect is the full correction of US-36: conversion, render,
// conflict-suffix resolution, S3 copy, verification, source deletion,
// record update to status validated, corrected event with the full diff -
// and a rollback leaving everything unchanged when any step fails.
func handleCorrect(re *core.RequestEvent, cfg *config.Config, s3 *storage.Client) error {
if !re.HasSuperuserAuth() {
return re.ForbiddenError("superusers only", nil)
}
id := re.Request.PathValue("id")
record, err := re.App.FindRecordById("documents", id)
if err != nil {
return re.JSON(404, map[string]any{"message": "document not found"})
}
pages := recordPages(record)
body, err := io.ReadAll(re.Request.Body)
if err != nil {
return re.JSON(400, map[string]any{"message": "unreadable body", "error": err.Error()})
}
var submitted map[string]any
if len(bytes.TrimSpace(body)) > 0 {
if err := json.Unmarshal(body, &submitted); err != nil {
return re.JSON(400, map[string]any{"message": "invalid JSON body", "error": err.Error()})
}
}
merged, convErrs, unknown, cleared := mergeSubmitted(cfg, FieldsFromRecord(cfg, record), submitted)
if len(convErrs) > 0 || len(unknown) > 0 {
return re.JSON(400, map[string]any{"message": "invalid values", "errors": append(convErrs, unknown...)})
}
tplName, rendered, evalErrs := evaluateMerged(cfg, merged, pages)
if len(evalErrs) > 0 {
return re.JSON(400, map[string]any{"message": "correction refused", "errors": evalErrs})
}
before := SnapshotFromRecord(cfg, record, true)
oldKey := record.GetString("file_key")
target := "files/" + rendered
ctx := re.Request.Context()
refiled := false
finalKey := oldKey
if target != oldKey {
finalKey, err = s3.ResolveConflict(ctx, target)
if err != nil {
return re.JSON(500, map[string]any{"message": "conflict resolution failed", "error": err.Error()})
}
if err := s3.Copy(ctx, oldKey, finalKey); err != nil {
return re.JSON(500, map[string]any{"message": "S3 copy failed", "error": err.Error()})
}
if !s3.Exists(ctx, finalKey) {
return re.JSON(500, map[string]any{"message": "target not verified after copy", "key": finalKey})
}
if err := s3.Remove(ctx, oldKey); err != nil {
_ = s3.Remove(ctx, finalKey) // best-effort rollback of the copy
return re.JSON(500, map[string]any{"message": "S3 source deletion failed", "error": err.Error()})
}
refiled = true
record.Set("file_key", finalKey)
}
for name := range cleared {
record.Set(name, nil)
}
for name, v := range merged {
record.Set(name, v)
}
record.Set("status", "validated")
after := SnapshotFromRecord(cfg, record, true)
// The record save and the corrected event commit together; on failure
// the S3 state is restored (US-36).
txErr := re.App.RunInTransaction(func(tx core.App) error {
if err := tx.Save(record); err != nil {
return err
}
return AppendEvent(tx, record.Id, "corrected", before, after)
})
if txErr != nil {
if refiled {
_ = s3.Copy(ctx, finalKey, oldKey)
_ = s3.Remove(ctx, finalKey)
}
return re.JSON(500, map[string]any{"message": "correction failed; everything rolled back", "error": txErr.Error()})
}
return re.JSON(200, map[string]any{
"id": record.Id,
"file_key": record.GetString("file_key"),
"status": "validated",
"template": tplName,
"rendered": rendered,
"fields": merged,
})
}
// recordPages decodes the stored page texts of a document (US-13).
func recordPages(record *core.Record) []string {
var pages []string
if raw := record.GetString("pages"); raw != "" {
if err := json.Unmarshal([]byte(raw), &pages); err != nil {
return nil
}
}
return pages
}
// mergeSubmitted converts the submitted values to their native types and
// merges them over the record values. A null or empty string clears the
// field. It reports per-field conversion errors, unknown fields, and the
// cleared names (the caller must unset them on the record).
func mergeSubmitted(cfg *config.Config, merged map[string]any, submitted map[string]any) (map[string]any, []string, []string, map[string]bool) {
var convErrs, unknown []string
cleared := map[string]bool{}
for name, raw := range submitted {
f, ok := cfg.Fields[name]
if !ok {
unknown = append(unknown, "unknown field: "+name)
continue
}
v, clear, err := convertUserValue(f, raw)
if err != nil {
convErrs = append(convErrs, fmt.Sprintf("field %s: %v", name, err))
continue
}
if clear {
cleared[name] = true
delete(merged, name)
continue
}
merged[name] = v
}
return merged, convErrs, unknown, cleared
}
// convertUserValue converts one submitted JSON value to the declared field
// type (US-35 mode B).
func convertUserValue(f config.Field, raw any) (v any, clear bool, err error) {
if raw == nil {
return nil, true, nil
}
switch f.Type {
case "text", "enum":
s, ok := raw.(string)
if !ok {
return nil, false, fmt.Errorf("(%s): value is not a string", f.Type)
}
if s == "" {
return nil, true, nil
}
return s, false, nil
case "number":
switch t := raw.(type) {
case float64:
return t, false, nil
case string:
if t == "" {
return nil, true, nil
}
n, perr := strconv.ParseFloat(t, 64)
if perr != nil {
return nil, false, fmt.Errorf("(number): %q is not numeric", t)
}
return n, false, nil
default:
return nil, false, fmt.Errorf("(number): value is not numeric")
}
case "date":
s, ok := raw.(string)
if !ok {
return nil, false, fmt.Errorf("(date): value is not a string")
}
if s == "" {
return nil, true, nil
}
for _, layout := range []string{"2006-01-02", time.RFC3339} {
if d, perr := time.Parse(layout, s); perr == nil {
return d, false, nil
}
}
return nil, false, fmt.Errorf("(date): cannot parse %q", s)
case "boolean":
b, ok := raw.(bool)
if !ok {
return nil, false, fmt.Errorf("(boolean): value is not a boolean")
}
return b, false, nil
}
return nil, false, fmt.Errorf("unsupported field type %q", f.Type)
}
// evaluateMerged mirrors cfg.Evaluate on an already-built field map: the
// mandatory check plus the template render against the stored pages. The
// rendered name is empty when any error remains.
func evaluateMerged(cfg *config.Config, merged map[string]any, pages []string) (string, string, []string) {
var errs []string
for _, f := range cfg.FieldList {
if f.Mandatory {
if v, ok := merged[f.Name]; !ok || v == nil {
errs = append(errs, "mandatory field missing: "+f.Name)
}
}
}
tplName := templateName(merged)
rendered := ""
if tpl, ok := cfg.Templates[tplName]; !ok {
errs = append(errs, "unknown template: "+tplName)
} else if r, rerr := tpl.Render(merged, pages); rerr != nil {
errs = append(errs, rerr.Error())
} else {
rendered = r
}
if len(errs) > 0 {
rendered = ""
}
return tplName, rendered, errs
}
func templateName(fields map[string]any) string {
if v, ok := fields["template"].(string); ok && v != "" {
return v
}
return "default"
}
func nonNil(errs []string) []string {
if errs == nil {
return []string{}
}
return errs
}
archivist/main.go
+6 -2
View File
@@ -65,7 +65,8 @@ func LoadEnvConfig() (*EnvConfig, error) {
return c, nil
}
func configPathFromArgs(args []string, defaultPath string) (string, error) {
func configPathFromArgs(args []strin
g, defaultPath string) (string, error) {
var configPath string
flags := pflag.NewFlagSet("archivist", pflag.ContinueOnError)
flags.SetInterspersed(true)
@@ -116,13 +117,16 @@ func main() {
}
// US-01: versioning recommendation, via the PocketBase logger
// (visible in the admin dashboard, Dashboard > Logs).
// (visible in the admin dashboard, Das
hboard > Logs).
if s3.StartupWarning != "" {
app.Logger().Warn(s3.StartupWarning, "bucket", envCfg.S3Bucket)
}
// US-04 / US-22 / US-23 / US-24: request hooks on documents and document_events.
pbapp.BindHooks(app, cfg, s3)
// US-31 / US-35 / US-36: the review-dashboard routes (config, pdf, dry-run, correct).
pbapp.BindRoutes(app, cfg, s3)
// Manual poll pass: a superuser-only route used by the E2E gate and the
// operator to trigger a pass without waiting for the cron (arch.md 10).