diff --git a/archivist/arch.md b/archivist/arch.md index 7e32521..c0be7bb 100644 --- a/archivist/arch.md +++ b/archivist/arch.md @@ -201,17 +201,41 @@ When the bucket is versioned, any deletion creates a delete marker and the data ## 7. Review Dashboard -The review dashboard is a static frontend — plain HTML/CSS/JS files, no build toolchain — served by PocketBase's built-in static file serving from the `pb_public/` directory shipped in the image next to `pb_data/`. The browser persists everything through the standard PocketBase API, whose request hooks already carry the correction, validation and deletion semantics (US-22 / US-23 / US-24): the dashboard never implements its own save path. Only what the standard API cannot do goes through dedicated JSON routes on the PocketBase router: +The review dashboard is a static frontend — plain HTML/CSS/JS files, no build toolchain, vendored dependencies only (no CDN: the application is 100% offline) — served by PocketBase's built-in static file serving from the `pb_public/` directory shipped in the image next to `pb_data/`. Two pages: the document list and the document detail. The browser persists everything through the standard PocketBase API, whose request hooks already carry the correction, validation and deletion semantics (US-22 / US-23 / US-24): the dashboard never implements its own save path. Only what the standard API cannot express goes through the dedicated JSON routes of section 7.3. -- File list: all files with status, filters, combined search over full text (FTS5) + metadata/fields; the search runs server-side through a dedicated search route querying the FTS5 index via `app.DB()`, no search logic in the browser. -- Detail: PDF preview streamed from S3 through `GET /api/archivist/documents/{id}/pdf` (a proxy route: the browser never reaches the S3 cluster), along with extracted fields; debug mode adds detailed log lines explaining every rule evaluation and action performed. -- Correction: performed directly from the standard PocketBase collections UI (or any PocketBase client) by editing the extracted fields. The application's `OnRecordUpdateRequest` hook on `documents` intercepts the save and appends the `corrected` event carrying the field diff — and nothing else. The UI edits one field at a time with no preview of the resulting file name, so the hook deliberately performs no template evaluation and no re-filing, and the status does not move: an inconsistent draft cannot trigger irreversible side effects. The full correction — all fields at once, dry-run of the resulting name, re-filing and validation — is the custom dashboard's job (US-3x). -- Validation: performed from the same interface by changing the status to `validated`; the hook appends the `validated` event, and any other status change is refused. A `pending` file can only leave the review queue through a full correction from the custom dashboard (US-3x). +### 7.1 Document list (US-32 / US-33) + +- Table: one row per document, paginated (50 by default), default columns `status` / `file_key` / `created`; a column chooser exposes every declared field plus `created`/`updated`, and the choice persists in the browser (`localStorage`). +- Sorting: click on a column header, ascending/descending, executed server-side through the standard PocketBase `sort` parameter. +- Per-column filters, typed from the field declaration: contains for `text`, dropdown for `enum` and `status` (vocabulary from the config metadata route), date range for `date`, tri-state for `boolean`; translated client-side into a PocketBase filter expression. +- Quick tabs with the per-status counts pre-filter the review queue (`pending` first). +- Combined search over full text (FTS5) + metadata/fields: the search runs server-side through a dedicated search route querying the FTS5 index via `app.DB()`, no search logic in the browser (US-4x). +- Bulk edit (US-33): row selection — page or whole filtered set — then one declared field and its new value in a control matching the type; applied as one standard PATCH per document, so the US-22 hook runs per record: a `corrected` event each, no re-filing, no status change; per-document success and failure are reported, one failure does not stop the rest. + +### 7.2 Document detail (US-34 to US-36) + +- Layout: two columns — the PDF preview embedded via the proxy route of section 7.3, and the correction form; below them, the document's event history, read-only, sorted by timestamp, payloads rendered as before/after diffs. +- Form: one typed control per declared field, pre-filled with the current values; mandatory fields marked; the classification errors of the `imported` event, if any, displayed at the top. +- Pipeline proposals (US-35): under each field whose value the pipeline would extract differently today, a discreet banner offers the proposed value with one-click adoption. The proposals come from the classification dry-run (section 7.3), re-run against the stored pages. +- Live name (US-35): as the form changes, the rendered target name under `files/` is recomputed by the render dry-run and displayed with its errors; nothing can be committed while errors remain. +- Exits, three and distinct: draft save (standard PATCH, US-22: values persist, no re-filing, no status change); full correction (US-36: the `/correct` route, re-filing and validation in one confirmed step); simple validation of a `processed` document (standard PATCH, US-23). A `pending` document only leaves the review queue through the full correction. + +### 7.3 Application routes + +Beyond the standard PocketBase API, authenticated only: + +- `GET /api/archivist/config` — the configuration metadata the UI needs: declared fields (name, type, mandatory), the enum vocabularies computed from the constants the rules assign, the template names. Read-only; the dashboard is entirely driven by it (columns, filters, controls). +- `GET /api/archivist/documents/{id}/pdf` — proxy route streaming the document from S3; the browser never reaches the S3 cluster. +- `POST /api/archivist/documents/{id}/dry-run` — two exclusive modes, one response shape (`fields`, `template`, `rendered`, `errors`), persisting nothing. Without a body: a classification re-run — the rule tree evaluated against the stored pages, returning the proposed fields (the source of the pipeline proposals). With field values: a template render — the values converted to their native types (per-field error on type mismatch), merged with the record values, rendered against the stored pages (placeholders may call `firstMatch`/`frDate` on the document text), returning the target name and its errors. The modes are never hybrid: user values never enter the rule-evaluation context, so the proposal always stays what the pipeline would do alone. +- `POST /api/archivist/documents/{id}/correct` — the full correction of US-36: transactional orchestration (conflict-suffix resolution, S3 copy, verification, old key deletion, record update with `file_key` and status `validated`, `corrected` event carrying the full diff, S3 rollback on mid-failure with the save refused). It reuses the engine internals (`FieldsFromRecord`, `ValidateValue`, `Render`, `ResolveConflict`, `DiffSnapshots`, `AppendEvent`), no parallel business logic. A route rather than a standard PATCH because the intent (re-file + validate) is not expressible in record data, the standard PATCH must stay side-effect-free for every client and the bulk edit (US-22 / US-33), and the operation is a multi-step transaction rather than a record save — the same line drawn by `POST /api/archivist/poll`. + +### 7.4 Standard collections UI and hooks + +- Correction: performed directly from the standard PocketBase collections UI (or any PocketBase client) by editing the extracted fields. The application's `OnRecordUpdateRequest` hook on `documents` intercepts the save and appends the `corrected` event carrying the field diff — and nothing else. The UI edits one field at a time with no preview of the resulting file name, so the hook deliberately performs no template evaluation and no re-filing, and the status does not move: an inconsistent draft cannot trigger irreversible side effects. The full correction — all fields at once, dry-run of the resulting name, re-filing and validation — is the dashboard's job (US-34 to US-36). +- Validation: performed from the same interface by changing the status to `validated`; the hook appends the `validated` event, and any other status change is refused. A `pending` file can only leave the review queue through a full correction from the dashboard (US-36). - Deletion: deleting a `documents` record from the same interface first deletes the S3 object at `file_key` (US-24); on S3 failure the deletion is refused and the record stays. The cascade delete of the `document_events` relation removes the history with the record; with versioning the object stays recoverable via its delete marker. -- Rule management: config display; dry-run on an existing document through `POST /api/archivist/documents/{id}/dry-run`, which renders the selected template from the proposed values without saving. The configuration is reloaded only on restart; there is no runtime reload. - -The static pages themselves carry no data and are public by nature; every fetch is authenticated against the PocketBase API and an unauthenticated visitor is redirected to the admin UI (`/_/`). Authentication: single account via PocketBase's standard auth. +The static pages themselves carry no data and are public by nature; every fetch is authenticated against the PocketBase API and an unauthenticated visitor is redirected to the admin UI (`/_/`). Authentication: single account via PocketBase's standard auth. The configuration is reloaded only on restart; there is no runtime reload. ## 8. Error Handling - Transient S3 error: retry with backoff, the object stays in `inbox/` and will be polled again. @@ -261,7 +285,7 @@ The scenarios form one narrative; each subtest names the US it proves: `POST /api/archivist/poll` (superuser only) runs one poll pass immediately. It exists so the gate and the operator trigger passes on demand instead of waiting for the cron; a fresh upload needs two calls because the poller waits for size/ETag stability across two passes (US-10). -The gate is a regression detector for dependency upgrades (PocketBase, minio-go, ledongthuc/pdf, expr, Garage), not an acceptance proof: acceptance criteria are reviewed on the stories themselves. `E2E_KEEP=1` leaves the containers running after the run for debugging. +When the dashboard routes land (US-30 to US-36), the gate grows matching assertions: the config metadata route, both dry-run modes, the `/correct` orchestration and the PDF proxy. The gate is a regression detector for dependency upgrades (PocketBase, minio-go, ledongthuc/pdf, expr, Garage), not an acceptance proof: acceptance criteria are reviewed on the stories themselves. `E2E_KEEP=1` leaves the containers running after the run for debugging. ## 11. Packaging diff --git a/archivist/user-stories.md b/archivist/user-stories.md index f7b3b44..4491162 100644 --- a/archivist/user-stories.md +++ b/archivist/user-stories.md @@ -176,7 +176,7 @@ Acceptance criteria: ## Epic: Data visualization (standard PocketBase UI) -The standard PocketBase collections UI is the nominal interface for reading and correcting documents. Field edits and status changes made through it (or through any PocketBase API client) are intercepted by the application's `OnRecordUpdateRequest` hook on `documents`, which records the change in the event history (US-22) and enforces the status rules (US-23) — but deliberately performs no re-filing: the UI edits one field at a time and shows no preview of the resulting file name, so an inconsistent draft cannot trigger irreversible side effects. The full correction — all fields at once, dry-run of the resulting name, re-filing and validation — is the custom dashboard's job (US-3x), and record deletions propagate to S3 (US-24). +The standard PocketBase collections UI is the nominal interface for reading and correcting documents. Field edits and status changes made through it (or through any PocketBase API client) are intercepted by the application's `OnRecordUpdateRequest` hook on `documents`, which records the change in the event history (US-22) and enforces the status rules (US-23) — but deliberately performs no re-filing: the UI edits one field at a time and shows no preview of the resulting file name, so an inconsistent draft cannot trigger irreversible side effects. The full correction — all fields at once, dry-run of the resulting name, re-filing and validation — is the review dashboard's job (US-34 to US-36), and record deletions propagate to S3 (US-24). ### US-20 — Browse documents in the collections UI As the user, @@ -213,7 +213,7 @@ Acceptance criteria: - Given a field edit is saved, when the record is persisted, then a `corrected` event is appended whose `before`/`after` diff (section-4 format) carries the changed fields only; `file_key` and `status` do not appear since they did not change. - Given a save where no declared field actually changed, when the hook runs, then no event is appended. - Given the save also changes the status, when the hook runs, then the status rules of US-23 apply on their own: `processed` → `validated` is appended as a separate `validated` event, anything else is refused. -- Given a `pending` document, when fields are edited this way, then it stays `pending` and its object stays in `pending/`; leaving the review queue requires the full correction of the custom dashboard (US-3x), whose form shows all fields and the resulting name before saving. +- Given a `pending` document, when fields are edited this way, then it stays `pending` and its object stays in `pending/`; leaving the review queue requires the full correction of the review dashboard (US-36), whose form shows all fields and the resulting name before saving. ### US-23 — Validate a document from the standard PocketBase UI As the user, @@ -239,7 +239,103 @@ Acceptance criteria: - Given the object at `file_key` is already gone from the bucket, when the hook runs, then the deletion is idempotent: the record is removed. - Given the bucket is versioned, when the object is deleted, then a delete marker keeps it recoverable; without versioning the deletion is final (same caveat as US-01). - Given a superuser issues the DELETE, when the hook runs, then the hook applies: like the append-only protection of US-04, it cannot rely on API rules alone. +## Epic: Review Dashboard (US-30 to US-36) + +The static frontend served from `pb_public/` (arch.md section 7): a document list and a document detail page, entirely driven by the configuration metadata route. No build toolchain, vendored dependencies only (100% offline). Every persistence goes through the standard PocketBase API and its hooks; the only application routes are those the standard API cannot express (config metadata, PDF proxy, dry-run, full correction). + +### US-30 — Serve the dashboard as static files + +As the user, +I want the review dashboard served by the application itself at the root URL, +so that no additional server or build step is needed. + +Acceptance criteria: + +- Given the image is built and running, when I open `http://:8090/`, then the dashboard pages are served from `pb_public/` — plain HTML/CSS/JS, no build toolchain, dependencies vendored locally, nothing loaded from a CDN. +- Given a page is opened, when it renders, then it carries no document data by itself; everything is loaded through authenticated API calls. +- Given an unauthenticated visitor, when a page requests data, then the API refuses it and the page redirects to the admin UI (`/_/`). + +### US-31 — Expose the configuration metadata + +As the dashboard, +I want the declared fields, their types and the enum vocabularies exposed by a read-only route, +so that the UI is entirely driven by the configuration. + +Acceptance criteria: + +- Given an authenticated request to `GET /api/archivist/config`, when it is served, then it returns the declared fields (name, type, mandatory), the enum vocabularies computed from the constants the rules assign, and the template names — and nothing sensitive (no S3 credentials). +- Given an unauthenticated request, when it is served, then it is refused. +- Given the configuration is reloaded only on restart, when the route is called, then it serves the in-memory compiled configuration (the same the pipeline uses). + +### US-32 — Browse documents in a configurable table + +As the user, +I want the documents in a table whose columns I choose, sortable and filterable per column, +so that I can work through the review queue efficiently. + +Acceptance criteria: + +- Given the list page, when it opens, then it shows a paginated table with `status`, `file_key` and `created` by default, and a column chooser exposing every declared field plus `created`/`updated`; the choice persists in the browser. +- Given a column header, when I click it, then the list is sorted by that column server-side and the state persists. +- Given a column, when I open its filter, then the control matches the field type (contains for `text`, dropdown for `enum` and `status`, range for `date`, tri-state for `boolean`) and the filter is applied server-side through a PocketBase filter expression. +- Given the list, when I open it, then quick tabs show the per-status counts and pre-filter the review queue (`pending` first). + +### US-33 — Change one field for a selection of documents + +As the user, +I want to select several documents and change one declared field for all of them, +so that a systematic correction does not require opening every document. + +Acceptance criteria: + +- Given the list, when I select rows (page or whole filtered set), then a bulk action proposes choosing a declared field and a new value in a control matching the field type, and shows the number of impacted documents. +- Given the bulk action is confirmed, when it runs, then it issues one standard PATCH per document — the US-22 hook applies per record: a `corrected` event each, no re-filing, no status change. +- Given one document of the selection fails, when the bulk action finishes, then the others are applied and the failure is reported with its error. + +### US-34 — Preview a document and correct each field + +As the user, +I want a page showing the PDF preview and a form for every field, +so that I can correct a document with its content in view. + +Acceptance criteria: + +- Given the detail page of a document, when it opens, then the PDF preview is embedded via `GET /api/archivist/documents/{id}/pdf` and the declared fields appear in typed controls pre-filled with the current values; mandatory fields are marked. +- Given a `pending` document, when the page opens, then the classification errors of its `imported` event are displayed at the top of the form. +- Given the page, when I look below the form, then the document's event history is listed read-only, sorted by timestamp, payloads rendered as before/after diffs. +- Given the form, when I save with the draft button, then the values persist through a standard PATCH (US-22: no re-filing, no status change). +- Given a `processed` document with no field change, when I use the validate button, then the US-23 transition applies. + +### US-35 — Propose the pipeline values and the resulting name live + +As the user, +I want, for each field, the value the pipeline would extract today when it differs from the current value, and the resulting file name recomputed as I type, +so that I see the effect of a correction before committing it. + +Acceptance criteria: + +- Given the detail page, when it opens, then it calls `POST /api/archivist/documents/{id}/dry-run` with no body; the route re-runs the rule tree against the stored pages and returns the proposed fields, the selected template, the rendered name and the errors, persisting nothing. +- Given a field whose proposed value differs from the current value, when the response arrives, then a discreet banner under the control offers the proposed value with one-click adoption. +- Given the form is modified, when the typing settles, then the page calls the dry-run route with the proposed field values; the route converts them to their native types (per-field error on mismatch), merges them with the record values, renders the template against the stored pages and returns the target name and its errors in the same response shape. +- Given the two modes, when either runs, then they are exclusive: user values never enter the rule-evaluation context, and no record, event or S3 object is touched. +- Given the dry-run returns errors, when the form still allows a draft save, then the full correction refuses to run while errors remain. + +### US-36 — Correct fully and validate + +As the user, +I want a single action that re-files the document under its corrected name and validates it, +so that a reviewed document leaves the queue in one confirmed step. + +Acceptance criteria: + +- Given the form, when I trigger the full correction, then `POST /api/archivist/documents/{id}/correct` runs the transactional orchestration: conflict-suffix resolution, S3 copy, target verification, deletion of the old key, record update (`file_key`, status `validated`), and a `corrected` event carrying the full diff. +- Given any step fails midway, when the route handles it, then the save is refused, the S3 state is restored, and the record and history are left unchanged. +- Given the rendered name equals the current `file_key`, when the correction runs, then the copy and delete are skipped, the status alone changes, and `file_key` is absent from the diff. +- Given a `pending` document, when the correction succeeds, then the `pending/` object is removed and the document leaves the review queue. +- Given an unauthenticated client, when it calls the route, then it is refused. +- Given the implementation, when it is reviewed, then it reuses the engine internals (`FieldsFromRecord`, `ValidateValue`, `Render`, `ResolveConflict`, `DiffSnapshots`, `AppendEvent`) with no parallel business logic. + ## Stories deferred to later epics -- US-3x (Custom review dashboard): file list with status, combined search (FTS5 + fields), detail with PDF preview streamed from S3, correction form pre-filled from the last event snapshot, rule dry-run. The dashboard owns the full correction: its form shows every field and the resulting file name (dry-run) before saving; saving a full correction re-files the object and sets the status to `validated` — the side effects US-22 deliberately keeps out of the standard UI. Validation remains that of US-23. +- US-5x (Config editor): pages to declare fields and build rules with user assistance — guided rule construction, test bench against existing documents through the dry-run routes. - US-4x (FTS5): full-text index synchronization via hooks.