From 1224ff916d7fe22003e5c80d273831928e7fafa8 Mon Sep 17 00:00:00 2001 From: napnap75 Date: Thu, 12 Nov 2026 19:35:56 +0100 Subject: [PATCH] =?UTF-8?q?Archivist=20:=20page=20Outils=20+=20outil=20?= =?UTF-8?q?=C2=AB=20Recherche=20des=20fichiers=20perdus=20=C2=BB=20(US-40,?= =?UTF-8?q?=20US-41)=20(#16)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Add tools page with find-lost-files consistency tool (US-40, US-41) Co-authored-by: napnap75 * tools page: hide the job panel when there is nothing to show, guard against undefined rendering Co-authored-by: napnap75 * inbox: leave non-PDF objects in place, never imported (US-12) Co-authored-by: napnap75 --------- Co-authored-by: Vibe Nuage Agent Co-authored-by: napnap75 --- archivist/README.md | 16 +- archivist/arch.md | 51 ++- archivist/e2e/e2e_test.go | 18 + archivist/internal/pbapp/setup.go | 39 +- archivist/internal/pipeline/pipeline.go | 31 +- archivist/internal/pipeline/pipeline_test.go | 24 ++ archivist/internal/storage/s3.go | 49 ++- archivist/internal/tools/routes.go | 55 +++ archivist/internal/tools/tools.go | 393 +++++++++++++++++++ archivist/internal/tools/tools_test.go | 274 +++++++++++++ archivist/main.go | 3 + archivist/pb_public/document.html | 1 + archivist/pb_public/index.html | 1 + archivist/pb_public/styles.css | 21 + archivist/pb_public/tools.html | 29 ++ archivist/pb_public/tools.js | 161 ++++++++ archivist/user-stories.md | 38 +- 17 files changed, 1185 insertions(+), 19 deletions(-) create mode 100644 archivist/internal/pipeline/pipeline_test.go create mode 100644 archivist/internal/tools/routes.go create mode 100644 archivist/internal/tools/tools.go create mode 100644 archivist/internal/tools/tools_test.go create mode 100644 archivist/pb_public/tools.html create mode 100644 archivist/pb_public/tools.js diff --git a/archivist/README.md b/archivist/README.md index d37bf54..7999111 100644 --- a/archivist/README.md +++ b/archivist/README.md @@ -14,6 +14,8 @@ The application is 100% offline: no cloud APIs, no data ever leaves your server. Duplicates (same sha256) are ignored: the `inbox/` copy is deleted, the original stays in place — no data is lost, the content is identical. +Files in `inbox/` that are not PDFs (no `%PDF-` header) are left as-is: they are never imported and stay in place until you remove or rename them. + ## Prerequisites - An S3 bucket, with three prefixes: `inbox/`, `pending/`, `files/`. The prefixes are created on first startup if they do not exist. Versioning is not required, but it is strongly recommended: without it, a file deleted by the application or by a bug is lost for good; with it, deleted objects stay recoverable via delete markers. The application logs a warning at startup when versioning is disabled (visible in the admin dashboard, Dashboard > Logs). @@ -119,6 +121,18 @@ A commented `config.jsonc` example file is provided in the repository. The dashb - Correction: a form to edit the fields. A draft save is always possible: it persists the values with no side effect (no template evaluation, no re-filing, no status change). Only the full correction — re-filing and validation — is refused while the rendered name still has errors. A corrected file is re-filed and becomes `validated`; a corrected `pending` file leaves the review queue. - Validation: direct transition from `processed` to `validated`. - History: every document has a complete, timestamped trace (import, correction, validation) readable from its detail view. +- Tools: maintenance tasks run by hand (see Tools below). + +## Tools + +The `Outils` page of the dashboard runs maintenance tasks by hand: one card per tool with its description, a run button, a live progress bar while the task runs (cancellable), and a summary with per-outcome counts when it ends. One task at a time. + +The first tool, `Recherche des fichiers perdus`, reconciles the bucket with the database when files have been moved by hand (e.g. through Nextcloud). It scans every object outside `inbox/`, computes each object's sha256 and compares it against the database: + +- the object is at the key its record knows: nothing happens; +- the object was moved outside the application: the record's `file_key` is corrected to the new key and a `relocated` event is appended to the document's history (the object itself never moves); +- the object is unknown of the database: it is moved to `inbox/` to be processed automatically by the poller; +- a record whose file has disappeared from the bucket is only reported in the summary, never acted upon. ## Nextcloud integration @@ -144,7 +158,7 @@ Without `E2E_IMAGE` the suite skips. The suite drives the poller through the sup ## Troubleshooting -- A file stays in `inbox/`: check the logs — admin dashboard, Dashboard > Logs (`docker logs archivist` shows startup errors). Frequent causes: unreadable PDF (goes to `pending/`), transient S3 error (retried on the next pass). +- A file stays in `inbox/`: check the logs — admin dashboard, Dashboard > Logs (`docker logs archivist` shows startup errors). Frequent causes: the file is not a PDF (left in place by design), unreadable PDF (goes to `pending/`), transient S3 error (retried on the next pass). - A file goes to `pending/`: open its detail view in the dashboard; the classification error(s) are displayed there. - The container does not start: invalid `config.jsonc`. The full error message is in the container logs; the application refuses to start with a partially valid configuration. - After editing `config.jsonc`: `docker restart archivist`. diff --git a/archivist/arch.md b/archivist/arch.md index 7021978..30612a3 100644 --- a/archivist/arch.md +++ b/archivist/arch.md @@ -56,13 +56,13 @@ One row per processed PDF file: The append-only `document_events` collection is the single source of truth for a document's history; no history is stored on the `documents` record itself. ### `document_events` -Append-only history: `document`, `event_type` (`imported`, `corrected`, `validated`), `payload` as JSON, `timestamp`, plus the standard `created`/`updated` autodate columns; the event's business timestamp is `timestamp`, mirrored in the payload. +Append-only history: `document`, `event_type` (`imported`, `corrected`, `validated`, `relocated`), `payload` as JSON, `timestamp`, plus the standard `created`/`updated` autodate columns; the event's business timestamp is `timestamp`, mirrored in the payload. The payload is self-contained: each event carries its own type, its own timestamp and the document state before and after, so a document's full history is reconstructed from this collection alone; the snapshots are stored as diffs — each `before`/`after` carries only the fields the event changes. All dates are RFC 3339; all S3 keys are complete, prefix included. Common payload keys: -- `event_type`: `imported` | `corrected` | `validated` +- `event_type`: `imported` | `corrected` | `validated` | `relocated` - `timestamp`: RFC 3339 date of the event - `before` / `after`: the event's diff, never full snapshots — a key appears on a side only when its value changes in the event: same value on both sides, it is omitted; absent from `before` and present in `after`, the field was added (every `after` key at import, since nothing existed before); present in `before` and absent from `after`, the field was deleted. `file_key` (the S3 key, unified with the `documents` record field) follows the same rule and appears only when the object actually moves; at import the status does not exist before, so `before` carries no `status`. A missing key means the value is unknown, there is no `null` @@ -157,6 +157,21 @@ The diff is derived by comparing the two snapshots: a field present in `before` } ``` +`relocated` — emitted by the consistency tools (section 7.6) when an object of the bucket was moved outside the application: the record's `file_key` is corrected to the key where the object was found (the object itself never moves), and the diff carries `file_key` alone: + +```json +{ + "event_type": "relocated", + "timestamp": "2026-02-01T10:12:44Z", + "before": { + "file_key": "files/BRED/Decembre 2025_Relevé de compte_Compte Thomas.pdf" + }, + "after": { + "file_key": "files/BRED/Archives/Decembre 2025_Relevé de compte_Compte Thomas.pdf" + } +} +``` + ## 5. Configuration File (JSONC) @@ -217,7 +232,7 @@ Validation on load: JSON schema, regex compilation, `expr` expression compilatio Executed by the PocketBase cron job (configurable interval, e.g. every minute): 1. **Poll**: list objects in `inbox/`; an object is a candidate only if no `documents` record references its key yet, and its size and ETag are stable across two consecutive polls (write-completion wait). The same pass also picks up interrupted filings: any `documents` record whose `file_key` is still under `inbox/` (a filing that was interrupted between record creation and the physical move) is completed as described below. -2. **Download**: fetch the object into memory (streamed), no disk writes. The sha256 is computed during download; if a `documents` record with the same hash already exists, the object is a duplicate: the `inbox/` key is deleted (a no-data-loss case — the original with the same content stays filed), a message is logged via the PocketBase logger (section 9), and processing stops there. By design, no `document_events` entry is recorded: the duplicate is not a document of its own, and the trace lives in the PocketBase logs only. +2. **Download**: fetch the object into memory (streamed), no disk writes. The sha256 is computed during download; if a `documents` record with the same hash already exists, the object is a duplicate: the `inbox/` key is deleted (a no-data-loss case — the original with the same content stays filed), a message is logged via the PocketBase logger (section 9), and processing stops there. By design, no `document_events` entry is recorded: the duplicate is not a document of its own, and the trace lives in the PocketBase logs only. An object that is not a PDF (its content does not start with the `%PDF-` header) is left as-is in `inbox/`: it is never imported, gets no record and no event; the skip is logged at Warn level once per unchanged content (the object is not re-downloaded on later passes until its ETag changes) and processing stops there. 3. **Extraction**: page-by-page text (one entry per page, for rule evaluation and the FTS index) via `ledongthuc/pdf`, including headers and footers 4. **Rule evaluation**: rules evaluated in order against the document pages and the fields set so far; full evaluation log written to the PocketBase logger at Debug level (section 9; the same trace feeds the built-in dry-run mode for debugging from the dashboard). 5. **Validation**: a result is valid if and only if: all rules have been processed with no error, all mandatory fields are identified, the selected template exists in the configuration, and the naming template evaluated with no error. Otherwise: route directly to `pending/`. @@ -230,7 +245,7 @@ When the bucket is versioned, any deletion creates a delete marker and the data ## 7. Review Dashboard -The review dashboard is a static frontend — plain HTML/CSS/JS files, no build toolchain, vendored dependencies only (no CDN: the application is 100% offline) — served by PocketBase's built-in static file serving from the `pb_public/` directory shipped in the image next to `pb_data/`. Two pages: the document list and the document detail. The browser persists everything through the standard PocketBase API, whose request hooks already carry the correction, validation and deletion semantics (US-22 / US-23 / US-24): the dashboard never implements its own save path. Only what the standard API cannot express goes through the dedicated JSON routes of section 7.3. +The review dashboard is a static frontend — plain HTML/CSS/JS files, no build toolchain, vendored dependencies only (no CDN: the application is 100% offline) — served by PocketBase's built-in static file serving from the `pb_public/` directory shipped in the image next to `pb_data/`. Three pages: the document list, the document detail, and the tools page (section 7.6). The browser persists everything through the standard PocketBase API, whose request hooks already carry the correction, validation and deletion semantics (US-22 / US-23 / US-24): the dashboard never implements its own save path. Only what the standard API cannot express goes through the dedicated JSON routes of section 7.3. ### 7.1 Document list (US-32 / US-33) @@ -278,6 +293,31 @@ The dashboard's visual design takes Beszel (https://github.com/henrygd/beszel) a - Status colors, the only domain additions to Beszel's tokens: `pending` maps to a warm badge on `--accent`, `validated` to a green from Beszel's chart scale (`hsl(140 50% 48%)` light / `hsl(140 49% 60%)` dark), `processed` to a neutral `--muted` badge. - Typography: Beszel ships InterVariable; archivist does not ship a font file — it uses the system font stack with Inter as the preferred family when installed, to keep the image lean. A vendored Inter subset remains an accepted option if rendering proves unsatisfactory. +### 7.6 Tools page (US-40, US-41) + +A third static page, `tools.html` + `tools.js`, linked from the dashboard's top bar: one card per tool, with its description and a run button; while a job runs, a progress bar (processed / total objects, current key) refreshed by polling; when it ends, a summary (per-outcome counts) plus the detail lists (missing keys, per-object errors). One job at a time; a running job can be cancelled at the next object boundary. The panel is restored on page reload from the latest job state. + +The tools are long-running maintenance tasks started by hand, the exact opposite of the cron poller: each run executes as a background job in a `internal/tools` registry (id, status `running`/`done`/`failed`/`cancelled`, progress, summary), served by dedicated routes and never blocking the pipeline. + +First tool, `find-lost-files` (US-41) — reconciling the bucket with the database when files have been moved by hand (e.g. through Nextcloud): + +1. List every object of the bucket except `inbox/` (the poller's drop zone). +2. For each object, stream-compute its sha256 and look up a `documents` record by hash: + - the record's `file_key` equals the object key: nothing to do (`ok`); + - the record's `file_key` no longer exists in the bucket: the file was moved outside the application — the record's `file_key` is corrected to the found key and a `relocated` event is appended (the object itself never moves); refused if another record already references the key (`corrected`); + - the record's `file_key` still exists: the scanned object is a stray copy — left in place and reported (`duplicates`); + - no record at all: the object is unknown of the database — moved to `inbox/` (conflict suffix on name collision) so the poller processes it automatically (`moved`). +3. After the scan, every record whose `file_key` exists nowhere in the bucket (and is not an interrupted filing under `inbox/`) is reported as missing, never acted upon (`missing`). + +Every outcome is logged through the PocketBase logger; per-object failures do not stop the scan and join the job's error list. + +Routes (superuser-only, same rules as section 7.3): + +- `GET /api/archivist/tools` — the tool descriptors (id, name, description) and the latest job state (running or last finished). +- `POST /api/archivist/tools/{id}/run` — starts the job; refused with 409 while one is running. +- `GET /api/archivist/tools/jobs/{id}` — the job's live state (progress, summary, lists). +- `POST /api/archivist/tools/jobs/{id}/cancel` — asks the job to stop at the next object boundary. + ## 8. Error Handling - Transient S3 error: retry with backoff, the object stays in `inbox/` and will be polled again. @@ -318,7 +358,7 @@ The scenarios form one narrative; each subtest names the US it proves: 1. Startup — health, collections migrated, versioning warning in the PocketBase logs (Garage has no versioning: the US-01 warning path is exercised for free), poll route refused without superuser auth (US-01, US-03, US-04). 2. Import — a valid simulated statement is classified and filed to `files/` with its `imported` event (US-10 to US-17). -3. Duplicate — the same content re-uploaded is deleted from `inbox/`, no record, no event (US-12). +3. Duplicate — the same content re-uploaded is deleted from `inbox/`, no record, no event; a non-PDF object uploaded to `inbox/` is left in place, never imported (US-12). 4. Pending — a missing mandatory date routes the document to `pending/` with `errors` in the imported event and the recognized fields kept (US-13 to US-15). 5. Correction — a field PATCH through the API persists the values with no side effect (no re-filing, no status change), `corrected` event carrying the field diff (US-22). 6. Validation — a status-only PATCH appends the `validated` event, `file_key` untouched (US-23). @@ -332,6 +372,7 @@ The scenarios form one narrative; each subtest names the US it proves: 14. Full correction — a pending document is re-filed and validated; value errors leave everything unchanged; an unchanged name skips the S3 move and the diff drops file_key (US-36). 15. Static dashboard — the review pages are served from pb_public/ with no external resource (US-30). 16. Enum lifecycle — the value-registration route extends the possible-values list and its refusals hold; a restart drops an unused added value, keeps a used one, and the `template` list follows the configuration (US-37). +17. Tools — the tools route serves the descriptors and refuses unauthenticated calls; a scan over a bucket with an in-place file, a relocated file, an unknown file and a deleted file produces the matching summary (ok, corrected with `relocated` event, moved to `inbox/`, missing reported) (US-40, US-41). `POST /api/archivist/poll` (superuser only) runs one poll pass immediately. It exists so the gate and the operator trigger passes on demand instead of waiting for the cron; a fresh upload needs two calls because the poller waits for size/ETag stability across two passes (US-10). diff --git a/archivist/e2e/e2e_test.go b/archivist/e2e/e2e_test.go index 7d3c7df..6fbd6aa 100644 --- a/archivist/e2e/e2e_test.go +++ b/archivist/e2e/e2e_test.go @@ -688,6 +688,24 @@ func TestE2E(t *testing.T) { } }) + // 3b. Non-PDF (US-12): an object that is not a PDF is left as-is in + // inbox/, never imported — no record, no event. + t.Run("non-pdf: a non-PDF object is left in place, not imported", func(t *testing.T) { + data := []byte("not a pdf at all\n") + if _, err := s3.PutObject(context.Background(), bucket, "inbox/notes.txt", + bytes.NewReader(data), int64(len(data)), minio.PutObjectOptions{}); err != nil { + t.Fatalf("PutObject notes.txt: %v", err) + } + poll(t) + poll(t) + if !s3Exists(t, "inbox/notes.txt") { + t.Fatal("non-PDF object was removed from inbox/") + } + if docs := listRecords(t, "documents", "(file_key='inbox/notes.txt')", ""); len(docs) != 0 { + t.Fatalf("non-PDF object was imported: %d record(s)", len(docs)) + } + }) + // 4. Pending (US-13 to US-15): a missing mandatory date routes the // document to pending/ with the causes in the imported event and the // recognized fields kept for the correction form. diff --git a/archivist/internal/pbapp/setup.go b/archivist/internal/pbapp/setup.go index 103f5d1..d8a668c 100644 --- a/archivist/internal/pbapp/setup.go +++ b/archivist/internal/pbapp/setup.go @@ -5,6 +5,7 @@ package pbapp import ( "errors" "fmt" + "slices" "ganigeorgiev.com/pocketbase/pocketbase/core" "github.com/napnap75/archivist/internal/config" @@ -12,6 +13,10 @@ import ( const eventsName = "document_events" +// EventTypes is the official event_type vocabulary. It grows with the +// application: existing collections are migrated at startup (ensureEvents). +var EventTypes = []string{"imported", "corrected", "validated", "relocated"} + // Setup ensures the collections and the superuser account (US-03, US-04, US-01). func Setup(app core.App, adminEmail, adminPassword string, cfg *config.Config) error { if err := ensureDocuments(app, cfg); err != nil { @@ -116,17 +121,18 @@ func isFixed(name string) bool { } func ensureEvents(app core.App) error { - if _, err := app.FindCollectionByNameOrId(eventsName); err == nil { - return nil + coll, err := app.FindCollectionByNameOrId(eventsName) + if err == nil { + return migrateEventTypes(app, coll) } docColl, err := app.FindCollectionByNameOrId("documents") if err != nil { return err } - coll := core.NewBaseCollection(eventsName) + coll = core.NewBaseCollection(eventsName) coll.Fields.Add( &core.RelationField{Name: "document", CollectionId: docColl.Id, MaxSelect: 1, CascadeDelete: true}, - &core.SelectField{Name: "event_type", MaxSelect: 1, Values: []string{"imported", "corrected", "validated"}}, + &core.SelectField{Name: "event_type", MaxSelect: 1, Values: EventTypes}, &core.JSONField{Name: "payload"}, &core.DateField{Name: "timestamp"}, &core.AutodateField{Name: "created", OnCreate: true}, @@ -141,6 +147,31 @@ func ensureEvents(app core.App) error { return nil } +// migrateEventTypes extends the event_type select of an existing collection +// with the event types the current version emits (e.g. "relocated", added +// with the consistency tools); the values already present are kept. +func migrateEventTypes(app core.App, coll *core.Collection) error { + field, ok := coll.Fields.GetByName("event_type").(*core.SelectField) + if !ok || field == nil { + return fmt.Errorf("%s: event_type is not a select field", eventsName) + } + changed := false + for _, v := range EventTypes { + if !slices.Contains(field.Values, v) { + field.Values = append(field.Values, v) + changed = true + } + } + if !changed { + return nil + } + if err := app.Save(coll); err != nil { + return fmt.Errorf("migrate %s event types: %w", eventsName, err) + } + app.Logger().Info("document_events: event types extended", "values", field.Values) + return nil +} + func ensureSuperuser(app core.App, email, password string) error { superusers, err := app.FindCollectionByNameOrId(core.CollectionNameSuperusers) if err != nil { diff --git a/archivist/internal/pipeline/pipeline.go b/archivist/internal/pipeline/pipeline.go index 6d4496a..8624388 100644 --- a/archivist/internal/pipeline/pipeline.go +++ b/archivist/internal/pipeline/pipeline.go @@ -3,6 +3,7 @@ package pipeline import ( + "bytes" "context" "encoding/json" "fmt" @@ -24,6 +25,18 @@ type pollState struct { var stable = map[string]pollState{} +// nonPDF remembers the objects already found not to be PDFs (key -> ETag at +// the check): they are left in place, so the pass skips them without +// re-downloading until their content changes. +var nonPDF = map[string]string{} + +var pdfMagic = []byte("%PDF-") + +// isPDF reports whether the content carries the PDF header. +func isPDF(data []byte) bool { + return bytes.HasPrefix(data, pdfMagic) +} + func RunPoll(app core.App, cfg *config.Config, s3 *storage.Client) { ctx := context.Background() @@ -48,19 +61,33 @@ func RunPoll(app core.App, cfg *config.Config, s3 *storage.Client) { continue // first sighting, or still uploading: wait for stability } delete(stable, o.Key) - if err := processOne(app, cfg, s3, ctx, o.Key); err != nil { + if err := processOne(app, cfg, s3, ctx, o.Key, o.ETag); err != nil { app.Logger().Error("poll: processing failed", "key", o.Key, "error", err) } } } -func processOne(app core.App, cfg *config.Config, s3 *storage.Client, ctx context.Context, key string) error { +func processOne(app core.App, cfg *config.Config, s3 *storage.Client, ctx context.Context, key, etag string) error { + // Already found not to be a PDF and unchanged since: left in place, + // nothing to do. + if nonPDF[key] == etag { + return nil + } + // US-12: download in memory, sha256 on the fly. data, hash, err := s3.Get(ctx, key) if err != nil { return fmt.Errorf("download: %w", err) } + // Not a PDF: the object is left as-is in inbox/, never imported — no + // record, no event; the skip is logged once per unchanged content. + if !isPDF(data) { + nonPDF[key] = etag + app.Logger().Warn("inbox: object is not a PDF, left in place", "key", key) + return nil + } + // US-12: duplicate rejection. dup, err := app.FindFirstRecordByFilter("documents", "sha256 = {:h}", dbx.Params{"h": hash}) if err == nil && dup != nil { diff --git a/archivist/internal/pipeline/pipeline_test.go b/archivist/internal/pipeline/pipeline_test.go new file mode 100644 index 0000000..4acc28c --- /dev/null +++ b/archivist/internal/pipeline/pipeline_test.go @@ -0,0 +1,24 @@ +package pipeline + +import "testing" + +// TestIsPDF covers the inbox rule: only content carrying the %PDF- header +// enters the pipeline; anything else is left in place (US-12). +func TestIsPDF(t *testing.T) { + cases := []struct { + name string + data []byte + want bool + }{ + {"pdf header", []byte("%PDF-1.7\n..."), true}, + {"pdf header with leading whitespace", []byte("\n%PDF-1.4"), false}, + {"plain text", []byte("not a pdf at all\n"), false}, + {"image content", []byte("\x89PNG\r\n\x1a\n..."), false}, + {"empty", nil, false}, + } + for _, c := range cases { + if got := isPDF(c.data); got != c.want { + t.Errorf("%s: isPDF = %v, want %v", c.name, got, c.want) + } + } +} diff --git a/archivist/internal/storage/s3.go b/archivist/internal/storage/s3.go index b683a55..babbc3a 100644 --- a/archivist/internal/storage/s3.go +++ b/archivist/internal/storage/s3.go @@ -61,6 +61,39 @@ type InboxObject struct { ETag string } +// ListAll lists every object of the bucket, all prefixes together. The +// inbox/ prefix is excluded: it is the poller's drop zone (US-10), the +// consistency tools never scan it. +func (c *Client) ListAll(ctx context.Context) ([]InboxObject, error) { + var out []InboxObject + for obj := range c.mc.ListObjects(ctx, c.Bucket, minio.ListObjectsOptions{Recursive: true}) { + if obj.Err != nil { + return nil, obj.Err + } + if strings.HasSuffix(obj.Key, "/") || strings.HasPrefix(obj.Key, "inbox/") { + continue + } + out = append(out, InboxObject{Key: obj.Key, Size: obj.Size, ETag: obj.ETag}) + } + return out, nil +} + +// Hash streams the object and computes its sha256 without holding the +// content in memory: the consistency tools hash every object of the bucket +// and never need the bytes themselves. +func (c *Client) Hash(ctx context.Context, key string) (string, error) { + obj, err := c.mc.GetObject(ctx, c.Bucket, key, minio.GetObjectOptions{}) + if err != nil { + return "", err + } + defer obj.Close() + h := sha256.New() + if _, err := io.Copy(h, obj); err != nil { + return "", err + } + return hex.EncodeToString(h.Sum(nil)), nil +} + // ListInbox lists the objects under the inbox/ prefix (US-10). func (c *Client) ListInbox(ctx context.Context) ([]InboxObject, error) { var out []InboxObject @@ -79,12 +112,7 @@ func (c *Client) ListInbox(ctx context.Context) ([]InboxObject, error) { // Get downloads the object into memory and computes its sha256 on the fly // (US-12: streamed in memory, no disk writes). func (c *Client) Get(ctx context.Context, key string) ([]byte, string, error) { - obj, err := c.mc.GetObject(ctx, c.Bucket, key, minio.GetObjectOptions{}) - if err != nil { - return nil, "", err - } - defer obj.Close() - data, err := io.ReadAll(obj) + data, err := c.getRaw(ctx, key) if err != nil { return nil, "", err } @@ -92,6 +120,15 @@ func (c *Client) Get(ctx context.Context, key string) ([]byte, string, error) { return data, hex.EncodeToString(sum[:]), nil } +func (c *Client) getRaw(ctx context.Context, key string) ([]byte, error) { + obj, err := c.mc.GetObject(ctx, c.Bucket, key, minio.GetObjectOptions{}) + if err != nil { + return nil, err + } + defer obj.Close() + return io.ReadAll(obj) +} + func (c *Client) Exists(ctx context.Context, key string) bool { _, err := c.mc.StatObject(ctx, c.Bucket, key, minio.StatObjectOptions{}) return err == nil diff --git a/archivist/internal/tools/routes.go b/archivist/internal/tools/routes.go new file mode 100644 index 0000000..430a1c2 --- /dev/null +++ b/archivist/internal/tools/routes.go @@ -0,0 +1,55 @@ +// Routes.go binds the tools-page application routes: the tool list, the +// run trigger, the live job state and the cancel action. Superuser-only, +// like every /api/archivist route. +package tools + +import ( + "ganigeorgiev.com/pocketbase/pocketbase/core" +) + +// BindRoutes registers the tools routes on the PocketBase router. +func BindRoutes(app core.App, s3 S3Store) { + app.OnServe().BindFunc(func(e *core.ServeEvent) error { + e.Router.GET("/api/archivist/tools", func(re *core.RequestEvent) error { + if !re.HasSuperuserAuth() { + return re.ForbiddenError("superusers only", nil) + } + job := LatestJob() + var state map[string]any + if job != nil { + state = job.State() + } + return re.JSON(200, map[string]any{"tools": Descriptors(), "job": state}) + }) + e.Router.POST("/api/archivist/tools/{id}/run", func(re *core.RequestEvent) error { + if !re.HasSuperuserAuth() { + return re.ForbiddenError("superusers only", nil) + } + job, err := Start(re.App, s3, re.Request.PathValue("id")) + if err != nil { + return re.JSON(409, map[string]any{"message": err.Error()}) + } + return re.JSON(200, job.State()) + }) + e.Router.GET("/api/archivist/tools/jobs/{id}", func(re *core.RequestEvent) error { + if !re.HasSuperuserAuth() { + return re.ForbiddenError("superusers only", nil) + } + job := JobByID(re.Request.PathValue("id")) + if job == nil { + return re.JSON(404, map[string]any{"message": "job not found"}) + } + return re.JSON(200, job.State()) + }) + e.Router.POST("/api/archivist/tools/jobs/{id}/cancel", func(re *core.RequestEvent) error { + if !re.HasSuperuserAuth() { + return re.ForbiddenError("superusers only", nil) + } + if !Cancel(re.Request.PathValue("id")) { + return re.JSON(404, map[string]any{"message": "no running job with this id"}) + } + return re.JSON(200, map[string]any{"status": "cancelling"}) + }) + return e.Next() + }) +} diff --git a/archivist/internal/tools/tools.go b/archivist/internal/tools/tools.go new file mode 100644 index 0000000..8ecfa55 --- /dev/null +++ b/archivist/internal/tools/tools.go @@ -0,0 +1,393 @@ +// Package tools implements the dashboard's tools page: long-running +// maintenance tasks started by hand, each executed as a background job with +// live progress and a final summary. +package tools + +import ( + "context" + "crypto/rand" + "database/sql" + "encoding/hex" + "errors" + "fmt" + "strings" + "sync" + "time" + + "ganigeorgiev.com/pocketbase/dbx" + "ganigeorgiev.com/pocketbase/pocketbase/core" + "github.com/napnap75/archivist/internal/pbapp" + "github.com/napnap75/archivist/internal/storage" +) + +// Descriptor is one tool as the tools page displays it: identity, title, +// description. The list is static; new tools are added here. +type Descriptor struct { + ID string `json:"id"` + Name string `json:"name"` + Description string `json:"description"` +} + +var registry = []Descriptor{ + { + ID: "find-lost-files", + Name: "Recherche des fichiers perdus", + Description: "Parcourt tout le bucket (hors inbox/), recalcule l'empreinte sha256 de chaque objet et la compare à la base. " + + "Un objet à sa place connue ne bouge pas ; un objet déplacé hors de l'application voit son file_key corrigé avec un événement relocated dans l'historique ; " + + "un objet inconnu de la base est déplacé vers inbox/ pour être traité automatiquement ; " + + "les enregistrements dont le fichier a disparu du bucket sont signalés dans le résumé, sans action automatique.", + }, +} + +// Descriptors returns the tool list served to the dashboard. +func Descriptors() []Descriptor { + return registry +} + +func descriptor(id string) (Descriptor, bool) { + for _, d := range registry { + if d.ID == id { + return d, true + } + } + return Descriptor{}, false +} + +// S3Store is the surface of the storage client the tools use; a seam kept +// for unit tests. +type S3Store interface { + ListAll(ctx context.Context) ([]storage.InboxObject, error) + Hash(ctx context.Context, key string) (string, error) + Exists(ctx context.Context, key string) bool + Copy(ctx context.Context, srcKey, dstKey string) error + Remove(ctx context.Context, key string) error + ResolveConflict(ctx context.Context, desired string) (string, error) +} + +// Job is one tool run: identity, live progress, final summary. +type Job struct { + ID string + Tool string + Status string // running | done | failed | cancelled + StartedAt time.Time + FinishedAt time.Time + Total int + Processed int + Current string + Summary map[string]int + Missing []string + Errors []string + + mu sync.Mutex + cancel context.CancelFunc +} + +const ( + StatusRunning = "running" + StatusDone = "done" + StatusFailed = "failed" + StatusCancelled = "cancelled" +) + +// State snapshots the job for the API responses. +func (j *Job) State() map[string]any { + j.mu.Lock() + defer j.mu.Unlock() + state := map[string]any{ + "id": j.ID, + "tool": j.Tool, + "status": j.Status, + "started_at": j.StartedAt.Format(time.RFC3339), + "total": j.Total, + "processed": j.Processed, + "current": j.Current, + "summary": j.Summary, + "missing": j.Missing, + "errors": j.Errors, + } + if !j.FinishedAt.IsZero() { + state["finished_at"] = j.FinishedAt.Format(time.RFC3339) + } + return state +} + +func (j *Job) set(fn func()) { + j.mu.Lock() + fn() + j.mu.Unlock() +} + +func (j *Job) addError(err error, key string) { + j.set(func() { + j.Errors = append(j.Errors, key+": "+err.Error()) + if n, ok := j.Summary["errors"]; ok { + j.Summary["errors"] = n + 1 + } else { + j.Summary["errors"] = 1 + } + }) +} + +var ( + regMu sync.Mutex + jobs = map[string]*Job{} + running *Job +) + +// Start launches a tool as a background job. One job at a time: a second +// start while one is running is refused. +func Start(app core.App, s3 S3Store, toolID string) (*Job, error) { + d, ok := descriptor(toolID) + if !ok { + return nil, fmt.Errorf("unknown tool: %s", toolID) + } + regMu.Lock() + defer regMu.Unlock() + if running != nil { + return nil, fmt.Errorf("a job is already running: %s", running.Tool) + } + buf := make([]byte, 8) + if _, err := rand.Read(buf); err != nil { + return nil, err + } + ctx, cancel := context.WithCancel(context.Background()) + job := &Job{ + ID: hex.EncodeToString(buf), + Tool: d.ID, + Status: StatusRunning, + StartedAt: time.Now().UTC(), + Summary: map[string]int{}, + cancel: cancel, + } + if d.ID == "find-lost-files" { + job.Summary = map[string]int{"scanned": 0, "ok": 0, "corrected": 0, "moved": 0, "duplicates": 0, "errors": 0} + } + jobs[job.ID] = job + running = job + go func() { + err := runTool(ctx, app, s3, d.ID, job) + regMu.Lock() + if running == job { + running = nil + } + regMu.Unlock() + job.set(func() { + job.cancel = nil + job.FinishedAt = time.Now().UTC() + if ctx.Err() != nil { + job.Status = StatusCancelled + } else if err != nil { + job.Status = StatusFailed + job.Errors = append(job.Errors, err.Error()) + } else { + job.Status = StatusDone + } + job.Current = "" + }) + }() + return job, nil +} + +// Cancel asks the running job to stop at the next object boundary. +func Cancel(jobID string) bool { + regMu.Lock() + job := jobs[jobID] + regMu.Unlock() + if job == nil { + return false + } + job.mu.Lock() + cancel := job.cancel + job.mu.Unlock() + if cancel == nil { + return false + } + cancel() + return true +} + +// JobByID returns a job by id, nil when unknown. +func JobByID(id string) *Job { + regMu.Lock() + defer regMu.Unlock() + return jobs[id] +} + +// LatestJob returns the running job if any, otherwise the most recent one +// (the tools page restores its panel on reload). +func LatestJob() *Job { + regMu.Lock() + defer regMu.Unlock() + if running != nil { + return running + } + var latest *Job + for _, j := range jobs { + if latest == nil || j.StartedAt.After(latest.StartedAt) { + latest = j + } + } + return latest +} + +func runTool(ctx context.Context, app core.App, s3 S3Store, toolID string, job *Job) error { + switch toolID { + case "find-lost-files": + return runFindLostFiles(ctx, app, s3, job) + } + return fmt.Errorf("unknown tool: %s", toolID) +} + +// runFindLostFiles reconciles the bucket with the database (US-41): every +// object outside inbox/ is hashed and matched against documents by sha256. +// In place: nothing. Moved in the bucket outside the application: the +// record's file_key is corrected and a relocated event is appended. Unknown +// of the database: moved to inbox/ for automatic processing. A record whose +// file_key exists nowhere is only reported. +func runFindLostFiles(ctx context.Context, app core.App, s3 S3Store, job *Job) error { + objects, err := s3.ListAll(ctx) + if err != nil { + return fmt.Errorf("list bucket: %w", err) + } + keySet := make(map[string]bool, len(objects)) + for _, o := range objects { + keySet[o.Key] = true + } + job.set(func() { job.Total = len(objects) }) + + for _, o := range objects { + if ctx.Err() != nil { + return nil + } + job.set(func() { job.Current = o.Key }) + if err := reconcileObject(ctx, app, s3, job, o.Key, keySet); err != nil { + app.Logger().Error("find-lost-files: object failed", "key", o.Key, "error", err) + job.addError(err, o.Key) + } + job.set(func() { job.Processed++ }) + } + + // Records whose file_key exists nowhere: reported, never acted upon. + // Records still under inbox/ are interrupted filings (US-11), the + // poller's business, not orphans. + records, err := app.FindAllRecords("documents") + if err != nil { + return fmt.Errorf("list documents: %w", err) + } + missing := []string{} + for _, r := range records { + key := r.GetString("file_key") + if key == "" || keySet[key] || strings.HasPrefix(key, "inbox/") { + continue + } + missing = append(missing, key) + app.Logger().Warn("find-lost-files: record file missing from bucket", "record", r.Id, "file_key", key) + } + job.set(func() { + job.Missing = missing + job.Summary["missing"] = len(missing) + }) + job.set(func() { + app.Logger().Info("find-lost-files: done", "scanned", job.Summary["scanned"], "ok", job.Summary["ok"], + "corrected", job.Summary["corrected"], "moved", job.Summary["moved"], + "duplicates", job.Summary["duplicates"], "errors", job.Summary["errors"], "missing", len(missing)) + }) + return nil +} + +// findRecord wraps FindFirstRecordByFilter: a missing record is a nil +// result, not an error. +func findRecord(app core.App, filter string, params dbx.Params) (*core.Record, error) { + record, err := app.FindFirstRecordByFilter("documents", filter, params) + if err != nil && !errors.Is(err, sql.ErrNoRows) { + return nil, err + } + return record, nil +} + +func reconcileObject(ctx context.Context, app core.App, s3 S3Store, job *Job, key string, keySet map[string]bool) error { + hash, err := s3.Hash(ctx, key) + if err != nil { + return fmt.Errorf("hash: %w", err) + } + job.set(func() { job.Summary["scanned"]++ }) + + record, err := findRecord(app, "sha256 = {:h}", dbx.Params{"h": hash}) + if err != nil { + return fmt.Errorf("lookup by sha256: %w", err) + } + + if record == nil { + // Unknown of the database: back to inbox/ for automatic processing. + finalKey, err := s3.ResolveConflict(ctx, "inbox/"+baseName(key)) + if err != nil { + return fmt.Errorf("conflict resolution: %w", err) + } + if err := moveObject(ctx, s3, key, finalKey); err != nil { + return err + } + job.set(func() { job.Summary["moved"]++ }) + app.Logger().Info("find-lost-files: unknown object moved to inbox", "from", key, "to", finalKey) + return nil + } + + oldKey := record.GetString("file_key") + if oldKey == key { + job.set(func() { job.Summary["ok"]++ }) + return nil + } + if keySet[oldKey] { + // The record points at an object that still exists: this scanned + // object is a stray copy, left in place and reported. + job.set(func() { job.Summary["duplicates"]++ }) + app.Logger().Warn("find-lost-files: stray copy left in place", "key", key, "record", record.Id, "file_key", oldKey) + return nil + } + + // The file was moved in the bucket outside the application: the record + // follows it, a relocated event keeps the trace. No other record may + // already reference the new key. + other, err := findRecord(app, "file_key = {:k} && id != {:id}", dbx.Params{"k": key, "id": record.Id}) + if err != nil { + return fmt.Errorf("lookup by file_key: %w", err) + } + if other != nil { + return fmt.Errorf("another record (%s) already references %s", other.Id, key) + } + before := pbapp.Snapshot{"file_key": oldKey} + after := pbapp.Snapshot{"file_key": key} + txErr := app.RunInTransaction(func(tx core.App) error { + record.Set("file_key", key) + if err := tx.Save(record); err != nil { + return err + } + return pbapp.AppendEvent(tx, record.Id, "relocated", before, after, nil) + }) + if txErr != nil { + return fmt.Errorf("relocation save: %w", txErr) + } + job.set(func() { job.Summary["corrected"]++ }) + app.Logger().Info("find-lost-files: file_key corrected", "record", record.Id, "from", oldKey, "to", key) + return nil +} + +// moveObject copies src to dst, verifies the copy, then deletes the source. +func moveObject(ctx context.Context, s3 S3Store, src, dst string) error { + if err := s3.Copy(ctx, src, dst); err != nil { + return fmt.Errorf("copy: %w", err) + } + if !s3.Exists(ctx, dst) { + return fmt.Errorf("target %s not verified after copy", dst) + } + if err := s3.Remove(ctx, src); err != nil { + return fmt.Errorf("remove source: %w", err) + } + return nil +} + +func baseName(key string) string { + if i := strings.LastIndex(key, "/"); i >= 0 { + return key[i+1:] + } + return key +} diff --git a/archivist/internal/tools/tools_test.go b/archivist/internal/tools/tools_test.go new file mode 100644 index 0000000..c328c26 --- /dev/null +++ b/archivist/internal/tools/tools_test.go @@ -0,0 +1,274 @@ +package tools + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "fmt" + "sort" + "strings" + "testing" + "time" + + "ganigeorgiev.com/pocketbase/pocketbase" + "ganigeorgiev.com/pocketbase/pocketbase/core" + "github.com/napnap75/archivist/internal/config" + "github.com/napnap75/archivist/internal/pbapp" + "github.com/napnap75/archivist/internal/storage" +) + +// fakeS3 is an in-memory bucket for the tool tests. +type fakeS3 struct { + objects map[string][]byte +} + +func (f *fakeS3) ListAll(ctx context.Context) ([]storage.InboxObject, error) { + var keys []string + for k := range f.objects { + if strings.HasPrefix(k, "inbox/") { + continue + } + keys = append(keys, k) + } + sort.Strings(keys) + out := make([]storage.InboxObject, 0, len(keys)) + for _, k := range keys { + out = append(out, storage.InboxObject{Key: k, Size: int64(len(f.objects[k]))}) + } + return out, nil +} + +func (f *fakeS3) Hash(ctx context.Context, key string) (string, error) { + data, ok := f.objects[key] + if !ok { + return "", fmt.Errorf("no such key %q", key) + } + sum := sha256.Sum256(data) + return hex.EncodeToString(sum[:]), nil +} + +func (f *fakeS3) Exists(ctx context.Context, key string) bool { + _, ok := f.objects[key] + return ok +} + +func (f *fakeS3) Copy(ctx context.Context, srcKey, dstKey string) error { + data, ok := f.objects[srcKey] + if !ok { + return fmt.Errorf("no such key %q", srcKey) + } + f.objects[dstKey] = data + return nil +} + +func (f *fakeS3) Remove(ctx context.Context, key string) error { + delete(f.objects, key) + return nil +} + +func (f *fakeS3) ResolveConflict(ctx context.Context, desired string) (string, error) { + if !f.Exists(ctx, desired) { + return desired, nil + } + for n := 1; n < 1000; n++ { + ext := "" + if i := strings.LastIndex(desired, "."); i >= 0 { + ext = desired[i:] + } + candidate := fmt.Sprintf("%s (%d)%s", strings.TrimSuffix(desired, ext), n, ext) + if !f.Exists(ctx, candidate) { + return candidate, nil + } + } + return "", fmt.Errorf("cannot resolve conflict on %q", desired) +} + +func newTestApp(t *testing.T) core.App { + t.Helper() + app := pocketbase.NewWithConfig(pocketbase.Config{DefaultDataDir: t.TempDir()}) + if err := app.Bootstrap(); err != nil { + t.Fatalf("bootstrap PocketBase: %v", err) + } + t.Cleanup(func() { app.ClearBootstrap() }) + cfg := &config.Config{ + Fields: map[string]config.Field{ + "documentType": {Name: "documentType", Type: "text"}, + }, + FieldList: []config.Field{{Name: "documentType", Type: "text"}}, + } + if err := pbapp.Setup(app, "admin@example.net", "password-123456", cfg); err != nil { + t.Fatalf("setup: %v", err) + } + return app +} + +func addDocument(t *testing.T, app core.App, sha, fileKey string) *core.Record { + t.Helper() + coll, err := app.FindCollectionByNameOrId("documents") + if err != nil { + t.Fatalf("find documents: %v", err) + } + rec := core.NewRecord(coll) + rec.Set("sha256", sha) + rec.Set("file_key", fileKey) + rec.Set("status", "processed") + if err := app.Save(rec); err != nil { + t.Fatalf("save document: %v", err) + } + return rec +} + +func runScan(t *testing.T, app core.App, s3 S3Store) *Job { + t.Helper() + job := &Job{ + Tool: "find-lost-files", + Status: StatusRunning, + StartedAt: time.Now().UTC(), + Summary: map[string]int{"scanned": 0, "ok": 0, "corrected": 0, "moved": 0, "duplicates": 0, "errors": 0}, + } + if err := runFindLostFiles(context.Background(), app, s3, job); err != nil { + t.Fatalf("runFindLostFiles: %v", err) + } + return job +} + +func TestFindLostFilesInPlace(t *testing.T) { + app := newTestApp(t) + s3 := &fakeS3{objects: map[string][]byte{"files/a.pdf": []byte("content-a")}} + addDocument(t, app, mustHash("content-a"), "files/a.pdf") + + job := runScan(t, app, s3) + if job.Summary["ok"] != 1 || job.Summary["corrected"] != 0 || job.Summary["moved"] != 0 { + t.Fatalf("summary = %v, want ok=1", job.Summary) + } + if len(job.Missing) != 0 { + t.Fatalf("missing = %v, want empty", job.Missing) + } + if _, ok := s3.objects["files/a.pdf"]; !ok { + t.Fatalf("object moved: %v", s3.objects) + } +} + +func TestFindLostFilesRelocated(t *testing.T) { + app := newTestApp(t) + s3 := &fakeS3{objects: map[string][]byte{"files/new/a.pdf": []byte("content-a")}} + rec := addDocument(t, app, mustHash("content-a"), "files/old/a.pdf") + + job := runScan(t, app, s3) + if job.Summary["corrected"] != 1 { + t.Fatalf("summary = %v, want corrected=1", job.Summary) + } + fresh, err := app.FindRecordById("documents", rec.Id) + if err != nil { + t.Fatal(err) + } + if got := fresh.GetString("file_key"); got != "files/new/a.pdf" { + t.Fatalf("file_key = %q, want files/new/a.pdf", got) + } + events, err := app.FindAllRecords("document_events") + if err != nil { + t.Fatal(err) + } + if len(events) != 1 { + t.Fatalf("events = %d, want 1", len(events)) + } + if events[0].GetString("event_type") != "relocated" { + t.Fatalf("event_type = %q, want relocated", events[0].GetString("event_type")) + } + if _, ok := s3.objects["files/new/a.pdf"]; !ok { + t.Fatalf("object should not move: %v", s3.objects) + } +} + +func TestFindLostFilesUnknownGoesToInbox(t *testing.T) { + app := newTestApp(t) + s3 := &fakeS3{objects: map[string][]byte{"files/unknown.pdf": []byte("content-x")}} + + job := runScan(t, app, s3) + if job.Summary["moved"] != 1 { + t.Fatalf("summary = %v, want moved=1", job.Summary) + } + if _, ok := s3.objects["inbox/unknown.pdf"]; !ok { + t.Fatalf("object not in inbox: %v", s3.objects) + } +} + +func TestFindLostFilesStrayDuplicateLeftInPlace(t *testing.T) { + app := newTestApp(t) + s3 := &fakeS3{objects: map[string][]byte{ + "files/orig.pdf": []byte("content-a"), + "files/stray.pdf": []byte("content-a"), + }} + addDocument(t, app, mustHash("content-a"), "files/orig.pdf") + + job := runScan(t, app, s3) + if job.Summary["duplicates"] != 1 || job.Summary["ok"] != 1 { + t.Fatalf("summary = %v, want duplicates=1 ok=1", job.Summary) + } + if _, ok := s3.objects["files/stray.pdf"]; !ok { + t.Fatalf("stray copy should stay: %v", s3.objects) + } +} + +func TestFindLostFilesMissingReported(t *testing.T) { + app := newTestApp(t) + s3 := &fakeS3{objects: map[string][]byte{}} + addDocument(t, app, mustHash("content-a"), "files/gone.pdf") + + job := runScan(t, app, s3) + if len(job.Missing) != 1 || job.Missing[0] != "files/gone.pdf" { + t.Fatalf("missing = %v, want [files/gone.pdf]", job.Missing) + } + if job.Summary["missing"] != 1 { + t.Fatalf("summary = %v, want missing=1", job.Summary) + } +} + +func TestFindLostFilesInboxExcluded(t *testing.T) { + app := newTestApp(t) + s3 := &fakeS3{objects: map[string][]byte{ + "inbox/upload.pdf": []byte("content-inbox"), + "files/a.pdf": []byte("content-a"), + }} + addDocument(t, app, mustHash("content-a"), "files/a.pdf") + addDocument(t, app, mustHash("content-inbox"), "inbox/upload.pdf") + + job := runScan(t, app, s3) + if job.Summary["scanned"] != 1 { + t.Fatalf("summary = %v, want scanned=1 (inbox excluded)", job.Summary) + } + if len(job.Missing) != 0 { + t.Fatalf("missing = %v, want empty (inbox records are the poller's business)", job.Missing) + } + if _, ok := s3.objects["inbox/upload.pdf"]; !ok { + t.Fatalf("inbox object should stay: %v", s3.objects) + } +} + +func TestEventTypesIncludeRelocated(t *testing.T) { + app := newTestApp(t) + coll, err := app.FindCollectionByNameOrId("document_events") + if err != nil { + t.Fatal(err) + } + field, ok := coll.Fields.GetByName("event_type").(*core.SelectField) + if !ok { + t.Fatalf("event_type is not a select field") + } + for _, v := range []string{"imported", "corrected", "validated", "relocated"} { + found := false + for _, value := range field.Values { + if value == v { + found = true + } + } + if !found { + t.Errorf("event_type values %v miss %q", field.Values, v) + } + } +} + +func mustHash(content string) string { + sum := sha256.Sum256([]byte(content)) + return hex.EncodeToString(sum[:]) +} diff --git a/archivist/main.go b/archivist/main.go index 6ed07db..1eede46 100644 --- a/archivist/main.go +++ b/archivist/main.go @@ -10,6 +10,7 @@ import ( "github.com/napnap75/archivist/internal/pbapp" "github.com/napnap75/archivist/internal/pipeline" "github.com/napnap75/archivist/internal/storage" + "github.com/napnap75/archivist/internal/tools" "github.com/spf13/pflag" ) @@ -125,6 +126,8 @@ func main() { pbapp.BindHooks(app, cfg, s3) // US-31 / US-35 / US-36: the review-dashboard routes (config, pdf, dry-run, correct). pbapp.BindRoutes(app, cfg, s3) + // US-40 / US-41: the tools-page routes (list, run, progress, cancel). + tools.BindRoutes(app, s3) // Manual poll pass: a superuser-only route used by the E2E gate and the // operator to trigger a pass without waiting for the cron (arch.md 10). diff --git a/archivist/pb_public/document.html b/archivist/pb_public/document.html index 7a511c1..d97c775 100644 --- a/archivist/pb_public/document.html +++ b/archivist/pb_public/document.html @@ -12,6 +12,7 @@

← Archivist

diff --git a/archivist/pb_public/index.html b/archivist/pb_public/index.html index 43ec657..3171107 100644 --- a/archivist/pb_public/index.html +++ b/archivist/pb_public/index.html @@ -12,6 +12,7 @@

Archivist — Relecture

diff --git a/archivist/pb_public/styles.css b/archivist/pb_public/styles.css index 3e512fe..e208388 100644 --- a/archivist/pb_public/styles.css +++ b/archivist/pb_public/styles.css @@ -215,3 +215,24 @@ h3 { margin: 0 0 .5rem; font-size: .9rem; } .diff { font-size: 13px; margin-top: .15rem; word-break: break-all; } .diff .k { color: var(--muted-foreground); } .diff .arrow { color: var(--muted-foreground); padding: 0 .2rem; } + +/* Tools page (arch.md 7.6): tool cards, progress bar, summary grid. */ +.muted { color: var(--muted-foreground); } +.mono { font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; font-size: 12px; word-break: break-all; } +.toolcard { margin-bottom: 1rem; } +.toolhead { display: flex; align-items: center; justify-content: space-between; gap: 1rem; } +.toolhead h3 { margin: 0; font-size: 1rem; } +.tooldesc { margin: .5rem 0 0; color: var(--muted-foreground); } +.jobcard { margin-bottom: 1rem; } +.jobhead { display: flex; align-items: center; justify-content: space-between; gap: 1rem; } +.jobhead h3 { margin: 0; font-size: 1rem; } +.progress { height: .5rem; border-radius: 999px; background: var(--muted); overflow: hidden; margin: .75rem 0 .35rem; } +.progressbar { height: 100%; background: var(--status-validated); transition: width .4s ease; } +.progressinfo { color: var(--muted-foreground); font-size: 12px; margin-bottom: .75rem; word-break: break-all; } +.summarygrid { display: grid; grid-template-columns: repeat(auto-fit, minmax(9rem, 1fr)); gap: .75rem; margin: .5rem 0; } +.summaryitem { display: grid; gap: .1rem; padding: .6rem .75rem; border: 1px solid var(--border); border-radius: var(--radius); background: var(--secondary); } +.summarynum { font-size: 1.25rem; font-weight: 600; } +.summarylabel { color: var(--muted-foreground); font-size: 12px; } +.jobdetails { margin-top: .5rem; } +.jobdetails summary { cursor: pointer; color: var(--muted-foreground); } +.mono-list { margin: .35rem 0 0 1rem; padding: 0; } diff --git a/archivist/pb_public/tools.html b/archivist/pb_public/tools.html new file mode 100644 index 0000000..738a3ca --- /dev/null +++ b/archivist/pb_public/tools.html @@ -0,0 +1,29 @@ + + + + + +Archivist — Outils + + + + +
+
+

← Archivist

+ +
+
+
+

Outils

+

Tâches de maintenance lancées à la demande. Une seule tâche à la fois.

+
+ +
+ + + + diff --git a/archivist/pb_public/tools.js b/archivist/pb_public/tools.js new file mode 100644 index 0000000..d1ea0e4 --- /dev/null +++ b/archivist/pb_public/tools.js @@ -0,0 +1,161 @@ +"use strict"; + +// Tools page (US-40, US-41): one card per tool with its description and a +// run button, live progress while the job runs, final summary afterwards. +// The job panel is restored from GET /api/archivist/tools on reload. + +var pollTimer = null; +var currentJobId = null; + +function $(id) { return document.getElementById(id); } + +var SUMMARY_LABELS = { + scanned: "Fichiers analysés", + ok: "OK (à leur place)", + corrected: "Corrigés (déplacés)", + moved: "Déplacés vers inbox/", + duplicates: "Copies parasites", + missing: "Introuvables", + errors: "Erreurs" +}; + +var SUMMARY_ORDER = ["scanned", "ok", "corrected", "moved", "duplicates", "missing", "errors"]; + +function esc(v) { + return String(v == null ? "" : v) + .replace(/&/g, "&").replace(//g, ">") + .replace(/"/g, """).replace(/'/g, "'"); +} + +async function init() { + initTheme("themeBtn"); + var res = await api("GET", "/api/archivist/tools"); + if (res.status !== 200) return; + renderTools(res.data.tools || []); + var job = res.data && res.data.job; + if (job && job.id) { + renderJob(job); + if (job.status === "running") poll(job.id); + } else { + hideJob(); + } +} + +// hideJob clears and hides the progress/summary block: nothing to show +// when no job is running and no result is worth restoring. +function hideJob() { + stopPoll(); + currentJobId = null; + var panel = $("jobpanel"); + panel.hidden = true; + panel.innerHTML = ""; +} + +function renderTools(tools) { + var host = $("tools"); + host.innerHTML = ""; + tools.forEach(function (tool) { + var card = document.createElement("div"); + card.className = "card toolcard"; + card.innerHTML = + '

' + esc(tool.name) + "

" + + '
' + + '

' + esc(tool.description) + "

"; + card.querySelector("button").addEventListener("click", function () { run(tool.id); }); + host.appendChild(card); + }); +} + +async function run(toolId) { + var res = await api("POST", "/api/archivist/tools/" + encodeURIComponent(toolId) + "/run"); + if (res.status !== 200) { + $("jobpanel").hidden = false; + $("jobpanel").innerHTML = '

' + + esc(errMsg(res)) + "

"; + return; + } + renderJob(res.data); + poll(res.data.id); +} + +function poll(jobId) { + currentJobId = jobId; + if (pollTimer) clearInterval(pollTimer); + pollTimer = setInterval(async function () { + var res = await api("GET", "/api/archivist/tools/jobs/" + encodeURIComponent(jobId)); + if (res.status === 404) { + hideJob(); + return; + } + if (res.status !== 200) { + stopPoll(); + return; + } + renderJob(res.data); + if (res.data.status !== "running") stopPoll(); + }, 1000); +} + +function stopPoll() { + if (pollTimer) clearInterval(pollTimer); + pollTimer = null; +} + +function renderJob(job) { + job = job || {}; + var panel = $("jobpanel"); + panel.hidden = false; + var running = job.status === "running"; + var processed = job.processed || 0; + var total = job.total || 0; + var pct = total > 0 ? Math.round((processed / total) * 100) : 0; + + var html = '
'; + if (running) { + html += '

Tâche en cours…

' + + '
'; + } else { + var label = job.status === "done" ? "Tâche terminée" + : job.status === "cancelled" ? "Tâche annulée" + : job.status === "failed" ? "Tâche échouée" + : job.status ? "Tâche " + job.status : "Tâche"; + html += '

' + esc(label) + "

"; + } + + html += '
' + + '
' + + '
' + processed + " / " + total + " fichiers analysés" + + (running && job.current ? " — " + '' + esc(job.current) + "" : "") + "
"; + + var summary = job.summary || {}; + html += '
'; + SUMMARY_ORDER.forEach(function (key) { + if (summary[key] === undefined) return; + html += '
' + summary[key] + "" + + '' + esc(SUMMARY_LABELS[key] || key) + "
"; + }); + html += "
"; + + if ((job.missing || []).length) { + html += '
Fichiers introuvables dans le bucket (' + job.missing.length + ')
    '; + job.missing.forEach(function (k) { html += '
  • ' + esc(k) + "
    • "; }); + html += "
"; + } + if ((job.errors || []).length) { + html += '
Erreurs (' + job.errors.length + ')
    '; + job.errors.forEach(function (e) { html += '
  • ' + esc(e) + "
    • "; }); + html += "
"; + } + html += "
"; + panel.innerHTML = html; + + var cancelBtn = $("cancelBtn"); + if (cancelBtn && job.status === "running") { + cancelBtn.addEventListener("click", async function () { + cancelBtn.disabled = true; + await api("POST", "/api/archivist/tools/jobs/" + encodeURIComponent(job.id) + "/cancel"); + }); + } +} + +init(); diff --git a/archivist/user-stories.md b/archivist/user-stories.md index b8b5682..d8fbeaa 100644 --- a/archivist/user-stories.md +++ b/archivist/user-stories.md @@ -54,7 +54,7 @@ so that the history of a document remains an immutable, append-only source of tr Acceptance criteria: -- Given the `document_events` collection does not exist, when the application starts, then it is created with the columns `document` (relation to `documents`, cascade delete with the record), `event_type` (select: `imported`, `corrected`, `validated`), `payload` (JSON), `timestamp` (date), plus the standard PocketBase `created`/`updated` autodate columns. +- Given the `document_events` collection does not exist, when the application starts, then it is created with the columns `document` (relation to `documents`, cascade delete with the record), `event_type` (select: `imported`, `corrected`, `validated`, `relocated`), `payload` (JSON), `timestamp` (date), plus the standard PocketBase `created`/`updated` autodate columns. - Given the collection exists, when the application starts, then its schema is verified and left unchanged. - Given any API request that tries to update or delete a `document_events` record, when it is served, then it is rejected by the application's request hooks (`OnRecordUpdateRequest` / `OnRecordDeleteRequest` for the collection, bound to superusers as well). API rules alone cannot enforce this: superusers bypass all API rules, and no rule value blocks everyone (`null` = superusers only, empty string = public). - Given a record is appended, when the write succeeds, then it is never modified afterwards by the application itself. @@ -108,6 +108,8 @@ Acceptance criteria: - Given an eligible candidate, when it is downloaded, then it is streamed in memory with no disk write and its sha256 is computed on the fly. - Given the computed sha256 matches an existing `documents` record, when the download completes, then the `inbox/` key is deleted (a no-data-loss case — the original with the same content stays filed; with versioning enabled the deleted copy is additionally recoverable via delete markers), the duplicate is logged in the PocketBase logs only (Dashboard > Logs; no `document_events` entry, by design), and processing stops for that object. - Given the sha256 is new, when the download completes, then processing continues to extraction. +- Given an object under `inbox/` whose content does not start with the `%PDF-` header, when it is processed, then it is left as-is in `inbox/`: it is never imported, gets no record and no event, and the skip is logged (PocketBase logger, Warn level, once per unchanged content). +- Given an object already skipped as non-PDF, when a later pass lists it with an unchanged ETag, then it is not re-downloaded; when its content changes (new ETag), then it is checked again. ### US-13 — Extract the text page by page @@ -353,6 +355,40 @@ Acceptance criteria: - Given the route, when it is called with an unknown or non-enum field, an empty value, the reserved dummy `-----`, or the `template` field, then it is refused; the list of `template` follows the configuration alone. - Given a startup, when the cleanup pass runs, then every enum value no document uses anymore is removed from the list; the dummy `-----` is kept only when the list would otherwise be empty; removals are logged. +## Epic: Tools (dashboard) + +The tools page (arch.md section 7.6): maintenance tasks started by hand from a dedicated page — one card per tool with its description, a run button, live progress while the job runs, and a final summary. One job at a time, cancellable. The first tool reconciles the bucket with the database when files have been moved by hand (e.g. through Nextcloud). + +### US-40 — A tools page with run, progress and summary + +As the user, +I want a page listing the maintenance tools, each with a description, a run button, a live progress display and a final summary, +so that one-off maintenance tasks are visible and controllable instead of hidden scripts. + +Acceptance criteria: + +- Given the dashboard, when I open the tools page, then each tool appears as a card with its name, its description and a run button. +- Given a tool is started, when the job runs, then the page shows a progress bar (processed / total objects, the key currently being examined), refreshed by polling, and a cancel button that stops the job at the next object boundary. +- Given the job ends, when the page refreshes, then it shows a summary with the per-outcome counts plus the detail lists (missing keys, per-object errors), and the panel is restored after a page reload. +- Given a job is already running, when a tool is started, then the request is refused with an explicit message. +- Given an unauthenticated visitor, when any tools route is called, then it is refused. + +### US-41 — Find the files lost in the bucket + +As the user, +I want a tool that hashes every object of the bucket and reconciles it with the database, +so that files moved by hand are re-attached to their record and unknown files re-enter the pipeline. + +Acceptance criteria: + +- Given the tool runs, when it scans, then it lists every object of the bucket except `inbox/`, stream-computes each object's sha256 and looks up a `documents` record by hash. +- Given an object whose record's `file_key` equals its key, when it is scanned, then nothing happens (`ok`). +- Given an object whose record's `file_key` no longer exists in the bucket, when it is scanned, then the record's `file_key` is corrected to the found key (the object itself never moves), a `relocated` event is appended with the `file_key` diff, and the correction is refused when another record already references the key (`corrected`). +- Given an object whose record's `file_key` still exists elsewhere, when it is scanned, then the stray copy is left in place and reported (`duplicates`). +- Given an object no record knows, when it is scanned, then it is moved to `inbox/` (conflict suffix on name collision) so the poller processes it automatically (`moved`). +- Given a record whose `file_key` exists nowhere in the bucket and is not an interrupted filing under `inbox/`, when the scan ends, then it is reported in the summary (`missing`) with its key, never acted upon. +- Given a per-object failure (S3 error, lookup error), when it occurs, then it is logged, counted in the summary and listed, and the scan continues with the next object. + ## Stories deferred to later epics - US-5x (Config editor): pages to declare fields and build rules with user assistance — guided rule construction, test bench against existing documents through the dry-run routes.