archivist: runtime logs through PocketBase logger (Dashboard > Logs); stderr kept for pre-bootstrap fatals

archivist/README.md

@@ -16,7 +16,7 @@ Duplicates (same sha256) are ignored: the `inbox/` copy is deleted, the original
 
 ## Prerequisites
 
-- An S3 bucket, with three prefixes: `inbox/`, `pending/`, `files/`. The prefixes are created on first startup if they do not exist. Versioning is not required, but it is strongly recommended: without it, a file deleted by the application or by a bug is lost for good; with it, deleted objects stay recoverable via delete markers. The application logs a warning at startup when versioning is disabled.
+- An S3 bucket, with three prefixes: `inbox/`, `pending/`, `files/`. The prefixes are created on first startup if they do not exist. Versioning is not required, but it is strongly recommended: without it, a file deleted by the application or by a bug is lost for good; with it, deleted objects stay recoverable via delete markers. The application logs a warning at startup when versioning is disabled (visible in the admin dashboard, Dashboard > Logs).
 - An S3 user with read/write access to the bucket.
 - Docker.
 
@@ -113,9 +113,13 @@ A commented `config.jsonc` example file is provided in the repository. The dashb
 
 Mount the S3 bucket as Nextcloud external storage. Users drop files into `inbox/`, browse and share from `files/`. Archivist and Nextcloud work on the same objects; neither is the exclusive interface of the other.
 
+## Logs
+
+- Startup errors (missing environment variable, invalid `config.jsonc`) are written to the container's stderr: read them with `docker logs archivist`.
+- Runtime activity — imports, duplicates, recoveries, and the rule-evaluation trace at Debug level — is logged through PocketBase's logger and browsable from the admin dashboard: Dashboard > Logs, filterable by level and message. Retention and the minimum log level are configured from the standard PocketBase logs settings.
 ## Troubleshooting
 
-- A file stays in `inbox/`: check the container logs (`docker logs archivist`). Frequent causes: unreadable PDF (goes to `pending/`), transient S3 error (retried on the next pass).
+- A file stays in `inbox/`: check the logs — admin dashboard, Dashboard > Logs (`docker logs archivist` shows startup errors). Frequent causes: unreadable PDF (goes to `pending/`), transient S3 error (retried on the next pass).
 - A file goes to `pending/`: open its detail view in the dashboard; the classification error(s) are displayed there.
-- The container does not start: invalid `config.jsonc`. The full error message is in the logs; the application refuses to start with a partially valid configuration.
+- The container does not start: invalid `config.jsonc`. The full error message is in the container logs; the application refuses to start with a partially valid configuration.
 - After editing `config.jsonc`: `docker restart archivist`.

archivist/arch.md

@@ -26,6 +26,7 @@ Everything runs offline on the server infrastructure: no cloud APIs, no outbound
 | Config validation | `santhosh-tekuri/jsonschema` (schema generated from structs via `invopop/jsonschema`) |
 | Database | SQLite (embedded in PocketBase) |
 | Full-text search | SQLite FTS5 index synchronized via hooks |
+| Logging | PocketBase `app.Logger()` (Dashboard > Logs); stderr before bootstrap |
 
 ## 3. S3 Layout
 
@@ -183,9 +184,9 @@ Validation on load: JSON schema, regex compilation, `expr` expression compilatio
 Executed by the PocketBase cron job (configurable interval, e.g. every minute):
 
 1. **Poll**: list objects in `inbox/`; an object is a candidate only if no `documents` record references its key yet, and its size and ETag are stable across two consecutive polls (write-completion wait). The same pass also picks up interrupted filings: any `documents` record whose `file_key` is still under `inbox/` (a filing that was interrupted between record creation and the physical move) is completed as described below.
-2. **Download**: fetch the object into memory (streamed), no disk writes. The sha256 is computed during download; if a `documents` record with the same hash already exists, the object is a duplicate: the `inbox/` key is deleted (a no-data-loss case — the original with the same content stays filed), a message is logged in the application logs, and processing stops there. By design, no `document_events` entry is recorded: the duplicate is not a document of its own, and the trace lives in the application logs only.
+2. **Download**: fetch the object into memory (streamed), no disk writes. The sha256 is computed during download; if a `documents` record with the same hash already exists, the object is a duplicate: the `inbox/` key is deleted (a no-data-loss case — the original with the same content stays filed), a message is logged via the PocketBase logger (section 9), and processing stops there. By design, no `document_events` entry is recorded: the duplicate is not a document of its own, and the trace lives in the PocketBase logs only.
 3. **Extraction**: page-by-page text (one entry per page, for rule evaluation and the FTS index) via `ledongthuc/pdf`, including headers and footers
-4. **Rule evaluation**: rules evaluated in order against the document pages and the fields set so far; full evaluation log written to the application logs (built-in dry-run mode for debugging from the dashboard).
+4. **Rule evaluation**: rules evaluated in order against the document pages and the fields set so far; full evaluation log written to the PocketBase logger at Debug level (section 9; the same trace feeds the built-in dry-run mode for debugging from the dashboard).
 5. **Validation**: a result is valid if and only if: all rules have been processed with no error, all mandatory fields are identified, the selected template exists in the configuration, and the naming template evaluated with no error. Otherwise: route directly to `pending/`.
 6. **Record**: create the `documents` record (sha256, current inbox key, pages, fields, status) and add an `imported` event including the original key (`before.file_key`). The record exists before any physical move, so an interrupted filing leaves a state the poller recovers automatically on its next pass.
 7. **Filing**:
@@ -212,3 +213,17 @@ Authentication: single account via PocketBase's standard auth.
 - Unreadable PDF: `pending` status, reason recorded in the `errors` key of the `imported` event payload.
 - Target key conflict (two files renamed identically, whether filing to `files/` or moving to `pending/`): automatic incremental suffix, event logged.
 - Invalid config: error message displayed and the application does not start.
+## 9. Logging
+
+Two channels, split by lifecycle:
+
+- Before the PocketBase application is bootstrapped (fatal startup errors: missing environment variable, invalid `config.jsonc`, unreachable S3; pre-bootstrap startup traces): stderr, so `docker logs` shows the cause under the `--restart unless-stopped` restart loop (US-05).
+- After the bootstrap: PocketBase's `app.Logger()` — structured records (message plus key/value attributes) stored in the PocketBase logs database, separate from the application data, browsable and filterable from the admin dashboard (Dashboard > Logs). Retention and minimum level are configured from the standard PocketBase logs settings panel.
+
+Levels used:
+
+- `Error`: a failed operation (S3 error, event append failure).
+- `Warn`: a degraded but non-fatal state (versioning disabled, schema column removed, stuck record whose inbox object disappeared).
+- `Info`: pipeline outcomes (document imported, duplicate removed, interrupted filing recovered).
+- `Debug`: the per-rule evaluation trace of section 6, step 4.
+

archivist/internal/pbapp/hooks.go

@@ -3,7 +3,6 @@ package pbapp
 import (
 	"encoding/json"
 	"fmt"
-	"log"
 	"time"
 
 	"github.com/napnap75/archivist/internal/config"
@@ -100,7 +99,7 @@ func handleDocumentUpdate(app *core.App, cfg *config.Config, s3 *storage.Client,
 	before := SnapshotFromRecord(cfg, orig, true)
 	after := SnapshotFromRecord(cfg, newRec, true)
 	if err := AppendEvent(app, newRec.Id, "corrected", before, after, nil); err != nil {
-		log.Printf("corrected event for %s: %v", newRec.Id, err)
+		app.Logger().Error("corrected event append failed", "document", newRec.Id, "error", err)
 	}
 	return e.Next()
 }

archivist/internal/pbapp/setup.go

@@ -5,7 +5,6 @@ package pbapp
 import (
 	"errors"
 	"fmt"
-	"log"
 
 	"github.com/napnap75/archivist/internal/config"
 	"github.com/pocketbase/pocketbase/core"
@@ -81,7 +80,7 @@ func ensureDocuments(app *core.App, cfg *config.Config) error {
 			continue
 		}
 		if _, declared := cfg.Fields[f.Name]; !declared && !isFixed(f.Name) {
-			log.Printf("documents: removing column %q (removed from config)", f.Name)
+			app.Logger().Warn("documents: column removed", "field", f.Name, "reason", "removed from config")
 			coll.Fields.RemoveByName(f.Name)
 		}
 	}

archivist/internal/pipeline/pipeline.go

@@ -6,7 +6,6 @@ import (
 	"context"
 	"encoding/json"
 	"fmt"
-	"log"
 	"strings"
 
 	"github.com/ledongthuc/pdf"
@@ -33,7 +32,7 @@ func RunPoll(app *core.App, cfg *config.Config, s3 *storage.Client) {
 
 	objects, err := s3.ListInbox(ctx)
 	if err != nil {
-		log.Printf("poll: list inbox: %v", err)
+		app.Logger().Error("poll: list inbox failed", "error", err)
 		return
 	}
 
@@ -50,7 +49,7 @@ func RunPoll(app *core.App, cfg *config.Config, s3 *storage.Client) {
 		}
 		delete(stable, o.Key)
 		if err := processOne(app, cfg, s3, ctx, o.Key); err != nil {
-			log.Printf("poll: process %s: %v", o.Key, err)
+			app.Logger().Error("poll: processing failed", "key", o.Key, "error", err)
 		}
 	}
 }
@@ -68,7 +67,7 @@ func processOne(app *core.App, cfg *config.Config, s3 *storage.Client, ctx conte
 		if err := s3.Remove(ctx, key); err != nil {
 			return fmt.Errorf("duplicate cleanup: %w", err)
 		}
-		log.Printf("duplicate %s (sha256 %s) removed from inbox, original is %s", key, hash, dup.GetString("file_key"))
+		app.Logger().Info("duplicate removed from inbox", "key", key, "sha256", hash, "original", dup.GetString("file_key"))
 		return nil
 	}
 
@@ -118,9 +117,9 @@ func processOne(app *core.App, cfg *config.Config, s3 *storage.Client, ctx conte
 
 	// US-16: the imported event, appended once the filing is done.
 	if err := pbapp.AppendImportedEvent(app, record.Id, key, newKey, valid, fields, errs); err != nil {
-		log.Printf("imported event for %s: %v", key, err)
+		app.Logger().Error("imported event append failed", "key", key, "error", err)
 	}
-	log.Printf("imported %s -> %s (status %s)", key, newKey, record.GetString("status"))
+	app.Logger().Info("document imported", "key", key, "file_key", newKey, "status", record.GetString("status"))
 	return nil
 }
 
@@ -158,7 +157,7 @@ func fileDocument(s3 *storage.Client, ctx context.Context, record *core.Record,
 func recoverStuck(app *core.App, cfg *config.Config, s3 *storage.Client, ctx context.Context) {
 	stuck, err := app.FindRecordsByFilter("documents", "file_key ~ {:p}", "", 100, 0, dbx.Params{"p": "inbox/"})
 	if err != nil {
-		log.Printf("recovery: %v", err)
+		app.Logger().Error("recovery: query failed", "error", err)
 		return
 	}
 	for _, record := range stuck {
@@ -171,26 +170,26 @@ func recoverStuck(app *core.App, cfg *config.Config, s3 *storage.Client, ctx con
 			}
 			tpl, ok := cfg.Templates[tplName]
 			if !ok {
-				log.Printf("recovery: record %s: unknown template %q", record.Id, tplName)
+				app.Logger().Error("recovery: unknown template", "record", record.Id, "template", tplName)
 				continue
 			}
 			var err error
 			rendered, err = tpl.Render(pbapp.FieldsFromRecord(cfg, record), nil)
 			if err != nil {
-				log.Printf("recovery: record %s: %v", record.Id, err)
+				app.Logger().Error("recovery: render failed", "record", record.Id, "error", err)
 				continue
 			}
 		}
 		newKey, err := fileDocument(s3, ctx, record, rendered)
 		if err != nil {
-			log.Printf("recovery: record %s (%s): %v", record.Id, record.GetString("file_key"), err)
+			app.Logger().Error("recovery: filing failed", "record", record.Id, "file_key", record.GetString("file_key"), "error", err)
 			continue
 		}
 		if err := app.Save(record); err != nil {
-			log.Printf("recovery: save record %s: %v", record.Id, err)
+			app.Logger().Error("recovery: save failed", "record", record.Id, "error", err)
 			continue
 		}
-		log.Printf("recovery: record %s completed filing to %s", record.Id, newKey)
+		app.Logger().Info("recovery: filing completed", "record", record.Id, "file_key", newKey)
 	}
 }
 

archivist/internal/storage/s3.go

@@ -7,7 +7,6 @@ import (
 	"encoding/hex"
 	"fmt"
 	"io"
-	"log"
 	"regexp"
 	"strconv"
 	"strings"
@@ -19,6 +18,10 @@ import (
 type Client struct {
 	mc     *minio.Client
 	Bucket string
+	// StartupWarning is non-empty when the bucket runs without versioning
+	// (or its versioning state could not be read): the caller logs it once
+	// through the PocketBase logger after bootstrap (arch.md section 9).
+	StartupWarning string
 }
 
 func New(endpoint, accessKey, secretKey, bucket string, useSSL bool) (*Client, error) {
@@ -43,9 +46,9 @@ func New(endpoint, accessKey, secretKey, bucket string, useSSL bool) (*Client, e
 	// performed by the application (or by a bug) is final; with it, delete
 	// markers keep deleted objects recoverable.
 	if vc, err := mc.GetBucketVersioning(ctx, bucket); err != nil {
-		log.Printf("warning: could not check bucket versioning: %v", err)
+		c.StartupWarning = fmt.Sprintf("could not check bucket versioning: %v", err)
 	} else if vc.Status != "Enabled" {
-		log.Printf("warning: bucket %q has versioning disabled; consider enabling it to reduce the risk of losing a file to a bug", bucket)
+		c.StartupWarning = "bucket has versioning disabled; consider enabling it to reduce the risk of losing a file to a bug"
 	}
 	return c, nil
 }

archivist/main.go

@@ -83,12 +83,23 @@ func main() {
 	}
 
 	app := pocketbase.New()
+	// The PocketBase logger and database are available only after bootstrap;
+	// everything logged before this point goes to stderr (arch.md section 9).
+	if err := app.Bootstrap(); err != nil {
+		log.Fatalf("pocketbase bootstrap: %v", err)
+	}
 
 	// US-03 / US-04 / US-01: collections migration and superuser account.
 	if err := pbapp.Setup(app, envCfg.AdminEmail, envCfg.AdminPassword, cfg); err != nil {
 		log.Fatalf("pocketbase setup: %v", err)
 	}
 
+	// US-01: versioning recommendation, via the PocketBase logger
+	// (visible in the admin dashboard, Dashboard > Logs).
+	if s3.StartupWarning != "" {
+		app.Logger().Warn(s3.StartupWarning, "bucket", envCfg.S3Bucket)
+	}
+
 	// US-22 / US-23 / US-04: request hooks on documents and document_events.
 	pbapp.BindHooks(app, cfg, s3)
 

archivist/user-stories.md

@@ -12,9 +12,9 @@ so that the container starts with no configuration file for infrastructure and s
 
 Acceptance criteria:
 
-- Given the container starts with all mandatory variables set (`ARCHIVIST_S3_ENDPOINT`, `ARCHIVIST_S3_ACCESS_KEY`, `ARCHIVIST_S3_SECRET_KEY`, `ARCHIVIST_S3_BUCKET`, `ARCHIVIST_ADMIN_EMAIL`, `ARCHIVIST_ADMIN_PASSWORD`), when the application starts, then it connects to the S3 server, verifies the bucket is reachable, and starts serving HTTP on the configured port; when bucket versioning is disabled, a warning is logged recommending the user to enable it to reduce the risk of losing a file to a bug (the application starts anyway).
+- Given the container starts with all mandatory variables set (`ARCHIVIST_S3_ENDPOINT`, `ARCHIVIST_S3_ACCESS_KEY`, `ARCHIVIST_S3_SECRET_KEY`, `ARCHIVIST_S3_BUCKET`, `ARCHIVIST_ADMIN_EMAIL`, `ARCHIVIST_ADMIN_PASSWORD`), when the application starts, then it connects to the S3 server, verifies the bucket is reachable, and starts serving HTTP on the configured port; when bucket versioning is disabled, a warning is emitted through the PocketBase logger (visible in the admin dashboard, Dashboard > Logs) recommending the user to enable it to reduce the risk of losing a file to a bug (the application starts anyway).
 - Given an optional variable is not set, when the application starts, then it uses its documented default (`ARCHIVIST_S3_USE_SSL=true`, `ARCHIVIST_POLL_INTERVAL=* * * * *`, `ARCHIVIST_HTTP_PORT=8090`).
-- Given a mandatory variable is missing or the S3 server is unreachable, when the application starts, then it exits with a non-zero code and a single explicit error message in the logs naming the faulty variable or resource.
+- Given a mandatory variable is missing or the S3 server is unreachable, when the application starts, then it exits with a non-zero code and a single explicit error message on stderr (visible in `docker logs`) naming the faulty variable or resource.
 - Given the admin account variables are set, when the application starts for the first time, then the PocketBase admin account is created with these credentials; on subsequent starts it is left unchanged unless the password differs, in which case it is updated.
 
 ### US-02 — Load and validate the functional configuration
@@ -83,7 +83,7 @@ Acceptance criteria:
 - Given the cron triggers, when the poller runs, then it lists all objects under `inbox/`.
 - Given an object is listed, when no `documents` record references its key yet, then it is a candidate and its size and ETag are recorded; it becomes eligible for download only if its size and ETag are unchanged on the next poll.
 - Given an object is listed, when a `documents` record already references its key, then it is skipped (interrupted filings are handled by US-11).
-- Given a transient S3 error occurs, when the poller runs, then the error is logged, the pass ends, and the next pass retries; no object is lost.
+- Given a transient S3 error occurs, when the poller runs, then the error is logged (PocketBase logger, Error level), the pass ends, and the next pass retries; no object is lost.
 
 ### US-11 — Recover interrupted filings
 
@@ -95,7 +95,7 @@ Acceptance criteria:
 
 - Given a poll pass runs, when a `documents` record exists with `file_key` under `inbox/` and a classification verdict already computed, then the filing procedure (US-17) runs again for that record.
 - Given the recovery succeeds, when the object is moved, then the record is updated exactly as a normal filing would (same events, same conflict resolution).
-- Given the inbox object of a stuck record no longer exists, when the recovery is attempted, then the record is flagged in the logs for manual inspection and never blocks the poll pass.
+- Given the inbox object of a stuck record no longer exists, when the recovery is attempted, then the record is flagged in the logs (PocketBase logger, Warn level) for manual inspection and never blocks the poll pass.
 
 ### US-12 — Download the object and reject duplicates
 
@@ -106,7 +106,7 @@ so that the same file never enters the pipeline twice.
 Acceptance criteria:
 
 - Given an eligible candidate, when it is downloaded, then it is streamed in memory with no disk write and its sha256 is computed on the fly.
-- Given the computed sha256 matches an existing `documents` record, when the download completes, then the `inbox/` key is deleted (a no-data-loss case — the original with the same content stays filed; with versioning enabled the deleted copy is additionally recoverable via delete markers), the duplicate is logged in the application logs only (no `document_events` entry, by design), and processing stops for that object.
+- Given the computed sha256 matches an existing `documents` record, when the download completes, then the `inbox/` key is deleted (a no-data-loss case — the original with the same content stays filed; with versioning enabled the deleted copy is additionally recoverable via delete markers), the duplicate is logged in the PocketBase logs only (Dashboard > Logs; no `document_events` entry, by design), and processing stops for that object.
 - Given the sha256 is new, when the download completes, then processing continues to extraction.
 
 ### US-13 — Extract the text page by page
@@ -135,7 +135,7 @@ Acceptance criteria:
 - Given a field assignment whose value does not match the declared field type (raw string to a `date`, non-numeric to a `number`), when it executes, then it is a rule error and the document is routed to `pending/`.
 - Given `firstMatch` finds no match, when its result is used, then it returns an empty string (parsing an empty string with `date()` is a rule error).
 - Given `delete` is executed, when the rule fires, then the named fields become undefined again for later rules and template rendering.
-- Given the whole evaluation, when it runs, then every rule evaluation and action is written in detail to the application logs (the same trace feeds the future dry-run mode).
+- Given the whole evaluation, when it runs, then every rule evaluation and action is written in detail to the PocketBase logger at Debug level (visible in Dashboard > Logs; the same trace feeds the future dry-run mode).
 
 ### US-15 — Decide the verdict