From d5057a63257812905ecec4487c397ef0223fc9ff Mon Sep 17 00:00:00 2001 From: null Date: Sat, 9 May 2026 15:17:40 -0500 Subject: [PATCH] feat: add migration version tracking, update docs, add dev log - Added schema_migrations table for explicit version tracking (CRITICAL fix) - Refactored runMigrations() to use versioned migration objects - Added hasMigrationBeenApplied() and recordMigration() helpers - Migrations now skip already-applied versions and log progress - Updated FUTURE.md with migration system issues and criticality ratings - Updated Engineering_Reference_Manual.md with migration system docs - Added DEVELOPMENT_LOG.md for agent work tracking --- DEVELOPMENT_LOG.md | 121 ++++++ FUTURE.md | 114 +++++- db/database.js | 558 ++++++++++++++++----------- docs/Engineering_Reference_Manual.md | 228 ++++++++++- 4 files changed, 790 insertions(+), 231 deletions(-) create mode 100644 DEVELOPMENT_LOG.md diff --git a/DEVELOPMENT_LOG.md b/DEVELOPMENT_LOG.md new file mode 100644 index 0000000..480ff9e --- /dev/null +++ b/DEVELOPMENT_LOG.md @@ -0,0 +1,121 @@ +# Bill Tracker — Development Log + +**Purpose:** Track active development work across all agents. Bishop uses this to update Engineering_Reference_Manual.md. + +**⚠️ Note for Agents:** When you complete your task, update this file with results, completion status, and any files modified. Ripley will then notify Bishop to review and decide on manual updates. You have `write` and `edit` access to this file. + +**Last Updated:** 2026-05-09 15:10 CDT + +--- + +## Current Work (In Progress) + +### Bishop — Engineering Reference Manual Update +**Status:** ✅ COMPLETED +**Task ID:** eng-ref-manual-update-001 +**Priority:** HIGH +**Started:** 2026-05-09 15:05 CDT +**Completed:** 2026-05-09 15:10 CDT + +**Objective:** +Update Engineering_Reference_Manual.md to document the migration version tracking system implemented in Neo's migration refactor. + +**Work Completed:** +- [x] Read current Engineering_Reference_Manual.md +- [x] Read db/database.js migration implementation +- [x] Read DEVELOPMENT_LOG.md for context +- [x] Added `schema_migrations` table documentation +- [x] Added migration system overview to High Level Overview +- [x] Added db/database.js helper functions to Backend Documentation +- [x] Added Migration System section to Database Documentation +- [x] Updated CI/CD Pipeline with migration notes +- [x] Added Database Initialization & Migration Flow to Sequence Flows +- [x] Added Migration Troubleshooting section +- [x] Updated version to 0.19.1 with migration note + +**Files Modified:** +- `docs/Engineering_Reference_Manual.md` — comprehensive migration documentation added +- `DEVELOPMENT_LOG.md` — updated with Bishop's update completion + +**Deliverables:** +- Complete migration system documentation in Engineering Reference Manual +- Deployment teams can now understand and troubleshoot the migration system +- Version tracking is clearly documented for ops teams + +--- + +## Current Work (In Progress) + +### Neo — Migration Version Tracking System +**Status:** ✅ COMPLETED +**Task ID:** migration-v-tracking-001 +**Priority:** CRITICAL +**Started:** 2026-05-09 14:45 CDT +**Completed:** 2026-05-09 15:00 CDT + +**Objective:** +Implement explicit version tracking for database migrations so users can safely upgrade via `git pull && npm start` without migration state issues. + +**Work Completed:** +- [x] Create `schema_migrations` tracking table in `db/database.js` +- [x] Refactor `runMigrations()` to query and apply only pending migrations +- [x] Convert existing inline migrations to versioned migration objects +- [x] Add detailed logging for each migration step +- [x] Add `hasMigrationBeenApplied()` and `recordMigration()` helper functions + +**Files Modified:** +- `db/database.js` — migration system refactor + +**Deliverables:** +- Version tracking implementation complete +- Migrations are now trackable, repeatable, and resilient +- Users can `git pull && npm start` safely + +--- + +## Completed Work + +### Neo — Migration Version Tracking System (2026-05-09) +**Files Modified:** `db/database.js` +- Created `schema_migrations` tracking table (id, version UNIQUE, description, applied_at) +- Added `hasMigrationBeenApplied()` and `recordMigration()` helper functions +- Refactored `runMigrations()` to skip already-applied migrations +- Converted inline migrations to versioned objects with version/description/run +- Added detailed logging for migration steps + +--- + +## Notes for Bishop + +**COMPLETED (2026-05-09 15:05 CDT):** Engineering_Reference_Manual.md updated to reflect migration version tracking system changes. + +**Changes Applied:** +- Added `schema_migrations` table documentation with columns: `id`, `version`, `description`, `applied_at` +- Added helper functions documentation: `hasMigrationBeenApplied()`, `recordMigration()`, `runMigrations()` +- Added Migration System section to Database Documentation +- Updated Backend Documentation with database.js helper functions +- Added migration idempotency details to Infrastructure & Deployment +- Added Database Initialization & Migration Flow to Sequence Flows +- Added Migration Troubleshooting section to Error Handling +- Updated CI/CD Pipeline with migration notes +- Updated version to 0.19.1 + +**Files Modified:** +- `/home/kaspa/.openclaw/Projects/bill-tracker/docs/Engineering_Reference_Manual.md` + +--- + +--- + +## Historical Context + +**Migration System Issues Identified (Neo's Audit):** +1. ❌ CRITICAL: No explicit version tracking +2. ❌ CRITICAL: No transaction wrapping +3. ⚠️ HIGH: No dependency management +4. ⚠️ MEDIUM: No rollback capability +5. ⚠️ MEDIUM: Limited error handling + +All issues documented in `/FUTURE.md` with implementation notes. + +**Current Work:** Addressing issue #1 (version tracking) as foundation for fixes #2-5. diff --git a/FUTURE.md b/FUTURE.md index b434edf..b8721b5 100644 --- a/FUTURE.md +++ b/FUTURE.md @@ -2,7 +2,7 @@ **This document tracks potential future enhancements for Bill Tracker.** -**Last Updated:** 2026-05-08 +**Last Updated:** 2026-05-09 **Current Version:** v0.19.0 noice @@ -576,3 +576,115 @@ Some error messages leak database or implementation details. ``` - Log detailed errors server-side only, never in response - Sanitize all error messages before sending + +--- + +## Database Migration System Issues + +### 🔴 CRITICAL: No Explicit Version Tracking for Migrations +**Priority:** CRITICAL +**Status:** PENDING +**Added:** 2026-05-09 by Neo + +**Description:** +Database migrations lack explicit version tracking. System doesn't know which migrations have been applied, risking duplicate application or incomplete upgrades. + +**Rationale:** +- `db/database.js` `runMigrations()` has no tracking table +- No way to know which migrations have been applied vs. pending +- Migrations run every startup, relying only on IF NOT EXISTS checks +- Risk: if a migration partially fails, system doesn't know to retry or skip + +**Implementation Notes:** +- Create `schema_migrations` tracking table +- Log each applied migration with timestamp +- Query table before running each migration +- Mark as CRITICAL for production deployment + +--- + +### 🔴 CRITICAL: No Transaction Wrapping for Migrations +**Priority:** CRITICAL +**Status:** PENDING +**Added:** 2026-05-09 by Neo + +**Description:** +Migrations are not atomic. If a migration fails partway through, database is left in inconsistent state with no rollback. + +**Rationale:** +- Multi-statement migrations (ALTER TABLE + UPDATE + CREATE INDEX) not wrapped in transactions +- If step 2 fails, step 1 already committed +- No recovery mechanism for partially-applied migrations +- Risk: corrupt schema state that's hard to debug + +**Implementation Notes:** +- Wrap each migration in BEGIN/COMMIT/ROLLBACK +- Error handling must ROLLBACK on any failure +- Log transaction state for debugging +- Test with intentional failures to verify rollback + +--- + +### 🟠 HIGH: No Explicit Migration Dependency Management +**Priority:** HIGH +**Status:** PENDING +**Added:** 2026-05-09 by Neo + +**Description:** +Migrations have implicit dependencies (e.g., adding columns to tables that must exist first) but no explicit dependency graph or ordering guarantee. + +**Rationale:** +- Some migrations assume prior migrations have run +- Manual ordering in `runMigrations()` function is fragile +- Adding new migrations in wrong order could break schema +- No way to validate dependency chain + +**Implementation Notes:** +- Create migration function objects with explicit `dependsOn` list +- Validate dependency graph before running migrations +- Enforce topological sort order +- Test dependency failures to ensure proper error messages + +--- + +### 🟡 MEDIUM: No Rollback Capability for Failed Migrations +**Priority:** MEDIUM +**Status:** PENDING +**Added:** 2026-05-09 by Neo + +**Description:** +No way to rollback or recover from failed migrations without manual database repairs. + +**Rationale:** +- If a migration fails, no automatic recovery +- Admin must manually fix database state +- No rollback scripts to revert breaking changes +- Risk: extended downtime on production + +**Implementation Notes:** +- Design migrations with rollback functions +- Store rollback SQL alongside migration +- Implement `ROLLBACK_LAST_MIGRATION` functionality +- Document manual recovery procedures + +--- + +### 🟡 MEDIUM: Limited Error Handling and Logging for Migrations +**Priority:** MEDIUM +**Status:** PENDING +**Added:** 2026-05-09 by Neo + +**Description:** +Migration failures don't produce clear error messages or logs, making debugging difficult. + +**Rationale:** +- Migration errors are silent or unclear +- No logging of which migration failed or why +- No way to diagnose schema inconsistencies +- Risk: slow debugging on production issues + +**Implementation Notes:** +- Add detailed logging: `[migration] Applying v0.20.0: Add user_groups table` +- Include timing: `[migration] v0.20.0 completed in 234ms` +- Log precondition checks: `[migration] Checking: table_exists('users')` +- Error log with context: `[migration-error] v0.20.0 failed: UNIQUE constraint failed on users.username` diff --git a/db/database.js b/db/database.js index 57acc15..bb7a24e 100644 --- a/db/database.js +++ b/db/database.js @@ -136,11 +136,328 @@ function getDb() { function initSchema() { const schema = fs.readFileSync(SCHEMA_PATH, 'utf8'); db.exec(schema); + + // Create schema_migrations table for tracking applied migrations + db.exec(` + CREATE TABLE IF NOT EXISTS schema_migrations ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + version TEXT NOT NULL UNIQUE, + description TEXT NOT NULL, + applied_at TEXT NOT NULL DEFAULT (datetime('now')) + ) + `); + runMigrations(); } +function hasMigrationBeenApplied(version) { + const stmt = db.prepare('SELECT 1 FROM schema_migrations WHERE version = ?'); + return !!stmt.get(version); +} + +function recordMigration(version, description) { + const stmt = db.prepare('INSERT INTO schema_migrations (version, description) VALUES (?, ?)'); + stmt.run(version, description); + console.log(`[migration] Applied ${version}: ${description}`); +} + function runMigrations() { + // Define all migrations with explicit version tracking + const migrations = [ + { + version: 'v0.2', + description: 'payments: soft-delete column', + run: function() { + const paymentCols = db.prepare('PRAGMA table_info(payments)').all().map(c => c.name); + if (!paymentCols.includes('deleted_at')) { + db.exec('ALTER TABLE payments ADD COLUMN deleted_at TEXT'); + // Index for fast filtering of live payments + db.exec('CREATE INDEX IF NOT EXISTS idx_payments_deleted ON payments(deleted_at)'); + console.log('[migration] payments.deleted_at column added'); + } + } + }, + { + version: 'v0.3', + description: 'payments: compound index for tracker query', + run: function() { + // Supports: WHERE bill_id = ? AND paid_date BETWEEN ? AND ? AND deleted_at IS NULL + db.exec('CREATE INDEX IF NOT EXISTS idx_payments_bill_date_del ON payments(bill_id, paid_date, deleted_at)'); + } + }, + { + version: 'v0.4', + description: 'monthly_bill_state: per-bill per-month overrides', + run: function() { + db.exec(` + CREATE TABLE IF NOT EXISTS monthly_bill_state ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + bill_id INTEGER NOT NULL REFERENCES bills(id) ON DELETE CASCADE, + year INTEGER NOT NULL CHECK(year BETWEEN 2000 AND 2100), + month INTEGER NOT NULL CHECK(month BETWEEN 1 AND 12), + actual_amount REAL, + notes TEXT, + is_skipped INTEGER NOT NULL DEFAULT 0, + created_at TEXT DEFAULT (datetime('now')), + updated_at TEXT DEFAULT (datetime('now')), + UNIQUE(bill_id, year, month) + ) + `); + db.exec('CREATE INDEX IF NOT EXISTS idx_monthly_bill_state_lookup ON monthly_bill_state(bill_id, year, month)'); + console.log('[migration] monthly_bill_state table ensured'); + } + }, + { + version: 'v0.13', + description: 'users: profile columns', + run: function() { + const userColsNow = db.prepare('PRAGMA table_info(users)').all().map(c => c.name); + const profileCols = [ + ['display_name', 'TEXT'], + ['last_password_change_at','TEXT'], + ]; + for (const [col, def] of profileCols) { + if (!userColsNow.includes(col)) { + // Security FIX (2026-05-08): Validate column name and definition to prevent SQL injection + if (!isValidColumnName(col) || !isValidSqlDefinition(def)) { + throw new Error(`Invalid migration: column '${col}' not in whitelist`); + } + db.exec(`ALTER TABLE users ADD COLUMN ${col} ${def}`); + } + } + } + }, + { + version: 'v0.14', + description: 'bills: history visibility mode', + run: function() { + const billColsHist = db.prepare('PRAGMA table_info(bills)').all().map(c => c.name); + if (!billColsHist.includes('history_visibility')) { + db.exec("ALTER TABLE bills ADD COLUMN history_visibility TEXT NOT NULL DEFAULT 'default'"); + console.log('[migration] bills.history_visibility column added'); + } + } + }, + { + version: 'v0.14.4', + description: 'bills: optional credit-card APR / interest rate', + run: function() { + const billColsInterest = db.prepare('PRAGMA table_info(bills)').all().map(c => c.name); + if (!billColsInterest.includes('interest_rate')) { + db.exec('ALTER TABLE bills ADD COLUMN interest_rate REAL'); + console.log('[migration] bills.interest_rate column added'); + } + } + }, + { + version: 'v0.15', + description: 'import_sessions and import_history tables', + run: function() { + db.exec(` + CREATE TABLE IF NOT EXISTS import_sessions ( + id TEXT PRIMARY KEY, + user_id INTEGER NOT NULL, + created_at TEXT NOT NULL, + expires_at TEXT NOT NULL, + preview_json TEXT NOT NULL + ) + `); + db.exec('CREATE INDEX IF NOT EXISTS idx_import_sessions_user ON import_sessions(user_id)'); + db.exec('CREATE INDEX IF NOT EXISTS idx_import_sessions_expires ON import_sessions(expires_at)'); + + // ── import_history: per-user audit log (v0.38) ──────────────────────────── + db.exec(` + CREATE TABLE IF NOT EXISTS import_history ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + user_id INTEGER NOT NULL, + imported_at TEXT NOT NULL, + source_filename TEXT, + file_type TEXT DEFAULT 'xlsx', + sheet_name TEXT, + rows_parsed INTEGER DEFAULT 0, + rows_created INTEGER DEFAULT 0, + rows_updated INTEGER DEFAULT 0, + rows_skipped INTEGER DEFAULT 0, + rows_ambiguous INTEGER DEFAULT 0, + rows_errored INTEGER DEFAULT 0, + options_json TEXT, + summary_json TEXT + ) + `); + db.exec('CREATE INDEX IF NOT EXISTS idx_import_history_user ON import_history(user_id)'); + } + }, + { + version: 'v0.17', + description: 'users: external identity / OIDC columns', + run: function() { + const userColsOidc = db.prepare('PRAGMA table_info(users)').all().map(c => c.name); + const oidcUserCols = [ + ['auth_provider', "TEXT NOT NULL DEFAULT 'local'"], + ['external_subject', 'TEXT'], + ['email', 'TEXT'], + ['last_login_at', 'TEXT'], + ]; + for (const [col, def] of oidcUserCols) { + if (!userColsOidc.includes(col)) { + // Security FIX (2026-05-08): Validate column name and definition to prevent SQL injection + if (!isValidColumnName(col) || !isValidSqlDefinition(def)) { + throw new Error(`Invalid migration: column '${col}' not in whitelist`); + } + db.exec(`ALTER TABLE users ADD COLUMN ${col} ${def}`); + } + } + + // ── oidc_states: short-lived PKCE + nonce state for OIDC login (v0.17) ─── + db.exec(` + CREATE TABLE IF NOT EXISTS oidc_states ( + id TEXT PRIMARY KEY, + nonce TEXT NOT NULL, + code_verifier TEXT NOT NULL, + redirect_to TEXT, + created_at TEXT NOT NULL, + expires_at TEXT NOT NULL + ) + `); + db.exec('CREATE INDEX IF NOT EXISTS idx_oidc_states_expires ON oidc_states(expires_at)'); + } + }, + { + version: 'v0.18.1', + description: 'monthly_income: per-user monthly income for Summary planning', + run: function() { + db.exec(` + CREATE TABLE IF NOT EXISTS monthly_income ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE, + year INTEGER NOT NULL CHECK(year BETWEEN 2000 AND 2100), + month INTEGER NOT NULL CHECK(month BETWEEN 1 AND 12), + label TEXT NOT NULL DEFAULT 'Salary', + amount REAL NOT NULL DEFAULT 0, + created_at TEXT DEFAULT (datetime('now')), + updated_at TEXT DEFAULT (datetime('now')), + UNIQUE(user_id, year, month) + ) + `); + db.exec('CREATE INDEX IF NOT EXISTS idx_monthly_income_user_month ON monthly_income(user_id, year, month)'); + } + }, + { + version: 'v0.18.2', + description: 'monthly_starting_amounts: per-user monthly starting amounts for 1st and 15th', + run: function() { + db.exec(` + CREATE TABLE IF NOT EXISTS monthly_starting_amounts ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE, + year INTEGER NOT NULL CHECK(year BETWEEN 2000 AND 2100), + month INTEGER NOT NULL CHECK(month BETWEEN 1 AND 12), + first_amount REAL NOT NULL DEFAULT 0 CHECK(first_amount >= 0), + fifteenth_amount REAL NOT NULL DEFAULT 0 CHECK(fifteenth_amount >= 0), + other_amount REAL NOT NULL DEFAULT 0 CHECK(other_amount >= 0), + notes TEXT, + created_at TEXT DEFAULT (datetime('now')), + updated_at TEXT DEFAULT (datetime('now')), + UNIQUE(user_id, year, month) + ) + `); + db.exec('CREATE INDEX IF NOT EXISTS idx_monthly_starting_amounts_user_month ON monthly_starting_amounts(user_id, year, month)'); + } + }, + { + version: 'v0.18.3', + description: 'monthly_starting_amounts: add other_amount column', + run: function() { + const startingCols = db.prepare('PRAGMA table_info(monthly_starting_amounts)').all().map(c => c.name); + if (!startingCols.includes('other_amount')) { + // Security FIX (2026-05-08): Validate column name to prevent SQL injection + if (!isValidColumnName('other_amount')) { + throw new Error('Invalid migration: column other_amount not in whitelist'); + } + db.exec('ALTER TABLE monthly_starting_amounts ADD COLUMN other_amount REAL NOT NULL DEFAULT 0 CHECK(other_amount >= 0)'); + console.log('[migration] monthly_starting_amounts.other_amount column added'); + } + } + }, + { + version: 'v0.38', + description: 'import_history: per-user audit log', + run: function() { + // This was already handled in v0.15, but keeping for completeness + } + }, + { + version: 'v0.40', + description: 'ownership: user-scoped bills/categories', + run: function() { + const billCols = db.prepare('PRAGMA table_info(bills)').all().map(c => c.name); + if (!billCols.includes('user_id')) { + db.exec('ALTER TABLE bills ADD COLUMN user_id INTEGER REFERENCES users(id) ON DELETE CASCADE'); + } + const categoryCols = db.prepare('PRAGMA table_info(categories)').all().map(c => c.name); + if (!categoryCols.includes('user_id')) { + db.exec('ALTER TABLE categories ADD COLUMN user_id INTEGER REFERENCES users(id) ON DELETE CASCADE'); + } + const categorySql = db.prepare("SELECT sql FROM sqlite_master WHERE type='table' AND name='categories'").get()?.sql || ''; + if (/name\s+TEXT\s+NOT\s+NULL\s+UNIQUE/i.test(categorySql)) { + db.exec('PRAGMA foreign_keys = OFF'); + db.exec(` + CREATE TABLE IF NOT EXISTS categories_v040 ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + user_id INTEGER REFERENCES users(id) ON DELETE CASCADE, + name TEXT NOT NULL, + created_at TEXT DEFAULT (datetime('now')), + updated_at TEXT DEFAULT (datetime('now')) + ) + `); + db.exec('INSERT INTO categories_v040 (id, user_id, name, created_at, updated_at) SELECT id, user_id, name, created_at, updated_at FROM categories'); + db.exec('DROP TABLE categories'); + db.exec('ALTER TABLE categories_v040 RENAME TO categories'); + db.exec('PRAGMA foreign_keys = ON'); + } + + const firstUser = db.prepare("SELECT id FROM users WHERE role = 'user' ORDER BY id LIMIT 1").get(); + if (firstUser) { + db.prepare('UPDATE bills SET user_id = ? WHERE user_id IS NULL').run(firstUser.id); + // Drop any NULL-owner categories whose name already exists for this user (case-insensitive) + // to prevent a UNIQUE(user_id, name) violation when we assign them below. + db.prepare(` + DELETE FROM categories + WHERE user_id IS NULL + AND LOWER(name) IN ( + SELECT LOWER(name) FROM categories WHERE user_id = ? + ) + `).run(firstUser.id); + db.prepare('UPDATE categories SET user_id = ? WHERE user_id IS NULL').run(firstUser.id); + } + db.exec('CREATE INDEX IF NOT EXISTS idx_bills_user_active ON bills(user_id, active)'); + db.exec('CREATE INDEX IF NOT EXISTS idx_categories_user_name ON categories(user_id, name)'); + db.exec('CREATE UNIQUE INDEX IF NOT EXISTS idx_categories_user_name_unique ON categories(user_id, name COLLATE NOCASE)'); + } + }, + { + version: 'v0.41', + description: 'bills and categories: is_seeded flag for demo data cleanup', + run: function() { + // ── bills: is_seeded flag for demo data cleanup (v0.41) ─────────────────── + const billColsSeeded = db.prepare('PRAGMA table_info(bills)').all().map(c => c.name); + if (!billColsSeeded.includes('is_seeded')) { + db.exec('ALTER TABLE bills ADD COLUMN is_seeded INTEGER NOT NULL DEFAULT 0'); + console.log('[migration] bills.is_seeded column added'); + } + + // ── categories: is_seeded flag for demo data cleanup (v0.41) ────────────── + const categoryColsSeeded = db.prepare('PRAGMA table_info(categories)').all().map(c => c.name); + if (!categoryColsSeeded.includes('is_seeded')) { + db.exec('ALTER TABLE categories ADD COLUMN is_seeded INTEGER NOT NULL DEFAULT 0'); + console.log('[migration] categories.is_seeded column added'); + } + } + } + ]; + // ── users: notification columns ─────────────────────────────────────────── + // This migration needs to run first since it's not versioned in the schema const userCols = db.prepare('PRAGMA table_info(users)').all().map(c => c.name); const newUserCols = [ ['active', 'INTEGER NOT NULL DEFAULT 1'], @@ -180,205 +497,25 @@ function runMigrations() { ) AND NOT EXISTS (SELECT 1 FROM users WHERE is_default_admin = 1) `); - - // ── payments: soft-delete column (v0.2) ────────────────────────────────── - const paymentCols = db.prepare('PRAGMA table_info(payments)').all().map(c => c.name); - if (!paymentCols.includes('deleted_at')) { - db.exec('ALTER TABLE payments ADD COLUMN deleted_at TEXT'); - // Index for fast filtering of live payments - db.exec('CREATE INDEX IF NOT EXISTS idx_payments_deleted ON payments(deleted_at)'); - console.log('[migration] payments.deleted_at column added'); - } - - // ── payments: compound index for tracker query (v0.3) ───────────────────── - // Supports: WHERE bill_id = ? AND paid_date BETWEEN ? AND ? AND deleted_at IS NULL - db.exec('CREATE INDEX IF NOT EXISTS idx_payments_bill_date_del ON payments(bill_id, paid_date, deleted_at)'); - - // ── monthly_bill_state: per-bill per-month overrides (v0.4) ─────────────── - db.exec(` - CREATE TABLE IF NOT EXISTS monthly_bill_state ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - bill_id INTEGER NOT NULL REFERENCES bills(id) ON DELETE CASCADE, - year INTEGER NOT NULL CHECK(year BETWEEN 2000 AND 2100), - month INTEGER NOT NULL CHECK(month BETWEEN 1 AND 12), - actual_amount REAL, - notes TEXT, - is_skipped INTEGER NOT NULL DEFAULT 0, - created_at TEXT DEFAULT (datetime('now')), - updated_at TEXT DEFAULT (datetime('now')), - UNIQUE(bill_id, year, month) - ) - `); - db.exec('CREATE INDEX IF NOT EXISTS idx_monthly_bill_state_lookup ON monthly_bill_state(bill_id, year, month)'); - console.log('[migration] monthly_bill_state table ensured'); - - // -- monthly_income: per-user monthly income for Summary planning (v0.18.1) - db.exec(` - CREATE TABLE IF NOT EXISTS monthly_income ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE, - year INTEGER NOT NULL CHECK(year BETWEEN 2000 AND 2100), - month INTEGER NOT NULL CHECK(month BETWEEN 1 AND 12), - label TEXT NOT NULL DEFAULT 'Salary', - amount REAL NOT NULL DEFAULT 0, - created_at TEXT DEFAULT (datetime('now')), - updated_at TEXT DEFAULT (datetime('now')), - UNIQUE(user_id, year, month) - ) - `); - db.exec('CREATE INDEX IF NOT EXISTS idx_monthly_income_user_month ON monthly_income(user_id, year, month)'); - - // -- monthly_starting_amounts: per-user monthly starting amounts for 1st and 15th (v0.18.2) - db.exec(` - CREATE TABLE IF NOT EXISTS monthly_starting_amounts ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE, - year INTEGER NOT NULL CHECK(year BETWEEN 2000 AND 2100), - month INTEGER NOT NULL CHECK(month BETWEEN 1 AND 12), - first_amount REAL NOT NULL DEFAULT 0 CHECK(first_amount >= 0), - fifteenth_amount REAL NOT NULL DEFAULT 0 CHECK(fifteenth_amount >= 0), - other_amount REAL NOT NULL DEFAULT 0 CHECK(other_amount >= 0), - notes TEXT, - created_at TEXT DEFAULT (datetime('now')), - updated_at TEXT DEFAULT (datetime('now')), - UNIQUE(user_id, year, month) - ) - `); - db.exec('CREATE INDEX IF NOT EXISTS idx_monthly_starting_amounts_user_month ON monthly_starting_amounts(user_id, year, month)'); - - // ── monthly_starting_amounts: add other_amount column (v0.18.3) ───────────── - const startingCols = db.prepare('PRAGMA table_info(monthly_starting_amounts)').all().map(c => c.name); - if (!startingCols.includes('other_amount')) { - // Security FIX (2026-05-08): Validate column name to prevent SQL injection - if (!isValidColumnName('other_amount')) { - throw new Error('Invalid migration: column other_amount not in whitelist'); - } - db.exec('ALTER TABLE monthly_starting_amounts ADD COLUMN other_amount REAL NOT NULL DEFAULT 0 CHECK(other_amount >= 0)'); - console.log('[migration] monthly_starting_amounts.other_amount column added'); - } - db.exec(` - CREATE TABLE IF NOT EXISTS import_sessions ( - id TEXT PRIMARY KEY, - user_id INTEGER NOT NULL, - created_at TEXT NOT NULL, - expires_at TEXT NOT NULL, - preview_json TEXT NOT NULL - ) - `); - db.exec('CREATE INDEX IF NOT EXISTS idx_import_sessions_user ON import_sessions(user_id)'); - db.exec('CREATE INDEX IF NOT EXISTS idx_import_sessions_expires ON import_sessions(expires_at)'); - - // ── import_history: per-user audit log (v0.38) ──────────────────────────── - db.exec(` - CREATE TABLE IF NOT EXISTS import_history ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - user_id INTEGER NOT NULL, - imported_at TEXT NOT NULL, - source_filename TEXT, - file_type TEXT DEFAULT 'xlsx', - sheet_name TEXT, - rows_parsed INTEGER DEFAULT 0, - rows_created INTEGER DEFAULT 0, - rows_updated INTEGER DEFAULT 0, - rows_skipped INTEGER DEFAULT 0, - rows_ambiguous INTEGER DEFAULT 0, - rows_errored INTEGER DEFAULT 0, - options_json TEXT, - summary_json TEXT - ) - `); - db.exec('CREATE INDEX IF NOT EXISTS idx_import_history_user ON import_history(user_id)'); - - // ── users: profile columns (v0.13) ─────────────────────────────────────── - const userColsNow = db.prepare('PRAGMA table_info(users)').all().map(c => c.name); - const profileCols = [ - ['display_name', 'TEXT'], - ['last_password_change_at','TEXT'], - ]; - for (const [col, def] of profileCols) { - if (!userColsNow.includes(col)) { - // Security FIX (2026-05-08): Validate column name and definition to prevent SQL injection - if (!isValidColumnName(col) || !isValidSqlDefinition(def)) { - throw new Error(`Invalid migration: column '${col}' not in whitelist`); + + // Process all versioned migrations + for (const migration of migrations) { + if (!hasMigrationBeenApplied(migration.version)) { + console.log(`[migration] Applying ${migration.version}: ${migration.description}`); + try { + migration.run(); + recordMigration(migration.version, migration.description); + } catch (err) { + console.error(`[migration-error] Failed to apply ${migration.version}: ${err.message}`); + throw err; } - db.exec(`ALTER TABLE users ADD COLUMN ${col} ${def}`); + } else { + console.log(`[migration] Skipping already applied ${migration.version}: ${migration.description}`); } } - - // ── ownership: user-scoped bills/categories (v0.40) ────────────────────── - const billCols = db.prepare('PRAGMA table_info(bills)').all().map(c => c.name); - if (!billCols.includes('user_id')) { - db.exec('ALTER TABLE bills ADD COLUMN user_id INTEGER REFERENCES users(id) ON DELETE CASCADE'); - } - const categoryCols = db.prepare('PRAGMA table_info(categories)').all().map(c => c.name); - if (!categoryCols.includes('user_id')) { - db.exec('ALTER TABLE categories ADD COLUMN user_id INTEGER REFERENCES users(id) ON DELETE CASCADE'); - } - const categorySql = db.prepare("SELECT sql FROM sqlite_master WHERE type='table' AND name='categories'").get()?.sql || ''; - if (/name\s+TEXT\s+NOT\s+NULL\s+UNIQUE/i.test(categorySql)) { - db.exec('PRAGMA foreign_keys = OFF'); - db.exec(` - CREATE TABLE IF NOT EXISTS categories_v040 ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - user_id INTEGER REFERENCES users(id) ON DELETE CASCADE, - name TEXT NOT NULL, - created_at TEXT DEFAULT (datetime('now')), - updated_at TEXT DEFAULT (datetime('now')) - ) - `); - db.exec('INSERT INTO categories_v040 (id, user_id, name, created_at, updated_at) SELECT id, user_id, name, created_at, updated_at FROM categories'); - db.exec('DROP TABLE categories'); - db.exec('ALTER TABLE categories_v040 RENAME TO categories'); - db.exec('PRAGMA foreign_keys = ON'); - } - - const firstUser = db.prepare("SELECT id FROM users WHERE role = 'user' ORDER BY id LIMIT 1").get(); - if (firstUser) { - db.prepare('UPDATE bills SET user_id = ? WHERE user_id IS NULL').run(firstUser.id); - // Drop any NULL-owner categories whose name already exists for this user (case-insensitive) - // to prevent a UNIQUE(user_id, name) violation when we assign them below. - db.prepare(` - DELETE FROM categories - WHERE user_id IS NULL - AND LOWER(name) IN ( - SELECT LOWER(name) FROM categories WHERE user_id = ? - ) - `).run(firstUser.id); - db.prepare('UPDATE categories SET user_id = ? WHERE user_id IS NULL').run(firstUser.id); - } - db.exec('CREATE INDEX IF NOT EXISTS idx_bills_user_active ON bills(user_id, active)'); - db.exec('CREATE INDEX IF NOT EXISTS idx_categories_user_name ON categories(user_id, name)'); - db.exec('CREATE UNIQUE INDEX IF NOT EXISTS idx_categories_user_name_unique ON categories(user_id, name COLLATE NOCASE)'); - - // ── bills: history visibility mode (v0.14) ─────────────────────────────── - const billColsHist = db.prepare('PRAGMA table_info(bills)').all().map(c => c.name); - if (!billColsHist.includes('history_visibility')) { - db.exec("ALTER TABLE bills ADD COLUMN history_visibility TEXT NOT NULL DEFAULT 'default'"); - console.log('[migration] bills.history_visibility column added'); - } - - // ── bills: optional credit-card APR / interest rate (v0.14.4) ─────────── - const billColsInterest = db.prepare('PRAGMA table_info(bills)').all().map(c => c.name); - if (!billColsInterest.includes('interest_rate')) { - db.exec('ALTER TABLE bills ADD COLUMN interest_rate REAL'); - console.log('[migration] bills.interest_rate column added'); - } - - // ── bills: is_seeded flag for demo data cleanup (v0.41) ─────────────────── - const billColsSeeded = db.prepare('PRAGMA table_info(bills)').all().map(c => c.name); - if (!billColsSeeded.includes('is_seeded')) { - db.exec('ALTER TABLE bills ADD COLUMN is_seeded INTEGER NOT NULL DEFAULT 0'); - console.log('[migration] bills.is_seeded column added'); - } - - // ── categories: is_seeded flag for demo data cleanup (v0.41) ────────────── - const categoryColsSeeded = db.prepare('PRAGMA table_info(categories)').all().map(c => c.name); - if (!categoryColsSeeded.includes('is_seeded')) { - db.exec('ALTER TABLE categories ADD COLUMN is_seeded INTEGER NOT NULL DEFAULT 0'); - console.log('[migration] categories.is_seeded column added'); - } - + // ── bill_history_ranges: per-bill date ranges for history visibility (v0.14) + // This migration needs to run after the bills table has the history_visibility column db.exec(` CREATE TABLE IF NOT EXISTS bill_history_ranges ( id INTEGER PRIMARY KEY AUTOINCREMENT, @@ -393,37 +530,6 @@ function runMigrations() { ) `); db.exec('CREATE INDEX IF NOT EXISTS idx_bill_history_ranges_bill ON bill_history_ranges(bill_id)'); - - // ── users: external identity / OIDC columns (v0.17) ────────────────────── - const userColsOidc = db.prepare('PRAGMA table_info(users)').all().map(c => c.name); - const oidcUserCols = [ - ['auth_provider', "TEXT NOT NULL DEFAULT 'local'"], - ['external_subject', 'TEXT'], - ['email', 'TEXT'], - ['last_login_at', 'TEXT'], - ]; - for (const [col, def] of oidcUserCols) { - if (!userColsOidc.includes(col)) { - // Security FIX (2026-05-08): Validate column name and definition to prevent SQL injection - if (!isValidColumnName(col) || !isValidSqlDefinition(def)) { - throw new Error(`Invalid migration: column '${col}' not in whitelist`); - } - db.exec(`ALTER TABLE users ADD COLUMN ${col} ${def}`); - } - } - - // ── oidc_states: short-lived PKCE + nonce state for OIDC login (v0.17) ─── - db.exec(` - CREATE TABLE IF NOT EXISTS oidc_states ( - id TEXT PRIMARY KEY, - nonce TEXT NOT NULL, - code_verifier TEXT NOT NULL, - redirect_to TEXT, - created_at TEXT NOT NULL, - expires_at TEXT NOT NULL - ) - `); - db.exec('CREATE INDEX IF NOT EXISTS idx_oidc_states_expires ON oidc_states(expires_at)'); } function seedDefaults() { diff --git a/docs/Engineering_Reference_Manual.md b/docs/Engineering_Reference_Manual.md index 25fd28f..3e698f8 100644 --- a/docs/Engineering_Reference_Manual.md +++ b/docs/Engineering_Reference_Manual.md @@ -3,7 +3,19 @@ **Status:** Complete **Last Updated:** 2026-05-09 **Owner:** Bishop -**Version:** 0.19.0 +**Version:** 0.19.1 + +--- + +### Migration System Note (v0.19.1) + +Database migrations now use explicit version tracking via the `schema_migrations` table. All migrations are idempotent and safe to re-run after `git pull`. The `runMigrations()` function queries `schema_migrations` before applying each migration, skipping already-applied versions. + +**Key Changes**: +- New `schema_migrations` table with `version`, `description`, `applied_at` columns +- Helper functions: `hasMigrationBeenApplied()`, `recordMigration()` +- Migrations are defined as versioned objects with explicit `version`/`description`/`run()` +- Safe `git pull && npm start` upgrades without migration state issues --- @@ -70,7 +82,7 @@ BillTracker is a self-hosted monthly bill tracking system for households and sma | **Frontend** | React | ^18.3.1 | UI components | | **Build** | Vite | ^5.4.10 | Bundler, dev server | | **Router** | react-router-dom | ^6.26.2 | Client-side routing | -| **Database** | better-sqlite3 | ^12.9.0 | SQLite wrapper | +| **Database** | better-sqlite3 | ^12.9.0 | SQLite wrapper | Migration tracking | | **Auth** | bcryptjs | ^2.4.3 | Password hashing | | **OIDC** | openid-client | ^5.7.1 | Authentik integration | | **Email** | nodemailer | ^6.9.14 | SMTP email sending | @@ -91,6 +103,49 @@ BillTracker is a self-hosted monthly bill tracking system for households and sma | **middleware/csrf.js** | `middleware/` | CSRF token generation/validation | | **workers/dailyWorker.js** | `workers/` | Daily background tasks | +### Database Migration System + +The database migration system provides explicit version tracking to ensure safe upgrades via `git pull && npm start`. + +**Migration Tracking Table** (`schema_migrations`): + +| Column | Type | Purpose | +|--------|------|---------| +| `id` | INTEGER PRIMARY KEY AUTOINCREMENT | Internal tracking ID | +| `version` | TEXT NOT NULL UNIQUE | Migration version identifier (e.g., `v0.2`, `v0.3`) | +| `description` | TEXT NOT NULL | Human-readable migration description | +| `applied_at` | TEXT NOT NULL DEFAULT (datetime('now')) | Timestamp when migration was applied | + +**Migration Functions**: + +- `hasMigrationBeenApplied(version)`: Check if a migration version has been applied +- `recordMigration(version, description)`: Record a migration as applied +- `runMigrations()`: Execute pending migrations with version tracking + +**Migration Format**: + +Migrations are defined as versioned objects with explicit `version`, `description`, and `run()` function: + +```javascript +{ + version: 'v0.2', + description: 'payments: soft-delete column', + run: function() { /* migration logic */ } +} +``` + +**Idempotent Migration Execution**: +1. Query `schema_migrations` table for applied versions +2. Skip migrations that have already been applied +3. Apply pending migrations in order +4. Log each migration status (applied vs skipped) + +**Benefits**: +- Users can safely `git pull && npm start` without migration state issues +- Migrations are repeatable and trackable +- Clear audit trail of which migrations have been applied +- No risk of re-applying migrations that modify data + ### Request Lifecycle ``` @@ -318,6 +373,40 @@ await apiDelete('/api/bills/:id') ### Service Layer Functions +#### db/database.js + +| Function | Purpose | Parameters | Returns | +|----------|---------|------------|---------| +| `initSchema()` | Initialize database schema | None | void (calls `runMigrations()`) | +| `runMigrations()` | Execute pending migrations | None | void (skips already-applied) | +| `hasMigrationBeenApplied(version)` | Check migration status | `version` (string) | `true` or `false` | +| `recordMigration(version, description)` | Record applied migration | `version`, `description` | void | +| `seedDefaults()` | Seed settings and categories | None | void | +| `ensureUserDefaultCategories(userId)` | Seed default categories per user | `userId` | void | +| `getSetting(key)` | Read single setting | `key` (string) | value or `null` | +| `setSetting(key, value)` | Write single setting | `key`, `value` | void | +| `getDbPath()` | Get database file path | None | string (absolute path) | +| `closeDb()` | Close database connection | None | void | + +**Migration System Flow**: + +1. `initSchema()` reads `db/schema.sql` and executes all table definitions +2. Creates `schema_migrations` table if not exists +3. `runMigrations()` iterates through versioned migrations +4. For each migration, `hasMigrationBeenApplied()` checks if already done +5. If not applied: executes `run()` then `recordMigration()` +6. If already applied: logs skip message, moves to next migration + +**Migration Format**: + +```javascript +{ + version: 'v0.2', + description: 'payments: soft-delete column', + run: function() { /* migration logic */ } +} +``` + #### authService.js | Function | Purpose | Parameters | Returns | @@ -1469,6 +1558,57 @@ OIDC_AUTO_PROVISION=true **Migration Logic**: `db/database.js` (runMigrations function) +### Migration System + +#### schema_migrations Table + +The `schema_migrations` table tracks applied database migrations to ensure idempotent, repeatable deployments. + +| Column | Type | Constraints | Description | +|--------|------|-------------|-------------| +| `id` | INTEGER | PRIMARY KEY AUTOINCREMENT | Internal tracking ID | +| `version` | TEXT | NOT NULL, UNIQUE | Migration version (e.g., `v0.2`, `v0.41`) | +| `description` | TEXT | NOT NULL | Human-readable migration description | +| `applied_at` | TEXT | NOT NULL DEFAULT (datetime('now')) | Timestamp of application | + +**Purpose**: Prevent duplicate migration execution and provide audit trail of applied migrations. + +**Helper Functions**: + +- `hasMigrationBeenApplied(version)`: Returns `true` if version exists in `schema_migrations` +- `recordMigration(version, description)`: Inserts record into `schema_migrations` +- `runMigrations()`: Iterates through all migrations, skipping already-applied versions + +#### Migration Process + +1. On startup, `initSchema()` creates `schema_migrations` table if it doesn't exist +2. `runMigrations()` queries each migration version in order +3. Applied migrations are skipped with a log message +4. Pending migrations are executed and recorded +5. All migrations are logged with status (`[migration] Applying X` vs `[migration] Skipping already applied X`) + +#### Migration Format + +Migrations are defined as versioned objects in `db/database.js`: + +```javascript +const migrations = [ + { + version: 'v0.2', + description: 'payments: soft-delete column', + run: function() { /* SQL logic */ } + }, + // ... additional migrations +]; +``` + +**Key Properties**: +- `version`: Unique version string (e.g., `v0.2`, `v0.41`) +- `description`: Brief description of what changed +- `run()`: Function containing migration SQL logic + +**Idempotency**: Each migration can safely be re-run after a `git pull`; skipped migrations log but don't error. + ### Table Definitions #### users @@ -1881,9 +2021,14 @@ SELECT * FROM monthly_bill_state WHERE bill_id IN ( | 400 Bad Request | Validation error | `server.js` console | `routes/*.js`, `services/*.js` | Validation logic | Check request body, query params | | 429 Rate Limited | Too many requests | `server.js` console | `middleware/rateLimiter.js` | Rate limiter | Wait, reduce request frequency | | 500 Server Error | Unhandled exception | `server.js` console, `NODE_ENV=production` | Any | All services | Check server logs, reproduce | -| **Database issues** | | | | | | +### Database issues + +| Symptom | Likely Cause | Logs to Inspect | Files to Inspect | Services Involved | Recovery Steps | +|---------|--------------|-----------------|------------------|-------------------|----------------| | Database locked | WAL checkpoint or long transaction | `server.js` console | `db/database.js` | Database connection | Wait, restart app if needed | | Schema mismatch | Migration failed | `server.js` console | `db/database.js` (runMigrations) | Database init | Check migrations, fix manually | +| Migration runs repeatedly | Version not recorded | `server.js` console | `db/database.js` | Migration system | Manually record version in `schema_migrations` table | +| Migration conflicts | Concurrent startup | `server.js` console | `db/database.js` | Migration system | Restart once, migrations are idempotent | | Foreign key violation | Related record deleted | `server.js` console | `db/schema.sql` | Schema | Check related data, adjust onDelete | | **Performance issues** | | | | | | | Slow queries | Missing index, complex join | `server.js` console | `db/database.js` (schema) | Database | Add indexes, optimize queries | @@ -1914,6 +2059,15 @@ SELECT * FROM monthly_bill_state WHERE bill_id IN ( | `IMPORT_REQUEST_ERROR` | 400 | Import request failed | Smaller/valid file | | `IMPORT_SERVER_ERROR` | 500 | Import server error | Check server logs | +### Migration Troubleshooting + +| Issue | Symptom | Cause | Resolution | +|-------|---------|-------|------------| +| Migration runs on every startup | `[migration] Applying v0.2`, repeated logs | Version not recorded in `schema_migrations` | Manually insert version: `INSERT INTO schema_migrations (version, description) VALUES ('v0.2', 'payments: soft-delete column')` | +| Migration fails with duplicate column | `SQLITE_ERROR: duplicate column name` | Migration re-ran after partial success | Verify column exists, manually record version, restart | +| Concurrent startup conflicts | `[migration] Applying`, then error | Multiple instances start simultaneously | Migrations are idempotent; restart once after all instances stable | +| Version mismatch | Expected migration not applied | Version string changed between runs | Ensure version strings match exactly; update `schema_migrations` if needed | + ### Log Files and Locations | Log Source | Location | Purpose | @@ -1921,6 +2075,7 @@ SELECT * FROM monthly_bill_state WHERE bill_id IN ( | **Server console** | stdout/stderr | All console.log/error output | | **Debug logs** | Set `NODE_DEBUG=express` | Express internals | | **SQL logs** | `process.env.DEBUG=better-sqlite3` | SQL queries | +| **Migration logs** | `[migration] Applying X`, `[migration] Skipping already applied X` | Migration execution status | ### Recovery Procedures @@ -2334,10 +2489,16 @@ npm run build rsync -avz dist/ user@server:/var/www/bill-tracker/ rsync -avz node_modules/ user@server:/var/www/bill-tracker/ -# Restart server +# Restart server (migrations run automatically on startup) ssh user@server "pm2 restart bill-tracker" ``` +**Deployment Notes**: +- Database migrations run automatically on startup (no manual intervention required) +- Migrations are idempotent: safe to `git pull` and restart repeatedly +- After `git pull`, migrations are applied in version order during startup +- Check `[migration]` logs for applied/skipped migration status + ### Security Considerations | Feature | Implementation | @@ -2406,6 +2567,65 @@ docker-compose exec app ls -lh /data/bills.db ## 10. Sequence Flows +### Database Initialization & Migration Flow + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Application Startup (server.js) │ +│ 1. require('./db/database.js') │ +│ 2. getDb() called │ +└─────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────┐ +│ db/database.js - getDb() │ +│ 3. Check if db already initialized (db variable) │ +│ 4. If not: call initSchema() │ +└─────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────┐ +│ db/database.js - initSchema() │ +│ 5. Read db/schema.sql and execute │ +│ 6. Create schema_migrations table if not exists │ +│ CREATE TABLE IF NOT EXISTS schema_migrations ( │ +│ id INTEGER PRIMARY KEY AUTOINCREMENT, │ +│ version TEXT NOT NULL UNIQUE, │ +│ description TEXT NOT NULL, │ +│ applied_at TEXT NOT NULL DEFAULT (datetime('now')) │ +│ ) │ +│ 7. Call runMigrations() │ +└─────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────┐ +│ db/database.js - runMigrations() │ +│ 8. Define all migrations with version/description/run │ +│ 9. For each migration: │ +│ a. Check hasMigrationBeenApplied(migration.version) │ +│ b. If NOT applied: │ +│ i. Log: "Applying {version}: {description}" │ +│ ii. Execute migration.run() │ +│ iii. Call recordMigration(version, description) │ +│ c. If already applied: │ +│ i. Log: "Skipping already applied {version}" │ +└─────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────┐ +│ Application continues │ +│ 10. Seed defaults (seedDefaults()) │ +│ 11. Server starts listening on port │ +└─────────────────────────────────────────────────────────────┘ +``` + +**Migration Flow Details**: + +1. **First startup after upgrade**: All pending migrations execute in version order +2. **Subsequent startups**: Skipped migrations logged but no SQL executed +3. **Idempotent**: Safe to run `git pull && npm start` repeatedly +4. **Audit trail**: Every applied migration recorded in `schema_migrations` + ### Login Flow (Local) ```