test(e2e): census gets its own project, ordered before the mutating probes
The census baselines were captured after api.probe had already created its adversarial bills (Zero/Big/<script>) in the scratch DB, so snapshot content depended on spec ordering — the first census diffs were probe artifacts, not product changes. Now: a dedicated 'census' project (read-only, pristine seed data) runs via project dependencies before 'probe' (api.probe/webauthn mutate state), and the default UI projects ignore the spec. Re-baselined from a prepare-db-fresh DB; probe chain 33/33 across three consecutive runs; default e2e 27 passed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
e7974cf4c0
commit
64ab37ae39
|
|
@ -6,7 +6,7 @@ button | Print / Save PDF
|
||||||
button | Refresh
|
button | Refresh
|
||||||
button | Reset filters
|
button | Reset filters
|
||||||
button | Search billsCtrl K
|
button | Search billsCtrl K
|
||||||
button | Tracker#
|
button | Tracker
|
||||||
input:checkbox | (unnamed)
|
input:checkbox | (unnamed)
|
||||||
input:number | (unnamed)
|
input:number | (unnamed)
|
||||||
link | About
|
link | About
|
||||||
|
|
@ -17,7 +17,7 @@ link | Release Notes
|
||||||
link | Skip to main content
|
link | Skip to main content
|
||||||
select | # months# months# months
|
select | # months# months# months
|
||||||
select | # months# months# months# months
|
select | # months# months# months# months
|
||||||
select | All bills<script>alert(#)</script>Adobe Creative CloudAmazon
|
select | All billsAdobe Creative CloudAmazon PrimeAMC A-ListApple iCl
|
||||||
select | All categoriesBeautyCoffeeCredit CardsDining OutEntertainmen
|
select | All categoriesBeautyCoffeeCredit CardsDining OutEntertainmen
|
||||||
select | JanuaryFebruaryMarchAprilMayJuneJulyAugustSeptemberOctoberNo
|
select | JanuaryFebruaryMarchAprilMayJuneJulyAugustSeptemberOctoberNo
|
||||||
select | LineArea
|
select | LineArea
|
||||||
|
|
@ -2,7 +2,7 @@ button | Current theme: Dark. Click to change.
|
||||||
button | Open Tanstack query devtools
|
button | Open Tanstack query devtools
|
||||||
button | Open user menu
|
button | Open user menu
|
||||||
button | Search billsCtrl K
|
button | Search billsCtrl K
|
||||||
button | Tracker#
|
button | Tracker
|
||||||
link | About
|
link | About
|
||||||
link | Analytics
|
link | Analytics
|
||||||
link | Bill Tracker
|
link | Bill Tracker
|
||||||
|
|
@ -1,13 +1,11 @@
|
||||||
button | #st bucket
|
button | #st bucket
|
||||||
button | #th bucket
|
button | #th bucket
|
||||||
button | <script>alert(#)</script>
|
|
||||||
button | AMC A-List
|
button | AMC A-List
|
||||||
button | Add Bill
|
button | Add Bill
|
||||||
button | Adobe Creative Cloud
|
button | Adobe Creative Cloud
|
||||||
button | Amazon Prime
|
button | Amazon Prime
|
||||||
button | Apple iCloud+
|
button | Apple iCloud+
|
||||||
button | Autopay
|
button | Autopay
|
||||||
button | Big
|
|
||||||
button | Cadence
|
button | Cadence
|
||||||
button | Capital One Quicksilver
|
button | Capital One Quicksilver
|
||||||
button | Car Insurance
|
button | Car Insurance
|
||||||
|
|
@ -33,8 +31,6 @@ button | Homeowners Insurance
|
||||||
button | Inactive
|
button | Inactive
|
||||||
button | Internet Provider
|
button | Internet Provider
|
||||||
button | Mortgage
|
button | Mortgage
|
||||||
button | Move <script>alert(#)</script> down
|
|
||||||
button | Move <script>alert(#)</script> up
|
|
||||||
button | Move AMC A-List down
|
button | Move AMC A-List down
|
||||||
button | Move AMC A-List up
|
button | Move AMC A-List up
|
||||||
button | Move Adobe Creative Cloud down
|
button | Move Adobe Creative Cloud down
|
||||||
|
|
@ -43,8 +39,6 @@ button | Move Amazon Prime down
|
||||||
button | Move Amazon Prime up
|
button | Move Amazon Prime up
|
||||||
button | Move Apple iCloud+ down
|
button | Move Apple iCloud+ down
|
||||||
button | Move Apple iCloud+ up
|
button | Move Apple iCloud+ up
|
||||||
button | Move Big down
|
|
||||||
button | Move Big up
|
|
||||||
button | Move Capital One Quicksilver down
|
button | Move Capital One Quicksilver down
|
||||||
button | Move Capital One Quicksilver up
|
button | Move Capital One Quicksilver up
|
||||||
button | Move Car Insurance down
|
button | Move Car Insurance down
|
||||||
|
|
@ -83,8 +77,6 @@ button | Move Student Loan down
|
||||||
button | Move Student Loan up
|
button | Move Student Loan up
|
||||||
button | Move Trash Service down
|
button | Move Trash Service down
|
||||||
button | Move Trash Service up
|
button | Move Trash Service up
|
||||||
button | Move Zero down
|
|
||||||
button | Move Zero up
|
|
||||||
button | Netflix
|
button | Netflix
|
||||||
button | Open Tanstack query devtools
|
button | Open Tanstack query devtools
|
||||||
button | Open user menu
|
button | Open user menu
|
||||||
|
|
@ -92,9 +84,8 @@ button | Search & filters# of # shown · no filters
|
||||||
button | Search billsCtrl K
|
button | Search billsCtrl K
|
||||||
button | Spotify
|
button | Spotify
|
||||||
button | Student Loan
|
button | Student Loan
|
||||||
button | Tracker#
|
button | Tracker
|
||||||
button | Trash Service
|
button | Trash Service
|
||||||
button | Zero
|
|
||||||
combobox | Filter by billing schedule
|
combobox | Filter by billing schedule
|
||||||
combobox | Filter by category
|
combobox | Filter by category
|
||||||
combobox | Use a bill template
|
combobox | Use a bill template
|
||||||
|
|
@ -7,7 +7,7 @@ button | Refresh calendar
|
||||||
button | Search billsCtrl K
|
button | Search billsCtrl K
|
||||||
button | Subscribe Calendar
|
button | Subscribe Calendar
|
||||||
button | Today
|
button | Today
|
||||||
button | Tracker#
|
button | Tracker
|
||||||
button | View #/#/#
|
button | View #/#/#
|
||||||
link | About
|
link | About
|
||||||
link | Analytics
|
link | Analytics
|
||||||
|
|
@ -52,7 +52,7 @@ button | Rename Subscriptions
|
||||||
button | Rename Utilities
|
button | Rename Utilities
|
||||||
button | Search billsCtrl K
|
button | Search billsCtrl K
|
||||||
button | Show Empty
|
button | Show Empty
|
||||||
button | Tracker#
|
button | Tracker
|
||||||
input:text | New category name...
|
input:text | New category name...
|
||||||
link | About
|
link | About
|
||||||
link | Analytics
|
link | Analytics
|
||||||
|
|
@ -6,7 +6,7 @@ button | ImportBring in existing data
|
||||||
button | Open Tanstack query devtools
|
button | Open Tanstack query devtools
|
||||||
button | Open user menu
|
button | Open user menu
|
||||||
button | Search billsCtrl K
|
button | Search billsCtrl K
|
||||||
button | Tracker#
|
button | Tracker
|
||||||
button | TransactionsReview & match
|
button | TransactionsReview & match
|
||||||
link | About
|
link | About
|
||||||
link | Analytics
|
link | Analytics
|
||||||
|
|
@ -5,7 +5,7 @@ button | Open Tanstack query devtools
|
||||||
button | Open user menu
|
button | Open user menu
|
||||||
button | Refresh
|
button | Refresh
|
||||||
button | Search billsCtrl K
|
button | Search billsCtrl K
|
||||||
button | Tracker#
|
button | Tracker
|
||||||
link | About
|
link | About
|
||||||
link | Analytics
|
link | Analytics
|
||||||
link | Bill Tracker
|
link | Bill Tracker
|
||||||
|
|
@ -7,7 +7,7 @@ button | Print
|
||||||
button | Required Minimum: help
|
button | Required Minimum: help
|
||||||
button | Search billsCtrl K
|
button | Search billsCtrl K
|
||||||
button | Time Savings: help
|
button | Time Savings: help
|
||||||
button | Tracker#
|
button | Tracker
|
||||||
button | ▲
|
button | ▲
|
||||||
button | ▼
|
button | ▼
|
||||||
combobox | Capital One Quicksilver$#
|
combobox | Capital One Quicksilver$#
|
||||||
|
|
@ -1,14 +1,14 @@
|
||||||
button | Add key
|
button | Add key
|
||||||
button | Change Password
|
button | Change Password
|
||||||
button | Current theme: Dark. Click to change.
|
button | Current theme: Dark. Click to change.
|
||||||
button | Last LoginJul # # #:# PMDesktop · Chrome on Linux#
|
button | Last LoginJul # # #:# AMDesktop · Chrome on Linux#
|
||||||
button | Open Tanstack query devtools
|
button | Open Tanstack query devtools
|
||||||
button | Open user menu
|
button | Open user menu
|
||||||
button | Save Privacy Settings
|
button | Save Privacy Settings
|
||||||
button | Save Profile
|
button | Save Profile
|
||||||
button | Search billsCtrl K
|
button | Search billsCtrl K
|
||||||
button | Set up
|
button | Set up
|
||||||
button | Tracker#
|
button | Tracker
|
||||||
input:password | (unnamed)
|
input:password | (unnamed)
|
||||||
input:text | Display name
|
input:text | Display name
|
||||||
link | About
|
link | About
|
||||||
|
|
@ -15,7 +15,7 @@ button | Search billsCtrl K
|
||||||
button | Send test
|
button | Send test
|
||||||
button | System
|
button | System
|
||||||
button | Theme: help
|
button | Theme: help
|
||||||
button | Tracker#
|
button | Tracker
|
||||||
combobox | MM/DD/YYYY
|
combobox | MM/DD/YYYY
|
||||||
combobox | Sunday
|
combobox | Sunday
|
||||||
combobox | Tracker
|
combobox | Tracker
|
||||||
|
|
@ -26,7 +26,7 @@ button | Restore Ramsey Order
|
||||||
button | Save Order
|
button | Save Order
|
||||||
button | Search billsCtrl K
|
button | Search billsCtrl K
|
||||||
button | Toggle plan details
|
button | Toggle plan details
|
||||||
button | Tracker#
|
button | Tracker
|
||||||
input:number | #
|
input:number | #
|
||||||
link | About
|
link | About
|
||||||
link | Analytics
|
link | Analytics
|
||||||
|
|
@ -17,7 +17,7 @@ button | Previous month
|
||||||
button | Search billsCtrl K
|
button | Search billsCtrl K
|
||||||
button | Shopping
|
button | Shopping
|
||||||
button | Shopping$# transactionavg $#/mo$# left
|
button | Shopping$# transactionavg $#/mo$# left
|
||||||
button | Tracker#
|
button | Tracker
|
||||||
button | Uncategorized
|
button | Uncategorized
|
||||||
button | Uncategorized$# transaction
|
button | Uncategorized$# transaction
|
||||||
button | View
|
button | View
|
||||||
|
|
@ -40,7 +40,7 @@ button | Search subscriptions# of # shown · no filters
|
||||||
button | Search transactionsall bank transactions · no filters
|
button | Search transactionsall bank transactions · no filters
|
||||||
button | Spotify
|
button | Spotify
|
||||||
button | Track Subscription
|
button | Track Subscription
|
||||||
button | Tracker#
|
button | Tracker
|
||||||
button | View recommendation details
|
button | View recommendation details
|
||||||
input:search | Search by merchant, description, or payee…
|
input:search | Search by merchant, description, or payee…
|
||||||
input:search | Search recommendations…
|
input:search | Search recommendations…
|
||||||
|
|
@ -1,15 +1,11 @@
|
||||||
button | Current theme: Dark. Click to change.
|
button | Current theme: Dark. Click to change.
|
||||||
button | Edit
|
button | Edit
|
||||||
button | Move <script>alert(#)</script> down
|
|
||||||
button | Move <script>alert(#)</script> up
|
|
||||||
button | Move AMC A-List down
|
button | Move AMC A-List down
|
||||||
button | Move AMC A-List up
|
button | Move AMC A-List up
|
||||||
button | Move Adobe Creative Cloud down
|
button | Move Adobe Creative Cloud down
|
||||||
button | Move Adobe Creative Cloud up
|
button | Move Adobe Creative Cloud up
|
||||||
button | Move Apple iCloud+ down
|
button | Move Apple iCloud+ down
|
||||||
button | Move Apple iCloud+ up
|
button | Move Apple iCloud+ up
|
||||||
button | Move Big down
|
|
||||||
button | Move Big up
|
|
||||||
button | Move Capital One Quicksilver down
|
button | Move Capital One Quicksilver down
|
||||||
button | Move Capital One Quicksilver up
|
button | Move Capital One Quicksilver up
|
||||||
button | Move Car Insurance down
|
button | Move Car Insurance down
|
||||||
|
|
@ -46,8 +42,6 @@ button | Move Student Loan down
|
||||||
button | Move Student Loan up
|
button | Move Student Loan up
|
||||||
button | Move Trash Service down
|
button | Move Trash Service down
|
||||||
button | Move Trash Service up
|
button | Move Trash Service up
|
||||||
button | Move Zero down
|
|
||||||
button | Move Zero up
|
|
||||||
button | Next month
|
button | Next month
|
||||||
button | Open Tanstack query devtools
|
button | Open Tanstack query devtools
|
||||||
button | Open user menu
|
button | Open user menu
|
||||||
|
|
@ -55,7 +49,7 @@ button | Previous month
|
||||||
button | Print / PDF
|
button | Print / PDF
|
||||||
button | Search billsCtrl K
|
button | Search billsCtrl K
|
||||||
button | Today
|
button | Today
|
||||||
button | Tracker#
|
button | Tracker
|
||||||
link | About
|
link | About
|
||||||
link | Analytics
|
link | Analytics
|
||||||
link | Bill Tracker
|
link | Bill Tracker
|
||||||
|
|
@ -12,11 +12,9 @@ button | Current theme: Dark. Click to change.
|
||||||
button | Debt
|
button | Debt
|
||||||
button | Due
|
button | Due
|
||||||
button | Due Soon, click to mark paid
|
button | Due Soon, click to mark paid
|
||||||
button | Edit <script>alert(#)</script>
|
|
||||||
button | Edit AMC A-List
|
button | Edit AMC A-List
|
||||||
button | Edit Adobe Creative Cloud
|
button | Edit Adobe Creative Cloud
|
||||||
button | Edit Apple iCloud+
|
button | Edit Apple iCloud+
|
||||||
button | Edit Big
|
|
||||||
button | Edit Capital One Quicksilver
|
button | Edit Capital One Quicksilver
|
||||||
button | Edit Car Insurance
|
button | Edit Car Insurance
|
||||||
button | Edit Car Payment
|
button | Edit Car Payment
|
||||||
|
|
@ -35,25 +33,18 @@ button | Edit Netflix
|
||||||
button | Edit Spotify
|
button | Edit Spotify
|
||||||
button | Edit Student Loan
|
button | Edit Student Loan
|
||||||
button | Edit Trash Service
|
button | Edit Trash Service
|
||||||
button | Edit Zero
|
|
||||||
button | Edit monthly starting amounts
|
button | Edit monthly starting amounts
|
||||||
button | Expected
|
button | Expected
|
||||||
button | Export or print this month
|
button | Export or print this month
|
||||||
button | Hide
|
button | Hide
|
||||||
button | Hide overdue panel
|
|
||||||
button | Keyboard shortcuts
|
button | Keyboard shortcuts
|
||||||
button | Last Month
|
button | Last Month
|
||||||
button | Late, click to mark paid
|
|
||||||
button | Move <script>alert(#)</script> down
|
|
||||||
button | Move <script>alert(#)</script> up
|
|
||||||
button | Move AMC A-List down
|
button | Move AMC A-List down
|
||||||
button | Move AMC A-List up
|
button | Move AMC A-List up
|
||||||
button | Move Adobe Creative Cloud down
|
button | Move Adobe Creative Cloud down
|
||||||
button | Move Adobe Creative Cloud up
|
button | Move Adobe Creative Cloud up
|
||||||
button | Move Apple iCloud+ down
|
button | Move Apple iCloud+ down
|
||||||
button | Move Apple iCloud+ up
|
button | Move Apple iCloud+ up
|
||||||
button | Move Big down
|
|
||||||
button | Move Big up
|
|
||||||
button | Move Capital One Quicksilver down
|
button | Move Capital One Quicksilver down
|
||||||
button | Move Capital One Quicksilver up
|
button | Move Capital One Quicksilver up
|
||||||
button | Move Car Insurance down
|
button | Move Car Insurance down
|
||||||
|
|
@ -90,8 +81,6 @@ button | Move Student Loan down
|
||||||
button | Move Student Loan up
|
button | Move Student Loan up
|
||||||
button | Move Trash Service down
|
button | Move Trash Service down
|
||||||
button | Move Trash Service up
|
button | Move Trash Service up
|
||||||
button | Move Zero down
|
|
||||||
button | Move Zero up
|
|
||||||
button | Next month
|
button | Next month
|
||||||
button | Open Tanstack query devtools
|
button | Open Tanstack query devtools
|
||||||
button | Open calendar for this month
|
button | Open calendar for this month
|
||||||
|
|
@ -101,25 +90,19 @@ button | Paid
|
||||||
button | Paid Date
|
button | Paid Date
|
||||||
button | Paid, click to mark unpaid
|
button | Paid, click to mark unpaid
|
||||||
button | Pay all due (#)
|
button | Pay all due (#)
|
||||||
button | Pay now
|
|
||||||
button | Pin Due
|
button | Pin Due
|
||||||
button | Previous month
|
button | Previous month
|
||||||
button | Review overdue bills# items ready for pay, skip, or snooze$#
|
|
||||||
button | Search & sort# of # shown · no filters
|
button | Search & sort# of # shown · no filters
|
||||||
button | Search billsCtrl K
|
button | Search billsCtrl K
|
||||||
button | Skip
|
|
||||||
button | Snooze
|
|
||||||
button | Status
|
button | Status
|
||||||
button | Today
|
button | Today
|
||||||
button | Tracker#
|
button | Tracker
|
||||||
button | Unpaid
|
button | Unpaid
|
||||||
button | Upcoming, click to mark paid
|
button | Upcoming, click to mark paid
|
||||||
button | —
|
button | —
|
||||||
checkbox | Select <script>alert(#)</script>
|
|
||||||
checkbox | Select AMC A-List
|
checkbox | Select AMC A-List
|
||||||
checkbox | Select Adobe Creative Cloud
|
checkbox | Select Adobe Creative Cloud
|
||||||
checkbox | Select Apple iCloud+
|
checkbox | Select Apple iCloud+
|
||||||
checkbox | Select Big
|
|
||||||
checkbox | Select Capital One Quicksilver
|
checkbox | Select Capital One Quicksilver
|
||||||
checkbox | Select Car Insurance
|
checkbox | Select Car Insurance
|
||||||
checkbox | Select Car Payment
|
checkbox | Select Car Payment
|
||||||
|
|
@ -138,13 +121,10 @@ checkbox | Select Netflix
|
||||||
checkbox | Select Spotify
|
checkbox | Select Spotify
|
||||||
checkbox | Select Student Loan
|
checkbox | Select Student Loan
|
||||||
checkbox | Select Trash Service
|
checkbox | Select Trash Service
|
||||||
checkbox | Select Zero
|
|
||||||
combobox | Filter by billing schedule
|
combobox | Filter by billing schedule
|
||||||
combobox | Filter by category
|
combobox | Filter by category
|
||||||
combobox | Sort by
|
combobox | Sort by
|
||||||
input:number | <script>alert(#)</script> payment amount
|
|
||||||
input:number | Apple iCloud+ payment amount
|
input:number | Apple iCloud+ payment amount
|
||||||
input:number | Big payment amount
|
|
||||||
input:number | Capital One Quicksilver payment amount
|
input:number | Capital One Quicksilver payment amount
|
||||||
input:number | Car Payment payment amount
|
input:number | Car Payment payment amount
|
||||||
input:number | Cell Phone payment amount
|
input:number | Cell Phone payment amount
|
||||||
|
|
@ -51,6 +51,8 @@ module.exports = defineConfig({
|
||||||
// probe-only: enables WebAuthn for the seeded user mid-test, which would
|
// probe-only: enables WebAuthn for the seeded user mid-test, which would
|
||||||
// race the password logins in these parallel UI projects
|
// race the password logins in these parallel UI projects
|
||||||
/webauthn\.probe\.spec\.js/,
|
/webauthn\.probe\.spec\.js/,
|
||||||
|
// census-project-only: snapshots depend on pristine seed data
|
||||||
|
/control-census\.spec\.js/,
|
||||||
],
|
],
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
|
|
@ -64,6 +66,8 @@ module.exports = defineConfig({
|
||||||
// probe-only: enables WebAuthn for the seeded user mid-test, which would
|
// probe-only: enables WebAuthn for the seeded user mid-test, which would
|
||||||
// race the password logins in these parallel UI projects
|
// race the password logins in these parallel UI projects
|
||||||
/webauthn\.probe\.spec\.js/,
|
/webauthn\.probe\.spec\.js/,
|
||||||
|
// census-project-only: snapshots depend on pristine seed data
|
||||||
|
/control-census\.spec\.js/,
|
||||||
],
|
],
|
||||||
},
|
},
|
||||||
// Find-mode diagnostics (adversarial API probe + authenticated a11y scan).
|
// Find-mode diagnostics (adversarial API probe + authenticated a11y scan).
|
||||||
|
|
@ -71,12 +75,23 @@ module.exports = defineConfig({
|
||||||
// so open findings here don't turn the default suite red. Run on demand:
|
// so open findings here don't turn the default suite red. Run on demand:
|
||||||
// `npm run test:e2e:probe`. Fold a spec into the default projects once its
|
// `npm run test:e2e:probe`. Fold a spec into the default projects once its
|
||||||
// findings are fixed (it becomes a passing regression guard).
|
// findings are fixed (it becomes a passing regression guard).
|
||||||
|
// Control census (Appendix E automation): read-only snapshots of every
|
||||||
|
// page's interactive controls. Own project ordered BEFORE the probe so the
|
||||||
|
// snapshots always see pristine seed data — api.probe/webauthn mutate the
|
||||||
|
// scratch DB (adversarial bills, passkey enrollment) and would otherwise
|
||||||
|
// make the baselines depend on spec ordering.
|
||||||
{
|
{
|
||||||
name: 'probe',
|
name: 'census',
|
||||||
testMatch: /(api\.probe|a11y\.authed|webauthn\.probe|control-census)\.spec\.js/,
|
testMatch: /control-census\.spec\.js/,
|
||||||
use: { ...devices['Desktop Chrome'] },
|
use: { ...devices['Desktop Chrome'] },
|
||||||
dependencies: ['setup'],
|
dependencies: ['setup'],
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
name: 'probe',
|
||||||
|
testMatch: /(api\.probe|a11y\.authed|webauthn\.probe)\.spec\.js/,
|
||||||
|
use: { ...devices['Desktop Chrome'] },
|
||||||
|
dependencies: ['setup', 'census'],
|
||||||
|
},
|
||||||
// Cross-browser (WebAuthn/Select behavior differs) — enable when ready:
|
// Cross-browser (WebAuthn/Select behavior differs) — enable when ready:
|
||||||
// { name: 'firefox', use: { ...devices['Desktop Firefox'] }, dependencies: ['setup'], testIgnore: /auth\.setup\.js/ },
|
// { name: 'firefox', use: { ...devices['Desktop Firefox'] }, dependencies: ['setup'], testIgnore: /auth\.setup\.js/ },
|
||||||
// { name: 'webkit', use: { ...devices['Desktop Safari'] }, dependencies: ['setup'], testIgnore: /auth\.setup\.js/ },
|
// { name: 'webkit', use: { ...devices['Desktop Safari'] }, dependencies: ['setup'], testIgnore: /auth\.setup\.js/ },
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue