CSJ v2.3 Phase 3 Checkpoint

Updated: 2026-04-13

Completed in this phase

• Added lifecycle threshold constant:
  • MISSING_CONFIRMATION_RUNS = 3
• Added direct URL verification helper:
  • verify_missing_job_url(job_record, collector=None)
• Reworked lifecycle pass so full unfiltered runs now classify absent jobs into:
  • closed
  • missing_unconfirmed
  • withdrawn_confirmed (verification path now implemented, though not yet observed live)
• Added lifecycle event/history writes for:
  • closed
  • missing_from_results
  • withdrawn_confirmed
  • reopened
• Updated dedup/skip update path so existing active jobs now also get:
  • lifecycle_status = active
  • first_missing_at = null
  • consecutive_missing_runs = 0

Important implementation detail

• Legacy records that were skipped in dedup originally only had status=active, not lifecycle_status=active.
• This caused a migration-style issue where many current files still showed null lifecycle state.
• Fixed by updating the dedup refresh path to backfill/reset lifecycle fields for active jobs still present in search results.

Live validation completed Command run: python3 ~/.hermes/skills/research/civil-service-jobs-collector/scripts/collector.py --details -w 5 --full --refresh

Observed live behavior after fixes

• Current lifecycle counts on disk:
  • active: 1245
  • closed: 105
  • missing_unconfirmed: 7
• This means the archive now distinguishes jobs that:
  • are still live in current search results
  • disappeared after expiry
  • disappeared before confirmed closure/withdrawal

Notes on the 7 missing_unconfirmed jobs

• Some have future closes_iso
• Some have no parseable closes_iso
• They are now being preserved as nuanced lifecycle states instead of being silently left plain active forever or falsely marked closed

Not yet fully validated live

• withdrawn_confirmed via repeated missing runs/direct URL verification threshold
• reopened event from a role reappearing in search results after prior missing state

Recommended next steps

1. Add supporting asset extraction (supporting_links, attachments, embeds)
2. Add fixture-based tests for lifecycle transitions and URL verification
3. Consider whether repeated missing_from_results runs should always emit a new history snapshot, or only on first missing / status transition