CSJ v2.3 Phase 1 Checkpoint

Updated: 2026-04-13

Completed so far

• Added v2.3 parser foundations in scripts/collector.py
• Added new constants:
  • HISTORY_DIR
  • EVENTS_FILE
  • PARSER_VERSION = "2.3"
• ensure_dirs() now creates csj_history/
• Added diff/hash helpers:
  • stable_json_dumps()
  • normalize_text_for_diff()
  • normalize_list_for_diff()
  • build_comparable_record()
  • hash_value()
  • compute_field_hashes()
  • compute_content_hash()
  • diff_job_records()
• Added field groups:
  • OPERATIONAL_FIELDS
  • CRITICAL_FIELDS
  • TEXT_FIELDS
• Updated normalize_job() to include:
  • parser_version
  • lifecycle_status
  • first_missing_at
  • consecutive_missing_runs
  • last_changed_at
  • field_hashes
  • content_hash
• Updated detail-fetch flow to load existing_record instead of only first_seen

Validated

1. Live limited run succeeded: python3 ~/.hermes/skills/research/civil-service-jobs-collector/scripts/collector.py --details -w 1 -n 1 --full --force
2. Output file now contains:
   • parser_version: "2.3"
   • lifecycle_status: "active"
   • populated field_hashes
   • populated content_hash
3. Cosmetic diff test passed:
   • whitespace-only text changes and reordered list values produced no meaningful diff

Not yet implemented

• history snapshot writes to csj_history/
• append-only event log writes to csj_events.jsonl
• lifecycle progression for missing-before-expiry jobs
• direct URL verification for suspected withdrawals
• supporting asset extraction
• fixture-based pytest suite

Recommended next step

• Implement history snapshot creation + event log append helpers, then wire them into record updates.