Update Dataset (PR → snapshot → steps → grapher)

Use this skill to run a complete dataset update with Claude Code subagents, keep a live progress checklist, and pause for approval at a checkpoint after every numbered workflow step before continuing.

Inputs

• <namespace>/<old_version>/<name>
• Get <new_version> as today's date by running date -u +"%Y-%m-%d"

Optional trailing args: - branch: The working branch name (defaults to current branch)

Assumptions: - All artifacts are written to workbench/<short_name>/. - Persist progress to workbench/<short_name>/progress.md and update it after each step.

Progress checklist (maintain, tick live, and persist to progress.md)

(Checkpoint rule: After you finish each item below that represents a workflow step, immediately run the CHECKPOINT procedure. Do not batch multiple steps before a checkpoint.) - [ ] Parse inputs and resolve: channel, namespace, version, short_name, old_version, branch - [ ] Clean workbench directory: delete workbench/<short_name> unless continuing existing update - [ ] Run ETL update workflow via etl-update subagent (help → dry run → approval → real run) - [ ] Create or reuse draft PR and work branch - [ ] Update snapshot and compare to previous version; capture summary - [ ] Meadow step: run + fix + diff + summarize - [ ] Garden step: run + fix + diff + summarize - [ ] Grapher step: run + verify (skip diffs), or explicitly mark N/A - [ ] CHECKPOINT — present consolidated summary and request approval - [ ] If approved, commit, push, and update PR description - [ ] Optional: run indicator upgrade on staging and persist report

Persistence: - After ticking each item, update workbench/<short_name>/progress.md with the current checklist state and a timestamp.

CHECKPOINT (mandatory user approval)

Always performed immediately after completing each numbered workflow step (1–6). Never start the next step until approval is granted.

Procedure (each time): 1. Present a concise summary of what just changed, key diffs/issues resolved, and what the next step will do. 2. Ask exactly: Proceed? reply: yes/no 3. Only continue if the user replies exactly yes (case-insensitive). Any other reply = no; stop and wait. 4. On approval: - Update progress checklist (tick the completed item) and write workbench/<short_name>/progress.md with timestamp. - Commit related changes (if any), push. - Update (or append to) the PR description: add a collapsed section titled with the step name (e.g., "Snapshot Update", "Meadow Update") containing the summary.

Mandatory per-step checkpoints (rule)

You MUST: - Stop after each workflow step (1–6) and run CHECKPOINT before starting the next (step 7 is optional and still requires a checkpoint if executed). - Never chain multiple steps inside a single approval. - Treat missing or ambiguous replies as no.

Workflow orchestration

0) Initial setup - Check if workbench/<short_name>/progress.md exists to determine if continuing existing update - If starting fresh: delete workbench/<short_name> directory if it exists - Create fresh workbench/<short_name> directory for artifacts

1) Run ETL update command (etl-update subagent) - Inputs: <namespace>/<old_version>/<short_name> plus any required flags - Perform help check, dry run, approval, then real execution; capture summary for later PR notes - CHECKPOINT (stop → summarize → ask → require yes) 2) Create PR and integrate update via subagent (etl-pr) - Inputs: <namespace>/<old_version>/<short_name> - Create or reuse draft PR, set up work branch, and incorporate the ETL update outputs - CHECKPOINT 3) Snapshot run & compare (snapshot-runner subagent) - Inputs: <namespace>/<new_version>/<short_name> and <old_version> - CHECKPOINT 4) Meadow step repair/verify (step-fixer subagent, channel=meadow) - Run, fix, re-run; produce diffs - Save diffs and summaries - CHECKPOINT 5) Garden step repair/verify (step-fixer subagent, channel=garden) - Run, fix, re-run; produce diffs - Save diffs and summaries - CHECKPOINT 6) Grapher step run/verify (step-fixer subagent, channel=grapher, add --grapher) - Skip diff - CHECKPOINT 7) Indicator upgrade (optional, staging only) - Use indicator-upgrader subagent with <short_name> <branch> - CHECKPOINT (if executed)

Guardrails and tips

• Never return empty tables or comment out logic as a workaround — fix the parsing/transformations instead.
• Column name changes: update garden processing code and metadata YAMLs (garden/grapher) to match schema changes.
• Indexing: avoid leaking index columns from reset_index(); format tables with tb.format(["country", "year"]) as appropriate.
• Metadata validation errors are guidance — update YAML to add/remove variables as indicated.

Artifacts (expected)

• workbench/<short_name>/snapshot-runner.md
• workbench/<short_name>/progress.md
• workbench/<short_name>/meadow_diff_raw.txt and meadow_diff.md
• workbench/<short_name>/garden_diff_raw.txt and garden_diff.md
• workbench/<short_name>/indicator_upgrade.json (if indicator-upgrader was used)

Example usage

• Minimal catalog URI with explicit old version:
• update-dataset data://snapshot/irena/2024-11-15/renewable_power_generation_costs 2023-11-15 update-irena-costs

Common issues when data structure changes

• SILENT FAILURES WARNING: Never return empty tables or comment code as workarounds!
• Column name changes: If columns are renamed/split (e.g., single cost → local currency + PPP), update:
• Python code references in the garden step
• Garden metadata YAML (e.g., food_prices_for_nutrition.meta.yml)
• Grapher metadata YAML (if it exists)
• Index issues: Check for unwanted index columns from reset_index() — ensure proper indexing with tb.format(["country", "year"]).
• Metadata validation: Use error messages as a guide — they show exactly which variables to add/remove from YAML files.