What a migration job does with its run: the manual update/3 behind MishkaCmsMigration.MigrationRun's :run_dry_run and :run_import, the actions AshOban's triggers of the same names run (MishkaCmsMigration.Workers.DryRunWorker, MishkaCmsMigration.Workers.ImportWorker). It walks the run's source as the admin who started the run and saves what came of it on the run.

The walk

The job connects once with the run's connection and its opened secret (MishkaCmsMigration.MigrationRun.open_secret/1, through MishkaCmsMigration.Source.connect/4) and reads the source's facts (extra/1), then walks the kinds in the order the source's kinds/0 gives them, and pages through each with list/3 until the source answers no cursor. Each row is normalized into an item (MishkaCmsMigration.Source.Item), decided (MishkaCmsMigration.Workers.Helpers.Decision) and settled (MishkaCmsMigration.Workers.Helpers.Writer): written in an import, reported in a dry run. After each page with rows the job saves how far it is on the run (MishkaCmsMigration.MigrationRun's :record_progress: the kind and the counts so far), and the run's own Ash notifier broadcasts it on "migration_run:<site_id>:updated".

An item that points at a CMS record an admin archived or deleted for good, such as a post at its tag, waits for the admin's decision on that record (MishkaCmsMigration.Workers.Helpers.Needs), skipped as "waits_for: <kinds>", once the record's own kind, walked first, met it.

An item that sits under another of its kind (MishkaCmsMigration.Source.Item's parent) waits until that parent is settled, on the same page or a later one, whichever the source lists first, and is settled right after it, so the parent's CMS record is there for the target to point at. Its children wait in turn, so a tree is settled from the top down. An item whose parent the kind never lists, or one in a loop of parents, is settled once the kind's last page is walked.

A parent an admin left out takes its children with it: an item whose parent was settled as "left_out_by_admin" or "parent_left_out" is skipped as "parent_left_out", in a dry run and in an import, however deep it sits and whatever its own plan was, unless the admin chose for the item itself (left it out, or mapped it to a CMS record) or its CMS record is archived or deleted for good and waits for the admin's decision, which what points at it waits for too. Its CMS record, if an earlier import wrote one, is kept. An item under one an admin mapped goes under the record it was mapped to.

A source id met again in the same kind, on the same page or a later one, is skipped as "duplicate_in_source" and not recorded: the first row with that id is the one written and remembered. A row the source cannot normalize fails alone, and the report says why, also when normalize/3 raises or exits: a plugin's mistake costs that item, not the run.

Every action the job runs on the run and its items carries the trigger's context (scope: context), so the policies' AshOban.Checks.AshObanInteraction bypass lets the job through where every actor is refused. A target writes as the admin alone, under the CMS's own policies.

Dry run and import

A dry run decides and reports; it writes nothing in the CMS and fires no event, and records what it would do with each item as the item's plan (MishkaCmsMigration.MigrationItem's :record_plan), which the Items page shows before anything is imported. Once it has walked every kind, it empties, in one update (:forget_plan), the plan of each item of the run's source host on its site that it did not meet (one whose plan it did not record since it started), so a first dry run and a Run again alike leave a plan only on the items the source lists now; a dry run that stops halfway leaves every item the plan it had. An import writes each item as the admin who pressed Import, and once every kind is walked its report and the source's facts are saved on the run and the report is handed to migration:import:finished (MishkaCmsMigration.Events).

Picked items

A job of picked items (MishkaCmsMigration.MigrationRun's "Picked items") is handed their source ids by kind (items) and the status its run goes back to (back_to). It walks only their kinds, in the source's order, so a picked category is settled before a picked post, and reads only their rows (MishkaCmsMigration.Source.rows/3, or the kind's pages from list/3 for a source without it), keeping the picked ones once normalized. An item the source no longer answers is left as it is. Each page is decided as a whole run's is, so an admin's choices and the run's options hold, after its needs are read (MishkaCmsMigration.Workers.Helpers.Needs): an item the walk would create or update while its parent, or what its record points at, has no CMS record yet is skipped with the reason, unless it is only its parent and an admin decided to import the item without it (MishkaCmsMigration.MigrationItem.without_parent?/1), and one under a parent an admin left out is skipped as "parent_left_out". It saves no progress, empties no plan, and once walked hands its report to MishkaCmsMigration.Report.merge/2 and the run to :finish_items; an import of picked items hands its own report to migration:import:finished. What fails a whole run fails a job of picked items only in part: a kind the source cannot answer, or that raises, fails its picked items alone, each recorded as failed with the reason and with what earlier runs kept of it (its title, slug, parent and CMS record), and the job goes on to the next kind; a source it cannot connect to fails every picked item so. The run then goes back where it was, as after any job of picked items. Every item it records is broadcast on "migration_item:<site_id>:updated" (MishkaCmsMigration.MigrationItem), which its actions' shared context (picked_items?) marks, so the Items page draws those rows again.

When the job stops

A run that is no longer waiting for its job — an admin cancelled it — cancels the job: at its start, when AshOban's worker finds the run out of the trigger's where; at the first page whose progress the run refuses for its status; and when it refuses the report for its status (AshOban.Errors.CancelJob, reason :not_running). Any other error saving the progress or the report fails the attempt, as an error of the walk does.

A source that refuses what the run connects with — the password (:unauthorized, WordPress's 401), a secret that can no longer be opened (:unreadable_secret), a file that is gone or not an export (:no_file, :unreadable_file) — fails the run at once through its :fail, as its on_error would, and the job ends without raising: trying again would meet the same refusal and hold the site for nothing. The run reads why in words (MishkaCmsMigration.MigrationRun.reason_label/1, its expanded row, the bell). Any other error from connect/2 or list/3, a raise outside a single item, or a source whose cursors never end — {:cursor_repeated, kind} when list/3 answers a cursor it already answered for that kind, {:too_many_pages, kind} after 10,000 pages of one kind — fails the job with its reason in words (MishkaCmsMigration.Report.reason/1). AshOban tries it again, and after the last attempt runs the trigger's on_error, the run's :fail, which keeps the reason; the run keeps its secret, so it can run again. A last attempt that never returns — Oban stops it at its timeout, or it crashes past every rescue — fails its run through Oban's own telemetry (MishkaCmsMigration.Workers.Helpers.Discarded).

Each attempt is a [:mishka_cms_migration, :run] telemetry span, and the walk tells telemetry of each kind the source has listed and each item it settles (see "Telemetry" in MishkaCmsMigration.Workers).