# `MishkaCmsMigration.Workers.Helpers.Pipeline`
[🔗](https://github.com/mishka-group/mishka_cms_migration/blob/v0.0.1/lib/mishka_cms_migration/workers/__helpers/pipeline.ex#L5)

What a `migration` job does with its run: the manual `update/3` behind
`MishkaCmsMigration.MigrationRun`'s `:run_dry_run` and `:run_import`, the actions AshOban's
triggers of the same names run (`MishkaCmsMigration.Workers.DryRunWorker`,
`MishkaCmsMigration.Workers.ImportWorker`). It walks the run's source as the admin who started
the run and saves what came of it on the run.

## The walk

The job connects once with the run's connection and its opened secret
(`MishkaCmsMigration.MigrationRun.open_secret/1`, through `MishkaCmsMigration.Source.connect/4`)
and reads the source's facts (`extra/1`), then
walks the kinds in the order the source's `kinds/0` gives them, and pages through each with
`list/3` until the source answers no cursor. Each row is normalized into an item
(`MishkaCmsMigration.Source.Item`), decided (`MishkaCmsMigration.Workers.Helpers.Decision`) and
settled (`MishkaCmsMigration.Workers.Helpers.Writer`): written in an import, reported in a dry
run. After each page with rows the job saves how far it is on the run
(`MishkaCmsMigration.MigrationRun`'s `:record_progress`: the kind and the counts so far), and the
run's own Ash notifier broadcasts it on `"migration_run:<site_id>:updated"`.

An item that points at a CMS record an admin archived or deleted for good, such as a post at its
tag, waits for the admin's decision on that record (`MishkaCmsMigration.Workers.Helpers.Needs`),
skipped as `"waits_for: <kinds>"`, once the record's own kind, walked first, met it.

An item that sits under another of its kind (`MishkaCmsMigration.Source.Item`'s `parent`) waits
until that parent is settled, on the same page or a later one, whichever the source lists first,
and is settled right after it, so the parent's CMS record is there for the target to point at.
Its children wait in turn, so a tree is settled from the top down. An item whose parent the kind
never lists, or one in a loop of parents, is settled once the kind's last page is walked.

A parent an admin left out takes its children with it: an item whose parent was settled as
`"left_out_by_admin"` or `"parent_left_out"` is skipped as `"parent_left_out"`, in a dry run and
in an import, however deep it sits and whatever its own plan was, unless the admin chose for the
item itself (left it out, or mapped it to a CMS record) or its CMS record is archived or deleted
for good and waits for the admin's decision, which what points at it waits for too. Its CMS
record, if an earlier import wrote one, is kept. An item under one an admin mapped goes under the
record it was mapped to.

A source id met again in the same kind, on the same page or a later one, is skipped as
`"duplicate_in_source"` and not recorded: the first row with that id is the one written and
remembered. A row the source cannot normalize fails alone, and the report says why, also when
`normalize/3` raises or exits: a plugin's mistake costs that item, not the run.

Every action the job runs on the run and its items carries the trigger's context
(`scope: context`), so the policies' `AshOban.Checks.AshObanInteraction` bypass lets the job
through where every actor is refused. A target writes as the admin alone, under the CMS's own
policies.

## Dry run and import

A dry run decides and reports; it writes nothing in the CMS and fires no event, and records what
it would do with each item as the item's plan (`MishkaCmsMigration.MigrationItem`'s
`:record_plan`), which the Items page shows before anything is imported. Once it has walked every
kind, it empties, in one update (`:forget_plan`), the plan of each item of the run's source host
on its site that it did not meet (one whose plan it did not record since it started), so a first
dry run and a Run again alike leave a plan only on the items the source lists now; a dry run that
stops halfway leaves every item the plan it had. An import writes each item as the admin who
pressed Import, and once every kind is walked its report and the source's
facts are saved on the run and the report is handed to `migration:import:finished`
(`MishkaCmsMigration.Events`).

## Picked items

A job of picked items (`MishkaCmsMigration.MigrationRun`'s "Picked items") is handed their source
ids by kind (`items`) and the status its run goes back to (`back_to`). It walks only their kinds,
in the source's order, so a picked category is settled before a picked post, and reads only
their rows (`c:MishkaCmsMigration.Source.rows/3`, or the kind's pages from `list/3` for a source
without it), keeping the picked ones once normalized. An item the source no longer answers is
left as it is. Each page is decided as a whole run's is, so an admin's choices and the run's
options hold, after its needs are read (`MishkaCmsMigration.Workers.Helpers.Needs`): an item the
walk would create or update while its parent, or what its record points at, has no CMS record
yet is skipped with the reason, unless it is only its parent and an admin decided to import the
item without it (`MishkaCmsMigration.MigrationItem.without_parent?/1`), and one under a parent an
admin left out is skipped as `"parent_left_out"`. It saves no progress, empties no plan, and once walked hands its report to
`MishkaCmsMigration.Report.merge/2` and the run to `:finish_items`; an import of picked items
hands its own report to `migration:import:finished`. What fails a whole run fails a job of picked
items only in part: a kind the source cannot answer, or that raises, fails its picked items
alone, each recorded as failed with the reason and with what earlier runs kept of it (its title,
slug, parent and CMS record), and the job goes on to the next kind; a source it cannot connect to
fails every picked item so. The run then goes back where it was, as after any job of picked
items. Every item it records is broadcast on
`"migration_item:<site_id>:updated"` (`MishkaCmsMigration.MigrationItem`), which its actions'
shared context (`picked_items?`) marks, so the Items page draws those rows again.

## When the job stops

A run that is no longer waiting for its job — an admin cancelled it — cancels the job: at its
start, when AshOban's worker finds the run out of the trigger's `where`; at the first page whose
progress the run refuses for its status; and when it refuses the report for its status
(`AshOban.Errors.CancelJob`, reason `:not_running`). Any other error saving the progress or the
report fails the attempt, as an error of the walk does.

A source that refuses what the run connects with — the password (`:unauthorized`, WordPress's
401), a secret that can no longer be opened (`:unreadable_secret`), a file that is gone or not an
export (`:no_file`, `:unreadable_file`) — fails the run at once through its `:fail`, as its
`on_error` would, and the job ends without raising: trying again would meet the same refusal and
hold the site for nothing. The run reads why in words
(`MishkaCmsMigration.MigrationRun.reason_label/1`, its expanded row, the bell). Any other error
from `connect/2` or `list/3`, a raise outside a single item, or a source whose cursors never end
— `{:cursor_repeated, kind}` when `list/3` answers a cursor it already answered for that kind,
`{:too_many_pages, kind}` after 10,000 pages of one kind — fails the job with its reason in words (`MishkaCmsMigration.Report.reason/1`). AshOban
tries it again, and after the last attempt runs the trigger's `on_error`, the run's `:fail`,
which keeps the reason; the run keeps its secret, so it can run again. A last attempt that never returns — Oban stops it at
its timeout, or it crashes past every rescue — fails its run through Oban's own telemetry
(`MishkaCmsMigration.Workers.Helpers.Discarded`).

Each attempt is a `[:mishka_cms_migration, :run]` telemetry span, and the walk tells telemetry of
each kind the source has listed and each item it settles (see "Telemetry" in
`MishkaCmsMigration.Workers`).

# `mode`

```elixir
@type mode() :: :dry_run | :import
```

Whether a job only reports, or writes.

# `update`

Walks the run of `changeset` in `opts[:mode]` as `context.actor` and answers the run as its
finish left it, an `AshOban.Errors.CancelJob` when the run stopped waiting, or the reason the
walk could not go on.

---

*Consult [api-reference.md](api-reference.md) for complete listing*