upload current sources

docs/predecessor-flow.md

@@ -0,0 +1,172 @@
+# Predecessor flow — how the Express implementation actually works
+
+Extracted 2026-09-10 by a full read of `~/projects/sap-apps/arena-plm`
+(reference-only). File:line cites refer to that repo, mostly
+`backend/src/arena-plm.js`. This record informs the rebuild's design; the
+durable behavioral tombstones harvested from it live in `KNOWLEDGE.md`, and
+the requirements it produced are in `PRD.md` (ARN-010, sharpened ARN-002/004/006).
+
+**Shape of the system:** not event-driven — an **operator-driven staging
+workbench**. Arena is the read-only source, SAP is the write target *and* the
+durable ledger. The webhook and poller exist but are effectively inert (§1).
+Real syncs are triggered by a human clicking Approve in the UI5 worklist.
+
+## 1. Ingestion — three paths, only one live
+
+**Webhook (`POST /api/arena/webhook`, routes/arena-webhook.js)** — mounted
+before the XSUAA gate, Basic auth (timing-safe), raw-text body of any
+content-type (delivery shape never pinned down), always answers 200 so Arena
+never retries. `WEBHOOK_CAPTURE_ONLY` logs+acks; otherwise
+`pollOutboundIntegration({dryRun})` with dryRun defaulting true. **The
+consumer never syncs**: `arena-plm.js:1252-1256` is a literal TODO — non-dry
+runs only mark event items reconciled in Arena. The webhook path writes
+nothing to SAP.
+
+**Outbound-integration pull queue (`arena-plm.js:1197-1263`)** — Arena holds
+the cursor: `GET /outboundintegrations/{guid}/events?itemsReconciled=false`,
+per-event `…/items`, then `PUT …/items/{itemGuid} {reconciled:true}`
+(idempotent). No local state.
+
+**Change poller (`routes/poll.js` → `arena-plm.js:1117-1191`)** — manual only
+(no scheduler exists in the repo). Watermark precedence: explicit `since` →
+newest `A_ChangeMaster.ChangeNumberValidFromDate` where
+`startswith(ReasonForChangeText,'ARENA ')` → `defaults.arenaChangeCutoff`;
+refuses non-dry runs with neither (`POLL/NO_CUTOFF`). Pages the whole Arena
+`/changes` feed client-side (no working server-side date filter), filters
+`effectiveDateTime > since`. Dedup via one bulk read of every `ARENA `-marked
+change master out of SAP into a Map. Real runs create change masters only.
+
+## 2. Staging & approval
+
+SQLite via `node:sqlite` on the **ephemeral CF filesystem** (`db.js`):
+`approvals(id, item_number, change_type PRODUCT|CHANGE_MASTER|BOM|DOCUMENTS,
+status PENDING|APPROVED|REJECTED, arena_data, sap_preview, sap_result, error,
+reject_reason, …)` — one row per (item × change type); queue lost on restart.
+
+- Seeding (`POST /api/approvals/seed`): `previewFromArena()` per item — all
+  four syncs dry-run under `Promise.allSettled` (concurrency 6). **No dedup**
+  on (item, changeType): re-seeding duplicates rows.
+- Approve: `bulk-approve` returns immediately; work runs in a detached async
+  IIFE guarded by a **module-level in-memory `postJob` singleton** (progress
+  source AND concurrency guard; forces `instances: 1`; on restart, in-flight
+  jobs leave rows PENDING forever). `chain-approve` is synchronous per item,
+  halts on first failure. Reject is a pure DB flip.
+- **On approval only `item_number` is passed** — the stored `sap_preview` is
+  never replayed; every sync re-fetches Arena and recomputes. What was
+  approved is not necessarily what gets written (→ rebuild ARN-010).
+- **`REJECTED` is overloaded**: human rejection AND SAP write failure land in
+  the same status, distinguishable only by `error` vs `reject_reason`; no
+  retry path from REJECTED except re-seeding (→ rebuild ARN-002 states).
+
+## 3. Sync execution — the SAP write chain
+
+Order from `config/sync-chain-order.json`: PRODUCT → CHANGE_MASTER → BOM →
+DOCUMENTS (CM's material object record needs the material; BOM's
+`EngineeringChangeDocForEdit` needs the CM). Services as in PRD interface
+contracts (V4 product, V2 product fallback, BOM write `;v=0002` / read v1,
+change master `;v=0002`, CV attachment).
+
+**PRODUCT (`:1336-1663`)** — exists-check (V4 GET, 404 ⇒ create path).
+Create: `ProductType` strictly from Arena attr "SAP Product Type" (missing ⇒
+`422 SYNC/NO_PRODUCT_TYPE` — deliberately "not a material"); UOM via config +
+hardcoded ISO map; plants from attr "SAP Plant" through `plantMapping`; weight
+g→×1000, dims mm→×1000; description truncated 40. Deep insert with
+`_ProductDescription`, `_ProductPlant` only when no `AvailabilityCheckType`;
+else per-plant V2 `A_ProductPlant` POSTs; `ProcurementType` requires `MRPType`
+in the same `ProductPlantSupplyPlanning` PATCH. Best-effort CM link keyed on
+`startswith(ChangeNumberDescription,'ECO:')` — a different key than everything
+else (landmine). Update path: `ProductType`/`BaseUnit` immutable;
+`ProductDescription` PATCH by full key (nav PATCH → 501), POST fallback on
+404/405.
+
+**CHANGE_MASTER (`:1794-1873`)** — EFFECTIVE revision with `change.number`
+via `/revisions` (`/items/{guid}/changes` 404s — does not exist).
+Double-condition dedup: `ReasonForChangeText eq 'ARENA <eco>'` AND a material
+object record for *this* item. Create: `A_ChangeMaster` with
+`ChangeNumberDescription: "ECO: title"[0..40]`, `/Date(ms)/` valid-from,
+`ReasonForChangeText: "ARENA <eco>"`; then object-type assignments Material +
+MaterialBOM (`IsActiveForChangeNumber: true` mandatory — else 201 +
+silently discarded), then the material object record. Errors returned, not
+thrown.
+
+**BOM (`:1665-1792`)** — first mapped plant wins; lines `quantity > 0`, item
+numbers (i+1)*10 zero-padded 4; category via `bomlineByCategory` default 'L'.
+Opportunistic ECM lookup (bare catch — silently drops ECM and creates the BOM
+plain if the object-type assignment fails: traceability quietly lost).
+Component pre-check: if all components missing in SAP → early return, no
+orphan header (SAP's real error is an opaque consistency-check message).
+Header POST with `EngineeringChangeDocForEdit` (never
+`EngineeringChangeDocument` — creatable=false, silently dropped), validity end
+`/Date(253402214400000)/`. Items POSTed serially; each item re-reads the
+header on the v1 read service; ECM BOMs must carry `HeaderChangeDocument`.
+`BOMItemIsSalesRelevant:'X'` hardcoded. Partial failure is a return value.
+**Not idempotent — re-approval creates a second BOM header.**
+
+**DOCUMENTS (`:1875-1913`)** — URL links only, no bytes: filters files to
+`storageMethodName === 'WEB' && file.location` (blob-stored files silently
+skipped), then V2 function import `CreateUrlAsAttachment` with hardcoded
+`BusinessObjectTypeName BUS1001006`, `MIMEType 'text/url'`, description
+"<item> <category> Rev <edition>"[0..40]. The editable
+`arena-doc-category-to-sap-doctype.json` config is **ignored by code**.
+**Not idempotent — re-approval duplicates attachments.**
+
+**Cross-cutting:** CSRF token+cookie cached per service, invalidated only on
+403 with exactly one retry; PRODUCT services reject bare-root token fetch
+(seed from `Product?$top=1` / `A_Product?$top=1`). Arena 401 → re-login, one
+retry. No backoff, no retry queue, no transactions. Idempotency exists only
+for CHANGE_MASTER (double-check) and weakly for PRODUCT (exists→PATCH).
+
+## 4. Arena reads
+
+Session `POST /login {email, password, workspaceId}` → `arena_session_id`
+header, cached 24 min vs 25-min expiry, single-flight promise lock, one 401
+retry. Endpoints used by sync: `/items?number=` (resolves EFFECTIVE),
+`/items/{guid}`, `/items/{guid}/revisions`, `/items/{guid}/bom`,
+`/items/{guid}/files`, `/changes/{guid}`, `/changes?limit&offset`,
+`/changes/{guid}/items`, `/settings/items/categories`, outbound-integration
+trio. Write-back (`POST /changes/statuschanges`) is parked behind
+`ARENA_WRITEBACK_ENABLED`.
+
+**Load-bearing subtlety — WORKING-revision overlay (`:508-552`):** the seven
+SAP classification attributes (SAP Object Type / Product Type / Product
+Category / Sub-Category / Product Family / Business Unit / Plant) are only
+writable on the WORKING revision (Arena error 3224), while `/items?number=`
+returns the EFFECTIVE revision — so those seven are merged from WORKING over
+EFFECTIVE. The overlay is best-effort: a transient Arena failure silently
+loses "SAP Product Type" and turns a valid sync into `NO_PRODUCT_TYPE`.
+
+## 5. Config actually read vs dead
+
+Live (read by sync code, hot-reloaded, git-backed via external
+`sap-apps-config` repo): `defaults.json`, `plant-mapping.json`,
+`uom-arena-to-sap.json`, `material-group-by-product-type.json`,
+`availability-check-by-product-type.json`, `lifecycle-to-xplant-status.json`,
+`bomline-category-by-arena-category.json`, `sync-chain-order.json`.
+
+Dead (loaded, zero read sites — confirms the KNOWLEDGE.md landmine):
+`arena-category-to-sap.json`, `procurement-type.json` (code uses a hardcoded
+map keyed on the first char of Arena "Item Type"),
+`catalog-to-planning-strategy.json`. Ignored-but-editable:
+`arena-doc-category-to-sap-doctype.json`. Hardcoded despite the "no
+hardcoding" comment: ISO UOM map, status map, `ARENA ` marker, procurement
+map, service paths, working-attrs list, BOM validity end, sales-relevant
+flag, Language 'EN', 40-char truncations, BUS1001006.
+
+## 6. Fragilities the rebuild must not inherit
+
+1. In-memory `postJob` singleton = progress + concurrency guard; restart
+   orphans PENDING rows; forces `instances: 1`.
+2. Fire-and-forget approval worker; no resume, no cancellation.
+3. Seed dedup absent; BOM and DOCUMENTS syncs non-idempotent.
+4. `REJECTED` overload (human vs SAP failure).
+5. Silent ECM degradation (bare catches around ECO lookup, object-type
+   assignment, revision overlay).
+6. Two disagreeing CM lookup keys (`ChangeNumberDescription` startswith vs
+   `ReasonForChangeText` eq; 40-char truncation can split them).
+7. The `ARENA <eco>` free-text marker is the entire ledger — dedup key,
+   watermark, and status all hang on an editable 40-char field.
+8. O(history) paging on every poll (Arena full feed; up to 100k SAP change
+   masters).
+9. Arena credentials base64 in SQLite on the ephemeral filesystem; the whole
+   approvals queue dies on restart/redeploy.
+10. Approved preview is decorative — execution re-fetches and recomputes.