diff --git a/apps/costco-watch/build/README.md b/apps/costco-watch/build/README.md index 71b23af..0d65766 100644 --- a/apps/costco-watch/build/README.md +++ b/apps/costco-watch/build/README.md @@ -5,9 +5,12 @@ Source: https://git.alexzaw.dev/alexz/costco-watch (working copy: ~/projects/costco-watch). Build + push with ./build.sh. Gotchas: -- TARGET_URL must go through the costco-tire cloudflared proxy AND that - hostname needs a Cloudflare Transform Rule adding browser headers - (User-Agent, Accept, Accept-Language, Sec-Fetch-*). Without it Costco stalls - the request until timeout and the checker reports "error"/"blocked". +- Costco stalls (no response, just a hang) on requests lacking Sec-Fetch-* + headers. watch.js sends a browser header set in BROWSER_HEADERS - do not + strip it. Any single Sec-Fetch-* header is sufficient; UA/Accept alone are + not. Cloudflare Transform Rules CANNOT substitute for this: Sec-* headers are + on Cloudflare's restricted list, and cloudflared can't add headers at all. +- TARGET_URL goes through the costco-tire cloudflared ingress, which sets + httpHostHeader: tires.costco.com. - State (last stock value, transition history) lives in ${APP_DATA_DIR}/data; wiping it means the next check can re-alert. diff --git a/apps/costco-watch/config.json b/apps/costco-watch/config.json index ac65393..1506ef9 100644 --- a/apps/costco-watch/config.json +++ b/apps/costco-watch/config.json @@ -13,7 +13,7 @@ "automation", "utilities" ], - "description": "Polls a Costco tire product page on an interval and pushes an ntfy alert the moment it flips from out-of-stock to in-stock. Points at the costco-tire cloudflared proxy, which needs a Cloudflare Transform Rule attaching browser-like request headers - Costco stalls non-browser header sets.", + "description": "Polls a Costco tire product page on an interval and pushes an ntfy alert the moment it flips from out-of-stock to in-stock. Sends browser fetch-metadata headers itself - Costco stalls requests without Sec-Fetch-*, and neither cloudflared nor Cloudflare Transform Rules can supply them (Sec-* is on Cloudflare's restricted list). No Cloudflare-side configuration needed.", "short_desc": "Alerts when a Costco tire is back in stock", "author": "alexz", "source": "https://git.alexzaw.dev/alexz/costco-watch", @@ -81,4 +81,4 @@ "updated_at": 1754995000000, "deprecated": false, "min_tipi_version": "4.5.0" -} +} \ No newline at end of file diff --git a/apps/costco-watch/metadata/description.md b/apps/costco-watch/metadata/description.md index 7c92a86..3b269e8 100644 --- a/apps/costco-watch/metadata/description.md +++ b/apps/costco-watch/metadata/description.md @@ -3,19 +3,28 @@ Polls a Costco tire product page on an interval and pushes an ntfy alert when the item flips from out-of-stock to in-stock. -## Why it points at a proxy, not costco.com +## The header gotcha -Costco's bot mitigation silently stalls requests carrying a non-browser header -set (the connection completes TLS, then nothing comes back - no 403, just a -hang). The page loads fine through `costco-tire.alexzaw.dev`, a cloudflared -ingress rule with `httpHostHeader: tires.costco.com`, **provided** a Cloudflare -Transform Rule attaches browser-like request headers (`User-Agent`, `Accept`, -`Accept-Language`, `Sec-Fetch-*`, `Upgrade-Insecure-Requests`) for that -hostname. cloudflared itself cannot add headers - its `originRequest` block has -no such option - which is why the rule lives at the edge. +Costco's bot mitigation silently stalls requests that lack browser +fetch-metadata headers - TLS completes, then nothing comes back, no 403, just a +hang until timeout. Bisecting against the live site showed the gate is the +`Sec-Fetch-*` family: **any single one** of `Sec-Fetch-Site`, `Sec-Fetch-Mode` +or `Sec-Fetch-Dest` is enough to get a normal response, even with no +`User-Agent` at all. User-Agent, Accept and Accept-Language on their own make +no difference. -If the checker starts reporting `blocked`, that Transform Rule is the first -thing to verify. +This service therefore sends a browser header set itself (see +`BROWSER_HEADERS` in `watch.js`). Two things that do **not** work as fixes: + +- cloudflared cannot add headers - its `originRequest` block has no such + option, it only forwards what the client sent. +- Cloudflare Transform Rules cannot set `Sec-*` headers - they're on + Cloudflare's restricted list, so the edge can't supply the one header family + that matters. + +`TARGET_URL` still points at the `costco-tire.alexzaw.dev` cloudflared ingress +(which sets `httpHostHeader: tires.costco.com`), but no Cloudflare-side +configuration is required. ## Config