costco-watch: client-side browser headers, drop Transform Rule requirement
This commit is contained in:
@@ -5,9 +5,12 @@ Source: https://git.alexzaw.dev/alexz/costco-watch (working copy:
|
|||||||
~/projects/costco-watch). Build + push with ./build.sh.
|
~/projects/costco-watch). Build + push with ./build.sh.
|
||||||
|
|
||||||
Gotchas:
|
Gotchas:
|
||||||
- TARGET_URL must go through the costco-tire cloudflared proxy AND that
|
- Costco stalls (no response, just a hang) on requests lacking Sec-Fetch-*
|
||||||
hostname needs a Cloudflare Transform Rule adding browser headers
|
headers. watch.js sends a browser header set in BROWSER_HEADERS - do not
|
||||||
(User-Agent, Accept, Accept-Language, Sec-Fetch-*). Without it Costco stalls
|
strip it. Any single Sec-Fetch-* header is sufficient; UA/Accept alone are
|
||||||
the request until timeout and the checker reports "error"/"blocked".
|
not. Cloudflare Transform Rules CANNOT substitute for this: Sec-* headers are
|
||||||
|
on Cloudflare's restricted list, and cloudflared can't add headers at all.
|
||||||
|
- TARGET_URL goes through the costco-tire cloudflared ingress, which sets
|
||||||
|
httpHostHeader: tires.costco.com.
|
||||||
- State (last stock value, transition history) lives in ${APP_DATA_DIR}/data;
|
- State (last stock value, transition history) lives in ${APP_DATA_DIR}/data;
|
||||||
wiping it means the next check can re-alert.
|
wiping it means the next check can re-alert.
|
||||||
|
|||||||
@@ -13,7 +13,7 @@
|
|||||||
"automation",
|
"automation",
|
||||||
"utilities"
|
"utilities"
|
||||||
],
|
],
|
||||||
"description": "Polls a Costco tire product page on an interval and pushes an ntfy alert the moment it flips from out-of-stock to in-stock. Points at the costco-tire cloudflared proxy, which needs a Cloudflare Transform Rule attaching browser-like request headers - Costco stalls non-browser header sets.",
|
"description": "Polls a Costco tire product page on an interval and pushes an ntfy alert the moment it flips from out-of-stock to in-stock. Sends browser fetch-metadata headers itself - Costco stalls requests without Sec-Fetch-*, and neither cloudflared nor Cloudflare Transform Rules can supply them (Sec-* is on Cloudflare's restricted list). No Cloudflare-side configuration needed.",
|
||||||
"short_desc": "Alerts when a Costco tire is back in stock",
|
"short_desc": "Alerts when a Costco tire is back in stock",
|
||||||
"author": "alexz",
|
"author": "alexz",
|
||||||
"source": "https://git.alexzaw.dev/alexz/costco-watch",
|
"source": "https://git.alexzaw.dev/alexz/costco-watch",
|
||||||
@@ -81,4 +81,4 @@
|
|||||||
"updated_at": 1754995000000,
|
"updated_at": 1754995000000,
|
||||||
"deprecated": false,
|
"deprecated": false,
|
||||||
"min_tipi_version": "4.5.0"
|
"min_tipi_version": "4.5.0"
|
||||||
}
|
}
|
||||||
@@ -3,19 +3,28 @@
|
|||||||
Polls a Costco tire product page on an interval and pushes an ntfy alert when
|
Polls a Costco tire product page on an interval and pushes an ntfy alert when
|
||||||
the item flips from out-of-stock to in-stock.
|
the item flips from out-of-stock to in-stock.
|
||||||
|
|
||||||
## Why it points at a proxy, not costco.com
|
## The header gotcha
|
||||||
|
|
||||||
Costco's bot mitigation silently stalls requests carrying a non-browser header
|
Costco's bot mitigation silently stalls requests that lack browser
|
||||||
set (the connection completes TLS, then nothing comes back - no 403, just a
|
fetch-metadata headers - TLS completes, then nothing comes back, no 403, just a
|
||||||
hang). The page loads fine through `costco-tire.alexzaw.dev`, a cloudflared
|
hang until timeout. Bisecting against the live site showed the gate is the
|
||||||
ingress rule with `httpHostHeader: tires.costco.com`, **provided** a Cloudflare
|
`Sec-Fetch-*` family: **any single one** of `Sec-Fetch-Site`, `Sec-Fetch-Mode`
|
||||||
Transform Rule attaches browser-like request headers (`User-Agent`, `Accept`,
|
or `Sec-Fetch-Dest` is enough to get a normal response, even with no
|
||||||
`Accept-Language`, `Sec-Fetch-*`, `Upgrade-Insecure-Requests`) for that
|
`User-Agent` at all. User-Agent, Accept and Accept-Language on their own make
|
||||||
hostname. cloudflared itself cannot add headers - its `originRequest` block has
|
no difference.
|
||||||
no such option - which is why the rule lives at the edge.
|
|
||||||
|
|
||||||
If the checker starts reporting `blocked`, that Transform Rule is the first
|
This service therefore sends a browser header set itself (see
|
||||||
thing to verify.
|
`BROWSER_HEADERS` in `watch.js`). Two things that do **not** work as fixes:
|
||||||
|
|
||||||
|
- cloudflared cannot add headers - its `originRequest` block has no such
|
||||||
|
option, it only forwards what the client sent.
|
||||||
|
- Cloudflare Transform Rules cannot set `Sec-*` headers - they're on
|
||||||
|
Cloudflare's restricted list, so the edge can't supply the one header family
|
||||||
|
that matters.
|
||||||
|
|
||||||
|
`TARGET_URL` still points at the `costco-tire.alexzaw.dev` cloudflared ingress
|
||||||
|
(which sets `httpHostHeader: tires.costco.com`), but no Cloudflare-side
|
||||||
|
configuration is required.
|
||||||
|
|
||||||
## Config
|
## Config
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user