A retail brand I worked with discovered a competitor undercutting them by 4% on their three best selling SKUs. The discovery came from a weekly spreadsheet export, which meant the undercut had been live for six days and had already cost them the Buy Box on two listings. Their pricing team was not slow. Their data was.
That gap between what a marketplace looks like right now and what your systems believe is true is where margin quietly disappears. Amazon repricers move in minutes, promotional badges appear and vanish within the same hour, and stock levels shift constantly. If your monitoring loop runs once a day from a single server IP, you are not monitoring prices. You are collecting historical trivia.
This guide walks through building a price monitoring pipeline in Python that actually holds up: how to structure the fetch layer, how to rotate exit IPs without wrecking session consistency, how to parse defensively, and how to detect real price changes instead of drowning in noise.
What "Real Time" Actually Means Here
Before writing code, define your refresh budget honestly. True per-second monitoring across thousands of ASINs is neither necessary nor achievable at sane cost. What most teams need is tiered polling:
- Tier 1 (high velocity): your top revenue SKUs and the competitor listings that directly cannibalise them. Poll every 5 to 15 minutes.
- Tier 2 (watchlist): category rivals, secondary SKUs, seasonal items. Poll hourly.
- Tier 3 (long tail): broad catalogue coverage for trend analysis. Poll once or twice daily.
This tiering matters because it drives your entire infrastructure spend. A thousand ASINs at 10 minute intervals is 144,000 requests per day. The same thousand at 6 hour intervals is 4,000. Deciding this first prevents you from over-engineering, and it tells you how much proxy bandwidth you actually need.
Architecture: Five Components, Loosely Coupled
A maintainable price monitor separates concerns cleanly. When Amazon changes its markup (and it will), you should only need to touch one module.
1. The Scheduler
A queue that emits scrape jobs based on tier. Redis with a sorted set keyed on next_check_at works well and is trivially horizontally scalable. Avoid cron loops that fire all jobs simultaneously: staggered dispatch produces a smoother request profile and looks far less mechanical.
2. The Fetch Layer
This is where proxies live. Each job gets an outbound identity: an exit IP, a consistent user agent, an accept-language header matching the marketplace locale, and a cookie jar. The critical design decision is that identity is bundled, not assembled randomly per request. An IP geolocated in Ohio sending Accept-Language: de-DE while requesting amazon.com with a French postcode cookie is a fingerprint contradiction, and contradictions are what detection systems score on.
import asyncio, random, httpxfrom dataclasses import dataclass@dataclassclass Identity: proxy: str # http://user:pass@host:port user_agent: str accept_language: strasync def fetch(identity: Identity, url: str) -> str | None: headers = { "User-Agent": identity.user_agent, "Accept-Language": identity.accept_language, "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Upgrade-Insecure-Requests": "1", } timeout = httpx.Timeout(20.0, connect=10.0) async with httpx.AsyncClient( proxy=identity.proxy, headers=headers, timeout=timeout, follow_redirects=True, http2=True ) as client: try: r = await client.get(url) except (httpx.ProxyError, httpx.ConnectTimeout, httpx.ReadTimeout): return None if r.status_code != 200 or "[email protected]" in r.text: return None # soft block / robot check page return r.text
Two details worth stealing. First, HTTP/2 is enabled: a client that negotiates HTTP/1.1 while presenting a modern Chrome user agent is an obvious mismatch. Second, a 200 response is not a success. Amazon serves its robot check page with a 200 status, so you must inspect the body. Counting HTTP status codes alone will make your dashboards lie to you.
3. The Parse Layer
Parse defensively with a cascade of selectors rather than one brittle path. Amazon runs continuous layout experiments, so the same ASIN can return three different price containers depending on which bucket your request lands in.
from selectolax.parser import HTMLParserPRICE_SELECTORS = [ "span.a-price span.a-offscreen", "#corePriceDisplay_desktop_feature_div .a-price .a-offscreen", "#priceblock_ourprice", "#priceblock_dealprice",]def extract(html: str) -> dict: tree = HTMLParser(html) price = None for sel in PRICE_SELECTORS: node = tree.css_first(sel) if node and node.text(strip=True): price = node.text(strip=True) break seller = tree.css_first("#sellerProfileTriggerId") avail = tree.css_first("#availability span") return { "price_raw": price, "seller": seller.text(strip=True) if seller else None, "availability": avail.text(strip=True) if avail else None, "buybox_present": tree.css_first("#add-to-cart-button") is not None, }
Normalise currency separately from extraction. A German listing returns 19,99 € and a US listing returns $19.99; storing raw strings and parsing them at write time keeps a locale bug from corrupting your entire price history.
Also capture who holds the Buy Box, not just the number. A price drop from a third party seller with two units of stock is a different signal than a drop from Amazon Retail, and any repricing rule that treats them identically will hurt you.
4. Storage and Change Detection
Store append-only observations, never overwrite a current price. You want a time series: (asin, marketplace, seller, price_cents, currency, availability, buybox, observed_at). Then compute changes with a window function rather than in application code.
The subtlety is noise suppression. Marketplaces flicker: you will see a price of zero, a missing Buy Box, or a stale cached value for a single poll and then normality returns. Require confirmation before firing an alert, for example two consecutive observations from two different exit IPs agreeing on the new value. This single rule eliminates the majority of false positives that cause pricing teams to stop trusting the tool.
5. Alerting and Actioning
Route confirmed deltas into whatever your team already uses: Slack, a webhook into your repricer, or a daily digest for Tier 3 movements. Include the previous value, the delta as a percentage, the seller, and a timestamp. Alerts without context get muted within a week.
Common Mistakes That Kill These Projects
Hammering from one IP range. The fastest way to get nothing but robot check pages is 500 sequential requests from a single datacenter subnet. Concurrency should scale with pool diversity, not with your CPU count.
Ignoring the postcode. Amazon prices and delivery promises are partly location dependent. If you do not pin a delivery postcode via cookie and route through an exit IP in the matching region, your "price change" may just be a different fulfilment context.
Retrying blindly. A blocked request retried immediately on the same identity burns bandwidth and reinforces the block. Retry on a fresh identity, with jittered backoff, and cap attempts at three.
No success rate telemetry. Log block rate per exit IP, per subnet, and per marketplace. Silent degradation is the norm in scraping: the pipeline keeps running, the data quietly thins out, and nobody notices until a pricing decision goes wrong. It also pays to validate endpoints before a big run rather than during it, and a quick pass through a proxy tester will tell you whether the geolocation and latency of each entry point match what you expect.
Treating variations as one product. Size and colour variants have separate ASINs and separate prices. Collapsing them produces averages that describe nothing.
Where Proxies Fit In
Everything above depends on one thing: the ability to present as many distinct, credible consumers requesting a public product page. Amazon does not need to ban you to break your monitor. It only needs to serve you a slightly different page, a cached price, or a robot check often enough that your coverage becomes unreliable.
That is why the exit layer is a data quality problem rather than a plumbing detail. Rotating residential proxy pools give each request an IP that belongs to a real consumer connection in a real location, which matters twice over: it keeps block rates low, and it means the localised price you scrape is the price an actual shopper in that market would see. For Tier 1 SKUs where you want a persistent cookie jar and a pinned postcode across several polls, sticky sessions on ISP or static residential IPs are usually the better fit, while high volume long tail sweeps run more economically over rotating pools. Mobile exits are worth reserving for the hardest targets rather than used as a default.
This is the practical reason to run monitoring on infrastructure built for it. EnigmaProxy offers residential, ISP, datacenter, and mobile pools under one account with predictable pricing, which lets you assign pool type per tier instead of overpaying to route your entire catalogue through premium IPs. Ethical sourcing matters here as well: a pool assembled without informed consent is both a compliance exposure and an operational risk, because those networks are the ones that get disrupted and take your data continuity with them.
When you size your plan, work backwards from the request maths. Estimate average page weight (Amazon product pages are heavy, often 500KB to 1.5MB uncompressed), multiply by daily request volume, then add roughly 20% for retries. That number, not a vague sense of scale, is what you should be budgeting against.
Strategic Insights: Where Price Intelligence Is Heading
Rendering costs are rising. More marketplace surfaces are moving pricing data behind client side hydration. Expect a hybrid pipeline: lightweight HTTP fetches for the majority of pages, and a headless browser tier reserved for the pages that genuinely require it, since browser requests cost several times more in bandwidth.
Detection is shifting to behavioural scoring. Static fingerprint checks are being supplemented with request cadence, navigation patterns, and IP reputation history. Perfectly regular polling intervals are becoming a signal in themselves, which makes jitter and staggered scheduling a technical requirement rather than politeness.
LLMs are absorbing the parsing layer. Extraction models can already read a product page and return structured fields without hand-written selectors, which cuts maintenance sharply. They do not reduce fetch volume at all, so the network layer becomes an even larger share of total cost.
Compliance expectations are tightening. Collecting publicly visible prices remains widely accepted practice, but auditability is increasingly expected: documented sourcing for your IP pool, respect for rate limits, and no collection of personal data. Teams that can evidence this will face fewer procurement obstacles.
Conclusion
A useful Amazon price monitor is not really a scraper. It is a scheduling problem, a data quality problem, and a network diversity problem wearing a scraper's clothes. Tier your polling so cost tracks business value. Bundle proxy, headers, and cookies into coherent identities. Parse with fallbacks and store append-only observations. Confirm changes across two independent exit IPs before you alert anyone, and instrument block rate so you notice degradation early.
Get those five things right and you move from six day old spreadsheets to actionable signals inside a quarter of an hour. The remaining variable is the network underneath, and pairing your pipeline with a provider like EnigmaProxy that offers geo-coverage and pool diversity in the professional tier is what keeps that signal arriving reliably, week after week.