Price monitoring · Grocery prices
Grocery price monitoring by store, postcode and promotion
A grocery price is a price at one store, for one delivery area, on one day — and often with a promotion and a prior price attached. Collect each store's public catalogue from an exit in its region, keep the store context on one IP, and the shelf you store is the shelf a local shopper sees.
Pay per GB · No subscription · Prices in EUR, excl. VAT
- €0.75
- per GB on ARC Basic, excl. VAT
- −21%
- at 1 TB (€0.59 / GB)
- 185+
- countries, city targeting
- Never
- expires — no subscription
Written for
- CPG revenue growth managers
- Retail pricing analysts
- Inflation and price-index researchers
- Quick-commerce category teams
The short answer
- Online grocery prices are usually set per store or per delivery area, and the store context is chosen through the retailer's own postcode or store selector — the exit IP keeps that context plausible, it does not replace it.
- Hold one sticky session per store context so the store cookie and the IP stay together for the whole category sweep.
- Store the promotion as the page shows it: shelf price, promo price, mechanic, the stated prior price and the unit price. In the EU a price-reduction announcement must state the lowest price of at least the previous 30 days.
- Most grocery catalogue pages are light HTML or JSON. Start on ARC Basic, block images, and move only the stricter banners to residential.
The problem
What goes wrong without the right IPs
The national page is not anyone's price
Without a store or postcode set, many grocery sites show a default store or a national reference price. Collect that and every store in your panel reports the same number, while the real prices differ underneath it.
The store context falls apart mid-sweep
The selected store usually lives in a cookie tied to a session. If the exit IP changes halfway through a category walk, some sites reset to the default store and the second half of your sweep quietly belongs to a different shop.
Promotions are read wrong
A multibuy, a member price and a temporary price cut look identical if you only scrape the largest number on the tile. Without the mechanic and the stated prior price, a promotion analysis is a guess.
Quick-commerce assortments flip within hours
A dark store's catalogue reflects what is physically on its shelves now. A product that is out of stock at 18:00 may be gone from the listing entirely, so one sweep a day misses most of the availability signal.
Product images dominate the bill
A category page carries dozens of packshots. Billing is per gigabyte, so a sweep that loads images costs many times what the prices inside it are worth to you.
What to collect
What you can collect
- Store or delivery-area ID
- The key every other field hangs off. Record which store the page said it was showing, not which store you meant to select.
- Shelf price and currency
- The regular price as displayed, before any promotion.
- Promotion price and mechanic
- "2 for 3", "−25%", "€1 off", member-only: the mechanic decides how the effective unit price is calculated.
- Stated prior price
- In the EU the reference a reduction is measured against. Storing it lets you compare the claim with your own 30-day history.
- Unit price and unit
- Price per kilogram, litre or piece — the only way to compare pack sizes and shrinkflation over time.
- Pack size and GTIN
- Matches the same product across banners and catches a size change behind an unchanged shelf price.
- Availability flag
- In stock, out of stock, or missing from the listing — three different states in quick commerce.
- Promotion validity dates
- Where the page states them, they tell you whether two observations are one promotion or two.
- Collection timestamp
- Grocery prices change on a weekly promotion cycle and sometimes intraday; without a timestamp the series cannot be aligned.
How it runs
How a collection run works
- 01
Build the store panel first
List the stores or delivery postcodes you need per banner, and the region each belongs to. Map each to a country and, where it is on our city list, the nearest metro — Hamburg, Lyon, Manchester, Chicago and so on.
- 02
Select the store the way a shopper does
Use the retailer's public postcode or store selector on a logged-out session, then read back which store the page confirms. If it does not confirm the store you asked for, discard the sweep rather than storing a default store's prices.
- 03
Hold the exit for the whole store sweep
Open one sticky session per store context and keep it for the category walk. A sweep of 60–100 category pages at a polite pace fits comfortably in 15 minutes.
One sticky exit per store context17 linesimport requestsGATEWAY = "proxy.arcproxies.io:8000"PASSWORD = "PASSWORD"def store_session(store_id: str, country: str, city: str | None = None) -> requests.Session: # Store cookie and exit IP stay together for the whole sweep. user = f"USERNAME_pool-arc_country-{country}" if city: user += f"_city-{city}" user += f"_session-{store_id}_sesstime-15" proxy = f"http://{user}:{PASSWORD}@{GATEWAY}" s = requests.Session() s.proxies = {"http": proxy, "https": proxy} return ss = store_session("st0412", "de", "hamburg") - 04
Parse promotions into fields, not strings
Split each tile into shelf price, promo price, mechanic, prior price and unit price at collection time. A string like "2 for €3 (€1.80/kg), lowest price 30 days: €1.99" is four facts; store it as four.
- 05
Block what you do not store
Disable images, fonts and third-party tags. Most grocery catalogues also expose the same data the page renders as JSON, which is lighter still — see reducing proxy bandwidth.
- 06
Schedule around the promotion cycle
Weekly promotions typically start on a fixed weekday per banner. Run a full sweep the morning the cycle starts and lighter availability checks during the week; quick-commerce availability needs several passes a day.
Setup
Recommended setup
Grocery catalogues are high-volume and mostly plain HTML or JSON, so the cheapest pool carries the bulk and residential is reserved for the banners that need it.
| Plan | Role | Session | Targeting | Price |
|---|---|---|---|---|
ARC Basic Rotating ISP | Default for store sweeps Rotating ISP exits at our lowest price per GB, with city targeting from the published metro list. Enough for most catalogue pages once images are blocked. | Sticky, 15 minutes per store | Country + city | €0.75 / GB €0.59 at 1 TB (−21%) |
ARC Fast Rotating residential | For stricter banners and quick commerce Residential exits for the retailers and delivery apps' web storefronts where ISP addresses see repeated failures you cannot otherwise explain. | Sticky per store or delivery area | Country + city | €2.00 / GB €1.12 at 1 TB (−44%) |
Prices are list prices in EUR, excl. VAT. Volume discounts apply automatically at checkout — full ladder on the pricing page.
Connect
One connection string per market
The country, city and sticky session are tokens in the username. Change the token, change the market; keep the session handle, keep the IP.
http://USERNAME_pool-arcfast_country-de_session-k3f9a1x2_sesstime-10:PASSWORD@proxy.arcproxies.io:8000Swap _country-de for any market you quote. The full grammar is on the residential proxies page.
Illustration
What a 40-store grocery panel costs
An illustration with round numbers — your pages, payloads and retry rate will differ. The arithmetic is the real price ladder.
Assumptions
- Stores in the panel
- 40
- SKUs tracked per store
- 3,000
- Category pages per sweep
- 63 (48 products a page)
- Cadence
- one full sweep a day
- Weight per page, images blocked
- 300 KB
- Bandwidth per month
- ≈ 23 GB
On ARC Basic
€17.25per month, excl. VAT
23 GB bought in one purchase at €0.75 / GB (list price).
The same 23 GB on ARC Fast: €46.00 at €2.00 / GB. Route only the jobs that need it through the pricier pool.
Intraday availability checks for quick commerce add to this; they are usually a handful of pages per store and run several times a day. Loading product images would multiply the figure several times over.
Bandwidth never expires, so a bigger single purchase is only ever cheaper per GB. See the ladder.
In depth
Grocery prices, in depth
Why grocery prices differ by store and postcode
Grocers price by store, by zone or by fulfilment centre, and the web shop shows the price for whichever of those your session is attached to. Quick-commerce apps go further: each delivery area is served by one dark store, and the catalogue is that building's stock.
Three different things decide which shelf you see, and only one of them is the network you come from. Getting the order right is the difference between a store-level panel and 40 copies of the default store.
| What decides the shelf | How it is set | What the exit IP contributes |
|---|---|---|
| Selected store or delivery postcode | The retailer's own store or postcode selector, usually stored in a cookie | Nothing directly — but a session in Hamburg selecting a Hamburg store looks like what it is |
| Default region before a store is chosen | Often inferred from the visitor's location | Decisive: an exit in the right country or metro lands you in the right default |
| Country site and currency | Domain, language and the visitor's country | Decisive for geo-redirects between national shops — see geo-targeting |
City targeting on our rotating pools covers a fixed list of metros per country, not individual postcodes. For a store in a smaller town, target the country (or the nearest listed metro) and let the store selector do the precise part.
Promotions, prior prices and unit prices: what to store
In the EU, the Price Indication Directive requires the selling price and the unit price to be shown, and since 28 May 2022 its Article 6a requires any announced price reduction to indicate the prior price — defined as the lowest price the trader applied in a period of at least 30 days before the reduction. Member States may set different rules for goods that deteriorate or expire rapidly, which matters for fresh produce.
In September 2024 the Court of Justice applied this to a grocery case: in C-330/23, Aldi Süd, it held that a reduction advertised as a percentage or as a "price highlight" must be calculated from that 30-day lowest price, not from the price immediately before the offer. The products at issue included bananas and pineapples; in the first example the reduction was measured from €1.69 when the 30-day low was €1.29.
For a monitoring team this is a data-model question rather than a legal one. If you keep a daily series per store and SKU, you can compute your own 30-day low and set it beside the prior price the page states. Where they diverge you have a finding to investigate, not a conclusion — national rules differ on fresh goods and new products, so take specifics to counsel.
- Shelf price — the regular price as displayed.
- Promo price and mechanic — percentage, fixed amount, multibuy, member-only.
- Stated prior price — exactly as printed, including the wording around it.
- Unit price — and the unit, because "per 100 g" and "per kg" both appear.
Quick commerce: monitoring a dark store's shelf
Quick-commerce catalogues are smaller than a supermarket's and change faster. The interesting signal is less the price than the assortment: which products a dark store lists at all, which are out of stock, and how often the same SKU disappears and returns.
That shape favours many small requests over few big ones. A check of the top 200 SKUs in one delivery area is a few category or search pages; doing it four times a day across 20 areas is still a modest bandwidth line on per-GB billing, where a per-request plan would charge for every one.
Only collect what the logged-out storefront shows for a postcode. If a delivery app requires an account to see prices for an area, that area is outside what we would support; check the acceptable use policy.
Grocery prices for inflation and price-index research
Online grocery prices are an established input to price measurement. The MIT Billion Prices Project, described by Cavallo and Rigobon in the Journal of Economic Perspectives (2016), was collecting 5 million prices a day from over 300 retailers in 50 countries by 2010, and found that online indexes co-move with official consumer price indexes in most countries.
Statistical offices do this too. Eurostat's practical guidelines on web scraping for the HICP (November 2020) set out a netiquette that is a good default for any research crawl: identify the scraper in the user-agent, pause between requests (the guidelines give one second as an example), run at night or off-peak, respect robots.txt and read the site's terms.
For a research panel, the exit network matters mainly for representativeness: a price index for Germany should be collected from German exits against German store contexts. The country and city tokens make that a property of the job rather than of the machine it runs on. The related online price indices page covers the index-building side.
Keeping a daily store sweep polite
- Spread, do not burst. A 40-store sweep run in parallel at full speed is 40 sessions hitting one retailer at once. Stagger store sessions and keep a pause between pages inside each.
- Back off on 429 and 503. A rate-limit response is the site telling you the pace; slow that store session down rather than rotating around it. See handling 429 rate limits.
- Collect off-peak. Early-morning sweeps catch the new promotion cycle and load the shop when shoppers are not using it.
- Fetch once, derive many. Store the raw page or JSON and re-parse it when your promotion model changes, instead of re-collecting.
These are the same habits the parent price monitoring page recommends for any competitor feed; grocery simply runs them at store granularity.
Works with anything that takes a proxy URL
- Python requests / httpx
- Scrapy
Playwright
Crawlee- Docker
FAQ
Questions
Something not covered? Write to contact@arcproxies.io.
Can a proxy set the postcode for an online grocery price?
Not on its own. Most grocery sites set the store or delivery area through their own postcode or store selector and keep it in a cookie. The exit IP decides the default region and the country site you land on; the selector decides the exact store. Use both, and verify the store the page confirms before storing prices.
Which cities can I target for store-level grocery monitoring?
ARC Basic, ARC Fast and ARC Ultra take a city token from a published list of metros, for example Hamburg, Munich, Lyon, Milan, Manchester, Warsaw, Toronto or Chicago. For stores outside those metros, target the country and select the store through the retailer's own selector.
What is the 30-day prior price rule for grocery promotions?
Article 6a of the EU Price Indication Directive, applicable since 28 May 2022, says an announced price reduction must indicate the prior price, meaning the lowest price the trader applied in at least the 30 days before the reduction. Member States may set different rules for perishable goods. This is a description of the rule, not legal advice.
How much bandwidth does a grocery price sweep use?
With images blocked, a category page is typically a few hundred kilobytes. A 40-store panel with 63 category pages per store, collected daily at 300 KB a page, is about 23 GB a month. Loading images would multiply that several times over.
Do I need residential IPs for grocery sites?
Often not. Many grocery catalogue pages are served the same way to any visitor, and ARC Basic covers them at our lowest price per GB with city targeting. Move a specific banner or quick-commerce storefront to ARC Fast when you see failures on it that you cannot otherwise explain.
How often should grocery prices be collected?
Once a day per store is enough for most shelf-price work, timed to the morning a banner's weekly promotion cycle starts. Quick-commerce availability changes within hours and usually needs several lighter checks a day.
Sources & review
Last reviewed by the ARC Proxies team. Prices are read from the live catalog at build time.
- [1]EUR-Lex: Directive (EU) 2019/2161 (Omnibus Directive), Article 2 inserting Article 6a into Directive 98/6/EC
- [2]European Commission: Price Indication Directive
- [3]Court of Justice of the European Union: Press release No 152/24: Judgment in Case C-330/23, Aldi Süd
- [4]Journal of Economic Perspectives: Cavallo & Rigobon, The Billion Prices Project: Using Online Prices for Measurement and Research (2016)
- [5]Eurostat: Practical guidelines on web scraping for the HICP (November 2020)
Price monitoring
More on price monitoring
Get started
Buy a gigabyte and try it on your own targets.
No subscription, no sales call. Code WELCOME15 takes 15% off your first purchase, and whatever you do not use is still there next quarter.