Web scraping · Public tenders
Public tender monitoring across EU and national portals
Public tender monitoring means reading every new contract notice, corrigendum and award that matches your sectors, across the EU's TED and the national portals below it. Use the official API where one exists, and spend proxy bandwidth only on the long tail of portals that have none.
Pay per GB · No subscription · Prices in EUR, excl. VAT
- €0.75
- per GB on ARC Basic, excl. VAT
- −21%
- at 1 TB (€0.59 / GB)
- 185+
- countries, city targeting
- Never
- expires — no subscription
Written for
- Bid and proposal teams
- Tender alert services
- Public sector sales teams
- Procurement researchers
The short answer
- Start with TED's own API. Searching published notices needs no key, and the full notice set is published as open data. A proxy adds nothing there.
- The gaps are below the EU thresholds and outside the EU. Contracts under the thresholds are advertised nationally, regionally or on a buyer's own profile, and that long tail is where a crawler earns its keep.
- Collect each national portal from an exit in its country. Several serve a reduced or redirected page to foreign visitors, and all of them are small public services that deserve a slow, steady crawl.
- Corrigenda move deadlines and change lots. Re-check open notices until they close, or your alert says a tender is due on a date that is no longer true.
- Tender documents behind a supplier registration are out of scope for automation. Collect the public notice; let the bid team register and download the documents themselves.
The problem
What goes wrong without the right IPs
Below-threshold notices never reach TED
Publication on TED is mandatory above the EU thresholds. Smaller contracts, often the ones an SME can actually win, appear only on national, regional or municipal portals, each with its own layout and search.
Hundreds of small portals, no common format
A single country can run a federal portal, regional platforms and buyer profiles on several vendors' e-procurement systems. Each needs its own parser, and each changes its templates without notice.
Corrigenda silently change the facts
A deadline extension, a new lot or a withdrawn notice arrives as a separate corrigendum. If you only read new notices, your database keeps the original deadline long after it moved.
The same tender appears twice
Above-threshold contracts are often published on TED and on the national portal. Without a matching rule on buyer, title, value and deadline, your alert fires twice and your pipeline counts double.
Public servers, modest capacity
Many regional portals run on infrastructure sized for human visitors. A burst of parallel requests from one address gets throttled, and a throttled night means a missed publication day.
What to collect
What you can collect
- Notice ID and notice type
- Distinguishes a contract notice from a prior information notice, a corrigendum or an award. The type decides whether the record opens, changes or closes an opportunity.
- Buyer name, country and region
- The filter most bid teams sort by. Store the buyer's own identifier where the portal prints one, so the same authority matches across portals.
- CPV codes
- The Common Procurement Vocabulary is how notices are classified by subject. Matching on CPV codes is far more reliable than keyword matching on titles in 24 languages.
- Estimated value and lots
- Qualifies the opportunity. Lots matter because a supplier can often bid for one lot of a tender it could never deliver whole.
- Procedure type
- Open, restricted, negotiated or a framework call-off. It tells the bid team how much work a response will take and whether they can still enter.
- Submission deadline and its history
- The field corrigenda change most often. Keep every value with the date you saw it, not just the latest.
- Document and portal links
- Where the bid team goes next. Store the link, not the documents; downloads often require a supplier account.
- Award details
- Winner, value and number of bids received, from award notices. This is the raw material for win-rate and competitor analysis.
How it runs
How a collection run works
- 01
Take TED from the API, not the website
Query published notices through the TED Search API by CPV code, country and publication date, or load the daily open-data packages. Both are free and neither needs a proxy. Record the notice IDs so you can recognise the same tender on a national portal later.
- 02
List the national and regional portals per country
For each target country, list the national portal, the regional platforms and any large buyers' own profiles. Check which offer RSS feeds, open-data exports or an API; use those first and crawl only what is left.
- 03
Sweep each portal's latest notices from inside its country
Fetch the newest-first listing through an exit in the portal's country, page until you reach notices you already hold, and stop. One sticky session per portal keeps any search state on one address.
Incremental sweep of one national portal24 linesimport timeimport httpxGATEWAY, PASSWORD = "proxy.arcproxies.io:8000", "PASSWORD"def portal_client(country: str, portal: str) -> httpx.Client: # Sticky ISP exit in the portal's country, one handle per portal user = f"USERNAME_pool-arc_country-{country}_session-{portal}_sesstime-15" return httpx.Client(proxy=f"http://{user}:{PASSWORD}@{GATEWAY}", timeout=30)def new_notices(client: httpx.Client, url: str, seen: set[str]): page = 1 while True: r = client.get(url, params={"sort": "published-desc", "page": page}) if r.status_code == 429: time.sleep(int(r.headers.get("Retry-After", 60))) continue ids = parse_notice_ids(r.text) # your parser for this portal fresh = [i for i in ids if i not in seen] yield from fresh if len(fresh) < len(ids) or not ids: break # reached notices we already hold page += 1 time.sleep(3) # a small public server: be slow - 04
Re-check open notices for corrigenda
Once a day, revisit every notice whose deadline has not passed. Store a changed deadline, a new lot or a cancellation as a new version of the record and alert on it as a change, not as a new tender.
- 05
Match across portals, then alert
Join national notices to TED notices on buyer, normalised title, estimated value and deadline. Alert once per tender, with links to every place it is published.
Setup
Recommended setup
Tender portals are light, public HTML. Keep the bulk on the cheapest pool and use residential exits only where a portal serves foreign or datacentre-looking visitors something else.
| Plan | Role | Session | Targeting | Price |
|---|---|---|---|---|
ARC Basic Rotating ISP | Default for portal sweeps and corrigendum checks Our lowest price per GB, with an exit in the portal's own country. Listing pages are small and a daily sweep across dozens of portals is bandwidth, not difficulty. | Sticky, 15 minutes per portal | Country | €0.75 / GB €0.59 at 1 TB (−21%) |
ARC Fast Rotating residential | For portals that treat ISP ranges differently Residential exits for the few platforms that answer an ISP range with an error or a reduced page. Move a portal here only after you see it fail. | Sticky, 15 minutes per portal | Country + city | €2.00 / GB €1.12 at 1 TB (−44%) |
Prices are list prices in EUR, excl. VAT. Volume discounts apply automatically at checkout — full ladder on the pricing page.
Connect
One connection string per market
The country, city and sticky session are tokens in the username. Change the token, change the market; keep the session handle, keep the IP.
http://USERNAME_pool-arcfast_country-de_session-k3f9a1x2_sesstime-10:PASSWORD@proxy.arcproxies.io:8000Swap _country-de for any market you quote. The full grammar is on the residential proxies page.
Illustration
What daily monitoring of 40 national and regional portals costs
An illustration with round numbers — your pages, payloads and retry rate will differ. The arithmetic is the real price ladder.
Assumptions
- Portals
- 40 across 8 countries
- Listing and notice pages per portal per day
- 800 at 100 KB
- Public notice attachments opened
- 300 a day at 2 MB
- TED
- via the free API, no proxy
- Cadence
- daily, 30 days
- Bandwidth per month
- ≈ 114 GB
On ARC Basic
€77.52per month, excl. VAT
114 GB bought in one purchase at €0.68 / GB (−9% volume rung, applied automatically).
The same 114 GB on ARC Fast: €155.04 at €1.36 / GB. Route only the jobs that need it through the pricier pool.
Attachments are the expensive part. Open them only for notices that already match your CPV filter, and never re-download a document whose link has not changed. More in how to reduce proxy bandwidth.
Bandwidth never expires, so a bigger single purchase is only ever cheaper per GB. See the ladder.
In depth
Public tenders, in depth
What TED covers, and what it leaves to national portals
TED (Tenders Electronic Daily) is the online edition of the Supplement to the Official Journal of the EU. According to the European Commission, it carries around 800,000 procurement notices a year, worth more than €815 billion, and the full set is available as open data in XML, with a CSV subset.
Since 25 October 2023, notices above the EU thresholds must use eForms, the structured standard set by Implementing Regulation (EU) 2019/1780. For a data team that is good news: one schema, machine-readable fields, and a public Search API that does not require a key for published notices.
The thresholds themselves are revised every two years; the values for 2026–2027 are set by Commission Delegated Regulation (EU) 2025/2152. Everything below them is published under national rules, which is why a complete feed for a sector always means crawling national and regional portals as well.
TED API or crawler: which source for which notice
| Source | What you get | Access | Proxy needed |
|---|---|---|---|
| TED Search API | All published EU notices, structured eForms fields | Free, no key for published notices | No |
| TED open data packages | Bulk XML of all notices, for backfills | Free download | No |
| National portals with feeds or exports | National notices, often including below-threshold | RSS, CSV or API where offered | Rarely |
| National and regional portals without feeds | Below-threshold and regional notices | Public HTML pages | Yes, an exit in the country |
| Buyer profile pages | Notices of one authority, sometimes only there | Public HTML pages | Yes, slow and polite |
Run the free sources first and let the crawler fill only the gaps. That keeps the proxy bill proportional to the part of the market nobody else structures for you. The general crawler patterns are on our web scraping page.
The UK after the Procurement Act 2023
The UK publishes its notices on its own Find a Tender service rather than TED. The Procurement Act 2023 came into force on 24 February 2025, and Find a Tender became the Central Digital Platform that buyers use to meet their transparency duties under the new regime.
Procurements started before that date continue under the old regulations, so for a while a UK feed carries notices under both regimes. Store which regime a notice falls under alongside its type, or your notice-type counts will not compare across the change.
Collecting tenders responsibly
- Public notices only. Anything behind a supplier login, including many tender document downloads, is for the bid team to fetch with their own account.
- One request at a time per portal, with a pause between pages. These are public services sized for people, and a crawl that slows them down is a crawl that will be blocked. See handling 429 rate limits.
- Read the portal's terms and robots.txt before you start; respecting robots.txt covers how to treat it.
- Contact names and phone numbers in notices are personal data. Store the buyer, not the person, unless you need the contact and have a lawful basis.
Our acceptable use policy applies to every request that goes through the network.
Works with anything that takes a proxy URL
- Python requests / httpx
- Scrapy
Crawlee
- n8n
- Docker
FAQ
Questions
Something not covered? Write to contact@arcproxies.io.
Do I need proxies to collect TED notices?
No. TED offers a public Search API that needs no key for published notices, and the full notice set is available as open data. Proxies are useful for the national, regional and buyer portals that publish below-threshold contracts and offer no feed.
Why are some tenders missing from TED?
Publication on TED is only mandatory for contracts above the EU thresholds, which are revised every two years. Smaller contracts are advertised under national rules, on national or regional portals or on the buyer's own profile, so they never appear on TED.
What are eForms?
eForms are the structured EU standard for procurement notices, introduced by Implementing Regulation (EU) 2019/1780 and mandatory for notices above the EU thresholds since 25 October 2023. They replaced the older standard forms and make notice fields machine-readable.
Can I automate downloading tender documents?
Only where the documents are public without an account. Many e-procurement platforms require supplier registration before download; automating around that is outside what we permit. Collect the public notice and let the bid team download documents with their own account.
How much bandwidth does tender monitoring use?
Listing and notice pages are light, typically around 100 KB. Forty portals swept daily with a few hundred public attachments opened comes to roughly 114 GB a month. Downloading every attachment for every notice is what multiplies it.
Sources & review
Last reviewed by the ARC Proxies team. Prices are read from the live catalog at build time.
- [1]European Commission: Tenders Electronic Daily
- [2]European Commission: eForms
- [3]Publications Office of the European Union: TED API documentation
- [4]European Commission, Public Buyers Community: Updated public procurement thresholds for 2026-2027 adopted by the European Commission
- [5]UK Government: Transforming Public Procurement
Web scraping
More on web scraping
Get started
Buy a gigabyte and try it on your own targets.
No subscription, no sales call. Code WELCOME15 takes 15% off your first purchase, and whatever you do not use is still there next quarter.