Every source has a baseline
Each source is measured against what it normally returns. A source that quietly delivers less is flagged within the hour, before the numbers built on it drift.
Data engineering
Data pipelines that keep running after the demo
Your product needs data from many websites, matched into one clean record and kept current, and the scripts you have keep breaking without telling anyone. We build and run the pipelines that collect public web data at national scale, match it across sources and turn it into data products, the same way we run Mycarro across ten countries.
Typical lengthweeks
From€10,000 · £8,700 · $11,500
Built onPostgreSQL, Playwright, Queues
When teams call us
What usually triggers the work
“Our collectors break every week, and we find out when a customer asks why the numbers look wrong.” “We have the same thing from five sources and can't tell which records are duplicates.” “Nobody trusts the warehouse, so every team keeps its own spreadsheet.”
Sites change their layout or start refusing automated requests, and a collector rarely fails loudly: it keeps running and returns less. Matching that was good enough for one source falls apart at ten. Prices, counts and indices start to drift, and nobody can say which source caused it.
Leaving it costs more than the fix. Wrong numbers reach customers and partners, a data product loses the trust it was sold on, and engineers spend their weeks repairing scrapers instead of building. When a buyer's lawyer asks where the data came from and on what terms, there is no written answer.
What we deliver
In your repository, demoed every week
- Collection from many sources at national scale, kept runningA collector per source with its own schedule and rate limit, and an hourly report comparing each source with its usual volume. You can see the health of every source, every hour.
- Deduplication and record matching across sourcesRecords for the same real thing (a car, a property, a product, a company) folded into one, with every merge decision stored. You can ask why any two records were joined or kept apart, and see both error rates measured.
- A warehouse and APIs your team and customers can rely onPostgreSQL schemas that keep history instead of overwriting it, with documented APIs and exports. You can trace any number back to the source records behind it.
- Data products on top: indices, valuations, alertsValuations against comparable records, market indices built from history, and alerts when something changes. You can see the comparables behind each valuation and the method behind each index.
How it holds up
The questions a reviewer will ask, answered in the build
Runs are safe to repeat
Collection, matching and pricing run as separate stages on queues. A failed run is retried without double-counting, and a fixed parser can be run again over data already collected.
Every number has a source
Each record keeps where and when it was seen, and what changed since. A valuation, an index or a removal can be explained line by line, to your team or to a customer.
Within the law
How we collect web data
Buyers' lawyers ask how web data was collected, and few suppliers answer in writing. These are the rules we apply to every source, for our own products and for clients.
- Public data only. We collect what anyone can see without an account. We don't log in, get round a paywall or solve CAPTCHAs to reach data.
- robots.txt is followed. If a source's robots.txt disallows the pages we need, we don't collect them, and if it changes to disallow us, we stop. We have retired collectors of our own for this reason.
- Gentle on the source. Each source has its own rate limit, set so our collection doesn't burden the site. If a source asks us to slow down or stop, we do.
- Terms read first. We go through each source's terms of service with you before building a collector. Where the terms restrict automated access or reuse, your lawyer decides whether that source is used.
- Personal data kept out. Collection is designed to leave out names, phone numbers and emails of private people. If a product truly needs personal data, we collect the minimum, with a lawful basis under GDPR that you and your DPO sign off, a retention period and deletion.
- Answers, not copies. The data products we build sell what the data means (prices, indices, trends, alerts), not copies of other sites' adverts, text or photos.
- What we refuse. Data behind a login or paywall, profiling of individual people, building marketing lists from personal data, and copying another company's database to republish it.
This is how we work, not legal advice. Each source is checked with you, and with your lawyer, before we build.
Price
What it costs
Data builds start at €10,000 (£8,700 · $11,500). The first step is a free 30-minute call about your sources and the questions the data has to answer. After it you get a fixed written quote with milestones, and it changes only if the scope does. Servers and other running costs sit in your own cloud account, and the quote estimates them per month.
Questions
Before you book
Is collecting web data legal?
It depends on the country, the source's terms and the data itself. In the EU, the database right can protect a site's collection of data even where single facts are free to use, and GDPR applies as soon as personal data is involved, public or not. UK and US law differ again. We don't give legal advice: we go through each source with you and your lawyer before we build, and drop any source they aren't comfortable with.
Do you sell the data you collect?
No. We build data platforms for clients, and the client owns them: the code, the collectors, the database and the data, in your own accounts. We don't resell what we collect for you, or use it in our own products.
Can you take over a pipeline someone else built?
Yes. We start by reading the code and running it as it is, then add a health report per source before changing anything, so we both see which sources are failing and how badly. From there you get a fixed quote to repair, rebuild or retire each part. We need read access to the repository and the cloud account, and a list of who depends on the data.
Who owns the code and the IP?
You do, from the first commit. We work in your repository and your accounts, and the contract assigns the IP to you in the present tense, not at some later date.
Who will actually work on our project?
The engineers named in the proposal, and you meet them before you sign. A key-person clause means nobody is swapped without your written consent.
How do time zones work for a UK or US team?
We work on Lisbon time, which is the same clock as London: Mon–Fri, 08:00–18:00 UK time. That leaves about four working hours a day in common with the US East Coast. You get a demo of working software every week and a short written update every working day.
What contracts do you sign?
A master services agreement with a statement of work for each project, or your own paper if you prefer. A mutual NDA before we see anything, and a GDPR data processing agreement. US clients get a W-8BEN-E from us. Which law governs the contract is answered on Contracts & security.
How do we pay?
The fixed-price first step is paid upfront. Builds are billed by milestone, after you have seen the work running. Monthly care, if you want it, is billed at the start of each month. We invoice in euros, pounds or dollars.
Free 30-minute call
Tell us what has to work, and by when
You talk to an engineer who would work on it, not a salesperson. You leave the call knowing what it would take and what the first step costs, or we tell you that you don't need us.