The global machine learning market was valued at $55.8 billion in 2024 and is projected to reach $282 billion by 2030. The computer cost of training frontier AI models has crossed $100 million for a single run. According to a 2025 IBM Institute for Business Value report, the cost of computing for AI rose 89 percent between 2023 and 2025, with executives citing training data sourcing as the critical driver of that increase.
This is the context in which data collection decisions are made. When the cost of training a model is measured in millions and the business value of a production-ready AI system is measured in competitive advantage, the data collection decisions made before a single training run begins are among the highest-leverage choices in the entire project. Getting them wrong does not just produce a worse model. It produces a model that cannot be fixed by retraining on the same data and cannot be trusted in production on the inputs it was not exposed to during training.
Why Data Collection Is the Highest-Leverage Decision in an ML Project
Most ML teams spend the majority of their engineering time on model architecture, training infrastructure, and evaluation pipelines. Data collection is often treated as a precondition rather than a design decision, something to be solved quickly so the real work can begin. This framing inverts the actual leverage structure of an ML project.
A model architecture can be changed in a day. Training infrastructure can be scaled in hours. Fixing a training dataset that was collected from the wrong sources, at insufficient volume, or with quality problems that only surface after the model is in production requires going back to the collection layer and starting again. The IBM 2025 report found that 43 percent of chief operations officers now identify data quality issues as their most significant data priority, with more than a quarter of organisations estimating they lose over five million dollars annually to poor data quality. In an AI context, where poor training data produces models that systematically underperform or behave incorrectly on real-world inputs, the downstream cost compounds further.
Gartner estimates the average annual cost of poor data quality across organisations at between $12.9 million and $15 million. For AI teams, this figure understates the problem: unlike a bad database record that affects one transaction, a bad training example affects every inference the model makes for the lifetime of its deployment.
The collection decision also determines what is possible at every later stage. A model cannot learn from data it was never exposed to. Gaps in geographic coverage, category coverage, or temporal coverage in the training corpus become systematic blind spots in the deployed model. Addressing those blind spots after training requires returning to the collection layer, not rerunning training on the same data.
The Four Data Collection Methods for Machine Learning
ML teams have four primary options for sourcing training data. Each has a distinct profile of what it produces, what it costs, and where it falls short for commercial AI applications.
1. Open datasets and public corpora
Open datasets are the starting point for most ML projects. Common Crawl, Wikipedia, Hugging Face Datasets, Kaggle, and domain-specific academic repositories provide large-scale data with no collection overhead. For research, prototyping, and benchmarking, open datasets are indispensable. The limitations emerge when a commercial AI application requires domain-specific data, current data, or data with proprietary commercial signals that public repositories do not contain.
A pricing intelligence model trained on a public dataset from 2022 does not reflect current market conditions. A retail recommendation engine trained on a public product catalogue does not capture the specific inventory, regional availability, and pricing structures of the markets it needs to serve. Open datasets establish a useful baseline but rarely constitute sufficient training data for production of commercial AI systems on their own.
2. Official APIs
Platform APIs provide structured, authenticated access to data from specific services. Where they exist and provide the data needed, APIs are the cleanest collection method: the data is structured by the platform, rate limits are documented, and the terms of access are clear. The limitations are significant for ML use cases that require breadth across multiple platforms, historical depth, or data types that the platform chooses not to expose through its API.
Amazon's Product Advertising API, for example, is scoped to affiliate display use cases and prohibits bulk data collection or internal analytics. DigiKey's developer API requires registration and approval. Most major e-commerce and marketplace platforms either do not offer public APIs, restrict API access to their own seller ecosystem, or cap request volumes at levels that are insufficient for training-scale data collection. For ML teams that need product, pricing, or availability data across multiple platforms simultaneously, API coverage is fragmentary at best.
3. Web data collection at scale
Web data collection is the primary method for assembling domain-specific training corpora at commercial scale. It provides access to the full breadth of publicly available information across any platform, in any geography, and on any refresh cadence. For AI applications that need current, diverse, and platform-specific data, web collection is the only method that can deliver at training-relevant volumes without being constrained by platform API policies or the coverage limitations of public datasets.
The technical requirements for web data collection at ML training scale go beyond what a basic scraping script can satisfy. Anti-bot systems on commercially significant websites require residential proxy infrastructure, browser emulation, and CAPTCHA resolution. Parser stability across website structure changes requires ongoing maintenance. Geographic targeting for location-specific data requires proxy infrastructure across the relevant regions. And the output must be consistently structured, because inconsistent field formatting across sources creates noise that the model learns as a signal rather than an artefact. These requirements are addressed in more detail in the section on quality standards below.
4. Synthetic data
Synthetic data generation uses existing models or rule-based systems to produce training examples that augment or replace real-world data. It is particularly valuable for edge cases and rare events that are underrepresented in real data, for privacy-sensitive domains where real data cannot be used, and for scenarios where labelled data is expensive to produce manually. The limitation of synthetic data for commercial ML is that it reflects what the generating model already knows. A synthetic product catalogue generated by a language model reflects the model's prior over what product descriptions look like, not the actual distribution of product descriptions as they appear on Amazon or Walmart today. For applications that need to reflect current market reality, synthetic data can supplement but not replace real-world collection.
Talk to Syphoon about web data collection for your ML project.
What Makes Web-Collected Data Usable for Machine Learning
The quality of web-collected training data is not a single property. It is a set of dimensions, each of which affects model performance in a specific way. ML teams evaluating a data collection approach or provider need to assess quality across all of them, not just volume.
| Quality dimension | What it means | Effect of deficiency on model |
|---|---|---|
| Completeness | All required fields present for every record | Missing features produce gaps in the input space; model cannot generalise to records with those fields populated |
| Consistency | Same fields formatted the same way across all records and sources | Inconsistent formatting creates spurious variation that the model learns as a signal rather than an artefact |
| Accuracy | Field values correctly reflect the source data | Inaccurate training labels produce a model that systematically learns the wrong mapping |
| Freshness | Data collected recently enough to reflect current conditions | Stale data produces models that learned from historical patterns that no longer hold in production |
| Coverage | Sufficient representation across categories, geographies, and time periods | Underrepresented segments become blind spots: the model performs poorly on inputs from those segments |
| Structural stability | Output schema consistent even when source site structure changes | Parser breaks produce missing or malformed records that corrupt the training batch silently |
| Deduplication | No repeated records within or across sources | Duplicates cause the model to overweight certain examples, producing biased predictions |
Of these dimensions, structural stability deserves particular attention for ML teams using web-collected data. Most web scraping implementations are built against a snapshot of a website's HTML structure. When the website updates its front end, field values start returning incorrectly or not at all. In a traditional analytics context, this is noticed quickly because reports break visibly. In an ML training pipeline, malformed records may pass schema validation if they are not entirely missing, and the quality problem only surfaces when the trained model behaves unexpectedly on inputs that correspond to the corrupted records in the training set. The cost of discovering a data quality problem at evaluation or production time is significantly higher than catching it at collection time.
Why Web Collection Is the Primary Method for Commercial ML Applications
Commercial AI applications, particularly those in retail, e-commerce, market intelligence, and financial services, share a set of data requirements that distinguish them from research or general-purpose ML projects. They need data that is current, domain-specific, commercially grounded, and representative of the markets they will operate in. Open datasets rarely satisfy more than one of these requirements simultaneously, and official APIs rarely provide the breadth across platforms that production systems need.
Web collection addresses all four requirements. Current data is achievable because web collection can be scheduled at whatever refresh cadence the use case demands. Domain-specific data is achievable because collection can be targeted to the specific platforms, categories, and geographies relevant to the application. Commercial grounding comes naturally from collecting real market data rather than synthetic approximations. And market representativeness is achievable when the collection covers the actual platforms and regions where the model will operate.
The three ML application categories where this distinction matters most are pricing and demand forecasting, product understanding and recommendation, and market intelligence.
Pricing and demand forecasting
A pricing model or demand forecasting system trained on web-collected product data from Amazon, Walmart, and other major retailers learns from the actual price distribution and availability patterns that define the market. It learns how prices vary by region, how availability signals correlate with demand, and how competitor pricing responds to supply changes. None of this is available in public datasets or through restricted official APIs. The model's commercial value is directly proportional to the coverage and freshness of the underlying collection.
Product understanding and recommendation
Product recommendation systems, shopping assistants, and catalogue matching tools require training data that covers the actual product landscape of the markets they serve. This means product titles, descriptions, specifications, categories, and images from the platforms where the products are sold, not synthetic product descriptions generated from a language model's prior. The richer and more current the product data in the training corpus, the more accurately the model learns to understand product relationships, substitution patterns, and user intent.
Market intelligence
AI systems that provide market intelligence, competitive analysis, or trend forecasting require training data that reflects actual market conditions across time. Historical price series, availability patterns, new product launch signals, and category-level demand shifts are all derivable from structured web-collected e-commerce data at scale. The quality of the market intelligence the model produces is bounded by the coverage and freshness of the data it learned from.
Explore structured eCommerce data across 60 platforms for your ML application.
Structured E-commerce Data: The Highest-Value Web Input for Commercial ML
Among all web-collected data types, structured e-commerce data from major marketplace platforms has properties that make it particularly well-suited for ML training. Understanding these properties helps ML teams make more precise sourcing decisions rather than treating all web data as equivalent.
The first property is implicit labelling at scale. Every structured field in a product record is a label. A product's category breadcrumb is its classification label, usable directly for training a product taxonomy model. Its price relative to category average is a label for a pricing tier model. Its availability status is a label for a stock prediction model. Its review count and rating together are a proxy label for product quality and demand. This implicit labelling means that structured e-commerce data supports supervised learning tasks without the annotation overhead that text, image, or audio data typically requires.
The second property is commercial ground truth. Unlike synthetic data or academically constructed datasets, product data from live platforms reflects actual market conditions at the time of collection. The prices are what buyers are paying. The availability states are what the platform is showing to real customers. The category structures are how the platform's own taxonomy system classifies products. A model trained on this data learns from market reality rather than an approximation of it.
The third property is multi-platform compatibility. The same manufacturer part number, EAN, or product title may appear on Amazon, Walmart, Shopee, and TikTok Shop simultaneously at different prices and with different availability. When structured data from multiple platforms is collected with consistent field naming across sources, the cross-platform signals become training features in their own right. A model that learns how the same product is priced and positioned across platforms develops a richer understanding of market dynamics than one trained on a single source.
Syphoon collects structured product data from Amazon, Walmart, Shopee, TikTok Shop, Naver, and more than 55 additional e-commerce and marketplace platforms. All data is delivered with consistent field naming across platforms, covering pricing at volume break quantities, availability, specifications, seller and seller offer data, category paths, review metrics, and product images. For ML teams sourcing training data for retail, pricing, or market intelligence applications, this structured multi-platform data feeds directly into the collection layer of the pipeline without requiring custom parsers per platform or separate provider relationships for each marketplace.
From Collection to Training-Ready: What the Path Looks Like
Web-collected data does not go directly from collection to model training. The path from structured collection output to training-ready data involves several steps, and understanding them helps ML teams design their collection requirements more precisely.
Schema validation and field completeness checks
The first step after receiving collected data is validating that the expected fields are present and correctly formatted across all records. Any records with missing required fields are flagged for investigation: the problem may be a parser issue on a specific page type, a structural change on the source site, or a category of products where the field genuinely does not exist. Field completeness rates are the first quality signal that the collection layer is working correctly.
Deduplication
Web collection from overlapping sources, or collection of the same ASIN from multiple geographic targets, produces duplicate records that must be identified and resolved before training. Deduplication logic depends on what constitutes a duplicate in the context of the specific ML application. For a pricing model that needs one price per ASIN per location per day, records with the same ASIN and location collected on the same day are duplicates. For a model that needs to learn price variation across locations, the same ASIN at different ZIP codes is distinct.
Normalisation and type enforcement
Raw collected data frequently contains format inconsistencies that must be resolved before training. Prices may be returned as strings with currency symbols on some platforms and as floats on others. Category values may use different capitalisation conventions. Availability states may use different vocabulary across sources. Normalisation converts all fields to consistent types and value spaces, eliminating the spurious variation that the model would otherwise learn as a feature.
Train, validation, and test splits
For time-series-dependent applications like pricing models, the split between training, validation, and test data must respect temporal ordering. Using future data in the training set and past data in the test set produces artificially high evaluation metrics that do not hold in production. When the collection layer delivers timestamped records, the split can be implemented correctly as a temporal holdout rather than a random split.
Need structured web data for ML?
Join our Discord server
Connect with our team, discuss your use case, ask technical questions, and share feedback with a community of people working on similar problems.