Common Crawl: Data Formats, Access, Costs & Risks

What Is Common Crawl?

Why does Common Crawl matter to search engines, cybersecurity, market research, and artificial intelligence? Common Crawl is a nonprofit project at commoncrawl.org that collects publicly accessible web pages and releases the resulting crawl data as a web crawl dataset anyone can analyze. Think of Common Crawl as large snapshots of the open web, not a live copy of every website.

According to Common Crawl’s overview, its archive exceeds 10 PiB, and a typical crawl contains more than two billion pages. Recent archives can represent hundreds of tebibytes of uncompressed content.

This guide explains:

• How Common Crawl collects and organizes pages
• What WARC, WAT, and WET files contain
• How to query indexes without downloading an entire crawl
• Where cloud and processing costs arise
• How to filter crawl data for business or AI projects
• What copyright, privacy, and compliance teams should review

Source page reviewed in Chrome during article research. Follow the image link for the current page.

How Common Crawl Builds a Web Crawl Dataset

Common Crawl operates CCBot, an automated crawler built to visit public URLs. It checks a site’s robots.txt rules before requesting allowed pages. According to the project’s official FAQ, the crawler uses HTTP GET requests but does not execute JavaScript or use cookies.

A crawl is incomplete by design. A page that requires login, depends on client-side rendering, or was not selected during a crawl may be absent. Even a captured page may lack images, scripts, or later updates. Common Crawl is a sample of the web at particular times, not an authoritative backup service.

A crawl cycle:

1. CCBot builds a queue of candidate public URLs.
2. It checks each host’s robots instructions and applies crawl-rate controls.
3. It requests permitted resources and records HTTP responses.
4. The system packages responses and crawl metadata into compressed files.
5. Common Crawl publishes the archive and searchable indexes.

The result is organized into named releases such as CC-MAIN-2026-17. A release may contain page captures gathered over several days. For example, the April 2026 archive contained 2.19 billion pages, covered 43.2 million hosts, and represented 379.2 TiB of uncompressed content.

Expectation	Reality
Every public page is included	Each crawl is a large sample
Captures show the current site	Every record has a record time
Browser-rendered content is preserved	JavaScript is not executed during collection
A page appears only once	The same URL may occur in many crawl releases

Common Crawl Data Structure: WARC Files, WAT Files, and WET Files

First, choose between WARC, WAT, and WET files. Downloading raw WARC files for plain text is like ordering a warehouse for one box.

Common Crawl’s Get Started documentation divides its crawl data into three main formats:

Format	What it contains	Good fit for
WARC	Raw HTTP requests, responses, headers, payloads, and crawl records	Archiving, forensic analysis, page reconstruction, and custom parsing
WAT	Computed metadata derived from WARC records, including links and HTTP details	Link analysis, domain research, and metadata studies
WET	Extracted plaintext derived from eligible WARC responses	Language analysis, search experiments, and text-corpus preparation

A WARC record can preserve the target URL, record date, response status, MIME type, payload digest, HTTP headers, and returned content. It is the richest but most expensive format to move and process.

WAT files are smaller because they describe captures without reproducing every response body. They provide redirect patterns and outgoing links without requiring teams to parse billions of HTML documents.

WET files contain extracted text. They suit natural-language processing, but extraction is imperfect. Navigation labels, cookie notices, repeated footers, and malformed text can remain. WET is a starting point, not a clean business dataset.

The official Get Started page explains access locations and the WARC, WAT, and WET formats.

How to Use the Common Crawl Index Before Downloading

The Common Crawl index makes the dataset manageable. For captures from 500 company domains, do not download and locally search hundreds of terabytes.

The CDXJ Index is designed for looking up individual URLs and captures. Common Crawl publishes a separate index for each crawl rather than one global index covering every release. A CDXJ response can include:

• Original URL and normalized URL key
• Record timestamp
• HTTP status and MIME type
• Detected language and character encoding
• Payload digest
• WARC filename
• Byte offset and record length

The last three fields enable selective retrieval. With a WARC filename, offset, and length, an HTTP Range request can download only that compressed record. The official CDXJ guide provides query and range-request examples.

Common Crawl also provides a URL Index, formerly called the Columnar Index. It suits large analytical workloads that scan or aggregate many records. The trade-off:

Access approach	Best use	Main limitation
CDXJ API	A domain, URL pattern, or small record set	Rate limits and many requests at scale
CDXJ files on S3	Custom index processing	Requires storage-aware code
URL Index	Large scans and analytical queries	More setup than a simple API call
Complete file lists	Bulk crawl processing	Large compute and transfer requirements

Common Crawl asks users to pause between API calls, avoid parallel requests from one IP address, and avoid proxy networks. For production-scale work, process index files in AWS rather than using the public endpoint for batch processing.

How to Access Common Crawl Data and Estimate Cost

The crawl data itself is free. According to commoncrawl.org/get-started, it is hosted in the s3://commoncrawl/ bucket in AWS us-east-1. Users outside AWS can retrieve files without an AWS account through https://data.commoncrawl.org/.

Three access patterns:

1. Query and retrieve selected records. Search a crawl index, filter the results, then issue byte-range requests for matching WARC records. This is usually the best first project.

2. Download selected WET or WAT files. Use this when a representative sample suffices and exact URLs are not required in advance.

3. Process a full crawl in AWS. Run Spark, Hadoop, EMR, or another distributed system in us-east-1, close to the S3 bucket.

Cost area	What creates the bill	How to control it
Dataset access	HTTP access is free; S3 API access requires authentication	Start with HTTP for small tests
Compute	Parsing, decompression, deduplication, and model preparation	Filter through indexes before parsing content
Storage	Raw files, intermediate outputs, and backups	Retain derived fields instead of every payload
Network	Inter-region routing, cloud egress, load balancers, or Elastic IP traffic	Process in us-east-1 and review AWS network pricing
Operations	Failed jobs, retries, logs, and engineering time	Test on a few files and set budget alerts

“Free data” does not mean a free project. A full crawl can contain hundreds of tebibytes before compression. Start with a measurable question, a domain allowlist, and a storage ceiling. A small pilot often reveals that only WAT metadata or WET text is needed.

A Diagram-Ready Common Crawl Data Pipeline

A sound Common Crawl pipeline discards irrelevant material early. That lowers cost and reduces the amount of questionable or sensitive content entering downstream systems.

flowchart LR
    A[Choose crawl release] --> B[Query CDXJ or URL Index]
    B --> C[Filter domains, dates, status, MIME, language]
    C --> D[Retrieve WARC ranges or selected WAT/WET files]
    D --> E[Parse and normalize]
    E --> F[Deduplicate and score quality]
    F --> G[Remove sensitive, unsafe, or restricted content]
    G --> H[Human and compliance review]
    H --> I[Approved analytics, search, or AI dataset]
    I --> J[Track provenance, retention, and deletion]

A practical setup:

1. Define the business purpose, permitted sources, time range, and fields you need.
2. Choose one crawl release and query its index for candidate records.
3. Filter for successful responses, expected MIME types, relevant languages, and approved domains.
4. Retrieve only the matching ranges or derived files.
5. Normalize encodings, strip repeated page furniture, and reject parsing failures.
6. Deduplicate by digest, normalized URL, and near-duplicate text similarity.
7. Scan for personal data, credentials, malware indicators, hate content, and licensing restrictions.
8. Record the crawl ID, record time, source URL, digest, transformation steps, and deletion status.
9. Validate a human-reviewed sample before releasing the output to users or models.

For example, a cybersecurity team studying exposed login portals could begin with WAT metadata and status codes. A customer-support team building a product manual search tool should instead restrict retrieval to its own approved documentation domains. A market analyst might use WET text, but should keep publication dates and source URLs so claims can be checked.

Filtering Crawl Data for Useful Results

Raw Common Crawl data is noisy. It contains duplicate pages, machine-generated sites, expired domains, error documents, spam, and text in thousands of layouts. Most of the work lies here.

Use layered filters, not one opaque “quality score”:

• Scope: Restrict hosts, URL paths, record dates, and document types.
• HTTP quality: Prefer status 200 and investigate redirects instead of silently merging them.
• Content: Detect language, minimum text length, repeated templates, and malformed extraction.
• Duplication: Combine exact payload digests with near-duplicate detection.
• Safety: Detect personal data, secrets, malware, sexual content, and abusive material.
• Rights: Apply domain exclusions, takedown lists, contractual limits, and the Common Crawl Opt-Out Registry.
• Provenance: Preserve enough metadata to trace every retained item to its source record.

Consider four projects. A fraud team can examine WAT links to find clusters of newly created domains that redirect to the same destination. An SEO team can compare historical title text from selected company sites. A research group can measure terminology changes in public policy pages. An LLM team can build a domain-limited corpus, but only after removing duplicates, low-quality text, personal information, and excluded sources.

Set acceptance metrics before processing. A pilot might require at least 95% correct language labels, fewer than 2% parsing failures, and zero known secrets in a reviewed sample. These are project thresholds, not Common Crawl guarantees.

Check	What to verify	Why it matters
Business purpose	Each retained field supports an approved use	Prevents unnecessary collection
Sample accuracy	Humans review random and high-risk records	Finds errors hidden by averages
Source traceability	Crawl, URL, timestamp, and digest are retained	Supports investigation and deletion
Exclusion updates	Opt-outs and internal blocklists are reapplied	Archived data can outlive new restrictions
Retention	Raw and derived data have deletion dates	Reduces privacy and security exposure

Robots, Copyright, Privacy, and Compliance

Robots rules govern crawler access; they do not grant a copyright license. A page being public, present in Common Crawl, or allowed by robots.txt does not automatically permit every downstream use in every country.

Common Crawl says CCBot identifies itself as CCBot/2.0, respects robots exclusions, and does not bypass paywalls or log in. A publisher can block future crawling with:

User-agent: CCBot
Disallow: /

Common Crawl also maintains an Opt-Out Registry for legal exclusion requests. Users should apply that registry and their suppression lists before each processing run. They should also enable investigation and removal of source records from derived systems.

Before using crawl data, ask:

• Does copyright law or a license permit the planned copying and transformation?
• Could the records contain names, contact details, health information, account data, or other personal information?
• Is the organization acting as a controller, processor, or independent data user under applicable privacy law?
• Do GDPR deletion rights, California privacy rules, sector contracts, or local retention laws apply?
• Will the result expose source text, or only publish aggregate findings?
• Can the team locate every derivative copy after an opt-out or valid legal request?

For HIPAA, FedRAMP, SOC 2, or ISO 27001 programs, do not assume an open dataset belongs inside an approved environment. Document its origin, scan it as untrusted input, restrict access, encrypt stored files, log transformations, and apply retention controls. Common Crawl provides infrastructure, not a compliance determination or legal clearance. Legal counsel should review high-risk commercial and AI uses.

Common Crawl as LLM Training Data

Common Crawl became widely known because filtered versions have been used in large language model research. Its official history says Google’s C4 dataset was constructed from one crawl snapshot and that filtered Common Crawl supplied a large share of GPT-3’s training tokens. The same page cites a 2024 Mozilla Foundation study finding that at least 64% of 47 surveyed language models released from 2019 through 2023 used filtered Common Crawl data.

The word “filtered” carries most of the weight. Common Crawl is raw source material, not ready-to-use LLM training data. An LLM pipeline may need:

• Document extraction and language identification
• Exact and semantic deduplication
• Quality and spam scoring
• Personal-data and secret removal
• Toxicity and unsafe-content controls
• Copyright and opt-out policy enforcement
• Benchmark contamination checks
• Source balancing to reduce domain and language bias

Turning a public web archive into a useful model corpus is impressive. It is also unsettling when provenance disappears. A responsible team keeps source lineage, publishes a dataset card, records exclusions, and tests whether the model reproduces personal or copyrighted passages.

Most businesses need not train a foundation model from Common Crawl. Safer alternatives often work better:

Goal	Better starting point
Answer questions about company policies	Retrieval over approved internal documents
Build product support search	A selected corpus from owned documentation
Study web-wide language or link patterns	Filtered Common Crawl data
Fine-tune a narrow classifier	Licensed, labeled examples
Train a general-purpose foundation model	Large governed corpus with Common Crawl as one reviewed source

Final Thoughts

Common Crawl is a vast, free web crawl dataset containing sampled public web pages. Its WARC files preserve raw responses, WAT files expose derived metadata, and WET files provide extracted text. Searchable indexes locate specific captures without downloading an entire crawl.

Begin small: choose one release, query the index, retrieve a limited set, and measure quality and cost. Keep provenance from the first experiment. Filter aggressively. Treat every downloaded record as untrusted input.

Separate technical access from permission to use. Robots rules, copyright, privacy law, contractual duties, and removal requests address different questions. Common Crawl can supply the raw material for useful research, security analysis, search, and AI. The organization must still decide what enters its business systems.