Repositories list

• cc-webgraph

  Public
  Tools to construct and process Common Crawl webgraphs

  pagerankwebgraphcommoncrawl+ 3common-crawlcentrality-measureswebgraph-framework

  Java

  •

  Apache License 2.0

  •4•102•2•0•Updated Dec 4, 2025Dec 4, 2025

• cc-index-table

  Public
  Index Common Crawl archives in tabular format

  sqlsparkcolumnar-storage+ 3aws-athenaapache-parquetcommoncrawl

  Java

  •

  Apache License 2.0

  •14•124•7•1•Updated Dec 4, 2025Dec 4, 2025

• nutch

  Public

  commoncrawl/nutch’s past year of commit activity
  Common Crawl fork of Apache Nutch

  javabig-datahadoop+ 2web-crawlercommoncrawl

  Java

  •

  Apache License 2.0

  •1.3k•39•6•1•Updated Dec 4, 2025Dec 4, 2025

• crawler-commons

  Public
  A set of reusable Java components that implement functionality common to any web crawler

  Java

  •

  Apache License 2.0

  •88•2•0•2•Updated Dec 4, 2025Dec 4, 2025

• ia-hadoop-tools

  Public
  Web archiving tools on Hadoop

  Java

  •28•4•2•1•Updated Dec 3, 2025Dec 3, 2025

• ia-web-commons

  Public
  Web archiving utility library

  cdx-fileswarc-fileswat-files

  Java

  •

  Apache License 2.0

  •74•11•4•1•Updated Dec 3, 2025Dec 3, 2025

• cc-crawl-statistics

  Public
  Statistics of Common Crawl monthly archives mined from URL index files

  statisticscommoncrawlcommon-crawl

  Python

  •

  Apache License 2.0

  •16•203•2•0•Updated Dec 3, 2025Dec 3, 2025

• cc-citations

  Public
  Scientific articles using or citing Common Crawl data

  bibtexbibliographyopendata

  Jupyter Notebook

  •5•28•0•0•Updated Dec 1, 2025Dec 1, 2025

• web-languages

  Public
  Crowd-sourced lists of urls to help Common Crawl crawl under-resourced languages. See https://github.com/commoncrawl/web-languages-code/ for the code

  crawlinglanguage-detectiondataset

  84•65•3•1•Updated Nov 25, 2025Nov 25, 2025

• cc-webgraph-statistics

  Public
  Statistics of Common Crawl monthly Web Graphs

  Python

  •

  Apache License 2.0

  •1•5•2•0•Updated Nov 24, 2025Nov 24, 2025

• cc-notebooks

  Public
  Various Jupyter notebooks about Common Crawl data

  jupyter-notebookaws-athenacommoncrawl+ 3common-crawlwebarchivingwebgraph-framework

  Jupyter Notebook

  •

  Apache License 2.0

  •10•60•0•0•Updated Nov 22, 2025Nov 22, 2025

• warcio-s3

  Public
  Streaming WARC/ARC library for fast web archive IO

  Python

  •

  Apache License 2.0

  •65•0•0•1•Updated Nov 21, 2025Nov 21, 2025

• whirlwind-python

  Public
  A whirlwind tour of Common Crawl's data using Python

  pythontutorialarchive+ 1warc

  Python

  •

  Apache License 2.0

  •6•29•0•0•Updated Nov 20, 2025Nov 20, 2025

• webarchive-indexing

  Public
  Tools for bulk indexing of WARC/ARC files on Hadoop, EMR or local file system.

  cc-index

  Python

  •

  MIT License

  •11•6•0•1•Updated Nov 19, 2025Nov 19, 2025

• whirlwind-python-notebook

  Public
  A jupyter notebook illistrating the basics of Common Crawl's datasets.

  awss3open-datasets+ 1sagemaker

  Jupyter Notebook

  •

  Apache License 2.0

  •0•1•0•1•Updated Nov 17, 2025Nov 17, 2025

• cc-pyspark

  Public
  Process Common Crawl data with Python and Spark

  sparkpysparksparksql+ 5wetcommoncrawlcommon-crawlwarc-fileswat-files

  Python

  •

  MIT License

  •90•448•4•2•Updated Nov 13, 2025Nov 13, 2025

• cc-monitoring

  Public
  Code that monitors Common Crawl infrastructure

  Python

  •0•2•0•0•Updated Nov 7, 2025Nov 7, 2025

• cc-index-annotations

  Public
  Example code to join an annotation to a host or url index

  Python

  •0•1•0•0•Updated Nov 7, 2025Nov 7, 2025

• robotstxt-experiments

  Public
  How is the Robots Exclusion Protocol (robots.txt) used in the WWW? This projects tries to get some insights mining Common Crawl's robots.txt captures of the years 2016 – 2024.

  robots-txtrobotstxt

  Jupyter Notebook

  •

  MIT License

  •0•0•0•0•Updated Oct 23, 2025Oct 23, 2025

• presentations

  Public
  A collection of public presentations from the Common Crawl Foundation

  1•9•0•0•Updated Oct 22, 2025Oct 22, 2025

• cc-host-index

  Public
  Tools for working with the host index

  Python

  •2•11•2•0•Updated Oct 18, 2025Oct 18, 2025

• cc-nutch-example

  Public
  Apache Nutch example project to archive content in WARC files

  Shell

  •

  Apache License 2.0

  •2•3•0•0•Updated Oct 11, 2025Oct 11, 2025

• language-detection-cld2

  Public
  Natural language detection, Java bindings for CLD2

  natural-languagelanguage-detectionlanguage-identification

  Java

  •

  Apache License 2.0

  •2•14•2•0•Updated Sep 26, 2025Sep 26, 2025

• cc-downloader

  Public
  A polite and user-friendly downloader for Common Crawl data

  rustdownloadercommoncrawl

  Rust

  •

  Apache License 2.0

  •4•63•1•2•Updated Aug 17, 2025Aug 17, 2025

• web-languages-code

  Public
  The code used to generate templates for the web-languages repo https://github.com/commoncrawl/web-languages

  Python

  •

  Apache License 2.0

  •2•2•0•2•Updated Jul 11, 2025Jul 11, 2025

• cc-warcinfo-index-builder

  Public
  Code to build an index that maps warcinfo-id to crawl / warc

  Python

  •0•0•0•0•Updated May 27, 2025May 27, 2025

• cc-host-index-media

  Public
  Media files used in the README.d of cc-host-index

  HTML

  •0•0•0•0•Updated Apr 20, 2025Apr 20, 2025

• wac2025-cc-annotator-poster

  Public
  A proof of concept pipeline for WARC annotation

  Rust

  •

  Apache License 2.0

  •0•1•0•0•Updated Apr 10, 2025Apr 10, 2025

• wac2025-webgraph-workshop

  Public
  Introduction to WebGraphs - Workshop at the IIPC Web Archiving Conference 2025

  Shell

  •

  MIT License

  •0•3•0•0•Updated Apr 10, 2025Apr 10, 2025

• arc2warc-conversion

  Public
  Experiences converting Common Crawl's ARC files from the crawls 2008 - 2012 to the WARC format

  arcwarcwebarchive+ 4webarchivingwarc-fileswarc-formatarc-files

  0•0•0•0•Updated Apr 3, 2025Apr 3, 2025