Name	Name	Last commit message	Last commit date
Latest commit gunesacar Remove the unused notebook. Jul 15, 2019 c2fbdf8 · Jul 15, 2019 History 95 Commits
crawl_utils @ 6f5a46a	crawl_utils @ 6f5a46a	Add `englehardt/crawl_utils` as a submodule.	Oct 2, 2018
notebooks	notebooks	Remove HTTP responses notebook.	Oct 9, 2018
.gitignore	.gitignore	Initial commit	Jun 22, 2018
.gitmodules	.gitmodules	Add `englehardt/crawl_utils` as a submodule.	Oct 2, 2018
Crawl-Data-Metrics.ipynb	Crawl-Data-Metrics.ipynb	Update notebook with the 2019-06 crawl.	Jul 15, 2019
LICENSE	LICENSE	Initial commit	Jun 22, 2018
README.md	README.md	Update README.md	Jul 11, 2019
analyze_crawl.py	analyze_crawl.py	Combine scripts that check visit_id's and fix alexa ranks into existi…	Oct 3, 2018
batch-process-crawl-data.sh	batch-process-crawl-data.sh	Add the script we use to automate crawl data preprocessing and the an…	Oct 4, 2018
batch-process.sh	batch-process.sh	Use LZ4HC -9 compression, update data paths.	Oct 3, 2018
compute_metrics.py	compute_metrics.py	Add notebook with the metric plots.	Aug 28, 2018
convert-bz2-to-lz4.sh	convert-bz2-to-lz4.sh	Add a script to batch process crawl data.	Jun 28, 2018
create_sample_dbs.py	create_sample_dbs.py	Add a module to create sample databases.	Sep 28, 2018
db_schema.py	db_schema.py	Print number of rows for each table	Jul 24, 2018
fix_alexa_ranks.py	fix_alexa_ranks.py	Use the correct URL structure for Alexa lists after May 2018.	Oct 1, 2018
manage_users.py	manage_users.py	Add a module to manage user credentials and access.	Sep 28, 2018
normalize_db.py	normalize_db.py	Print number of rows for each table	Jul 24, 2018
process_crawl_data.py	process_crawl_data.py	Stop unifying databases, health-check the 2018-11 stateless crawl.	Nov 28, 2018
util.py	util.py	Add notebook with the metric plots.	Aug 28, 2018

Repository files navigation

openwpm-data-release

Code for Princeton Web Census data release.

To process new census crawls:

Decompress the tar.bz2 archive.
Run python analyze_crawl.py crawl_dir out_dir

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

openwpm-data-release

About

Releases

Packages

Contributors 2

Languages

License

citp/openwpm-data-release

Folders and files

Latest commit

History

Repository files navigation

openwpm-data-release

About

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Contributors 2

Languages

Packages