WHOTS Metadata Scrapper

• This is a Scrapy Framework project for scraping some information from the Upper Ocean Processes Group / Project WHOTS - WHOI Hawaii Ocean Time-series Station website.

• The data is automatically saved at 00:00 on day-of-month 1. GitHub actions and saved at results/items.json

Prerequisites:

name: whots-metadata
channels:
  - conda-forge
  - defaults
dependencies:
  - scrapy
  - python

Installing:

conda create --prefix ./.env python scrapy -c conda-forge  

Usage:

• To run whots_metadata crawler:

conda activate ./.env      
cd src/whots_metadata
scrapy crawl whotsmetadata -O ../../results/items.json