Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update README.md #80

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Update README.md #80

wants to merge 1 commit into from

Conversation

urvishp80
Copy link
Contributor

Guide for (this) Scraper Repository

@urvishp80 urvishp80 requested a review from kouloumos September 16, 2024 13:45
Copy link
Member

@kouloumos kouloumos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While the updated README introduces additional content, it doesn't seem to provide significant extra value or guidance to the repository. The increase in lines doesn't appear to justify the changes made.

Comment on lines +30 to +34
- [Python 3.8+](https://www.python.org/downloads/)
- [Node.js 14+](https://nodejs.org/)
- [pip](https://pip.pypa.io/en/stable/)
- [yarn](https://classic.yarnpkg.com/en/docs/install/)
- [Elasticsearch](https://www.elastic.co/downloads/elasticsearch)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We recently moved away from nodejs implemented scrapers, right? so nodejs and yarn are probably not needed in this list. Also elasticsearch is installed as part of requirements.txt, so I feel that this line will create confusion for people.

Comment on lines +115 to +122
- To run the mailing list scrapers, use the following commands based on the type of documents you want to scrape:
- **For Linux Foundation Documents**

Ensure that the `URL` environment variable is set to the appropriate mailing list URL (e.g., `https://lists.linuxfoundation.org/pipermail/lightning-dev/` or `https://lists.linuxfoundation.org/pipermail/bitcoin-dev/`).
Run the following command:
```bash
node mailing-list/main.js
```
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a depreciated scraper

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants