Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rewrote the package, added download feature too #114

Merged
merged 2 commits into from
Oct 10, 2024
Merged

rewrote the package, added download feature too #114

merged 2 commits into from
Oct 10, 2024

Conversation

rohitnairtech
Copy link
Contributor

@rohitnairtech rohitnairtech commented Oct 9, 2024 •
edited
Loading

Key changes

  1. Added support for download
  2. Shuffle between userAgents to avoid detection
  3. Added default puppeteer configs
  4. Renamed scraper.js inside of google to index.js (dont want to write scrapers in the import lazy :) )
  5. Refactored winston logger
  6. Added axios package 📦 for downloading
  7. Added fs and path package to create directories and files in sync
  8. Removed _scrapePage _scrape _isScrollable _isButtonVisible _clickAllImages
  9. Added two new methods - downloadImages getImageUrl
  10. Added consistency to response for single string/array of string queries

And most importantly, the code works! 🎉

pevers
pevers approved these changes Oct 10, 2024
View reviewed changes
Copy link
Owner

@pevers pevers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great! Thank you so much. I'll get it merged and published

src/example.js
@@ -1,10 +1,10 @@
'use strict';

var Scraper = require('./google/scraper');
const Scraper = require('./google');
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, you can see that the first commit is 10 years old ;)

@pevers
Copy link
Owner

pevers commented Oct 10, 2024

Screw the tests, they were flaky anyway

@pevers pevers merged commit 5f650e0 into pevers:master Oct 10, 2024
0 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pevers pevers pevers approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@rohitnairtech @pevers