Skip to content

Conversation

@saattrupdan
Copy link
Owner

Changed

  • We now use the Boligsiden API instead of scraping the website. This makes the
    application much faster and more reliable, as we no longer have to deal with
    changes to the website layout. It also puts less load on the Boligsiden servers.

Fixes #3

@saattrupdan saattrupdan self-assigned this Apr 24, 2025
@saattrupdan saattrupdan requested a review from Copilot April 24, 2025 07:31
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR replaces the website scraping logic with API calls to the Boligsiden API to improve performance and reliability. Key changes include:

  • Removal of Selenium‑based webdriver functionality and corresponding tests.
  • Updated URL construction in the data models and scraper to use the API endpoint.
  • Adjustments in the CLI and changelog to reflect the new API-based approach.

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated no comments.

Show a summary per file
File Description
tests/test_webdriver.py Removed outdated webdriver tests.
tests/test_scraper.py Removed tests for the scraper’s old extraction logic.
tests/test_data_models.py Updated URL strings to use the API endpoint.
src/bolig_ping/webdriver.py Removed the obsolete webdriver module.
src/bolig_ping/scraper.py Replaced Selenium scraping with direct API requests.
src/bolig_ping/data_models.py Revised URL construction and property type formatting.
src/bolig_ping/cli.py Adjusted input normalization by updating city name formatting.
pyproject.toml Removed Selenium dependency.
README.md Updated the code coverage badge.
CHANGELOG.md Updated changelog to describe API usage changes.
Comments suppressed due to low confidence (3)

src/bolig_ping/data_models.py:82

  • The URL construction assumes the base URL always contains a query parameter separator. Consider using a URL builder (e.g., urllib.parse) for more robust query parameter handling if the base URL format changes.
url += "&" + "&".join(f"{key}={value}" for key, value in non_trivial_arguments.items())

tests/test_scraper.py:1

  • The removal of tests for the extract_number function and other scraper logic may leave gaps in test coverage for the new API-based functionality. Consider adding tests that verify the API response handling in scrape_results.
"""Tests for the `scraper` module."""

src/bolig_ping/cli.py:141

  • The city name normalization logic has been changed from replacing spaces and non-ASCII characters to simply replacing dashes with spaces. Verify that this transformation meets the API requirements and does not introduce unintended formatting issues.
cities=[c.replace("-", " ").lower() for c in city],

@saattrupdan saattrupdan merged commit a8c3757 into main Apr 24, 2025
4 checks passed
@saattrupdan saattrupdan deleted the feat/use-api branch April 24, 2025 07:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use API instead of scraping

2 participants