Task 1 - Otodom scraper #4
Conversation
otodom/task_1/wk/fin.py
Outdated
| headers = { | ||
| "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36" | ||
| } |
There was a problem hiding this comment.
we should define constans at the beggining of the file with capital letters
otodom/task_1/wk/fin.py
Outdated
| regex = re.compile("Idź do strony [0-9]+$") | ||
| result = doc.find_all("a", {"aria-label": regex}) | ||
| pages_n = max(list(map(lambda x: int(x.string), result))) |
There was a problem hiding this comment.
well, definetly there is a better approach
You just could find the nav, and form its a childrens take the last element. The best way is always to find the most general solution when looking for the elements
otodom/task_1/wk/fin.py
Outdated
| if not response.ok: | ||
| with open("db.json", "w", encoding="utf-8") as file: | ||
| json.dump(scrapped_data, file, ensure_ascii=False, indent=4) | ||
| print("Already scrapped data saved in db.json") | ||
| print("Error occured on page " + str(i) + ". Aborting.") | ||
| sys.exit(1) |
There was a problem hiding this comment.
it may happen that for some reason 1 of 100 times the server response will simply fail on their site (5XX code). So do we really should exit on such circumstances?
otodom/task_1/wk/fin.py
Outdated
| tags = record.find_all("p") | ||
| for tag in tags: | ||
| if tag.has_attr("title"): |
There was a problem hiding this comment.
just find a tag with an attribute title instead of all
otodom/task_1/wk/fin.py
Outdated
| record_dict["promoted"] = ( | ||
| True if len(record.find_all("p", string="Podbite")) > 0 else False | ||
| ) |
There was a problem hiding this comment.
i dived into the html code. What if the site loaded in english? Or we would like to scrap in english? Then maybe it wouldn't work.
Finding single element p whose parent is a span i think would be more error-omitting approach
otodom/task_1/wk/fin.py
Outdated
| if len(zl) > 1: | ||
| record_dict["price"] = (zl[0].string + ", " + zl[1].string).replace("\xa0", " ") | ||
| else: | ||
| record_dict["price"] = "" |
There was a problem hiding this comment.
no need to scrap price /m^2 since we can simply divide price/area and get it
otodom/task_1/wk/fin.py
Outdated
| record_dict["estate_agency"] = ( | ||
| False if len(record.find_all("p", string="Oferta prywatna")) > 0 else True | ||
| ) |
There was a problem hiding this comment.
if there is an estate agency we would like to get its name
otodom/task_1/wk/generate_link.py
Outdated
| distance = settings["distance_radius"] | ||
| if distance != "None": | ||
| url += "?distanceRadius=" + str(distance) | ||
|
|
||
| # price | ||
| min_p = settings["price_min"] | ||
| max_p = settings["price_max"] | ||
| if min_p != "None": | ||
| url += "&priceMin=" + str(min_p) | ||
| if max_p != "None": | ||
| url += "&priceMax=" + str(max_p) |
No description provided.