upwork-glrv

Scrape glrv for upwork client

Questions

Notes on main.py

Targets

Delivery 7

• https://www.literv.com/
• https://www.jerrystrailers.com/
• https://wolfecampers.com/
• https://www.northtrailrv.com/
• https://www.epicrvliquidators.com/

Delivery 6

• https://www.myrvta.com/

• https://www.schroedersbodyshop.com/

  • No obvious inventory. Found this link: https://www.schroedersbodyshop.com/skylinerv/

    which says: search for “Skyline RV and Auto Sales” so is this the target site?

    https://www.skylinetravelcentre.com/

• https://www.suncitytrailers.com/

• https://daviscampers.com/

  Details of changes in .py file

• https://www.whiteautosalesandrvs.com/

  Couldn't debug due to vpn not working with selenium

Delivery 5

• https://greattimervs.com/

• https://www.fmrv.com/

• https://www.terrystruck.com/

• http://www.bobsrvsales.com/

• http://www.swrv.com/

Delivery 4

• https://www.bestcampersales.com/

  

• http://amosrv.com/inventory/

• https://www.gloucesterrv.net/

• https://boerv.com/

• https://www.olrv.com/

Delivery 3

• https://gcrvc.com/

• https://abcmotors.com/

• www.pplmotorhomes.com

  • Follows this pattern in pagination: https://www.pplmotorhomes.com/used-rvs-for-sale/all-available-vehicles?sortBy=price%20desc&startIndex=24 , as each page has 24 items.

    So this is its logic for pagination:

    per_page = 24   #? PPLMOTORHOMES only
    for i in range(1, min(10, max_page+1) if QUICK_TEST else max_page+1):
        url = competitor_url % ((i-1) * per_page)   # set off is 0, p2 = 24, p3 = 48, ...
        driver.get(url)

• https://batesrv.com/

  • Location not available:

    

• https://lakecorv.com/

---

Delivery 2

Drive URL

• http://www.brucelitton.com/

  • Home shows only 3 items. has /New-Inventory and /Used-Inventory pages
  • Should I add those as a list in competitor_url_list
  • In /new some items have *** AVAILABLE IN STOCK ***, others have SOLD!! values, should I scrape them? if yes to which key?

• https://jjcamper.com/

  Inventory: https://jjcamper.com/inventory/

• https://campercorralms.com/

  • Some values return Consignment like vin, designation, or stocknumber. Should i replace with None ?

  • A price is listed is this msrp or sale price ?

    

• https://cooleysrvs.com/

  Broken

  

• https://www.easycampingrv.com/new-inventory

---

Delivery 1

• https://smithrv.com/inventory/

  • Catch TimeoutException in driver.get()

  

  • JS rendered content, pagination doesn't work, requires clicking on next at the end of the loop to work.

• https://www.glrv.com/rv-sales/

• https://lewistonrv.com/shop/

• https://www.buckeyerv.com/inventory-all/

• https://www.aokrvs.com/new-rvs

  • I extracted both designation & stocknumber (if available) from title.

  • I've included the part in the middle between year & stocknumber as rv_type - sometimes it's "5th Wheel" not sure if that's correct rv_type - but we can omit it if not very accurate.

  • 

  • If the part above title is model (I guess so) I can include it too.