Skip to content

Commit a1a3ef8

Browse files
mdmintzmdmintz
committed
Add CDP Mode examples
1 parent 3ec337a commit a1a3ef8

12 files changed

+652
-0
lines changed

‎examples/cdp_mode/ReadMe.md

Lines changed: 301 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,301 @@
1+
<!-- SeleniumBase Docs -->
2+
3+
## [<img src="https://seleniumbase.github.io/img/logo6.png" title="SeleniumBase" width="32">](https://github.com/seleniumbase/SeleniumBase/) CDP Mode 🐙
4+
5+
🐙 <b translate="no">SeleniumBase</b> <b translate="no">CDP Mode</b> (Chrome Devtools Protocol Mode) is a special mode inside of <b><a href="https://github.com/seleniumbase/SeleniumBase/blob/master/help_docs/uc_mode.md" translate="no"><span translate="no">SeleniumBase UC Mode</span></a></b> that lets bots appear human while controlling the browser with the <b translate="no">CDP-Driver</b>. Although regular <span translate="no">UC Mode</span> can't perform <span translate="no">WebDriver</span> actions while the <code>driver</code> is disconnected from the browser, the <span translate="no">CDP-Driver</span> can still perform actions (while maintaining its cover).
6+
7+
👤 <b translate="no">UC Mode</b> avoids bot-detection by first disconnecting WebDriver from the browser at strategic times, calling special <code>PyAutoGUI</code> methods to bypass CAPTCHAs (as needed), and finally reconnecting the <code>driver</code> afterwards so that WebDriver actions can be performed again. Although this approach works for bypassing simple CAPTCHAs, more flexibility is needed for bypassing bot-detection on websites with advanced protection. (That's where <b translate="no">CDP Mode</b> comes in.)
8+
9+
🐙 <b translate="no">CDP Mode</b> is based on <a href="https://github.com/HyperionGray/python-chrome-devtools-protocol" translate="no">python-cdp</a>, <a href="https://github.com/HyperionGray/trio-chrome-devtools-protocol" translate="no">trio-cdp</a>, and <a href="https://github.com/ultrafunkamsterdam/nodriver" translate="no">nodriver</a>. <code>trio-cdp</code> was an early implementation of <code>python-cdp</code>, whereas <code>nodriver</code> is a modern implementation of <code>python-cdp</code>. (Refactored CDP code is imported from <a href="https://github.com/mdmintz/MyCDP" translate="no">MyCDP</a>.)
10+
11+
🐙 <b translate="no">CDP Mode</b> includes multiple updates to the above, such as:
12+
13+
* Sync methods. (Using `async`/`await` is not necessary!)
14+
* The ability to use WebDriver and CDP-Driver together.
15+
* Backwards compatibility for existing UC Mode scripts.
16+
* More configuration options when launching browsers.
17+
* More methods. (And bug-fixes for existing methods.)
18+
* Faster response time for support. (Eg. [Discord Chat](https://discord.gg/EdhQTn3EyE))
19+
20+
--------
21+
22+
### 🐙 <b translate="no">CDP Mode</b> initialization:
23+
24+
* `sb.activate_cdp_mode(url)`
25+
26+
> (Call that from a **UC Mode** script)
27+
28+
--------
29+
30+
### 🐙 <b translate="no">CDP Mode</b> examples:
31+
32+
> [SeleniumBase/examples/cdp_mode](https://github.com/seleniumbase/SeleniumBase/tree/master/examples/cdp_mode)
33+
34+
### 🔖 Example 1: (Pokemon site using Incapsula/Imperva protection with invisible reCAPTCHA)
35+
36+
> [SeleniumBase/examples/cdp_mode/raw_pokemon.py](https://github.com/seleniumbase/SeleniumBase/tree/master/examples/cdp_mode/raw_pokemon.py)
37+
38+
<div></div>
39+
<details>
40+
<summary> ▶️ (<b>Click to expand code preview</b>)</summary>
41+
42+
```python
43+
from seleniumbase import SB
44+
45+
with SB(uc=True, test=True, locale_code="en") as sb:
46+
url = "https://www.pokemon.com/us"
47+
sb.activate_cdp_mode(url)
48+
sb.sleep(1)
49+
sb.cdp.click_if_visible("button#onetrust-reject-all-handler")
50+
sb.cdp.click('a[href="https://www.pokemon.com/us/pokedex/"]')
51+
sb.sleep(1)
52+
sb.cdp.click('b:contains("Show Advanced Search")')
53+
sb.sleep(1)
54+
sb.cdp.click('span[data-type="type"][data-value="electric"]')
55+
sb.cdp.click("a#advSearch")
56+
sb.sleep(1)
57+
sb.cdp.click('img[src*="img/pokedex/detail/025.png"]')
58+
sb.cdp.assert_text("Pikachu", 'div[class*="title"]')
59+
sb.cdp.assert_element('img[alt="Pikachu"]')
60+
sb.cdp.scroll_into_view("div.pokemon-ability-info")
61+
sb.sleep(1)
62+
sb.cdp.flash('div[class*="title"]')
63+
sb.cdp.flash('img[alt="Pikachu"]')
64+
sb.cdp.flash("div.pokemon-ability-info")
65+
name = sb.cdp.get_text("label.styled-select")
66+
info = sb.cdp.get_text("div.version-descriptions p.active")
67+
print("*** %s: ***\n* %s" % (name, info))
68+
sb.sleep(2)
69+
sb.cdp.highlight_overlay("div.pokemon-ability-info")
70+
sb.sleep(2)
71+
sb.cdp.click('a[href="https://www.pokemon.com/us/play-pokemon/"]')
72+
sb.cdp.click('h3:contains("Find an Event")')
73+
location = "Concord, MA, USA"
74+
sb.cdp.type('input[data-testid="location-search"]', location)
75+
sb.sleep(1)
76+
sb.cdp.click("div.autocomplete-dropdown-container div.suggestion-item")
77+
sb.cdp.click('img[alt="search-icon"]')
78+
sb.sleep(2)
79+
events = sb.cdp.select_all('div[data-testid="event-name"]')
80+
print("*** Pokemon events near %s: ***" % location)
81+
for event in events:
82+
print("* " + event.text)
83+
sb.sleep(2)
84+
```
85+
86+
</details>
87+
88+
### 🔖 Example 2: (Hyatt site using Kasada protection)
89+
90+
> [SeleniumBase/examples/cdp_mode/raw_hyatt.py](https://github.com/seleniumbase/SeleniumBase/tree/master/examples/cdp_mode/raw_hyatt.py)
91+
92+
<div></div>
93+
<details>
94+
<summary> ▶️ (<b>Click to expand code preview</b>)</summary>
95+
96+
```python
97+
from seleniumbase import SB
98+
99+
with SB(uc=True, test=True, locale_code="en") as sb:
100+
url = "https://www.hyatt.com/"
101+
sb.activate_cdp_mode(url)
102+
sb.sleep(1)
103+
sb.cdp.click_if_visible('button[aria-label="Close"]')
104+
sb.sleep(0.5)
105+
sb.cdp.click('span:contains("Explore")')
106+
sb.sleep(1)
107+
sb.cdp.click('a:contains("Hotels & Resorts")')
108+
sb.sleep(2.5)
109+
location = "Anaheim, CA, USA"
110+
sb.cdp.press_keys("input#searchbox", location)
111+
sb.sleep(1)
112+
sb.cdp.click("div#suggestion-list ul li a")
113+
sb.sleep(1)
114+
sb.cdp.click('div.hotel-card-footer button')
115+
sb.sleep(1)
116+
sb.cdp.click('button[data-locator="find-hotels"]')
117+
sb.sleep(4)
118+
hotel_names = sb.cdp.select_all(
119+
'div[data-booking-status="BOOKABLE"] [class*="HotelCard_header"]'
120+
)
121+
hotel_prices = sb.cdp.select_all(
122+
'div[data-booking-status="BOOKABLE"] div.rate-currency'
123+
)
124+
sb.assert_true(len(hotel_names) == len(hotel_prices))
125+
print("Hyatt Hotels in %s:" % location)
126+
print("(" + sb.cdp.get_text("ul.b-color_text-white") + ")")
127+
if len(hotel_names) == 0:
128+
print("No availability over the selected dates!")
129+
for i, hotel in enumerate(hotel_names):
130+
print("* %s: %s => %s" % (i + 1, hotel.text, hotel_prices[i].text))
131+
```
132+
133+
</details>
134+
135+
### 🔖 Example 3: (BestWestern site using DataDome protection)
136+
137+
* [SeleniumBase/examples/cdp_mode/raw_bestwestern.py](https://github.com/seleniumbase/SeleniumBase/tree/master/examples/cdp_mode/raw_bestwestern.py)
138+
139+
<div></div>
140+
<details>
141+
<summary> ▶️ (<b>Click to expand code preview</b>)</summary>
142+
143+
```python
144+
from seleniumbase import SB
145+
146+
with SB(uc=True, test=True, locale_code="en") as sb:
147+
url = "https://www.bestwestern.com/en_US.html"
148+
sb.activate_cdp_mode(url)
149+
sb.sleep(1.5)
150+
sb.cdp.click_if_visible("div.onetrust-close-btn-handler")
151+
sb.sleep(0.5)
152+
sb.cdp.click("input#destination-input")
153+
sb.sleep(1.5)
154+
location = "Palm Springs, CA, USA"
155+
sb.cdp.press_keys("input#destination-input", location)
156+
sb.sleep(0.6)
157+
sb.cdp.click("ul#google-suggestions li")
158+
sb.sleep(0.6)
159+
sb.cdp.click("button#btn-modify-stay-update")
160+
sb.sleep(1.5)
161+
sb.cdp.click("label#available-label")
162+
sb.sleep(4)
163+
print("Best Western Hotels in %s:" % location)
164+
summary_details = sb.cdp.get_text("#summary-details-column")
165+
dates = summary_details.split("ROOM")[0].split("DATES")[-1].strip()
166+
print("(Dates: %s)" % dates)
167+
flip_cards = sb.cdp.select_all(".flipCard")
168+
for i, flip_card in enumerate(flip_cards):
169+
hotel = flip_card.query_selector(".hotelName")
170+
price = flip_card.query_selector(".priceSection")
171+
if hotel and price:
172+
print("* %s: %s => %s" % (
173+
i + 1, hotel.text.strip(), price.text.strip())
174+
)
175+
```
176+
177+
</details>
178+
179+
(<b>Note:</b> Extra <code translate="no">sb.sleep()</code> calls have been added to prevent bot-detection because some sites will flag you as a bot if you perform actions too quickly.)
180+
181+
(<b>Note:</b> Some sites may IP-block you for 36 hours or more if they catch you using regular <span translate="no">Selenium WebDriver</span>. Be extra careful when creating and/or modifying automation scripts that run on them.)
182+
183+
--------
184+
185+
### 🐙 CDP Mode API / Methods
186+
187+
(Some method args have been left out for simplicity. Eg: <code translate="no">timeout</code>)
188+
189+
```python
190+
sb.cdp.get(url)
191+
sb.cdp.reload()
192+
sb.cdp.refresh()
193+
sb.cdp.add_handler(event, handler)
194+
sb.cdp.find_element(selector)
195+
sb.cdp.find_all(selector)
196+
sb.cdp.find_elements_by_text(text, tag_name=None)
197+
sb.cdp.select(selector)
198+
sb.cdp.select_all(selector)
199+
sb.cdp.click_link(link_text)
200+
sb.cdp.tile_windows(windows=None, max_columns=0)
201+
sb.cdp.get_all_cookies(*args, **kwargs)
202+
sb.cdp.set_all_cookies(*args, **kwargs)
203+
sb.cdp.save_cookies(*args, **kwargs)
204+
sb.cdp.load_cookies(*args, **kwargs)
205+
sb.cdp.clear_cookies(*args, **kwargs)
206+
sb.cdp.sleep(seconds)
207+
sb.cdp.bring_active_window_to_front()
208+
sb.cdp.get_active_element()
209+
sb.cdp.get_active_element_css()
210+
sb.cdp.click(selector)
211+
sb.cdp.click_active_element()
212+
sb.cdp.click_if_visible(selector)
213+
sb.cdp.mouse_click(selector)
214+
sb.cdp.nested_click(parent_selector, selector)
215+
sb.cdp.get_nested_element(parent_selector, selector)
216+
sb.cdp.flash(selector)
217+
sb.cdp.focus(selector)
218+
sb.cdp.highlight_overlay(selector)
219+
sb.cdp.remove_element(selector)
220+
sb.cdp.remove_from_dom(selector)
221+
sb.cdp.remove_elements(selector)
222+
sb.cdp.scroll_into_view(selector)
223+
sb.cdp.send_keys(selector, text)
224+
sb.cdp.press_keys(selector, text)
225+
sb.cdp.type(selector, text)
226+
sb.cdp.evaluate(expression)
227+
sb.cdp.js_dumps(obj_name)
228+
sb.cdp.maximize()
229+
sb.cdp.minimize()
230+
sb.cdp.medimize()
231+
sb.cdp.set_window_rect()
232+
sb.cdp.reset_window_size()
233+
sb.cdp.get_window()
234+
sb.cdp.get_text()
235+
sb.cdp.get_title()
236+
sb.cdp.get_current_url()
237+
sb.cdp.get_origin()
238+
sb.cdp.get_page_source()
239+
sb.cdp.get_user_agent()
240+
sb.cdp.get_cookie_string()
241+
sb.cdp.get_locale_code()
242+
sb.cdp.get_screen_rect()
243+
sb.cdp.get_window_rect()
244+
sb.cdp.get_window_size()
245+
sb.cdp.get_window_position()
246+
sb.cdp.get_element_rect(selector)
247+
sb.cdp.get_element_size(selector)
248+
sb.cdp.get_element_position(selector)
249+
sb.cdp.get_gui_element_rect(selector)
250+
sb.cdp.get_gui_element_center(selector)
251+
sb.cdp.get_document()
252+
sb.cdp.get_flattened_document()
253+
sb.cdp.get_element_attributes(selector)
254+
sb.cdp.get_element_html(selector)
255+
sb.cdp.set_attributes(selector, attribute, value)
256+
sb.cdp.internalize_links()
257+
sb.cdp.is_element_present(selector)
258+
sb.cdp.is_element_visible(selector)
259+
sb.cdp.assert_element(selector)
260+
sb.cdp.assert_element_present(selector)
261+
sb.cdp.assert_text(text, selector="html")
262+
sb.cdp.assert_exact_text(text, selector="html")
263+
sb.cdp.save_screenshot(name, folder=None, selector=None)
264+
```
265+
266+
--------
267+
268+
### 🐙 CDP Mode WebElement API / Methods
269+
270+
```python
271+
element.clear_input()
272+
element.click()
273+
element.flash()
274+
element.focus()
275+
element.highlight_overlay()
276+
element.mouse_click()
277+
element.mouse_drag(destination)
278+
element.mouse_move()
279+
element.query_selector(selector)
280+
element.querySelector(selector)
281+
element.query_selector_all(selector)
282+
element.querySelectorAll(selector)
283+
element.remove_from_dom()
284+
element.save_screenshot(*args, **kwargs)
285+
element.save_to_dom()
286+
element.scroll_into_view()
287+
element.select_option()
288+
element.send_file(*file_paths)
289+
element.send_keys(text)
290+
element.set_text(value)
291+
element.type(text)
292+
element.get_position()
293+
element.get_html()
294+
element.get_js_attributes()
295+
```
296+
297+
--------
298+
299+
<img src="https://seleniumbase.github.io/cdn/img/sb_text_f.png" alt="SeleniumBase" title="SeleniumBase" align="center" width="335">
300+
301+
<div><a href="https://github.com/seleniumbase/SeleniumBase"><img src="https://seleniumbase.github.io/cdn/img/sb_logo_gs.png" alt="SeleniumBase" title="SeleniumBase" width="335" /></a></div>

‎examples/cdp_mode/__init__.py

Whitespace-only changes.

‎examples/cdp_mode/raw_async.py

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
import asyncio
2+
import time
3+
from seleniumbase.core import sb_cdp
4+
from seleniumbase.undetected import cdp_driver
5+
6+
7+
async def main():
8+
driver = await cdp_driver.cdp_util.start()
9+
page = await driver.get("https://www.priceline.com/")
10+
time.sleep(3)
11+
print(await page.evaluate("document.title"))
12+
element = await page.select('[data-testid*="endLocation"]')
13+
await element.click_async()
14+
time.sleep(1)
15+
await element.send_keys_async("Boston")
16+
time.sleep(2)
17+
18+
if __name__ == "__main__":
19+
# Call an async function with awaited methods
20+
loop = asyncio.new_event_loop()
21+
loop.run_until_complete(main())
22+
23+
# Call everything without using async / await
24+
driver = loop.run_until_complete(cdp_driver.cdp_util.start())
25+
page = loop.run_until_complete(driver.get("https://www.pokemon.com/us"))
26+
time.sleep(3)
27+
print(loop.run_until_complete(page.evaluate("document.title")))
28+
element = loop.run_until_complete(page.select("span.icon_pokeball"))
29+
loop.run_until_complete(element.click_async())
30+
time.sleep(1)
31+
print(loop.run_until_complete(page.evaluate("document.title")))
32+
time.sleep(1)
33+
34+
# Call CDP methods via the simplified CDP API
35+
page = loop.run_until_complete(driver.get("https://www.priceline.com/"))
36+
sb = sb_cdp.CDPMethods(loop, page, driver)
37+
sb.sleep(3)
38+
sb.internalize_links() # Don't open links in a new tab
39+
sb.click("#link_header_nav_experiences")
40+
sb.sleep(2)
41+
sb.remove_element("msm-cookie-banner")
42+
sb.sleep(1)
43+
sb.press_keys('input[data-test-id*="search"]', "Amsterdam")
44+
sb.sleep(2)
45+
sb.click('span[data-test-id*="autocomplete"]')
46+
sb.sleep(5)
47+
print(sb.get_title())

‎examples/cdp_mode/raw_bestwestern.py

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
from seleniumbase import SB
2+
3+
with SB(uc=True, test=True, locale_code="en") as sb:
4+
url = "https://www.bestwestern.com/en_US.html"
5+
sb.activate_cdp_mode(url)
6+
sb.sleep(1.5)
7+
sb.cdp.click_if_visible("div.onetrust-close-btn-handler")
8+
sb.sleep(0.5)
9+
sb.cdp.click("input#destination-input")
10+
sb.sleep(1.5)
11+
location = "Palm Springs, CA, USA"
12+
sb.cdp.press_keys("input#destination-input", location)
13+
sb.sleep(0.6)
14+
sb.cdp.click("ul#google-suggestions li")
15+
sb.sleep(0.6)
16+
sb.cdp.click("button#btn-modify-stay-update")
17+
sb.sleep(1.5)
18+
sb.cdp.click("label#available-label")
19+
sb.sleep(4)
20+
print("Best Western Hotels in %s:" % location)
21+
summary_details = sb.cdp.get_text("#summary-details-column")
22+
dates = summary_details.split("ROOM")[0].split("DATES")[-1].strip()
23+
print("(Dates: %s)" % dates)
24+
flip_cards = sb.cdp.select_all(".flipCard")
25+
for i, flip_card in enumerate(flip_cards):
26+
hotel = flip_card.query_selector(".hotelName")
27+
price = flip_card.query_selector(".priceSection")
28+
if hotel and price:
29+
print("* %s: %s => %s" % (
30+
i + 1, hotel.text.strip(), price.text.strip())
31+
)

0 commit comments

Comments
 (0)