Skip to content

[Feature]: Option to disable inspect.stack() calls for performance optimization #2744

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
jl-martins opened this issue Feb 15, 2025 · 5 comments · Fixed by #2835 · May be fixed by #2836
Closed

[Feature]: Option to disable inspect.stack() calls for performance optimization #2744

jl-martins opened this issue Feb 15, 2025 · 5 comments · Fixed by #2835 · May be fixed by #2836

Comments

@jl-martins
Copy link

jl-martins commented Feb 15, 2025 •
edited
Loading

🚀 Feature Request

Summary

playwright-python currently makes frequent calls to inspect.stack, which significantly impacts performance in certain use cases, such as web scraping with scrapy-playwright. This feature request proposes adding an option to disable stack inspections to improve execution speed when debugging information is not required.

Use case

Although Playwright is primarily designed for end-to-end testing, some projects leverage it for web scraping, where performance is a key concern. In my case, I am using scrapy-playwright, and profiling results from cProfile indicate that inspect.stack calls contribute to roughly 25% of the total execution time, as illustrated in the following icicle and call graph:

Image

Image

As far as I can tell, these calls are primarily used for debugging and are not critical for normal execution.

Proposed solution

Introduce an environment variable (e.g.: PW_INSPECT_STACK) that allows users to disable stack inspections when debugging is not required. The default value would be 1 to preserve the current behavior.

Motivation

  • Performance Boost: Reducing unnecessary function calls can significantly speed up Playwright for scraping-heavy applications.
  • Flexibility: Users who require stack traces for debugging can leave the option enabled.
  • Backward Compatibility: The default behavior remains unchanged, ensuring no disruption for existing users.
@luisferreira93
Copy link

+1

@mxschmitt
Copy link
Member

Do you have in comparison how much time the actual page.goto/click/fill etc's took? Lets say 1s in average with network resources around different origins and sometimes even iframes etc. This should be way larger and demonstrate that its not worth investing into that end (inspector.stack).

@neoncube2
Copy link
Contributor

I think I might be affected by this, too.

In my case, I load a page, check to make sure that a certain amount of buttons are visible, then execute a few thousand page.evaluate() calls in a tight loop.

Profiling results

Image

Use case

I have a game written in JavaScript, and I'm using Pytorch to train an AI to play the game. I use Playwright to load the page and then play the game via JS calls: await evaluate('doGameAction', action_index)

As an aside: Is evaluate() truly async? It seems like it uses a lot of CPU, and I'm guessing that internally it may involve a loop with a bunch of wait() calls to see if the browser has finished executing the JS code.

Thanks! :)

@neoncube2
Copy link
Contributor

I think I might have a fix for this that improves performance and doesn't require a special flag! :)

I'm currently running some tests, and if they're successful, I'll probably open a PR within the next few days, God willing.

This was referenced Apr 29, 2025
Merged
Open
@neoncube2
Copy link
Contributor

I've opened two PRs to fix this:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
4 participants
@neoncube2 @jl-martins @mxschmitt @luisferreira93