Design proposal for test coverage #22827
Comments
eleanorjboyd
added
area-testing
needs proposal
|
Do we have a timeline for implementation of the same, or is the design still under discussion and not firm enough to be on roadmap? |
|
Coverage has been implemented and is merged so you can try it now on the pre-release of the python extension! |
|
@eleanorjboyd , Hi, I've upgraded Python extension to |
|
is your vscode version: ^1.94.0-20240913? |
|
This one: I've run "Check for updates", but no updates available yet. |
|
Ah ok sorry- I didn't realize it but the coverage change requires vscode insiders due to some extra dependencies. To do so you can download insiders here and it will work from there https://code.visualstudio.com/insiders/. Otherwise this will be part of the next release- so sometime next week you will be able to access it on stable vscode |
|
Thanks, sounds great! I'll wait for the next stable VSCode release and try it out. |
|
Is this also functional for unittests via Django? I'm getting coverage.py is installed. |
|
@msmitherdc I will need to investigate coverage and Django together- thank you for surfacing this! |
|
link to issue to track Django and Coverage together: #24199 |
|
@eleanorjboyd , But I don't see "Run with coverage" option. |
|
For the future reference: I've switched to pre-release version of Python extension |
|
I have switched to the pre-release version of Python extension v2024.17.2024100202 (pre-release) but still getting the error while running with coverage UNITTEST ERROR: TEST_RUN_PIPE is not set at the time of unittest trying to send data. Please confirm this environment variable is not being changed or removed as it is required for successful test discovery and execution.TEST_RUN_PIPE =
Exception ignored in atexit callback: <function execute_eot_and_cleanup at 0x75097171edd0>
Traceback (most recent call last):
File " .vscode/extensions/ms-python.python-2024.17.2024100202-linux-x64/python_files/unittestadapter/execution.py", line 274, in execute_eot_and_cleanup
send_post_request(eot_payload, test_run_pipe)
File " .vscode/extensions/ms-python.python-2024.17.2024100202-linux-x64/python_files/unittestadapter/pvsc_utils.py", line 335, in send_post_request
raise VSCodeUnittestError(error_msg)
unittestadapter.pvsc_utils.VSCodeUnittestError: UNITTEST ERROR: TEST_RUN_PIPE is not set at the time of unittest trying to send data. Please confirm this environment variable is not being changed or removed as it is required for successful test discovery and execution.TEST_RUN_PIPE =
Found 1 test(s).
System check identified no issues (0 silenced).
ebz_env/ peg_env/lib/python3.10/site-packages/coverage/control.py:894: CoverageWarning: No data was collected. (no-data-collected)
self._warn("No data was collected.", slug="no-data-collected")
Traceback (most recent call last):
File " .vscode/extensions/ms-python.python-2024.17.2024100202-linux-x64/python_files/unittestadapter/pvsc_utils.py", line 342, in send_post_request
__writer.connect()
File " .vscode/extensions/ms-python.python-2024.17.2024100202-linux-x64/python_files/testing_tools/socket_manager.py", line 28, in connect
self._socket.connect(self.name)
FileNotFoundError: [Errno 2] No such file or directory
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File " .vscode/extensions/ms-python.python-2024.17.2024100202-linux-x64/python_files/unittestadapter/execution.py", line 404, in <module>
send_post_request(payload_cov, test_run_pipe)
File " .vscode/extensions/ms-python.python-2024.17.2024100202-linux-x64/python_files/unittestadapter/pvsc_utils.py", line 346, in send_post_request
raise VSCodeUnittestError(error_msg) from error
unittestadapter.pvsc_utils.VSCodeUnittestError: Error attempting to connect to extension named pipe /run/user/1002/python-test-results-234754ce8149ed53015a.sock[vscode-unittest]: [Errno 2] No such file or directory
Exception ignored in atexit callback: <function execute_eot_and_cleanup at 0x743d5fb2ed40>
Traceback (most recent call last):
File " .vscode/extensions/ms-python.python-2024.17.2024100202-linux-x64/python_files/unittestadapter/execution.py", line 274, in execute_eot_and_cleanup
send_post_request(eot_payload, test_run_pipe)
File " .vscode/extensions/ms-python.python-2024.17.2024100202-linux-x64/python_files/unittestadapter/pvsc_utils.py", line 346, in send_post_request
raise VSCodeUnittestError(error_msg) from error
unittestadapter.pvsc_utils.VSCodeUnittestError: Error attempting to connect to extension named pipe /run/user/1002/python-test-results-234754ce8149ed53015a.sock[vscode-unittest]: [Errno 2] No such file or directory |
|
And you don't override any variables? This message is a sign we can't find a env var we set to use for communication: UNITTEST ERROR: TEST_RUN_PIPE is not set at the time of unittest trying to send data. Please confirm this environment variable is not being changed or removed as it is required for successful test discovery and execution Secondly does it work with regular run and discovery? Both of these use the same TEST_RUN_PIPE so im surprised you can get those to work and not coverage |
|
The terminal command for coverage works fine , and also my tests are been discovered in VS code. Also, the Run Tests button component in the testing works fine, but Run Tests with coverage runs all the tests but at the end fails with above error .
|
|
Now if you try in either the released version (v2024.16.0) or pre-released version (v2024.17.0) . This error is coming even when the test are sucessful ----------------------------------------------------------------------
Ran 1 test in 1.386s
OK
Destroying test database for alias 'default'...
UNITTEST ERROR: TEST_RUN_PIPE is not set at the time of unittest trying to send data. Please confirm this environment variable is not being changed or removed as it is required for successful test discovery and execution.TEST_RUN_PIPE =
Exception ignored in atexit callback: <function execute_eot_and_cleanup at 0x7117c1e62ef0>
Traceback (most recent call last):
File ".vscode/extensions/ms-python.python-2024.16.0-linux-x64/python_files/unittestadapter/execution.py", line 274, in execute_eot_and_cleanup
send_post_request(eot_payload, test_run_pipe)
File ".vscode/extensions/ms-python.python-2024.16.0-linux-x64/python_files/unittestadapter/pvsc_utils.py", line 335, in send_post_request
raise VSCodeUnittestError(error_msg)
unittestadapter.pvsc_utils.VSCodeUnittestError: UNITTEST ERROR: TEST_RUN_PIPE is not set at the time of unittest trying to send data. Please confirm this environment variable is not being changed or removed as it is required for successful test discovery and execution.TEST_RUN_PIPE =
Found 1 test(s).
System check identified no issues (0 silenced).
Exception ignored in atexit callback: <function execute_eot_and_cleanup at 0x74ba1ac22e60>
Traceback (most recent call last):
File ".vscode/extensions/ms-python.python-2024.16.0-linux-x64/python_files/unittestadapter/execution.py", line 274, in execute_eot_and_cleanup
send_post_request(eot_payload, test_run_pipe)
File ".vscode/extensions/ms-python.python-2024.16.0-linux-x64/python_files/unittestadapter/pvsc_utils.py", line 346, in send_post_request
raise VSCodeUnittestError(error_msg) from error
unittestadapter.pvsc_utils.VSCodeUnittestError: Error attempting to connect to extension named pipe /run/user/1002/python-test-results-df021158d91dd91ca8bc.sock[vscode-unittest]: [Errno 2] No such file or directory
Finished running tests! |
|
I just tested out "Run With Coverage" on my python project. I have the sources for my app in a sub-folder so my However, when I use Run With Coverage it appear to inject I've not found any way around for this. Is it a bug? Or have I missed a configuration? |
|
See this section of the docs
|
|
I found that when I had Not sure why the debugger wouldn't break when |
|
the debugger doesn't work with coverage- it is just a general design problem as the two interfere. Understandable - hm let me think about how I could enable coverage without adding cov=. to make sure your args are not overwritten |
|
@thomthom can you open another issue and we can move the conversation there so I can address it specifically? And if you have a suggestion on the above problem of triggering discovery with pytest-cov without overwritten exisiting args let me know, thanks! |
With the creation of the coverage API in vscode core, it is now possible to enable coverage for the python extension specifically. Below is the proposed design for how test coverage will work and includes some outstanding questions. Comments welcome as we are still in the design stage.
UX
Users can configure their coverage arguments in a similar way to how all run/debug/discovery args for testing. With the redesign of how testing arguments work with the extension of custom arguments this will be the entry point for coverage arguments. There is a new TestRunProfile kind of type coverage which will allow users to configure coverage specific configs that we will use to run coverage.
Secondly the entrypoint to run will be the buttons and other surfaces implemented in core and the default coverage config will be used unless a different one selected.
Implementation:
To implement coverage, the coverage.py library will be used. Our proposal is to not ship coverage.py and instead take the same approach as with pytest where the user must install coverage.py in their environment to use. This means that to utalize, a user will need to 1- install coverage.py, 2- create a coverage config, 3- select to run coverage.
Engineering design:
User coverage args will be sent to the python subproccess in the same means as any test args
if coverage.py is installed, and the coverage args are provided, then coverage will be calculated as described below
using the coverage.py API the python subprocess would then build a coverage payload
the payload design would include:
This would mean that coverage would support line based coverage (statement is the finite unit for coverage by vscode but python's smallest unit is line)
Due to the fact that branch coverage only lists the covered branch, and not the branches missing coverage, branch coverage will not be included at this time. If total branches can be calculated this can be reconsidered as an additional coverage stat provided.
The coverage payload will be read by the extension and displayed to the users
Other Design Concept:
The following idea was also considered: instead of using payloads for communication instead the team considered using the coverage file, like .coverage converted to XML and then read by the extension to parse the coverage and display. The pro of this would be the increase in separation between coverage and the base testing implementation that might make it easier to break out the coverage work into its own extension in the future. The con to the idea is that it will require the testing subprocess to send via information over a socket/pipe and a file thus duplicating the means of communication. It would also mean that the extension would do the heavy lifting in regards to taking the XML coverage data and converting it into the correct information design instead of doing this work on the python side. This fact would be contrary to our goal of increasing our percentage of python in the extension. ultimately we are leaning towards the first design outlined but are still open to discussion.
Given the proposal the following questions still exist:
The text was updated successfully, but these errors were encountered: