Improve performance and output of pyenv virtualenvs
#502
Conversation
Use the code from pyenv-versions for efficiency and consistent output. The main performance problem was in the call to pyenv-virtualenv-prefix, which called pyenv-prefix, which then enumerated every virtual environment. This was done inside a loop, compounding the problem. Simply the virtual environment listing so that it does not have to call pyenv-virtualenv-prefix anymore.
This is useful for listing the “frinedly” virtualenv names instead of the short path which includes the Python version.
…vironments Having the current virtual environments listed as options in the competion is noisy since only bare Python versions, such as 3.11.1, make sense as suggested completions for `pyenv virtualenv [version]`.
This makes the suggested completetions cleaner.
|
I could use some help getting the tests correct. I made a few changes but I don't understand why the |
|
Seems like only two tests are failing 😄 I just randomly checked into virtualenv. I recently did some perf improvements on the main I am not sure if this applies to your situation as well, haven't looked much into it. The main idea I have is:
|
|
The tests do need some adjusting since I've changed the output a bit. I'm just having a hard time figuring out what exactly |
|
Hehe, and the test file hasn't been touched in 10 years 😆 I did not encounter |
|
The code itself that was the source of the performance issue is 10-12 years old. Not that old code is bad, but there were some inefficiencies in there. |
|
No it's not that the old code is bad. It was probably good at the time it was written. It's more of a problem that the code has design principles which made sense back then, but fast forward 10 years and it's not as intuitive anymore. And if you're unlucky, you're facing tests which were quick-fix set up 10 years ago, and suddenly you need to fix the problem of ... It's a challenge for sure 😄 |
bin/pyenv-virtualenvs
Outdated
| if [ -z "$only_aliases" ]; then | ||
| for env_path in "${venv_dir_entries[@]}"; do | ||
| if [ -d "${env_path}" ]; then | ||
| print_version "${env_path#"${PYENV_ROOT}"/versions/}" "${env_path}" | ||
| fi | ||
| virtualenv_prefix="$(pyenv-virtualenv-prefix "${path##*/}" 2>/dev/null || true)" | ||
| if [ -d "${virtualenv_prefix}" ]; then | ||
| print_version "${path##*/}" " (created from ${virtualenv_prefix})" | ||
| done | ||
| fi | ||
|
|
||
| if [ -z "$skip_aliases" ]; then | ||
| for env_path in "${version_dir_entries[@]}"; do | ||
| if [ -d "${env_path}" ] && [ -L "${env_path}" ]; then | ||
| print_version "${env_path#"${PYENV_ROOT}"/versions/}" "${env_path}" | ||
| fi | ||
| for venv_path in "${path}/envs/"*; do | ||
| venv="${path##*/}/envs/${venv_path##*/}" | ||
| virtualenv_prefix="$(pyenv-virtualenv-prefix "${venv}" 2>/dev/null || true)" | ||
| if [ -d "${virtualenv_prefix}" ]; then | ||
| print_version "${venv}" " (created from ${virtualenv_prefix})" | ||
| fi | ||
| done | ||
| fi | ||
| done | ||
| done | ||
| fi |
There was a problem hiding this comment.
What happens with conda installations? They are currently listed, now they seem to stop being so.
Not sure why they are listed and if that's intended. Technically, each one contains a "base" environment -- but the same could be said about any other Python installation.
There was a problem hiding this comment.
I don't know anything about or use conda, so I omitted those. 😊
I will work on adding that back in as they should be listed in the output.
There was a problem hiding this comment.
Reading through the previous code, it seems like the intent was to look for certain indicator files and if they are present, list the directory as a virtual environment. I don't really know if that's a good heuristic or not.
Calling pyenv-virtualenv-prefix was what caused the program to take so long to run. If we think this behavior should remain, I'll have to reimplement this functionality using a more efficient approach.
There was a problem hiding this comment.
Hmm, the changes in this PR may actually be doing a better job with this than the current code.
I installed miniconda3 and created a new env.
pyenv install miniconda3-latest
pyenv shell miniconda3-latest
conda create -n my_conda_env
I only see my_conda_env in the second output.
Current output running master
> pyenv virtualenvs
3.10.2/envs/pyls-3.10.2 (created from /Users/sdoran/.pyenv/versions/3.10.2)
3.10.3/envs/ansible-2.12-3.10.3 (created from /Users/sdoran/.pyenv/versions/3.10.3)
3.10.3/envs/boto-3.10.4 (created from /Users/sdoran/.pyenv/versions/3.10.3)
3.10.3/envs/lexicon-3.10.3 (created from /Users/sdoran/.pyenv/versions/3.10.3)
3.10.4/envs/ansible-2.12-3.10.4 (created from /Users/sdoran/.pyenv/versions/3.10.4)
3.10.4/envs/ansible-dev-3.10.4 (created from /Users/sdoran/.pyenv/versions/3.10.4)
3.10.4/envs/flake8-3.10.4 (created from /Users/sdoran/.pyenv/versions/3.10.4)
3.10.4/envs/jello-3.10.4 (created from /Users/sdoran/.pyenv/versions/3.10.4)
3.10.4/envs/jsbeautifier-3.10.4 (created from /Users/sdoran/.pyenv/versions/3.10.4)
3.10.4/envs/youtube-dl-3.10.4 (created from /Users/sdoran/.pyenv/versions/3.10.4)
3.10.6/envs/ipython-3.10.6 (created from /Users/sdoran/.pyenv/versions/3.10.6)
3.10.6/envs/shiv-3.10.6 (created from /Users/sdoran/.pyenv/versions/3.10.6)
3.10.6/envs/speedtest-cli-3.10.6 (created from /Users/sdoran/.pyenv/versions/3.10.6)
3.11.1/envs/ansible-2.14-3.11.1 (created from /Users/sdoran/.pyenv/versions/3.11.1)
3.11.1/envs/ipython-3.11.1 (created from /Users/sdoran/.pyenv/versions/3.11.1)
3.11.1/envs/titlecase-3.11.1 (created from /Users/sdoran/.pyenv/versions/3.11.1)
3.11.1/envs/uptime-kuma-3.11.1 (created from /Users/sdoran/.pyenv/versions/3.11.1)
3.11.3/envs/ipython-3.11.3 (created from /Users/sdoran/.pyenv/versions/3.11.3)
3.11.3/envs/tasmota (created from /Users/sdoran/.pyenv/versions/3.11.3)
3.11.7/envs/tasmota-3.11.7 (created from /Users/sdoran/.pyenv/versions/3.11.7)
3.12.0/envs/ansible-2.16-3.12.0 (created from /Users/sdoran/.pyenv/versions/3.12.0)
3.12.0/envs/ipython-3.12.0 (created from /Users/sdoran/.pyenv/versions/3.12.0)
3.12.2/envs/borg (created from /Users/sdoran/.pyenv/versions/3.12.2)
3.12.2/envs/yt-dlp (created from /Users/sdoran/.pyenv/versions/3.12.2)
3.12.4/envs/qrcode (created from /Users/sdoran/.pyenv/versions/3.12.4)
3.12.8/envs/ansible-lint (created from /Users/sdoran/.pyenv/versions/3.12.8)
3.12.8/envs/cowsay (created from /Users/sdoran/.pyenv/versions/3.12.8)
3.12.8/envs/pyright-3.12.8 (created from /Users/sdoran/.pyenv/versions/3.12.8)
3.12.8/envs/ruff-3.12.8 (created from /Users/sdoran/.pyenv/versions/3.12.8)
3.13.1/envs/asciinema (created from /Users/sdoran/.pyenv/versions/3.13.1)
3.13.1/envs/diceware (created from /Users/sdoran/.pyenv/versions/3.13.1)
3.13.1/envs/ipython-3.13.1 (created from /Users/sdoran/.pyenv/versions/3.13.1)
3.6.15/envs/ipython-3.6.15 (created from /Users/sdoran/.pyenv/versions/3.6.15)
3.7.12/envs/appomni-dev-3.7.12 (created from /Users/sdoran/.pyenv/versions/3.7.12)
3.7.12/envs/ipython-3.7.12 (created from /Users/sdoran/.pyenv/versions/3.7.12)
3.8.12/envs/ipython-3.8.12 (created from /Users/sdoran/.pyenv/versions/3.8.12)
3.9.9/envs/ipython-3.9.9 (created from /Users/sdoran/.pyenv/versions/3.9.9)
ansible-2.12-3.10.3 (created from /Users/sdoran/.pyenv/versions/3.10.3)
ansible-2.12-3.10.4 (created from /Users/sdoran/.pyenv/versions/3.10.4)
ansible-2.14-3.11.1 (created from /Users/sdoran/.pyenv/versions/3.11.1)
ansible-2.16-3.12.0 (created from /Users/sdoran/.pyenv/versions/3.12.0)
ansible-dev-3.10.4 (created from /Users/sdoran/.pyenv/versions/3.10.4)
ansible-lint (created from /Users/sdoran/.pyenv/versions/3.12.8)
appomni-dev-3.7.12 (created from /Users/sdoran/.pyenv/versions/3.7.12)
asciinema (created from /Users/sdoran/.pyenv/versions/3.13.1)
borg (created from /Users/sdoran/.pyenv/versions/3.12.2)
boto-3.10.4 (created from /Users/sdoran/.pyenv/versions/3.10.3)
cowsay (created from /Users/sdoran/.pyenv/versions/3.12.8)
diceware (created from /Users/sdoran/.pyenv/versions/3.13.1)
flake8-3.10.4 (created from /Users/sdoran/.pyenv/versions/3.10.4)
ipython-3.10.6 (created from /Users/sdoran/.pyenv/versions/3.10.6)
ipython-3.11.1 (created from /Users/sdoran/.pyenv/versions/3.11.1)
ipython-3.11.3 (created from /Users/sdoran/.pyenv/versions/3.11.3)
ipython-3.12.0 (created from /Users/sdoran/.pyenv/versions/3.12.0)
ipython-3.13.1 (created from /Users/sdoran/.pyenv/versions/3.13.1)
ipython-3.6.15 (created from /Users/sdoran/.pyenv/versions/3.6.15)
ipython-3.7.12 (created from /Users/sdoran/.pyenv/versions/3.7.12)
ipython-3.8.12 (created from /Users/sdoran/.pyenv/versions/3.8.12)
ipython-3.9.9 (created from /Users/sdoran/.pyenv/versions/3.9.9)
jello-3.10.4 (created from /Users/sdoran/.pyenv/versions/3.10.4)
jsbeautifier-3.10.4 (created from /Users/sdoran/.pyenv/versions/3.10.4)
lexicon-3.10.3 (created from /Users/sdoran/.pyenv/versions/3.10.3)
miniconda3-latest (created from /Users/sdoran/.pyenv/versions/miniconda3-latest)
pyls-3.10.2 (created from /Users/sdoran/.pyenv/versions/3.10.2)
pyright-3.12.8 (created from /Users/sdoran/.pyenv/versions/3.12.8)
qrcode (created from /Users/sdoran/.pyenv/versions/3.12.4)
ruff-3.12.8 (created from /Users/sdoran/.pyenv/versions/3.12.8)
shiv-3.10.6 (created from /Users/sdoran/.pyenv/versions/3.10.6)
speedtest-cli-3.10.6 (created from /Users/sdoran/.pyenv/versions/3.10.6)
tasmota (created from /Users/sdoran/.pyenv/versions/3.11.3)
tasmota-3.11.7 (created from /Users/sdoran/.pyenv/versions/3.11.7)
titlecase-3.11.1 (created from /Users/sdoran/.pyenv/versions/3.11.1)
uptime-kuma-3.11.1 (created from /Users/sdoran/.pyenv/versions/3.11.1)
youtube-dl-3.10.4 (created from /Users/sdoran/.pyenv/versions/3.10.4)
yt-dlp (created from /Users/sdoran/.pyenv/versions/3.12.2)
New output from this branch
3.10.2/envs/pyls-3.10.2
3.10.3/envs/ansible-2.12-3.10.3
3.10.3/envs/boto-3.10.4
3.10.3/envs/lexicon-3.10.3
3.10.4/envs/ansible-2.12-3.10.4
3.10.4/envs/ansible-dev-3.10.4
3.10.4/envs/flake8-3.10.4
3.10.4/envs/jello-3.10.4
3.10.4/envs/jsbeautifier-3.10.4
3.10.4/envs/youtube-dl-3.10.4
3.10.6/envs/ipython-3.10.6
3.10.6/envs/shiv-3.10.6
3.10.6/envs/speedtest-cli-3.10.6
3.11.1/envs/ansible-2.14-3.11.1
3.11.1/envs/ipython-3.11.1
3.11.1/envs/titlecase-3.11.1
3.11.1/envs/uptime-kuma-3.11.1
3.11.3/envs/ipython-3.11.3
3.11.3/envs/tasmota
3.11.7/envs/tasmota-3.11.7
3.12.0/envs/ansible-2.16-3.12.0
3.12.0/envs/ipython-3.12.0
3.12.2/envs/borg
3.12.2/envs/yt-dlp
3.12.4/envs/qrcode
3.12.8/envs/ansible-lint
3.12.8/envs/cowsay
3.12.8/envs/pyright-3.12.8
3.12.8/envs/ruff-3.12.8
3.13.1/envs/asciinema
3.13.1/envs/diceware
3.13.1/envs/ipython-3.13.1
3.6.15/envs/ipython-3.6.15
3.7.12/envs/appomni-dev-3.7.12
3.7.12/envs/ipython-3.7.12
3.8.12/envs/ipython-3.8.12
3.9.9/envs/ipython-3.9.9
miniconda3-latest/envs/my_conda_env
ansible-2.12-3.10.3 --> /Users/sdoran/.pyenv/versions/3.10.3/envs/ansible-2.12-3.10.3
ansible-2.12-3.10.4 --> /Users/sdoran/.pyenv/versions/3.10.4/envs/ansible-2.12-3.10.4
ansible-2.14-3.11.1 --> /Users/sdoran/.pyenv/versions/3.11.1/envs/ansible-2.14-3.11.1
ansible-2.16-3.12.0 --> /Users/sdoran/.pyenv/versions/3.12.0/envs/ansible-2.16-3.12.0
ansible-dev-3.10.4 --> /Users/sdoran/.pyenv/versions/3.10.4/envs/ansible-dev-3.10.4
ansible-lint --> /Users/sdoran/.pyenv/versions/3.12.8/envs/ansible-lint
appomni-dev-3.7.12 --> /Users/sdoran/.pyenv/versions/3.7.12/envs/appomni-dev-3.7.12
asciinema --> /Users/sdoran/.pyenv/versions/3.13.1/envs/asciinema
borg --> /Users/sdoran/.pyenv/versions/3.12.2/envs/borg
boto-3.10.4 --> /Users/sdoran/.pyenv/versions/3.10.3/envs/boto-3.10.4
cowsay --> /Users/sdoran/.pyenv/versions/3.12.8/envs/cowsay
diceware --> /Users/sdoran/.pyenv/versions/3.13.1/envs/diceware
flake8-3.10.4 --> /Users/sdoran/.pyenv/versions/3.10.4/envs/flake8-3.10.4
ipython-3.10.6 --> /Users/sdoran/.pyenv/versions/3.10.6/envs/ipython-3.10.6
ipython-3.11.1 --> /Users/sdoran/.pyenv/versions/3.11.1/envs/ipython-3.11.1
ipython-3.11.3 --> /Users/sdoran/.pyenv/versions/3.11.3/envs/ipython-3.11.3
ipython-3.12.0 --> /Users/sdoran/.pyenv/versions/3.12.0/envs/ipython-3.12.0
ipython-3.13.1 --> /Users/sdoran/.pyenv/versions/3.13.1/envs/ipython-3.13.1
ipython-3.6.15 --> /Users/sdoran/.pyenv/versions/3.6.15/envs/ipython-3.6.15
ipython-3.7.12 --> /Users/sdoran/.pyenv/versions/3.7.12/envs/ipython-3.7.12
ipython-3.8.12 --> /Users/sdoran/.pyenv/versions/3.8.12/envs/ipython-3.8.12
ipython-3.9.9 --> /Users/sdoran/.pyenv/versions/3.9.9/envs/ipython-3.9.9
jello-3.10.4 --> /Users/sdoran/.pyenv/versions/3.10.4/envs/jello-3.10.4
jsbeautifier-3.10.4 --> /Users/sdoran/.pyenv/versions/3.10.4/envs/jsbeautifier-3.10.4
lexicon-3.10.3 --> /Users/sdoran/.pyenv/versions/3.10.3/envs/lexicon-3.10.3
pyls-3.10.2 --> /Users/sdoran/.pyenv/versions/3.10.2/envs/pyls-3.10.2
pyright-3.12.8 --> /Users/sdoran/.pyenv/versions/3.12.8/envs/pyright-3.12.8
qrcode --> /Users/sdoran/.pyenv/versions/3.12.4/envs/qrcode
ruff-3.12.8 --> /Users/sdoran/.pyenv/versions/3.12.8/envs/ruff-3.12.8
shiv-3.10.6 --> /Users/sdoran/.pyenv/versions/3.10.6/envs/shiv-3.10.6
speedtest-cli-3.10.6 --> /Users/sdoran/.pyenv/versions/3.10.6/envs/speedtest-cli-3.10.6
tasmota --> /Users/sdoran/.pyenv/versions/3.11.3/envs/tasmota
tasmota-3.11.7 --> /Users/sdoran/.pyenv/versions/3.11.7/envs/tasmota-3.11.7
titlecase-3.11.1 --> /Users/sdoran/.pyenv/versions/3.11.1/envs/titlecase-3.11.1
uptime-kuma-3.11.1 --> /Users/sdoran/.pyenv/versions/3.11.1/envs/uptime-kuma-3.11.1
youtube-dl-3.10.4 --> /Users/sdoran/.pyenv/versions/3.10.4/envs/youtube-dl-3.10.4
yt-dlp --> /Users/sdoran/.pyenv/versions/3.12.2/envs/yt-dlp
| ln -s "venv27" "${PYENV_ROOT}/versions/venv27" | ||
| ln -s "venv33" "${PYENV_ROOT}/versions/venv33" |
There was a problem hiding this comment.
AFAICS the links are created in the current directory. That doesn't seem correct.
There's a dedicated function to create aliases.
| 2.7.6/envs/venv27 | ||
| * 3.3.3/envs/venv33 |
There was a problem hiding this comment.
Diagnosed this failure. The logic does not equate "xxx" with "yyy/envs/xxx". If you set current version to "venv33", the alias will be the highlighted entry, not the full name.
To diagnose stuff in tests, I set PS4 to the long value from the launcher and run the command with PYENV_DEBUG=1. Then make sure there's a command later that prints its output in some form (in this case, assert_output failure handling logic).
| stub pyenv-virtualenv-prefix "venv27 : echo \"${PYENV_ROOT}/versions/2.7.6\"" | ||
| stub pyenv-virtualenv-prefix "venv33 : echo \"${PYENV_ROOT}/versions/3.3.3\"" |
There was a problem hiding this comment.
Not sure if this change improves anything. As a rule of thumb, one should not change test cases unless you've changed the behavior in them -- to avoid accidentally losing test cover.
| OUT | ||
|
|
||
| unstub pyenv-version-name | ||
| unstub pyenv-virtualenv-prefix | ||
| # unstub pyenv-version-name |
There was a problem hiding this comment.
unstub checcks if the specific stubbed executable was called exactly according to the sequence set up with stub calls. If not, it fails and prints the plan and the fact.
The implementation is in test_helper.bash and the stubs/stub script which the former calls.
| @@ -17,7 +17,7 @@ set -e | |||
| # Provide pyenv completions | |||
| if [ "$1" = "--complete" ]; then | |||
| echo --unset | |||
| exec pyenv-virtualenvs --bare | |||
| exec pyenv-virtualenvs --bare --only-aliases | |||
There was a problem hiding this comment.
This is a design change -- so this should be discussed separately. Envs are supposed to be accessible under both full name and alias. I'm not sure why that is; aliases are internally called "compat", indicating that they were added for compatibility with... something.
There was a problem hiding this comment.
I'll remove that change for now and we can discuss elsewhere. What's the best place to have a design discussion?
|
Thank you for the feedback @native-api. I'll address this later in the week. |
|
I had a crazy week and a full weekend away from home. I will work on this during the week. |
This was removed by accident.
For consistency, use conditional expressions instead of arithmetic evaluation when comparing bash versions.
The new implementation is not using realpath.
|
Thanks for all the feedback. I didn't get to fixing up the tests. I need to get some sleep for now. |
The biggest change is reusing most of the code used for
pyenv versionsinpyenv virtualenvs. This dramatically improves performance while making the output consistent.Related to #490. Maybe fixes the problem?
With 128 virtual environments, it currently takes about 13 seconds to run
pyenv virtualenvs --bare.Output
With the changes in this branch it is 236ms:
Output
Output formatting changes
This PR also includes changes to the output format to make it more consistent with
pyenv versions.pyenv versions output
There is now an
--only-aliasesflag. I found this useful to make the shell completions less verbose.The completions for
pyenv virtualenvnow omit aliases and envs. I believe only installed Python versions should be suggested instead of current virtual environments, which are aliases.