Updated Github Windows 2022 runners break Vims test suite

[chrisbra]

Description

This is a copy from vim/vim#16546

It seems the Github Windows runners has been updated to https://github.com/actions/runner-images/releases/tag/win22%2F20250127.1 and since then it breaks the Vim terminal_tests, e.g. here

Failures: 
	From test_terminal.vim:
	Found errors in Test_terminal_duplicate_eof_arg():
	Run 1, 17:43:32 - 17:43:32 in  0.872468 seconds:
	command line..script D:/a/vim/vim/src/testdir/runtest.vim[617]..function RunTheTest[57]..Test_terminal_duplicate_eof_arg line 16: Expected 123 but got 1
	Run 2, 17:43:34 - 17:43:35 in  0.238029 seconds:
	command line..script D:/a/vim/vim/src/testdir/runtest.vim[658]..function RunTheTest[57]..Test_terminal_duplicate_eof_arg line 16: Expected 123 but got 1
	Run 3, 17:43:39 - 17:43:39 in  0.238754 seconds:
	command line..script D:/a/vim/vim/src/testdir/runtest.vim[658]..function RunTheTest[57]..Test_terminal_duplicate_eof_arg line 16: Expected 123 but got 1
	Flaky test failed too often, giving up
	Found errors in Test_terminal_eof_arg():
	Run 1, 17:43:39 - 17:43:39 in  0.249702 seconds:
	command line..script D:/a/vim/vim/src/testdir/runtest.vim[617]..function RunTheTest[57]..Test_terminal_eof_arg line 14: Expected 123 but got 1
	Run 2, 17:43:41 - 17:43:42 in  0.237895 seconds:
	command line..script D:/a/vim/vim/src/testdir/runtest.vim[658]..function RunTheTest[57]..Test_terminal_eof_arg line 14: Expected 123 but got 1
	Run 3, 17:43:46 - 17:43:46 in  0.237767 seconds:
	command line..script D:/a/vim/vim/src/testdir/runtest.vim[658]..function RunTheTest[57]..Test_terminal_eof_arg line 14: Expected 123 but got 1
	Flaky test failed too often, giving up
	Found errors in Test_terminal_eof_arg_win32_ctrl_z():
	Run 1, 17:43:46 - 17:43:51 in  5.080960 seconds:
	command line..script D:/a/vim/vim/src/testdir/runtest.vim[617]..function RunTheTest[57]..Test_terminal_eof_arg_win32_ctrl_z[7]..WaitForAssert[2]..<SNR>3_WaitForCommon[11]..<lambda>27 line 1: Pattern '\\^Z' does not match 'hello'
	command line..script D:/a/vim/vim/src/testdir/runtest.vim[617]..function RunTheTest[57]..Test_terminal_eof_arg_win32_ctrl_z line 8: Pattern '\\^Z' does not match 'hello'
	Run 2, 17:43:53 - 17:43:58 in  5.084397 seconds:
	command line..script D:/a/vim/vim/src/testdir/runtest.vim[658]..function RunTheTest[57]..Test_terminal_eof_arg_win32_ctrl_z[7]..WaitForAssert[2]..<SNR>3_WaitForCommon[11]..<lambda>28 line 1: Pattern '\\^Z' does not match 'hello'
	command line..script D:/a/vim/vim/src/testdir/runtest.vim[658]..function RunTheTest[57]..Test_terminal_eof_arg_win32_ctrl_z line 8: Pattern '\\^Z' does not match 'hello'
	Run 3, 17:44:02 - 17:44:07 in  5.083964 seconds:
	command line..script D:/a/vim/vim/src/testdir/runtest.vim[658]..function RunTheTest[57]..Test_terminal_eof_arg_win32_ctrl_z[7]..WaitForAssert[2]..<SNR>3_WaitForCommon[11]..<lambda>29 line 1: Pattern '\\^Z' does not match 'hello'
	command line..script D:/a/vim/vim/src/testdir/runtest.vim[658]..function RunTheTest[57]..Test_terminal_eof_arg_win32_ctrl_z line 8: Pattern '\\^Z' does not match 'hello'
	Run 4, 17:44:13 - 17:44:18 in  5.082484 seconds:
	command line..script D:/a/vim/vim/src/testdir/runtest.vim[658]..function RunTheTest[57]..Test_terminal_eof_arg_win32_ctrl_z[7]..WaitForAssert[2]..<SNR>3_WaitForCommon[11]..<lambda>30 line 1: Pattern '\\^Z' does not match 'hello'
	command line..script D:/a/vim/vim/src/testdir/runtest.vim[658]..function RunTheTest[57]..Test_terminal_eof_arg_win32_ctrl_z line 8: Pattern '\\^Z' does not match 'hello'
	Run 5, 17:44:26 - 17:44:31 in  5.068240 seconds:
	command line..script D:/a/vim/vim/src/testdir/runtest.vim[658]..function RunTheTest[57]..Test_terminal_eof_arg_win32_ctrl_z[7]..WaitForAssert[2]..<SNR>3_WaitForCommon[11]..<lambda>31 line 1: Pattern '\\^Z' does not match 'hello'
	command line..script D:/a/vim/vim/src/testdir/runtest.vim[658]..function RunTheTest[57]..Test_terminal_eof_arg_win32_ctrl_z line 8: Pattern '\\^Z' does not match 'hello'
	Flaky test failed too often, giving up

Guessing from the test, there is an issue with Python?

The last successfull test was this commit here: patch 9.1.1055: make install fails because of a missing dependency, windows Image version: 0250120.2. The next one broke on the Windows builds and have an updated Windows Image version 20250127.1.

For the record, those 3 tests from test_terminal.vim fail:

func Test_terminal_eof_arg()
  call CheckPython(s:python)
  let g:test_is_flaky = 1

  call setline(1, ['print("hello")'])
  exe '1term ++eof=exit(123) ' .. s:python
  " MS-Windows echoes the input, Unix doesn't.
  if has('win32')
    call WaitFor({-> getline('$') =~ 'exit(123)'})
    call assert_equal('hello', getline(line('$') - 1))
  else
    call WaitFor({-> getline('$') =~ 'hello'})
    call assert_equal('hello', getline('$'))
  endif
  call assert_equal(123, bufnr()->term_getjob()->job_info().exitval)
  %bwipe!
endfunc

func Test_terminal_eof_arg_win32_ctrl_z()
  CheckMSWindows
  call CheckPython(s:python)
  let g:test_is_flaky = 1

  call setline(1, ['print("hello")'])
  exe '1term ++eof=<C-Z> ' .. s:python
  call WaitForAssert({-> assert_match('\^Z', getline(line('$') - 1))})
  call assert_match('\^Z', getline(line('$') - 1))
  %bwipe!
endfunc

func Test_terminal_duplicate_eof_arg()
  call CheckPython(s:python)
  let g:test_is_flaky = 1

  " Check the last specified ++eof arg is used and does not leak memory.
  new
  call setline(1, ['print("hello")'])
  exe '1term ++eof=<C-Z> ++eof=exit(123) ' .. s:python
  " MS-Windows echoes the input, Unix doesn't.
  if has('win32')
    call WaitFor({-> getline('$') =~ 'exit(123)'})
    call assert_equal('hello', getline(line('$') - 1))
  else
    call WaitFor({-> getline('$') =~ 'hello'})
    call assert_equal('hello', getline('$'))
  endif
  call assert_equal(123, bufnr()->term_getjob()->job_info().exitval)
  %bwipe!
endfunc

Expected behaviour

no failure

Version of Vim

v9.1.1057

Environment

Github Runner Windows Image 2022

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• Ubuntu 24.04
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• macOS 15
• macOS 15 Arm64
• Windows Server 2019
• Windows Server 2022
• Windows Server 2025

Image version and build link

Version: 20250127.1.0 (see links above)

Is it regression?

yes, see above

Expected behavior

successful tests

Actual behavior

Failing tests, my guess is some issue with Python?

Repro steps

Run the Vim test suite.

[RaviAkshintala]

Hi @chrisbra Thank you for bringing this issue to our attention. We will look into this issue closely and will update you after investigating.

[chrisbra]

This seems to be partly caused by a python 3.13 regression