Skip to content

Issues to discuss with the Visual CPP team

Jump to bottom Edit New page
Christian Soltenborn edited this page Oct 3, 2017 · 31 revisions

Microsoft has reached out to us for the sake of discussing ways to improve Google Test support based on our adapter. The following points are to be discussed with the Visual C++ team.

Improving the user experience

  • Make it easy to launch test executables exactly as if they were launched with "F5". Users have settings in the debug page of their project properties and expect GoogleTestAdapter to use these.
    • most issues were because working directory is not set properly (#123, #97, #71, #58, #57)
    • two issue result from environment variables (most likely $PATH) not set correctly (#63, #28)
    • ideally we would get the correct values from Project > Properties > Debug of test executable, but we don't do it because it seemed too difficult
    • our workaround are two test adapter options working directory and PATH extension (add stuff to $PATH), but this needs manual synchronisation
    • inheriting settings from Project > Properties > Debug could be a useful default, but it shouldn't be the only way. E.g. teams would want to have their shared configuration in a *.gta.runsettings file. We currently support $(SolutionDir) and $(ExecutableDir) in those files. Having also $(ProjectDir) would be really useful, but we currently don't know that value. Ideally we would support all macros available in the project properties.
    • We also need to consider VsTest.Console.exe here, where the test framework currently doesn't even deliver the solution dir (thus, a /TestSettings:<file> option needs to be used). If possible, we should aim at being able to execute tests with the same settings, no matter whether this happens within Visual Studio (as extension or NuGet dependency) or within VsTest.Console.exe (with options /UseVsixExtensions:true or /TestAdapterPath:<path to dlls>)
  • On VS2017, Google Test projects need to be built with VS setting Linker -> Debugging -> Generate debug info set to /DEBUG:FULL to receive test locations
  • It would be extremely nice to be able to execute tests from the test code
    • Think ReSharper C#
    • We haven't done anything with respect to this since it appears to be rather complex
      • Parsing C++ code
      • Enhancing the editor
      • Maybe not possible at all
        • Hollywood principle: Test framework calls us, we can't call the test framework
  • Google Test projects are difficult to set up (implemented in TAfGT, to be ported to GTA)
    • What about New project -> Google Test?
    • What about New C++ item -> Google Test Class?
      • Or New C++ item -> [[Type] Parameterized|Fixture] Google Test [with support for traits]?
  • Extend support for parameterized / data driven tests. Current state:
    • One test case per parameterized test case set is reported to VS
    • At test execution time, one test result per parameter value is reported to VS (all test results refer to same test case)
    • Drawback: after test discovery, it is not possible to execute test cases for single parameter values (all or nothing)
  • Improve VS integration of test adapters pulled in as NuGet dependency (according to MS, this is not possible)
    • Currently, our users can discover and run tests, but do not have options integration and toolbar

Improving the test adapter developer experience

  • Provide "support" for authors of test adapters
    • Example questions we would like to ask/would have asked in the past:
      • We haven't yet figured out why our adapter doesn't work with Run after build
      • Our adapter used to not receive Test run canceled events, and we don't know how we fixed this
      • DIA library is difficult to access
        • Different DLLs for different VS versions
        • Access via CLSID apparently stopped working with VS2017
        • Would be nice to know how to this "the right way" (and improve user experience by not breaking our adapter that often)
        • Our current approach is to be found here
      • Our "multi-version" support for test adapter framework is fragile / not future proof
    • A dedicated place to ask questions such as those would be nice
    • Could also make sure that issues such as this one (link broken :-) but content can also be found here) would be taken more seriously by MS
      • We spent quite a bit of time on this bug (including figuring out that it's not our bug, decompiling, providing sample solution, description of bug fix) and were rather annoyed about it being closed
  • Provide Azure resources to enable us to implement real end-to-end tests (maybe even UI tests)
    • Most quality issues arise from VS integration, which is pretty hard to test automatically
      • And cumbersome to test manually due to VS versions/editions supported
    • Is it possible to use coded UI tests against VS experimental instances? If yes: Provide appropriate VS licenses :-) (we don't do this for test stability reasons)
  • make output streams available (also when debugging)
    • IFrameworkHandle.LaunchProcessWithDebuggerAttached() returns id of running process => no way to access process output (to our knowledge)
    • currently we're jumping through hoops to start the process suspended with handles to output streams, attach the VS debugger, and continue the process.
    • We have a reliable workaround in place by now, but it would still be nice for future adapters to be able to launch processes in debug mode and still access their output

Improving the user experience (minor things, not requested by users)

  • Contribute more appropriate icons for toolbar (have been contributed by MS)
    • Current icons are ok, but do not integrate into VS as nicely as they could
    • We do not have any according skills
  • Contribute certificate for signing VSIX (we do not sign our extension; users can go for TAfGT if this is important for them)
    • Note that no one has requested this - might still give users a slightly better installation experience
Add a custom footer
Add a custom sidebar
Clone this wiki locally