Added test plan for TypeSpec extension

[xiaohua899]

fix #5444
Description

Added four test scenarios test plan to TypeSpec Extension:

• create-typespec-project-test-plan.md
• generate-from-typespec-test-plan.md
• import-typespec-from-openapi3-test-plan.md
• typespec-extension-basic-features-test-plan.md

@RodgeFu and @msyyc for notification.

[xiaohua899]

@microsoft-github-policy-service agree company="Microsoft"

[xiaohua899]

@RodgeFu We have updated by your comments, please re-review.

[xiaohua899]

@RodgeFu We have updated by your comments, please re-review.

[markcowl]

Do we intend to have this document checked in to the repo?

[RodgeFu]

Do we intend to have this document checked in to the repo?

Yes, so that vender can help to do manual test according to them when we need to sign-off release or merge PR impacting the UI considering we don't have UI test to cover these major user experience now.

The test doc looks good to me now except the one for generating code which is a little out of date. But considering we are making an improvement in that user experience now according to user feedback which will come soon, so I think we can have this merged and then update the "code generation" one after the improvement is done.

@markcowl , do you have any more suggestions/comments? or could you help to approve if it looks good to you? thanks.

[timotheeguerin]

I think some of the concern is that's a lot of image for a git repo that might need to be updated as the test plan change

[RodgeFu]

I think some of the concern is that's a lot of image for a git repo that might need to be updated as the test plan change

@msyyc , could you help to double check the images to see whether all of them are necessary for vender to follow to do the test? Timothee has a point that images have a higher maintain cost, so it's good if we can simplify the images but still make sure the doc is good enough for venders to follow to test the features. thanks.

After these user experience in UX is stabilized, we will expect any changes in these UX in the future should contains the corresponding change in these manual test plan (just like updating auto test for feature update now) and go through a manual test sign-off by venders which should also be helpful to make sure the doc is up-to-date. thanks.

[msyyc]

I think some of the concern is that's a lot of image for a git repo that might need to be updated as the test plan change

@msyyc , could you help to double check the images to see whether all of them are necessary for vender to follow to do the test? Timothee has a point that images have a higher maintain cost, so it's good if we can simplify the images but still make sure the doc is good enough for venders to follow to test the features. thanks.

After these user experience in UX is stabilized, we will expect any changes in these UX in the future should contains the corresponding change in these manual test plan (just like updating auto test for feature update now) and go through a manual test sign-off by venders which should also be helpful to make sure the doc is up-to-date. thanks.

Got it.
(1) We will try if we could reduce the number of images to least. For example, maybe we could reuse some images.
(2) In last sync meeting, I already told vendors to make an investigation about auto test based on the test doc. And as you said, we should make sure to update the test doc once there is new feature or change.

[timotheeguerin]

Is it also not possible to automate this instead

[RodgeFu]

Is it also not possible to automate this instead

we are doing some investigation into the auto-test for it as @msyyc mentioned. It will depend on the investigation result and how frequently these UX will be changed later to determine whether we want to add auto UI test (I have dealt with UI test a lot when in Sql which is actually painful to maintain because of its instability in nature). Before that, manual test would help to cover these user experience in vscode. thanks.

[markcowl]

I think the major concern is that this repo is the core repo for typespec and a test document does not seem to belong. If we need to check a test plan into someplace, I would suggest some other repo (perhaps typespec-azure-pr).

This just does not seem like it is of general interest to consumers of typespec, so does not seem to belong in the repo

[msyyc]

Update: after optimization, we reduce the image number used by docs from 80 to 55.

[RodgeFu]

I think the major concern is that this repo is the core repo for typespec and a test document does not seem to belong. If we need to check a test plan into someplace, I would suggest some other repo (perhaps typespec-azure-pr).

This just does not seem like it is of general interest to consumers of typespec, so does not seem to belong in the repo

I think we should call them "manual test cases" instead of "test plan" which would be more accurate. They are actually the same as automated test in logic if we don't look at the implementation (code/doc) and execution engine (machine/people). And a few detail reasons why I would prefer them to be along with the code:

• It's easier for people to find the corresponding test to "run" when they are changing the code. For example, if some people (internal or external) want to contribute to the UX part code in vscode, they can run these "manual test" to make sure nothing is broken when submitting their change. (just like they need to make sure all the automated tests pass when contributing into other features)
• It's also easier to make sure these test cases are always up to date with the code. For example, later on, when people make change impacting these UX experience in vscode, they need also update these tests(doc) in the same PR properly so that vender can use the updated test case(doc) in the PR to "run" the test and sign-off. Otherwise, these tests will fail if vender verifying using the original test case. (just like people needs to update automated test accordingly when they make some change impacting tests, otherwise the test execution will fail)

thx

[xiaohua899]

@RodgeFu /@msyyc Regarding the trigger of Import TypeSpec from OpenAPI3, I changed it to three options in the test plan, please review.