test(e2e): stabilize Amazon Q feature development tests #6520
Conversation
|
/runIntegrationTests |
chengoramazon
force-pushed
the
chengora/fix-tests
branch
from
824ecf8 to
2d4edce
Compare
|
/runIntegrationTests |
|
Failed test in E2E stable is unrelated to |
| beforeEach(async function () { | ||
| this.timeout(maxTestDuration) |
There was a problem hiding this comment.
Is there anyway we can avoid increasing the timeout? The default timeout is 5 minutes, so increasing it to 10 minutes in this beforeEach increases it up to 5 minutes per test in this block. Combined with the change above, this could add up to 50 minutes to execution time.
There was a problem hiding this comment.
I can revert the timeout increase given the logs showed each test can complete within 5 mins. Please note /dev is a long run which can go beyond, we may revisit this.
There was a problem hiding this comment.
Have you confirmed that these e2e tests are reliable without these increased timeouts? From the toolkits side, we are very reluctant to merge tests that cause inconsistencies in our CI due to the confusion and churn it creates.
There was a problem hiding this comment.
Confirmed. Tried a couple of runs which the new prompt can finish without timeout increase. The previous flakiness was mainly from a dead-end situation.
chengoramazon
force-pushed
the
chengora/fix-tests
branch
from
2d4edce to
1129a44
Compare
|
/runIntegrationTests |
| public getSelectedTab() { | ||
| const selectedTabId = this.mynahUI.getSelectedTabId() | ||
| const selectedTab = this.getTabs().find((tab) => tab.tabID === selectedTabId) | ||
|
|
||
| if (!selectedTab) { | ||
| assert.fail('Selected tab not found') | ||
| } | ||
| return selectedTab | ||
| } |
There was a problem hiding this comment.
Can you elaborate more on why this is needed?
There was a problem hiding this comment.
See reply above
| beforeEach(async function () { | ||
| tab = framework.createTab() |
There was a problem hiding this comment.
why are the two tabs here and line 226 different? are they always different?
There was a problem hiding this comment.
There is a new tab created when a /dev quick action triggered. The getSelectedTab can help get the current selected tab;
Problem
Amazon Q feature dev e2e test was disabled due to flakiness. Issue: #6513
Solution
getSelectedTabfunction inqTestingFrameworkto access newly created tab by quick action.feature/xbranches will not be squash-merged at release time.