Tests take too long to run

[srawlins]

Currently on my MacBook, pub run test takes around 300 seconds. This is far too long. It makes local testing slow, as well as Travis.

I suspect most of the waste comes from repeatedly resolving the Dart SDK, for each package in testing/, and for each test suite that documents each package in testing/. The time can be reduced by not resolving so much, or by converting any end-to-end test, which reads real packages from testing/, into a unit test, which uses a mock SDK or mock package graph, etc.

It's hard to pick a good acceptance criteria. If work is done to reduce the test duration to below 120 seconds, I think we'd be in a functional state.

[srawlins]

As individual test cases are refactored into unit tests, the duration will probably decrease by nil. Only when an entire testing/ package has been removed, or an entire test suite (group? test file?) has been removed, will real progress be made. To that end, here are some finer-grained stats:

• pub run test test/dartdoc_integration_test.dart: 210 seconds
• pub run test test/dartdoc_options_test.dart: 5 seconds
• pub run test test/dartdoc_test.dart: 210 seconds
• pub run test test/experiment_options_test.dart: 5 seconds
• pub run test test/grind_test.dart: 5 seconds
• pub run test test/html_generator_test.dart: 20 seconds
• pub run test test/io_utils_test.dart: 2 seconds
• pub run test test/markdown_processor_test.dart: 5 seconds
• pub run test test/model_special_cases_test.dart: 65 seconds
• pub run test test/model_test.dart: 24 seconds
• pub run test test/model_utils_test.dart: 5 seconds
• pub run test test/package_meta_test.dart: 5 seconds
• pub run test test/resource_loader_test.dart: 1 second
• pub run test test/source_linker_test.dart: 5 seconds
• pub run test test/template_test.dart: 1 second
• pub run test test/tool_runner_test.dart: 8 seconds
• pub run test test/utils_test.dart: 1 second
• pub run test test/warnings_test.dart: 5 seconds
• pub run test test/render/template_renderer_test.dart: 1 second

[srawlins]

In addition, just for tracking purposes, as of dartdoc 0.32.2, there were 581 test cases. Converting to unit-test-level tests should encourage more individual tests, with more meaningful failure states.

[srawlins]

Here are some updated stats on my same laptop, at bf7eb56. I'm about to set up a new laptop, so I'll run them on my new hardware as well at the same commit, as a new baseline. pub run test takes 342 seconds. Boo. But there are now 767 executed tests.

• pub run test test/end2end/dartdoc_integration_test.dart: 184 seconds
• pub run test test/dartdoc_test.dart: 133 seconds
• pub run test test/html_generator_test.dart: 9 seconds
• pub run test test/model_special_cases_test.dart: 45 seconds
• pub run test test/model_test.dart: 24 seconds

new:

• pub run test test/documentation_comment_test.dart: 17 seconds
• pub run test test/library_test.dart: 8 seconds
• pub run test test/mustachio/builder_test.dart: 16 seconds
• pub run test test/mustachio/parser_test.dart: 2 seconds
• pub run build_runner test -- test/mustachio/parser_test.dart: 9 seconds
• pub run test test/package_test.dart: 14 seconds

[srawlins]

Updated numbers:

• dart run test test/end2end/dartdoc_integration_test.dart: 144 seconds
• dart run test test/end2end/dartdoc_test.dart: 36 seconds
• dart run test test/end2end/model_special_cases_test.dart: 50 seconds
• dart run test test/end2end/model_test.dart: 14 seconds
• dart run test test/html_generator_test.dart: 3 seconds
• dart run test test/documentation_comment_test.dart: 9 seconds
• dart run test test/mustachio/aot_compiler_builder_test.dart: 4 seconds
• dart run test test/mustachio/aot_compiler_render_test.dart: 13 seconds
• dart run test test/mustachio/parser_test.dart: 2 seconds
• dart run test test/mustachio/renderers_output_test.dart: 24 seconds
• dart run test test/mustachio/runtime_renderer_builder_test.dart: 16 seconds
• dart run test test/mustachio/runtime_renderer_render_test.dart: 2 seconds
• dart run test test/package_test.dart: 4 seconds

The unit tests have really dropped! Total time is still 250 seconds, still too long, so this bug remains open. Probably mostly fixed by converting end2end tests. The hard part here I think is the @tool tests.

[sigurdm]

This seems obsolete.
On my cloudtop running dart test on main completes in less than a minute:

00:46 +1155 ~4: All tests passed!