We'd like to significantly improve resource autogrouping performance. It's not necessarily a blocker atm, but enabling fast autogrouping for very large resource graphs would have long-term benefits. Here are some suggested approaches one might take. Please feel free to suggest additional ideas as well.
-
Improve the core autogrouping algorithm.
I'm not a great algorithmist, and while I think it's correct, a smarter computer scientist might know of a better approach.
-
Add caching to the graph lookup operations that the algorithm uses.
Repeated lookups that are similar could be cached.
-
Add engineering approaches to minimize search size.
We might be able to extend the autogrouping API to statically know which resource kind's can pair together or not. This would reduce the search size. Remember to include the match against both same kind of resource and differing kind of resources.
-
Consider parser tricks to autogroup resources in the same name.
The resource allows either a single string name, or a list of strings as input the to $name of every resource. If a list is used, we should assume that we want to autogroup those items together (if it's possible to do so). This would offer speed up be initially reducing the graph size substantially.
-
Delta optimizations during graph transitions.
When swapping from one resource graph to a subsequent one, we essentially re-run the autogrouper. It might be helpful to look at the previous autogrouping result to determine if we can reuse some of that work. An algorithmist might want to be present to ensure we don't do something wildly more inefficient here.
-
Tests.
We'd definitely love to have more tests here both for correctness and performance. In particular, when we're doing hierarchical autogrouping (thing http:server:ui... related things) as well as send/recv things) we want to make sure not to break the engine.
Additional ideas and discussion welcome.
We'd like to significantly improve resource autogrouping performance. It's not necessarily a blocker atm, but enabling fast autogrouping for very large resource graphs would have long-term benefits. Here are some suggested approaches one might take. Please feel free to suggest additional ideas as well.
Improve the core autogrouping algorithm.
I'm not a great algorithmist, and while I think it's correct, a smarter computer scientist might know of a better approach.
Add caching to the graph lookup operations that the algorithm uses.
Repeated lookups that are similar could be cached.
Add engineering approaches to minimize search size.
We might be able to extend the autogrouping API to statically know which resource kind's can pair together or not. This would reduce the search size. Remember to include the match against both same kind of resource and differing kind of resources.
Consider parser tricks to autogroup resources in the same name.
The resource allows either a single string name, or a list of strings as input the to $name of every resource. If a list is used, we should assume that we want to autogroup those items together (if it's possible to do so). This would offer speed up be initially reducing the graph size substantially.
Delta optimizations during graph transitions.
When swapping from one resource graph to a subsequent one, we essentially re-run the autogrouper. It might be helpful to look at the previous autogrouping result to determine if we can reuse some of that work. An algorithmist might want to be present to ensure we don't do something wildly more inefficient here.
Tests.
We'd definitely love to have more tests here both for correctness and performance. In particular, when we're doing hierarchical autogrouping (thing http:server:ui... related things) as well as send/recv things) we want to make sure not to break the engine.
Additional ideas and discussion welcome.