Scalability limits in the Resource Processor

[TonyWildish-BH]

Description

In my scalability tests last year, I ran a script that attempted to create dozens of resources - workspaces in this case. With the Resource Processor at the default setting of a max pool size of one, my workspaces were all created, but of course I had to wait a long time, as only 5 processes were running at a time.

I tried enlarging the Resource Processor pool to see if I could create more resources in parallel. I just went into the Azure portal and manually increased the pool size from max 1 to max 4, then re-ran my tests. I saw that it did indeed try to create 20 workspaces in one go, but it failed with terraform errors, the APIs were being throttled by Azure, and resources were left in a bad state. Unfortunately, I no longer have the logs, so I can't give the precise message. However, I do recall that terraform was not handling the throttling well.

Anyway, my question is: How can I increase the parallelism of the Resource Processor without terraform falling over?

This likely requires two things:

• better retry-handling in terraform, so it doesn't just crash and burn when it doesn't need to. That will prevent the failures, but not speed things up.
• Relaxing the throttling requirements for the Azure APIs, which would speed things up.

Steps

The steps I have tried are:

1. Increase the size of the Resource Processor pool in the Azure portal.
2. Write a Bash script to use the tre CLI to create 20-30 workspaces in a tight loop, using --no-wait so they queue.
3. Watch as the workspaces fail in various stages, due to throttling of terraform API calls by Azure.

[marrobi]

Hi @TonyWildish-BH there are some scenarios where parallel operation's hit transient errors - #3177 (comment)

When it's Azure platform issues, all we can do is try work around by blocking parallel operations or throttling.

If you can reproduce the scenario and provide the specific errors then we look at next steps. Thanks.