Improve speed by parallelizing I/O

[ehmicky]

Most of the logic from next-on-netlify is I/O read/write operations. Those are currently performed serially. Performing those in parallel instead would significantly improve performance.

A queue should probably be used so that the OS max limits on open files is not reached.

@lindsaylevine

[ehmicky]

@ascorbic That is awesome!
I am wondering whether we might want to increase the max concurrency from 4?

[ascorbic]

@ehmicky Almost certainly, but I need to create a large site for benchmarking before doing it. I also tried it with jest-worker to run them in a worker pool, but that was slower on the small site (presumably because of the initial setup time). However it might be quicker for big sites, so we could switch between them. I'm going to see about creating a massive test site, unless anyone has one to hand.

[ehmicky]

Thanks for experimenting and tweaking this @ascorbic, this should provide with some huge performance benefits!

One thing to look for is the OS limit on the maximum number of open files. However, I think any value below 100 should be fairly safe in that respect.