Too many threads created during rendering

[kkoz]

We are experiencing issues where large numbers of simultaneous rendering requests are causing system crashes. We suspect the issue is related to the number of threads being created.

• By default. the number of threads created by a call to renderAsPackedInt can be as high as the number of cores (see https://github.com/glencoesoftware/openmicroscopy-private/blob/mainline_5_4/components/rendering/src/omeis/providers/re/RenderingStrategy.java#L154).
• The current ExecutorService being used has no upper bound on threads, so it will not prevent these from being created (see https://github.com/openjdk-mirror/jdk7u-jdk/blob/master/src/share/classes/java/util/concurrent/Executors.java#L194).
• However, there are by default #CPUs*2 worker verticles running in a standard omero-ms-image-region deployment (see https://github.com/glencoesoftware/omero-ms-image-region/blob/0.4.x/src/main/java/com/glencoesoftware/omero/ms/image/region/ImageRegionMicroserviceVerticle.java#L123).
• In total this could result in 2*(#CPUs)^2 concurrent threads, which may account for the system issues.
• Because the number of worker verticles already exceeds the number of CPUs, it may be wise to use a single thread executor so we will have at most 2*CPUs concurrent threads rendering (see https://github.com/openjdk-mirror/jdk7u-jdk/blob/master/src/share/classes/java/util/concurrent/Executors.java#L133).

[joshmoore]

@kkoz : potentially related -- ome/omero-renderer#10