Update cache and cache clear strategy

[ChrisJohnNOAA]

While working on #433, I came to the conclusion that we need a bigger update to our file caching strategy. The current approach with aggressive delete full cache for a given dataset is prone to race conditions if the server is in the middle of handling a request when that happens. There's likely a related race condition with a dataset reload in the middle of a request as well, so dataset loading logic may need to be updated.

We also should make a better distinction between request handling cache files (things written to disk during a request in order to allow us to reduce memory usage on the system), and processed dataset information like iso19115 files and subset variable precomputation information.

This would be a good time to re-evaluate using the request handling cache files, and maybe detect when the data is small enough that we can keep it in memory instead of writing to disk. While doing this work we should attempt to improve the request size estimations and load shedding logic.