Active key rotation

[bbockelm]

One of the challenges for the key rotation PR (#1748) is deciding when it is "OK" to start signing tokens with the latest generated key.

The answer is certainly not "immediately" -- it takes 10-15 minutes for a key to propagate from registry to director due to caching effects.

In the first attempt at #1748, we tried keeping a copy of the "previous key" in memory with the idea that the previous key, after some time period, would become the most recent key. That was also futile as, if the server restarts, we have no clue how long the "latest" has been the latest.

I think a spin on that "previous key" idea is going to be the best option. However, instead of keeping the previous key in memory, we should persist to the server's SQLite database.

The table should have a row per key ID keeping the following data:

• First observed
• Last observed
• First upload to registry (hopefully this is quite close to the "first observed"!)

Then, the active key can be used if it's been X minutes since the upload to the registry (should be a configurable parameter with a default of 20 minutes). Keep the database updates out of any hot paths -- instead, we only need to synchronize it every few minutes. An opportune time would be around when config.RefreshKeys() is invoked.

Please coordinate with @patrickbrophy to ensure this goes into a generic "server" database, not the per-service type database.

[bbockelm]

Additional note -- we should be careful to execute this logic only for the server case. I don't want to accidentally introduce a database for the client.