Add StackExchange.Utils.Configuration (#15)

* Add StackExchange.Utils.Configuration

This commit adds a helper library that performs substitution and prefixing for `IConfiguration`-based configuration sources. It allows a value in the configuration tree to reference other values within the configuration system using a placeholder syntax `${key}` or `${nested:key}`. In addition, prefixing allows a "namespace" prefix to be applied to subset of configuration, making it possible to segment configuration into logical areas.

This is particularly useful for storing secrets in a different, secure location but in a way that it makes it easy to compose configuration like connection strings with dealing with it inside the application. E.g. consider the following JSON:

**appsettings.json**
```json
{
    "ConnectionStrings": {
        "Database": "Server=srv01;Database=db01;User ID=${secrets:Database:UserId};Password=${secrets:Database:Password}"
    }
}
```

**secrets.json**
```json
{
    "Database": {
        "UserId": "User123",
        "Password": "Password123!"
    }
}
```

This instructs the substitution provider to lookup the keys `secrets:Database:UserId` and `secrets:Database:Password` from configuration again and replaces them in the value returned for `ConnectionStrings:Database`.

To support this a `ConfigurationBuilder` is configured as follows:

```c#
var configuration = new ConfigurationBuilder()
    .WithPrefix(
        "secrets",
        // everything in this configuration builder will be prefixed with "secrets:"
        c => c.AddJsonFile("secrets.json")
    )
    .WithSubstitution(
        // values in this configuration builder will have substitutions
        // replaced prior to being returned to the caller
        c => c.AddJsonFile("appsettings.json")
    )
    .Build();
```

Here we're loading a JSON file called `secrets.json` (it could equally be any source supported by the `IConfiguration` system) and prefixing it with `secrets:`. Then, we load `appsettings.json` with support for substitutions. If a caller asks the `IConfiguration` that is produced for a connection string:

```c#
var connectionString = configuration.GetConnectionString("Database");
```

That value will be returned as `Server=srv01;Database=db01;User ID=User123;Password=Password123!`.

- Build scripts updated to build all projects
- Builds scripts updated to support Linux/MacOS
- `.gitignore` tweaks for Rider
- Move to `LangVersion = latest`
- `version.json` in each packages directories
- Use `Microsoft.NETFramework.ReferenceAssemblies` for net462 builds on Linux

* Typos in doc

* Re-enable Kestrel tests that were conditionally compiled out and fix the build breaks

MacOS doesn't support a PKCS12 certificate without a password so `certificate.pfx` has been updated to add a password (`password`). Also MacOS does not support ALPN so HTTP/2 over TLS is not currently available - this commit skips those tests on MacOS.

* Clarifying comment

* Fix up build scripts to use `Build.csproj` instead of individual project files

* Argument validation + tests

* Support `Set` in the prefix and substitution providers, add tests

Author: deanward81

Readme.md

@@ -58,5 +58,77 @@ settings.ProfileRequest = request => MiniProfiler.Current.CustomTiming("http", r
 settings.ProfileGeneral = name => MiniProfiler.Current.Step(name);
 ```
 
-#### License
+### StackExchange.Utils.Configuration
+`StackExchange.Utils.Configuration` is a helper library that performs substitution and prefixing for `IConfiguration`-based configuration sources. It allows a value in the configuration tree to reference other values within the configuration system using a placeholder syntax `${key}` or `${nested:key}`. In addition, prefixing allows a "namespace" prefix to be applied to a subset of configuration, making it possible to segment configuration into logical areas.
+
+This is particularly useful for storing secrets in a different, secure location but in a way that it makes it easy to compose configuration values like connection strings without dealing with it inside the application. E.g. consider the following files:
+
+**appsettings.json**
+```json
+{
+    "ConnectionStrings": {
+        "Database": "Server=srv01;Database=db01;User ID=${secrets:Database:UserId};Password=${secrets:Database:Password}"
+    }
+}
+```
+
+**secrets.json**
+```json
+{
+    "Database": {
+        "UserId": "User123",
+        "Password": "Password123!"
+    }
+}
+```
+
+This instructs the substitution provider to lookup the keys `secrets:Database:UserId` and `secrets:Database:Password` from the configuration system and replaces them in the value returned for `ConnectionStrings:Database`.
+
+To support this a `ConfigurationBuilder` is configured as follows:
+
+```c#
+var configuration = new ConfigurationBuilder()
+    .WithPrefix(
+        "secrets",
+        // everything in this configuration builder will be prefixed with "secrets:"
+        c => c.AddJsonFile("secrets.json")
+    )
+    .WithSubstitution(
+        // values in this configuration builder will have substitutions
+        // replaced prior to being returned to the caller
+        c => c.AddJsonFile("appsettings.json")
+    )
+    .Build();
+```
+
+Here we're loading a JSON file called `secrets.json` (it could equally be any source supported by the `IConfiguration` system - ideally something secure like Azure KeyVault or Hashicorp's Vault) and prefixing it with `secrets:`. Then, we load `appsettings.json` with support for substitutions. If a caller asks the `IConfiguration` that is produced for a connection string:
+
+```c#
+var connectionString = configuration.GetConnectionString("Database");
+```
+
+That value will be returned as `Server=srv01;Database=db01;User ID=User123;Password=Password123!`.
+
+#### Substituting existing values
+In some cases it's useful to be able to substitute placeholders in existing strings. Support for that is provided by the `SubstitutionHelper`:
+
+```c#
+var configuration = new ConfigurationBuilder()
+    .WithPrefix(
+        "secrets",
+        // everything in this configuration builder will be prefixed with "secrets:"
+        c => c.AddJsonFile("secrets.json")
+    )
+    .Build();
+
+var value = "Server=srv01;Database=db01;User ID=${secrets:Database:UserId};Password=${secrets:Database:Password}";
+var valueWithSubstitution = SubstitutionHelper.ReplaceSubstitutionPlaceholders(value, configuration);
+```
+
+This will do exactly the same as if the value was substituted within the configuration system itself - the returned value will be `Server=srv01;Database=db01;User ID=User123;Password=Password123!`.
+
+#### Notes
+If a value has substitution placeholders that could not be replaced they are left intact - only placeholder keys that can be located in the configuration system are replaced.
+
+### License
 StackExchange.Utils is licensed under the [MIT license](https://github.com/StackExchange/StackExchange.Utils/blob/master/LICENSE.txt).

StackExchange.Utils.sln

@@ -25,9 +25,11 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution
 		build.ps1 = build.ps1
 		global.json = global.json
 		nuget.config = nuget.config
-		version.json = version.json
+		build.sh = build.sh
 	EndProjectSection
 EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "StackExchange.Utils.Configuration", "src\StackExchange.Utils.Configuration\StackExchange.Utils.Configuration.csproj", "{7BADB8FA-9723-4B10-B71C-427F2431D4FE}"
+EndProject
 Global
 	GlobalSection(SolutionConfigurationPlatforms) = preSolution
 		Debug|Any CPU = Debug|Any CPU
@@ -42,13 +44,18 @@ Global
 		{33716E0F-FE40-4A7A-9F58-1026EF7EBCD2}.Debug|Any CPU.Build.0 = Debug|Any CPU
 		{33716E0F-FE40-4A7A-9F58-1026EF7EBCD2}.Release|Any CPU.ActiveCfg = Release|Any CPU
 		{33716E0F-FE40-4A7A-9F58-1026EF7EBCD2}.Release|Any CPU.Build.0 = Release|Any CPU
+		{7BADB8FA-9723-4B10-B71C-427F2431D4FE}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{7BADB8FA-9723-4B10-B71C-427F2431D4FE}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{7BADB8FA-9723-4B10-B71C-427F2431D4FE}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{7BADB8FA-9723-4B10-B71C-427F2431D4FE}.Release|Any CPU.Build.0 = Release|Any CPU
 	EndGlobalSection
 	GlobalSection(SolutionProperties) = preSolution
 		HideSolutionNode = FALSE
 	EndGlobalSection
 	GlobalSection(NestedProjects) = preSolution
 		{168B503A-428F-499D-99A7-8EFC47A5FEDF} = {F0CFAC4D-516B-45DC-8F66-D58E3B1C04E1}
 		{33716E0F-FE40-4A7A-9F58-1026EF7EBCD2} = {9133A680-3A8F-4662-AA58-B59BBDD0A60E}
+		{7BADB8FA-9723-4B10-B71C-427F2431D4FE} = {F0CFAC4D-516B-45DC-8F66-D58E3B1C04E1}
 	EndGlobalSection
 	GlobalSection(ExtensibilityGlobals) = postSolution
 		SolutionGuid = {F211D702-85D2-4159-9B42-60B6177497B7}

appveyor.yml

@@ -17,11 +17,11 @@ nuget:
   disable_publish_on_pr: true
 
 build_script:
-  - ps: .\build.ps1 -PullRequestNumber "$env:APPVEYOR_PULL_REQUEST_NUMBER" -CreatePackages $true
+  - ps: ./build.ps1 -PullRequestNumber "$env:APPVEYOR_PULL_REQUEST_NUMBER" -CreatePackages $true
 
 test: off
 artifacts:
-- path: .\.nupkgs\*.nupkg
+- path: ./.nupkgs/*.nupkg
 
 deploy:
 - provider: NuGet

build.ps1

@@ -10,10 +10,7 @@ Write-Host "  CreatePackages: $CreatePackages"
 Write-Host "  RunTests: $RunTests"
 Write-Host "  dotnet --version:" (dotnet --version)
 
-$packageOutputFolder = "$PSScriptRoot\.nupkgs"
-$projectsToBuild =
-    'StackExchange.Utils.Http'
-
+$packageOutputFolder = Join-Path $PSScriptRoot ".nupkgs"
 $testsToRun =
     'StackExchange.Utils.Tests'
 
@@ -23,40 +20,30 @@ if ($PullRequestNumber) {
 }
 
 Write-Host "Building solution..." -ForegroundColor "Magenta"
-dotnet restore ".\StackExchange.Utils.sln" /p:CI=true
-dotnet build ".\StackExchange.Utils.sln" -c Release /p:CI=true
+dotnet build ./Build.csproj -c Release /p:CI=true
 Write-Host "Done building." -ForegroundColor "Green"
 
 if ($RunTests) {
-    foreach ($project in $testsToRun) {
-        Write-Host "Running tests: $project (all frameworks)" -ForegroundColor "Magenta"
-        Push-Location ".\tests\$project"
-
-        dotnet test -c Release
-        if ($LastExitCode -ne 0) {
-            Write-Host "Error with tests, aborting build." -Foreground "Red"
-            Pop-Location
-            Exit 1
-        }
-
-        Write-Host "Tests passed!" -ForegroundColor "Green"
-	    Pop-Location
+    Write-Host "Running tests (all frameworks)" -ForegroundColor "Magenta"
+    dotnet test ./Build.csproj -c Release
+    if ($LastExitCode -ne 0) {
+        Write-Host "Error with tests, aborting build." -Foreground "Red"
+        Exit 1
     }
+    Write-Host "Tests passed!" -ForegroundColor "Green"
 }
 
 if ($CreatePackages) {
-    mkdir -Force $packageOutputFolder | Out-Null
+    if (!(Test-Path $packageOutputFolder)) {
+        New-Item -ItemType Directory $packageOutputFolder | Out-Null
+    }
     Write-Host "Clearing existing $packageOutputFolder..." -NoNewline
-    Get-ChildItem $packageOutputFolder | Remove-Item
+    Get-ChildItem $packageOutputFolder -ErrorAction Ignore | Remove-Item
     Write-Host "done." -ForegroundColor "Green"
 
     Write-Host "Building all packages" -ForegroundColor "Green"
 
-    foreach ($project in $projectsToBuild) {
-        Write-Host "Packing $project (dotnet pack)..." -ForegroundColor "Magenta"
-        dotnet pack ".\src\$project\$project.csproj" --no-build -c Release /p:PackageOutputPath=$packageOutputFolder /p:NoPackageAnalysis=true /p:CI=true
-        Write-Host ""
-    }
+    dotnet pack ./Build.csproj --no-build -c Release -o $packageOutputFolder /p:NoPackageAnalysis=true /p:CI=true
 }
 
 Write-Host "Done."

build.sh

@@ -0,0 +1,2 @@
+#!/bin/sh
+pwsh -NoProfile -NoLogo -ExecutionPolicy unrestricted -Command "[System.Threading.Thread]::CurrentThread.CurrentCulture = ''; [System.Threading.Thread]::CurrentThread.CurrentUICulture = '';& './build.ps1' $*; exit $LASTEXITCODE"

src/Directory.Build.props

@@ -5,7 +5,7 @@
     <OutputPath>bin\$(Configuration)\</OutputPath>
     <DebugSymbols>true</DebugSymbols>
     <DebugType>embedded</DebugType>
-    <LangVersion>7.3</LangVersion>
+    <LangVersion>latest</LangVersion>
     <GenerateDocumentationFile>true</GenerateDocumentationFile>
     <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
 

src/StackExchange.Utils.Configuration/CompositeConfigurationProvider.cs

@@ -0,0 +1,45 @@
+using System;
+using System.Collections.Generic;
+using System.Collections.Immutable;
+using System.Linq;
+using Microsoft.Extensions.Configuration;
+using Microsoft.Extensions.Primitives;
+
+namespace StackExchange.Utils
+{
+    internal abstract class CompositeConfigurationProvider : IConfigurationProvider
+    {
+        protected CompositeConfigurationProvider(IConfigurationRoot configurationRoot)
+        {
+            ConfigurationRoot = configurationRoot;
+        }
+
+        protected IConfigurationRoot ConfigurationRoot { get; }
+
+        public virtual IEnumerable<string> GetChildKeys(IEnumerable<string> earlierKeys, string parentPath)
+            => ConfigurationRoot.Providers.SelectMany(x => x.GetChildKeys(earlierKeys, parentPath));
+        
+        public IChangeToken GetReloadToken() => new CompositeChangeToken(
+            ConfigurationRoot.Providers.Select(x => x.GetReloadToken()).ToImmutableArray()
+        );
+
+        private bool _initialLoad = true;
+        public void Load()
+        {
+            if (_initialLoad)
+            {
+                _initialLoad = false;
+                return;
+            }
+            ConfigurationRoot.Reload();
+        }
+
+        public virtual void Set(string key, string value) => ConfigurationRoot[key] = value;
+
+        public virtual bool TryGet(string key, out string value)
+        {
+            value = ConfigurationRoot[key];
+            return value != null;
+        }
+    }
+}

src/StackExchange.Utils.Configuration/ConfigurationBuilderExtensions.cs

@@ -0,0 +1,44 @@
+using System;
+using Microsoft.Extensions.Configuration;
+
+namespace StackExchange.Utils
+{
+    /// <summary>
+    /// Extension methods for <see cref="IConfigurationBuilder"/> to provide prefixing
+    /// and substitution support in <see cref="IConfiguration"/>.
+    /// </summary>
+    public static class ConfigurationBuilderExtensions
+    {
+        /// <summary>
+        /// Wraps a child <see cref="IConfigurationBuilder"/> with support for substituting values
+        /// of the form '${secrets:Database:Password}' with configuration from elsewhere in the
+        /// configuration system. The key in braces is looked and used to rewrite the value returned
+        /// from configuration. 
+        /// </summary>
+        public static IConfigurationBuilder WithSubstitution(this IConfigurationBuilder builder, Action<IConfigurationBuilder> action)
+        {
+            if (action == null) throw new ArgumentNullException(nameof(action));
+            
+            var childBuilder = new ConfigurationBuilder();
+            action(childBuilder);
+            builder.Add(new SubstitutingConfigurationSource(childBuilder));
+            return builder;
+        }
+
+        /// <summary>
+        /// Wraps a child <see cref="IConfigurationBuilder"/> with support for a prefix on all keys
+        /// contained within it. This can be used to effectively namespace a set of configuration values
+        /// and is useful when used in combination with <see cref="WithSubstitution"/>.
+        /// </summary>
+        public static IConfigurationBuilder WithPrefix(this IConfigurationBuilder builder, string prefix, Action<IConfigurationBuilder> action)
+        {
+            if (string.IsNullOrEmpty(prefix)) throw new ArgumentNullException(nameof(prefix));
+            if (action == null) throw new ArgumentNullException(nameof(action));
+            
+            var childBuilder = new ConfigurationBuilder();
+            action(childBuilder);
+            builder.Add(new PrefixedConfigurationSource(prefix, childBuilder));
+            return builder;
+        }
+    }
+}

src/StackExchange.Utils.Configuration/PrefixedConfigurationProvider.cs

@@ -0,0 +1,56 @@
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using Microsoft.Extensions.Configuration;
+
+namespace StackExchange.Utils
+{
+    internal class PrefixedConfigurationProvider : CompositeConfigurationProvider
+    {
+        private readonly string _prefixWithDelimiter;
+        public const char Delimiter = ':';
+        
+        public PrefixedConfigurationProvider(string prefix, IConfigurationRoot configurationRoot) : base(configurationRoot)
+        {
+            _prefixWithDelimiter = prefix + Delimiter;
+        }
+
+        public override IEnumerable<string> GetChildKeys(IEnumerable<string> earlierKeys, string parentPath)
+        {
+            var earlierKeyList = earlierKeys.ToList();
+            foreach (var provider in ConfigurationRoot.Providers)
+            {
+                foreach (var childKey in provider.GetChildKeys(earlierKeyList, parentPath))
+                {
+                    yield return _prefixWithDelimiter + childKey;
+                }
+            }
+        }
+
+        public override void Set(string key, string value)
+        {
+            if (!key.StartsWith(_prefixWithDelimiter) || key.Length == _prefixWithDelimiter.Length)
+            {
+                return;
+            }
+            
+            // TODO: make this moar efficient!
+            // slice off the prefix so we can fetch from our underlying providers
+            base.Set(key.AsSpan().Slice(_prefixWithDelimiter.Length).ToString(), value);
+        }
+        
+        public override bool TryGet(string key, out string value)
+        {
+            if (!key.StartsWith(_prefixWithDelimiter) || key.Length == _prefixWithDelimiter.Length)
+            {
+                value = null;
+                return false;
+            }
+
+            // TODO: make this moar efficient!
+            // slice off the prefix so we can fetch from our underlying providers
+            var keyWithoutPrefix = key.AsSpan().Slice(_prefixWithDelimiter.Length);
+            return base.TryGet(keyWithoutPrefix.ToString(), out value);
+        }
+    }
+}

src/StackExchange.Utils.Configuration/PrefixedConfigurationSource.cs

@@ -0,0 +1,18 @@
+using Microsoft.Extensions.Configuration;
+
+namespace StackExchange.Utils
+{
+    internal class PrefixedConfigurationSource : IConfigurationSource
+    {
+        private readonly IConfigurationBuilder _childBuilder;
+        private readonly string _prefix;
+
+        public PrefixedConfigurationSource(string prefix, ConfigurationBuilder childBuilder)
+        {
+            _childBuilder = childBuilder;
+            _prefix = prefix;
+        }
+
+        public IConfigurationProvider Build(IConfigurationBuilder builder) => new PrefixedConfigurationProvider(_prefix, _childBuilder.Build());
+    }
+}

src/StackExchange.Utils.Configuration/StackExchange.Utils.Configuration.csproj

@@ -0,0 +1,13 @@
+<Project Sdk="Microsoft.NET.Sdk">
+
+  <PropertyGroup>
+    <OutputType>Library</OutputType>
+    <TargetFramework>netcoreapp3.1</TargetFramework>
+  </PropertyGroup>
+
+  <ItemGroup>
+    <PackageReference Include="Microsoft.Extensions.Configuration" Version="3.1.6" />
+    <PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" Version="3.1.6" />
+  </ItemGroup>
+
+</Project>