October 2023 Pre-Release Build (#227)

* Update solution gitignore Update solution gitignore * Initial commit for function initial commit for fhir-service-event-functions * Update function app to use service bus Update fhir resource created function app to trigger from service bus * Initial commit * Cleanup of vs files from repo * Cleanup of old duplicate files * Refactor into different functions Reorganize and refactor initial fhir resource created export function into two functions connected via a service bus queue * itial commit * Initial AzureAppConfiguration Migration * Minor fix for Azure App Config * initial separation of functions * merge + reorg of processmessage * Embeded the returned content bundle in the reporting bundle * reorg of github repository * added apim yaml file (#165) * Mass Rename (#164) * Logic app workflow file upload (#169) * Update App Config (#170) * clean * Update App Config * Update for Keyvault through AppConfig * Update process message (#180) update processmessage to log more information to facilitate integration testing prior to the connectathon and also allow non-error level validation message to not prevent ingestion of the bundle * Nicholas/2023/update export flatten format (#185) * Update process message update processmessage to log more information to facilitate integration testing prior to the connectathon and also allow non-error level validation message to not prevent ingestion of the bundle * update flatten logic to be proper json * initial prototype for sorting datalake into folders * Nicholas/2023/combine export event (#187) * Update process message update processmessage to log more information to facilitate integration testing prior to the connectathon and also allow non-error level validation message to not prevent ingestion of the bundle * Add skip validation flag to processmessage * initial combine * update processmessage to return Content-Type header with application/fhir+json * Always export as flatten in addition to main export settings * Removed old commented out code to prevent confusion * added changelog * added root .gitignore, removed .vs * Repo clean-up, change to 2 folders, minor readme add, and Flu serv bundle comparison with summary (#213) * flu-serv bundle comparison before and after with fhir responses * flu-serv comparison added more responses for bundles, before and after * flu-serv comparison added readme with report * repo clean-up in 2 folders, and added minor readme * added to reamde * Nicholas/2023/combine export event (#216) * Update process message update processmessage to log more information to facilitate integration testing prior to the connectathon and also allow non-error level validation message to not prevent ingestion of the bundle * Add skip validation flag to processmessage * initial combine * update processmessage to return Content-Type header with application/fhir+json * Always export as flatten in addition to main export settings * Removed old commented out code to prevent confusion * reorg of combined event/export * update gitignore for pubxml * Rename ProcessExport to DataExport * Nicholas/2023/update message bundle processing (#217) * Update ProcessMessage for ingesting entire bundle FHIR-275 * update to folder structure and smart basicauth endpoint FHIR-265 * Update dataexport to use service principal for EDAV FHIR-265 * Add ability to export data to multiple locations (#218) Added ability to export data to multiple locations based on a config file which filters based off of resource profile used * Update CHANGELOG.md * post sync cleanup of deprecated folders * Mikeschi (#221) * September Connectathon 2023 Pre-Release Build (#219) * Update solution gitignore Update solution gitignore * Initial commit for function initial commit for fhir-service-event-functions * Update function app to use service bus Update fhir resource created function app to trigger from service bus * Initial commit * Cleanup of vs files from repo * Cleanup of old duplicate files * Refactor into different functions Reorganize and refactor initial fhir resource created export function into two functions connected via a service bus queue * itial commit * Initial AzureAppConfiguration Migration * Minor fix for Azure App Config * initial separation of functions * merge + reorg of processmessage * Embeded the returned content bundle in the reporting bundle * reorg of github repository * added apim yaml file (#165) * Mass Rename (#164) * Logic app workflow file upload (#169) * Update App Config (#170) * clean * Update App Config * Update for Keyvault through AppConfig * Update process message (#180) update processmessage to log more information to facilitate integration testing prior to the connectathon and also allow non-error level validation message to not prevent ingestion of the bundle * Nicholas/2023/update export flatten format (#185) * Update process message update processmessage to log more information to facilitate integration testing prior to the connectathon and also allow non-error level validation message to not prevent ingestion of the bundle * update flatten logic to be proper json * initial prototype for sorting datalake into folders * Nicholas/2023/combine export event (#187) * Update process message update processmessage to log more information to facilitate integration testing prior to the connectathon and also allow non-error level validation message to not prevent ingestion of the bundle * Add skip validation flag to processmessage * initial combine * update processmessage to return Content-Type header with application/fhir+json * Always export as flatten in addition to main export settings * Removed old commented out code to prevent confusion * added changelog * added root .gitignore, removed .vs * Repo clean-up, change to 2 folders, minor readme add, and Flu serv bundle comparison with summary (#213) * flu-serv bundle comparison before and after with fhir responses * flu-serv comparison added more responses for bundles, before and after * flu-serv comparison added readme with report * repo clean-up in 2 folders, and added minor readme * added to reamde * Nicholas/2023/combine export event (#216) * Update process message update processmessage to log more information to facilitate integration testing prior to the connectathon and also allow non-error level validation message to not prevent ingestion of the bundle * Add skip validation flag to processmessage * initial combine * update processmessage to return Content-Type header with application/fhir+json * Always export as flatten in addition to main export settings * Removed old commented out code to prevent confusion * reorg of combined event/export * update gitignore for pubxml * Rename ProcessExport to DataExport * Nicholas/2023/update message bundle processing (#217) * Update ProcessMessage for ingesting entire bundle FHIR-275 * update to folder structure and smart basicauth endpoint FHIR-265 * Update dataexport to use service principal for EDAV FHIR-265 * Add ability to export data to multiple locations (#218) Added ability to export data to multiple locations based on a config file which filters based off of resource profile used * Update CHANGELOG.md * post sync cleanup of deprecated folders --------- Co-authored-by: mslentz2 <[email protected]> Co-authored-by: mslentz2 <[email protected]> Co-authored-by: aaronegrant <[email protected]> Co-authored-by: Brian Sokolowski <[email protected]> Co-authored-by: lrp4 <[email protected]> Co-authored-by: Cosmin Potocean <[email protected]> * Removed JSON configuration files previously used for manual deployments, but were flagged by Fortify as security risks. --------- Co-authored-by: bennettn4 <[email protected]> Co-authored-by: mslentz2 <[email protected]> Co-authored-by: mslentz2 <[email protected]> Co-authored-by: aaronegrant <[email protected]> Co-authored-by: Brian Sokolowski <[email protected]> Co-authored-by: lrp4 <[email protected]> Co-authored-by: Cosmin Potocean <[email protected]> * Nicholas/2023/update process message response (#224) * Update Response code If bundle submitted to process-messaege is invalid and fails fhir profile validation, return 422 * Update response code again * Nicholas/2023/add data purge (#225) * Initial DataPurge * Finish first implementation of purge function * Nicholas/2023/fortify high fixes (#226) * Add validation on passed through bearer token * delete old manual deployment config * TestUpdate to bearer token auth header * update to clean bearertoken again * reshuffled header cleaning * cleanup * clean old manual deploy configs * Update CHANGELOG.md --------- Co-authored-by: mslentz2 <[email protected]> Co-authored-by: mslentz2 <[email protected]> Co-authored-by: aaronegrant <[email protected]> Co-authored-by: Brian Sokolowski <[email protected]> Co-authored-by: lrp4 <[email protected]> Co-authored-by: Cosmin Potocean <[email protected]> Co-authored-by: mikeschi12 <[email protected]>
CDCgov · Oct 27, 2023 · 48d8fde · 48d8fde
1 parent 9649c05
commit 48d8fde
Show file tree

Hide file tree

Showing 25 changed files with 179 additions and 1,517 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,7 +4,13 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [0.2.0] - 2023-05-06
+## [0.3.0] - 2023-10-27
+	- Added DataPurge function as a timed function to remove records from the fire after a period of minimum retention
+	- Added manual bulk uploading utility to repo
+	- Updated $process-message http response codes
+	- Includes various fixes for compliance
+
+## [0.2.0] - 2023-10-02
 	- Functional version used at September 2023 Connectathon
 	- Moved functionality of event-triggered processing merged into single function app with data exporting
 	- Removed now deprecated event function app

diff --git a/...EX.FHIR.Function/CDC.DEX.FHIR.Function.DataExport/CDC.DEX.FHIR.Function.DataExport.csproj b/...EX.FHIR.Function/CDC.DEX.FHIR.Function.DataExport/CDC.DEX.FHIR.Function.DataExport.csproj
@@ -8,6 +8,7 @@
     <PackageReference Include="Azure.Identity" Version="1.8.2" />
     <PackageReference Include="Azure.Messaging.ServiceBus" Version="7.13.1" />
     <PackageReference Include="Azure.Storage.Blobs" Version="12.15.0" />
+    <PackageReference Include="Hl7.Fhir.R4" Version="5.3.0" />
     <PackageReference Include="JsonFlatten" Version="1.0.2" />
     <PackageReference Include="Microsoft.Azure.Functions.Extensions" Version="1.1.0" />
     <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.ServiceBus" Version="5.9.0" />

diff --git a/source/CDC.DEX.FHIR.Function/CDC.DEX.FHIR.Function.DataExport/DataPurge.cs b/source/CDC.DEX.FHIR.Function/CDC.DEX.FHIR.Function.DataExport/DataPurge.cs
@@ -0,0 +1,128 @@
+using System;
+using CDC.DEX.FHIR.Function.DataExport;
+using System.Net.Http;
+using Microsoft.Azure.WebJobs;
+using Microsoft.Azure.WebJobs.Host;
+using Microsoft.Extensions.Configuration;
+using Microsoft.Extensions.Logging;
+using Hl7.Fhir.Rest;
+using System.Net.Http.Headers;
+using CDC.DEX.FHIR.Function.SharedCode;
+using CDC.DEX.FHIR.Function.SharedCode.Util;
+using System.Threading.Tasks;
+using Hl7.Fhir.Model;
+using Hl7.Fhir.Serialization;
+using System.Linq;
+using Newtonsoft.Json.Linq;
+using static Hl7.Fhir.Model.Bundle;
+using System.Collections.Generic;
+
+namespace CDC.DEX.FHIR.Function.ProcessExport
+{
+    public class DataPurge
+    {
+
+        private readonly IHttpClientFactory httpClientFactory;
+        private readonly IConfiguration configuration;
+        /// <summary>
+        /// Constructor
+        /// </summary>
+        /// <param name="httpClientFactory">Http client factory for FhirResourceCreatedExportFunction</param>
+        /// <param name="configuration">App Configuration</param>
+        public DataPurge(IHttpClientFactory httpClientFactory, IConfiguration configuration)
+        {
+            this.httpClientFactory = httpClientFactory;
+            this.configuration = configuration;
+        }
+
+
+        /// <summary>
+        /// Function timed trigger entry point
+        /// </summary>
+        /// <param name="triggerSchedule">The timer schedule which the function is triggered off of, configured dynamically through config</param>
+        /// <param name="log">Function logger</param>
+        [FunctionName("DataPurge")]
+        public async System.Threading.Tasks.Task RunAsync([TimerTrigger("%Purge:Schedule%")] TimerInfo triggerSchedule, ILogger log)
+        {
+            log.LogInformation($"DataPurge Timer trigger function executed at: {DateTime.Now}");
+
+            var serializer = new FhirJsonSerializer(new SerializerSettings()
+            {
+                Pretty = true
+            });
+
+            string bearerToken;
+            using (HttpClient authclient = httpClientFactory.CreateClient())
+            {
+                // get auth token
+                bearerToken = await FhirServiceUtils.GetFhirServerToken(configuration, authclient);
+            }
+            var handler = new AuthorizationMessageHandler();
+            handler.Authorization = new AuthenticationHeaderValue("Bearer", bearerToken);
+
+            var settings = new FhirClientSettings
+            {
+                PreferredFormat = ResourceFormat.Json,
+            };
+
+            var client = new FhirClient(configuration["BaseFhirUrl"], settings, handler);
+
+            //get each profile config
+            string testDestinationConfig = configuration["Export:DestinationConfig"];
+            JObject testDestinationConfigJSON = JObject.Parse(testDestinationConfig);
+
+            Dictionary<string,string> resourceIdsToDelete = new Dictionary<string,string>();
+
+            foreach (JObject profileConfig in testDestinationConfigJSON["Mappings"].Values<JObject>())
+            {
+                //take each profile and add it to a search query,
+                //but only if it has a DaysToRetain (if it's missing or less than 1, never purge)
+
+                if (profileConfig.ContainsKey("DaysToRetain") && profileConfig["DaysToRetain"].Value<int>() >= 0)
+                {
+
+                    int daysToRetain = profileConfig["DaysToRetain"].Value<int>();
+
+                    var searchParam = new SearchParams().Where($"_lastUpdated=le{DateTime.UtcNow.AddDays(-daysToRetain).ToString("s")}");
+
+                    foreach (string profilePath in profileConfig["ProfilePathsToFilter"].Values<string>())
+                    {
+                        searchParam.Where($"_profile={profilePath}");
+                    }
+
+                    Bundle results = await client.SearchAsync<Bundle>(searchParam);
+
+                    //int resultCount = 0;
+
+                    do
+                    {
+                        foreach (EntryComponent entryComponent in results.Entry)
+                        {
+                            resourceIdsToDelete.Add(entryComponent.Resource.Id, entryComponent.Resource.TypeName);
+                        }
+                        //resultCount += results.Entry.Count();
+                    }
+                    while ((results = await client.ContinueAsync(results)) != null);
+
+
+                    //log.LogInformation(searchParam.ToUriParamList().ToQueryString());
+                    //log.LogInformation(resultCount.ToString());
+                    //log.LogInformation(serializer.SerializeToString(results));
+
+                }
+            }
+
+            // delete the history and the resource
+            foreach(string resourceId in resourceIdsToDelete.Keys)
+            {
+                SearchParams searchParams = new SearchParams();
+                searchParams.Add("_id", resourceId);
+                searchParams.Add("hardDelete", "true");
+                await client.DeleteAsync(resourceIdsToDelete[resourceId], searchParams);
+                log.LogInformation($"Hard Delete Successful: {resourceIdsToDelete[resourceId]}/{resourceId}");
+            }
+
+
+        }
+    }
+}
diff --git a/...nction.DataExport/Properties/PublishProfiles/fhir-service-data-export - Zip Deploy.pubxml b/...nction.DataExport/Properties/PublishProfiles/fhir-service-data-export - Zip Deploy.pubxml
diff --git a/...aExport/Properties/PublishProfiles/fhir-service-event-export-combined - Zip Deploy.pubxml b/...aExport/Properties/PublishProfiles/fhir-service-event-export-combined - Zip Deploy.pubxml
diff --git a/...Export/Properties/PublishProfiles/fhir-service-event-export-combined - Zip Deploy1.pubxml b/...Export/Properties/PublishProfiles/fhir-service-event-export-combined - Zip Deploy1.pubxml
diff --git a/...on.DataExport/Properties/PublishProfiles/fhir-service-export-function - Zip Deploy.pubxml b/...on.DataExport/Properties/PublishProfiles/fhir-service-export-function - Zip Deploy.pubxml
diff --git a/...roperties/ServiceDependencies/fhir-service-data-export - Zip Deploy/appInsights1.arm.json b/...roperties/ServiceDependencies/fhir-service-data-export - Zip Deploy/appInsights1.arm.json