feat: Added dataset parameters to dataflow debug settings and added tests

[ClementVaillantCodit]

• Added dataset parameters to dataflow debug settings
  • For information, debugSettings.DatasetParameters is of type BinaryData. Data is basically a JSON representation of a Dictionary<string, Dictionary<string, object>>, where the high level dictionary contains the keys of the dataset names, and the low level dictionary contains the actual dataset parameter keys and values
• Added tests, @stijnmoreels since DataSetParameters property is internal, not sure how I could assert that if a dataset parameter is added to a not empty dictionary of parameters, it is added to the dictionary and not overridden. Any suggestions for this please? Should I expose the RunDataFlowOptions.DataSetParameters via a method?

[netlify]

✅ Deploy Preview for arcus-testing canceled.

Name	Link
🔨 Latest commit	386cc4e
🔍 Latest deploy log	https://app.netlify.com/sites/arcus-testing/deploys/674fffa3c96320000803b830

[stijnmoreels]

Great that you found the time to work on this a bit!
Besides these comments and related to your question on 'how to test this', I would not make the DataSetParameters public as asserting on the existence of those parameters in that dictionary does not garantuee that the dataset parameter was 'correctly added'. That can only be determined if those parameters are actually used with a real dataset.

Is it an idea to make the container name of the datasets in the TemporaryDataFactoryDataFlow test fixture configurable with dataset parameters? Or some other metadata stuff (can be anything). The same test fixture can then expose a method to retrieve back that metadata so that the test can assert that indeed adding dataset parameters result in a changed dataset.

So:

• Add integration test with at least three dataset parameters to verify that multiple parameters for the same dataset and for different datasets can be added correctly.
• Update docs\preview\02-Features\06-Integration/01-data-factory.mdx with the new options.AddDataSetParameter(...) method.

[ClementVaillantCodit]

@stijnmoreels I'd appreciate a fresh review on the newly pushed changes please. Added new methods specific to run the dataflow with dataflow params, really unsure if that's a good way to go.
Looking forward to read your comments!

[arcus-automation]

🧪 Code coverage summary

Metric	Value
Line coverage	🟢 91.6%
Branch coverage	🟢 80.9%

Great job! 😎 Your code coverage is higher than my caffeine levels! ☕

[stijnmoreels]

Already a big step forward on testing out whether we can add dataset parameters. Great job!
The remaining comments are all about the same thing: placing the right kind of information in the test body as currently the parameters are a bit hidden away in the TemporaryDataFactoryDataFlow while they are in fact the leading role in the test. As a test reader, it should be clear that we can provide parameters and that this results in a changed interaction with the DataSet.

Folder paths are I think a good way of doing tihs, indeed, as you provide many sub folders - and, so - many parameters. Only: these sub paths should be generated by the test and provided by the test, instead of being the responsibility of the fixture.

Some comments here already give you inspiration, but contact me if you want to have a chat if something is not clear. This is mostly about limiting duplication/code and place the right focus in the test.

Thx!

[stijnmoreels]

We're almost there, some final remarks on duplication and readability. Biggest chunk of work is behind us. Great job!

[stijnmoreels]

I think we're there. Thx again!