Refine examples/ for versioned schemas

[mferrera]

There are a few ways we need to improve examples/ as we move into versioned schemas. Files in this directory are used both in tests and in documentation.

• examples/0.8.0 contains metadata examples that are manually updated
  • These should be automatically generated
  • We may only care about the latest version of metadata (?)
  • These are important to tests. This fixture is used in quite a bit of tests:

  • fmu-dataio/tests/conftest.py

    Lines 448 to 456 in 5aedcfb

    	@pytest.fixture(scope="session")
    	def metadata_examples():
    	"""Parse all metadata examples.
    	Returns:
    	Dict: Dictionary with filename as key, file contents as value.
    	"""
    	return _metadata_examples()

  • fmu-dataio/tests/utils.py

    Lines 45 to 49 in 5aedcfb

    	def _metadata_examples():
    	return {
    	path.name: _isoformat_all_datetimes(_parse_yaml(path))
    	for path in Path(".").absolute().glob("examples/0.8.0/*.yml")
    	}

• examples/s/d/nn contains a mocked Ert ensemble that runs scripts and generates metadata
  • The file structure could be improved
  • We should generate more file types (ideally, one per content type)
  • Has at least one reference in tests

  • fmu-dataio/tests/test_integration/conftest.py

    Lines 78 to 83 in 5aedcfb

    	shutil.copy(
    	source_root
    	/ "examples/s/d/nn/xcase/realization-0/iter-0/rms"
    	/ "output/maps/props/poro_average.gri",
    	tmp_path / "ert/output/maps/props/poro_average.gri",
    	)

  • Is featured quite a bit in docs

  • fmu-dataio/docs/src/examples.rst

    Lines 31 to 40 in 5aedcfb

    	.. literalinclude:: ../../examples/s/d/nn/xcase/realization-0/iter-0/rms/bin/export_faultpolygons.py
    	:language: python
    	Press + to see generated YAML file.
    	.. toggle::
    	.. literalinclude:: ../../examples/s/d/nn/xcase/realization-0/iter-0/share/results/polygons/.volantis_gp_top--faultlines.pol.yml
    	:language: yaml

  • fmu-dataio/.readthedocs.yml

    Line 9 in 5aedcfb

    - bash examples/run_examples.sh

  • fmu-dataio/.github/workflows/fmudataio-documention.yml

    Line 28 in 5aedcfb

    run: sh examples/run_examples.sh

We should combine these two things into a simpler, cohesive, but comprehensive way of managing documentation, examples, and testing altogether.

It's possible that this should be part of the update-schemas script.

[mferrera]

It is possible to generate a new example directory for every schema version, and store those examples forever. Is this something we want to do?

[slangeveld]

What I was thinking so far was to create a folder structure like this

/examples
   schemas/
      templates/
      0.8.0/
      0.8.1/
      .
      .
      .
      1.0.0/    

where the folder templates would coutain templates for the examples:

/templates
   case_template.yml
   aggregated_surface_depth_template.yml
   surface_depth.yml
   (etc..)

Then update-schema could use the templates, create a new folder and generate yml-files for each new schema.

We should maybe delete some of the schema example folders after some time, but I guess that is not something we need to decide on yet.

Not sure where to place the files from examples/s/d/nn into the picture yet..

[mferrera]

One way I had thought of doing this is to have a script (or set of scripts) that generates metadata per content type. So basically do a full

export_data = ExportData(..)
...export()...

And do this for several (in the long run, all) content types.

fmu-dataio/src/fmu/dataio/_models/fmu_results/enums.py

Lines 28 to 57 in 6c66f57

	class Content(str, Enum):
	"""The content type of a given data object."""
	depth = "depth"
	facies_thickness = "facies_thickness"
	fault_lines = "fault_lines"
	fault_properties = "fault_properties"
	field_outline = "field_outline"
	field_region = "field_region"
	fluid_contact = "fluid_contact"
	khproduct = "khproduct"
	lift_curves = "lift_curves"
	named_area = "named_area"
	parameters = "parameters"
	pinchout = "pinchout"
	property = "property"
	pvt = "pvt"
	regions = "regions"
	relperm = "relperm"
	rft = "rft"
	seismic = "seismic"
	simulationtimeseries = "simulationtimeseries"
	subcrop = "subcrop"
	thickness = "thickness"
	time = "time"
	timeseries = "timeseries"
	transmissibilities = "transmissibilities"
	velocity = "velocity"
	volumes = "volumes"
	wellpicks = "wellpicks"

examples/s/d/nn contains a few of these scripts now, that generate data based upon the latest models.

Then, when update-schemas is run, and we see a version change (maybe it is just for major/minor versions), it invokes these exports scripts, exports the metadata to examples/x.y.z/, and renames the metadata files to human readable ones. (It can probably reach deeper into code and just call the metadata export function).

Then we can always keeps our testing and examples up-to-date with the latest version of the schema/models. For the documentation build we may have to see if we can do something clever like have a symlink "latest" in examples/ that points to the most recent version, and see if docs can read into the symlink and build examples in from there.

Maybe we should have a sync? There's a fair bit going on here, but it's important for the longevity of the current testing regime we have which will start breaking down when we bump versions