You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/source/rst_doc_files/classes/datageneration/datagenerator.rst
+15-16
Original file line number
Diff line number
Diff line change
@@ -4,10 +4,10 @@ Datagenerator
4
4
=============
5
5
6
6
The :class:`~f3dasm.datageneration.DataGenerator` class is the main class of the :mod:`~f3dasm.datageneration` module.
7
-
It is used to generate :attr:`~f3dasm.design.ExperimentData.output_data` for the :class:`~f3dasm.design.ExperimentData` by taking a :class:`~f3dasm.design.ExperimentSample` object.
7
+
It is used to generate :attr:`~f3dasm.ExperimentData.output_data` for the :class:`~f3dasm.ExperimentData` by taking a :class:`~f3dasm.ExperimentSample` object.
8
8
9
9
The :class:`~f3dasm.datageneration.DataGenerator` can serve as the interface between the
10
-
:class:`~f3dasm.design.ExperimentData` object and any third-party simulation software.
10
+
:class:`~f3dasm.ExperimentData` object and any third-party simulation software.
11
11
12
12
.. image:: ../../../img/f3dasm-datageneration.png
13
13
:width:70%
@@ -17,27 +17,27 @@ The :class:`~f3dasm.datageneration.DataGenerator` can serve as the interface bet
17
17
|
18
18
19
19
Use the simulator in the data-driven process
20
-
------------------------------------------
20
+
--------------------------------------------
21
21
22
-
In order to run your simulator on each of the :class:`~f3dasm.design.ExperimentSample` of your :class:`~f3dasm.design.ExperimentData`, you follow these steps:
22
+
In order to run your simulator on each of the :class:`~f3dasm.ExperimentSample` of your :class:`~f3dasm.ExperimentData`, you follow these steps:
23
23
In this case, we are utilizing a one of the :ref:`benchmark-functions` to mock a simulator.
24
24
25
-
We provide the datagenerator to the :meth:`~f3dasm.design.ExperimentData.evaluate` function with the :class:`~f3dasm.datageneration.DataGenerator` object as an argument.
25
+
We provide the datagenerator to the :meth:`~f3dasm.ExperimentData.evaluate` function with the :class:`~f3dasm.datageneration.DataGenerator` object as an argument.
Any key-word arguments that need to be passed down to the :class:`~f3dasm.datageneration.DataGenerator` can be passed in the :code:`kwargs` argument of the :meth:`~f3dasm.design.ExperimentData.evaluate` function.
33
+
Any key-word arguments that need to be passed down to the :class:`~f3dasm.datageneration.DataGenerator` can be passed in the :code:`kwargs` argument of the :meth:`~f3dasm.ExperimentData.evaluate` function.
34
34
35
35
36
-
There are three methods available of handeling the :class:`~f3dasm.design.ExperimentSample` objects:
36
+
There are three methods available of handeling the :class:`~f3dasm.ExperimentSample` objects:
37
37
38
-
* :code:`sequential`: regular for-loop over each of the :class:`~f3dasm.design.ExperimentSample` objects in order
39
-
* :code:`parallel`: utilizing the multiprocessing capabilities (with the `pathos <https://pathos.readthedocs.io/en/latest/pathos.html>`_ multiprocessing library), each :class:`~f3dasm.design.ExperimentSample` object is run in a separate core
40
-
* :code:`cluster`: utilizing the multiprocessing capabilities, each :class:`~f3dasm.design.ExperimentSample` object is run in a separate node. After completion of an sample, the node will automatically pick the next available sample. More information on this mode can be found in the :ref:`cluster-mode` section.
38
+
* :code:`sequential`: regular for-loop over each of the :class:`~f3dasm.ExperimentSample` objects in order
39
+
* :code:`parallel`: utilizing the multiprocessing capabilities (with the `pathos <https://pathos.readthedocs.io/en/latest/pathos.html>`_ multiprocessing library), each :class:`~f3dasm.ExperimentSample` object is run in a separate core
40
+
* :code:`cluster`: utilizing the multiprocessing capabilities, each :class:`~f3dasm.ExperimentSample` object is run in a separate node. After completion of an sample, the node will automatically pick the next available sample. More information on this mode can be found in the :ref:`cluster-mode` section.
41
41
42
42
43
43
Implement your simulator
@@ -65,16 +65,16 @@ Once you have created the ``data_generator`` object, you can plug-in a pre-proc
65
65
66
66
pre-processing
67
67
^^^^^^^^^^^^^^
68
-
The preprocessing method is used to create a simulator input file from the information in the :class:`~f3dasm.design.ExperimentSample`.
68
+
The preprocessing method is used to create a simulator input file from the information in the :class:`~f3dasm.ExperimentSample`.
69
69
70
70
71
71
This method should adhere to a few things:
72
72
73
-
* The first argument of the function needs to be ``experiment_sample`` of type :class:`~f3dasm.design.ExperimentSample`.
73
+
* The first argument of the function needs to be ``experiment_sample`` of type :class:`~f3dasm.ExperimentSample`.
74
74
* The method should return None.
75
75
* The method should create the input file ready for the simulator to process with the job_number as name (``experiment_sample.job_number``)
76
76
77
-
You can retrieve the parameters of the :class:`~f3dasm.design.ExperimentSample` object by calling the :meth:`~f3dasm.design.ExperimentSample.get` method.
77
+
You can retrieve the parameters of the :class:`~f3dasm.ExperimentSample` object by calling the :meth:`~f3dasm.ExperimentSample.get` method.
78
78
79
79
You can add the ``pre-process-function`` to the :class:`~f3dasm.datageneration.DataGenerator` object by passing it through the :meth:`~f3dasm.datageneration.DataGenerator.add_pre_process` method:
80
80
@@ -90,13 +90,12 @@ You can add the ``pre-process-function`` to the :class:`~f3dasm.datageneration.D
90
90
post-processing
91
91
^^^^^^^^^^^^^^^
92
92
93
-
* :meth:`~f3dasm.datageneration.DataGenerator.execute` - The main functon call to run the simulation. Returns None.
94
93
The post-processing method converts the output of the simulator to a ``results.pkl`` `pickle <https://docs.python.org/3/library/pickle.html>`_ file.
95
-
This ``results.pkl`` is then loaded into the :class:`~f3dasm.design.ExperimentData` object.
94
+
This ``results.pkl`` is then loaded into the :class:`~f3dasm.ExperimentData` object.
96
95
97
96
This method should adhere to a few things:
98
97
99
-
* The first argument of the function needs to be ``experiment_sample`` of type :class:`~f3dasm.design.ExperimentSample`.
98
+
* The first argument of the function needs to be ``experiment_sample`` of type :class:`~f3dasm.ExperimentSample`.
100
99
* The method should return None.
101
100
* The method read the output of the simulator (it's name is ``experiment_sample.job_number``) and convert it to a ``results.pkl`` file.
102
101
* This pickle file is stored in the current working directory.
Copy file name to clipboardExpand all lines: docs/source/rst_doc_files/classes/design/experimentdata.rst
+22-22
Original file line number
Diff line number
Diff line change
@@ -1,12 +1,12 @@
1
1
Experiment Data
2
2
===============
3
3
4
-
The :class:`~f3dasm.design.ExperimentData` object is the main object used to store implementations of a design-of-experiments,
4
+
The :class:`~f3dasm.ExperimentData` object is the main object used to store implementations of a design-of-experiments,
5
5
keep track of results, perform optimization and extract data for machine learning purposes.
6
6
7
7
All other processses of f3dasm use this object to manipulate and access data about your experiments.
8
8
9
-
The :class:`~f3dasm.design.ExperimentData` object consists of the following attributes:
9
+
The :class:`~f3dasm.ExperimentData` object consists of the following attributes:
10
10
11
11
- :ref:`domain <domain-format>`: The feasible :class:`~f3dasm.design.Domain` of the Experiment. Used for sampling and optimization.
12
12
- :ref:`input_data <input-data-format>`: Tabular data containing the input variables of the experiments as column and the experiments as rows.
@@ -25,7 +25,7 @@ The :class:`~f3dasm.design.ExperimentData` object consists of the following attr
25
25
26
26
Users of :mod:`f3dasm` are advised to not directly manipulate the attributes of the ExperimentData object. Instead, the methods of ExperimentData should be used to manipulate the data.
27
27
28
-
The :class:`~f3dasm.design.ExperimentData` object can be constructed in several ways:
28
+
The :class:`~f3dasm.ExperimentData` object can be constructed in several ways:
29
29
30
30
* :ref:`By providing your own data <experimentdata-own>`
31
31
* :ref:`Reconstructed from the project directory <experimentdata-file>`
@@ -37,15 +37,15 @@ The :class:`~f3dasm.design.ExperimentData` object can be constructed in several
37
37
ExperimentData from your own data
38
38
---------------------------------
39
39
40
-
You can construct a :class:`~f3dasm.design.ExperimentData` object by providing it :ref:`input_data <input-data-format>`, :ref:`output_data <output-data-format>`, a :ref:`domain <domain-format>` object and a :ref:`filename <filename-format>`.
40
+
You can construct a :class:`~f3dasm.ExperimentData` object by providing it :ref:`input_data <input-data-format>`, :ref:`output_data <output-data-format>`, a :ref:`domain <domain-format>` object and a :ref:`filename <filename-format>`.
41
41
42
42
.. code-block:: python
43
43
44
44
>>>from f3dasm import ExperimentData
45
45
>>> data = ExperimentData(domain, input_data, output_data)
46
46
47
47
48
-
The following sections will explain how to construct a :class:`~f3dasm.design.ExperimentData` object from your own data.
48
+
The following sections will explain how to construct a :class:`~f3dasm.ExperimentData` object from your own data.
49
49
50
50
.. _domain-format:
51
51
@@ -154,7 +154,7 @@ Several datatypes are supported for the ``output_data`` argument:
>>> data = ExperimentData(input_data="my_experiment_data.csv", domain=domain)
156
156
157
-
If you don't have output data yet, you can also construct an :class:`~f3dasm.design.ExperimentData` object without providing output data.
157
+
If you don't have output data yet, you can also construct an :class:`~f3dasm.ExperimentData` object without providing output data.
158
158
159
159
160
160
.. _filename-format:
@@ -172,7 +172,7 @@ You can provide a string or a path to a directory. If the directory does not exi
172
172
>>> project_dir ="folder/to/my_project_directory"
173
173
>>> data = ExperimentData(project_dir=project_dir)
174
174
175
-
You can also set the project directoy manually after creation with the :meth:`~f3dasm.design.ExperimentData.set_project_dir` method"
175
+
You can also set the project directoy manually after creation with the :meth:`~f3dasm.ExperimentData.set_project_dir` method"
176
176
177
177
.. code-block:: python
178
178
@@ -187,7 +187,7 @@ You can also set the project directoy manually after creation with the :meth:`~f
187
187
ExperimentData from project directory
188
188
-------------------------------------
189
189
190
-
If you already have constructed the :class:`~f3dasm.design.ExperimentData` object before, you can retrieve it from disk by calling the :meth:`~f3dasm.design.ExperimentData.from_file`
190
+
If you already have constructed the :class:`~f3dasm.ExperimentData` object before, you can retrieve it from disk by calling the :meth:`~f3dasm.ExperimentData.from_file`
191
191
classmethod with the path of project directory:
192
192
193
193
.. code-block:: python
@@ -200,7 +200,7 @@ classmethod with the path of project directory:
200
200
ExperimentData from a sampling
201
201
------------------------------
202
202
203
-
You can directly construct an :class:`~f3dasm.design.ExperimentData` object from a sampling strategy by using the :meth:`~f3dasm.design.ExperimentData.from_sampling` method.
203
+
You can directly construct an :class:`~f3dasm.ExperimentData` object from a sampling strategy by using the :meth:`~f3dasm.ExperimentData.from_sampling` method.
204
204
You have to provide the following arguments:
205
205
206
206
* A sampling function. To learn more about integrating your sampling function, please refer to :ref:`this <integrating-sampling>` section.
@@ -247,9 +247,9 @@ ExperimentData from a `hydra <https://hydra.cc/>`_ configuration file
If you are using `hydra <https://hydra.cc/>`_ for configuring your experiments, you can use it to construct
250
-
an :class:`~f3dasm.design.ExperimentData`objectfrom the information in the :code:`config.yaml`filewith the :meth:`~f3dasm.design.ExperimentData.from_yaml` method.
250
+
an :class:`~f3dasm.ExperimentData`objectfrom the information in the :code:`config.yaml`filewith the :meth:`~f3dasm.ExperimentData.from_yaml` method.
251
251
252
-
You can create an experimentdata :class:`~f3dasm.design.ExperimentData`objectin the same ways as described above, but now using the hydra configuration file.
252
+
You can create an experimentdata :class:`~f3dasm.ExperimentData`objectin the same ways as described above, but now using the hydra configuration file.
253
253
254
254
255
255
.. code-block:: yaml
@@ -276,7 +276,7 @@ You can create an experimentdata :class:`~f3dasm.design.ExperimentData` object i
276
276
The :class:`~f3dasm.design.Domain`object will be constructed using the :code:`domain` key in the :code:`config.yaml`file. Make sure you have the :code:`domain` key in your :code:`config.yaml`!
277
277
To see how to configure the :class:`~f3dasm.design.Domain`objectwith hydra, see :ref:`this <domain-from-yaml>` section.
278
278
279
-
Inside your python script, you can then create the :class:`~f3dasm.design.ExperimentData`objectwith the :meth:`~f3dasm.design.ExperimentData.from_yaml` method:
279
+
Inside your python script, you can then create the :class:`~f3dasm.ExperimentData`objectwith the :meth:`~f3dasm.ExperimentData.from_yaml` method:
280
280
281
281
.. code-block:: python
282
282
@@ -289,9 +289,9 @@ Inside your python script, you can then create the :class:`~f3dasm.design.Experi
289
289
290
290
.. note::
291
291
292
-
Make sure to pass the full :code:`config` to the :meth:`~f3dasm.design.ExperimentData.from_yaml` constructor!
292
+
Make sure to pass the full :code:`config` to the :meth:`~f3dasm.ExperimentData.from_yaml` constructor!
293
293
294
-
To create the :class:`~f3dasm.design.ExperimentData`objectwith the :meth:`~f3dasm.design.ExperimentData.from_sampling` method, you can use the following configuration:
294
+
To create the :class:`~f3dasm.ExperimentData`objectwith the :meth:`~f3dasm.ExperimentData.from_sampling` method, you can use the following configuration:
295
295
296
296
.. code-block:: yaml
297
297
:caption: config.yaml for from_sampling
@@ -320,7 +320,7 @@ To create the :class:`~f3dasm.design.ExperimentData` object with the :meth:`~f3d
320
320
To see how to configure the :class:`~f3dasm.design.Domain`objectwith hydra, see :ref:`this <domain-from-yaml>` section.
321
321
322
322
323
-
To create the :class:`~f3dasm.design.ExperimentData`objectwith the :meth:`~f3dasm.design.ExperimentData.from_file` method, you can use the following configuration:
323
+
To create the :class:`~f3dasm.ExperimentData`objectwith the :meth:`~f3dasm.ExperimentData.from_file` method, you can use the following configuration:
324
324
325
325
.. code-block:: yaml
326
326
:caption: config.yaml for from_file
@@ -331,7 +331,7 @@ To create the :class:`~f3dasm.design.ExperimentData` object with the :meth:`~f3d
331
331
Adding data after construction
332
332
------------------------------
333
333
334
-
If you have constructed your :class:`~f3dasm.design.ExperimentData`object, you can add ``input_data``, ``output_data``, a ``domain``or the ``filename`` using the :meth:`~f3dasm.design.ExperimentData.add` method:
334
+
If you have constructed your :class:`~f3dasm.ExperimentData`object, you can add ``input_data``, ``output_data``, a ``domain``or the ``filename`` using the :meth:`~f3dasm.ExperimentData.add` method:
335
335
336
336
.. code-block:: python
337
337
@@ -345,7 +345,7 @@ If you have constructed your :class:`~f3dasm.design.ExperimentData` object, you
345
345
346
346
.. warning::
347
347
348
-
You can only add data to an existing :class:`~f3dasm.design.ExperimentData`objectif the domain is the same as the existing domain.
348
+
You can only add data to an existing :class:`~f3dasm.ExperimentData`objectif the domain is the same as the existing domain.
349
349
350
350
351
351
Exporting
@@ -356,7 +356,7 @@ Exporting
356
356
Storing the ExperimentData object
357
357
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
358
358
359
-
The :class:`~f3dasm.design.ExperimentData`object can be exported to a collection of files using the :meth:`~f3dasm.design.ExperimentData.store` method.
359
+
The :class:`~f3dasm.ExperimentData`object can be exported to a collection of files using the :meth:`~f3dasm.ExperimentData.store` method.
360
360
361
361
.. code-block:: python
362
362
@@ -369,7 +369,7 @@ Inside the project directory, a subfolder `experiment_data` will be created cont
369
369
- :code:`output.csv`: The :attr:`~f3dasm.design.ExperimentData.output_data` table
370
370
- :code:`jobs.pkl`: The :attr:`~f3dasm.design.ExperimentData.jobs`object
371
371
372
-
These files are used to load the :class:`~f3dasm.design.ExperimentData`object again using the :meth:`~f3dasm.design.ExperimentData.from_file` method.
372
+
These files are used to load the :class:`~f3dasm.ExperimentData`object again using the :meth:`~f3dasm.ExperimentData.from_file` method.
373
373
374
374
.. code-block:: python
375
375
@@ -394,6 +394,6 @@ Storing to other datatypes
394
394
395
395
Alternatively, you can convert the input-and outputdata of your data-driven process to other well-known datatypes:
396
396
397
-
* :class:`~numpy.ndarray` (:meth:`~f3dasm.design.ExperimentData.to_numpy`); creates a tuple of two :class:`~numpy.ndarray` objects containing the input-and outputdata.
398
-
* :class:`~xarray.Dataset` (:meth:`~f3dasm.design.ExperimentData.to_xarray`); creates a :class:`~xarray.Dataset`object containing the input-and outputdata.
399
-
* :class:`~pd.DataFrame` (:meth:`~f3dasm.design.ExperimentData.to_pandas`); creates a tuple of two :class:`~pd.DataFrame`object containing the input-and outputdata.
397
+
* :class:`~numpy.ndarray` (:meth:`~f3dasm.ExperimentData.to_numpy`); creates a tuple of two :class:`~numpy.ndarray` objects containing the input-and outputdata.
398
+
* :class:`~xarray.Dataset` (:meth:`~f3dasm.ExperimentData.to_xarray`); creates a :class:`~xarray.Dataset`object containing the input-and outputdata.
399
+
* :class:`~pd.DataFrame` (:meth:`~f3dasm.ExperimentData.to_pandas`); creates a tuple of two :class:`~pd.DataFrame`object containing the input-and outputdata.
0 commit comments