You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+67-3Lines changed: 67 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -17,9 +17,9 @@ The development of `dagrs` follows the concept of Flow-based Programming.
17
17
18
18
### Flow based Programming
19
19
[Flow-based Programming](https://en.wikipedia.org/wiki/Flow-based_programming)(FBP) was invented by J. Paul Morrison in the early 1970s. It was initially implemented in software for a Canadian bank.
20
-
Over the years, it’s had various names but has always maintained its core principles of reducing development time and managing processes efficiently.
20
+
Over the years, it's had various names but has always maintained its core principles of reducing development time and managing processes efficiently.
21
21
22
-
FBP treats applications as networks of 'black box' processes that communicate by sending and receiving data, referred to as Information Packets, over predefined connections. It’s a component-oriented approach that fits well with modular software architecture.
22
+
FBP treats applications as networks of 'black box' processes that communicate by sending and receiving data, referred to as Information Packets, over predefined connections. It's a component-oriented approach that fits well with modular software architecture.
23
23
24
24
### Key Features of FBP
25
25
@@ -42,6 +42,69 @@ Dagrs leverages cutting-edge technologies to ensure functionality and performanc
42
42
-**[async_trait](https://crates.io/crates/async-trait)** - Type erasure for async trait methods.
43
43
44
44
45
+
## Advanced Features
46
+
47
+
### Conditional Node
48
+
Conditional nodes allow you to control task flow execution based on specific conditions. By implementing the `Condition` trait, you can define custom condition logic to determine whether to continue executing subsequent tasks. For example, you can create a validation node that only continues execution when input data meets specific conditions.
Loop DAG allows you to create task graphs with cyclic dependencies. This is particularly useful when you need to repeatedly execute certain tasks until specific conditions are met. For example, you can create an interactive system where certain tasks need to be executed repeatedly based on user input.
Dagrs allows you to design your own parser to define task graphs in custom configuration formats. By implementing the `Parser` trait, you can create parsers for various configuration formats (such as JSON, TOML, or your own custom format) to define task names, dependencies, and execution commands.
72
+
73
+
For example, in the [dagrs-sklearn](examples/dagrs-sklearn) example project, we provide a custom YAML parser implementation that demonstrates how to create a specialized parser for machine learning workflows. This parser extends the basic YAML format to support additional features specific to machine learning tasks.
74
+
75
+
```rust
76
+
pubstructYamlParser;
77
+
78
+
implParserforYamlParser {
79
+
fnparse_tasks(
80
+
&self,
81
+
file:&str,
82
+
specific_actions:HashMap<String, Box<dynAction>>,
83
+
) ->Result<(Graph, EnvVar), ParseError> {
84
+
// Custom parsing logic for machine learning workflow
85
+
Ok((graph, env_var))
86
+
}
87
+
}
88
+
```
89
+
90
+
## Examples
91
+
92
+
### dagrs-sklearn
93
+
The example [dagrs-sklearn](examples/dagrs-sklearn) shows how dagrs can help implement machine learning algorithms to train classifiers in a parallel manner. It provides:
94
+
1. A custom YAML parser for defining machine learning tasks
95
+
2. Integration with scikit-learn for data processing and model training
96
+
3. Example workflows for common machine learning tasks
97
+
98
+
This example shows a typical machine learning workflow with three stages:
99
+
- Data preprocessing
100
+
- Model training
101
+
- Model evaluation
102
+
103
+
Each stage is defined as a task with its dependencies and execution command. The custom YAML parser handles the configuration and builds the corresponding task graph.
104
+
105
+
For more detailed info about this example, please see the [notebook.ipynb](examples/dagrs-sklearn/examples/notebook.ipynb) jupyter notebook file.
106
+
107
+
45
108
## Contribution
46
109
47
110
The `dagrs` project relies on community contributions and aims to simplify getting started. To develop `dagrs`, clone the repository, then install all dependencies, run the test suite and try it out locally. Pick an issue, make changes, and submit a pull request for community review.
@@ -76,7 +139,7 @@ $ git commit -S -s -m 'This is my commit message'
76
139
77
140
### Rebase the branch
78
141
79
-
If you have a local git environment and meet the criteria below, one option is to rebase the branch and add your Signed-off-by lines in the new commits. Please note that if others have already begun work based upon the commits in this branch, this solution will rewrite history and may cause serious issues for collaborators (described in the git documentation under “The Perils of Rebasing”).
142
+
If you have a local git environment and meet the criteria below, one option is to rebase the branch and add your Signed-off-by lines in the new commits. Please note that if others have already begun work based upon the commits in this branch, this solution will rewrite history and may cause serious issues for collaborators (described in the git documentation under "The Perils of Rebasing").
0 commit comments