Skip to content

Commit cf5bf7a

Browse files
authored
Merge pull request #97 from 191220029/read-me
Update README.md
2 parents 38b0681 + 3558b50 commit cf5bf7a

File tree

1 file changed

+67
-3
lines changed

1 file changed

+67
-3
lines changed

README.md

Lines changed: 67 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,9 +17,9 @@ The development of `dagrs` follows the concept of Flow-based Programming.
1717

1818
### Flow based Programming
1919
[Flow-based Programming](https://en.wikipedia.org/wiki/Flow-based_programming)(FBP) was invented by J. Paul Morrison in the early 1970s. It was initially implemented in software for a Canadian bank.
20-
Over the years, its had various names but has always maintained its core principles of reducing development time and managing processes efficiently.
20+
Over the years, it's had various names but has always maintained its core principles of reducing development time and managing processes efficiently.
2121

22-
FBP treats applications as networks of 'black box' processes that communicate by sending and receiving data, referred to as Information Packets, over predefined connections. Its a component-oriented approach that fits well with modular software architecture.
22+
FBP treats applications as networks of 'black box' processes that communicate by sending and receiving data, referred to as Information Packets, over predefined connections. It's a component-oriented approach that fits well with modular software architecture.
2323

2424
### Key Features of FBP
2525

@@ -42,6 +42,69 @@ Dagrs leverages cutting-edge technologies to ensure functionality and performanc
4242
- **[async_trait](https://crates.io/crates/async-trait)** - Type erasure for async trait methods.
4343

4444

45+
## Advanced Features
46+
47+
### Conditional Node
48+
Conditional nodes allow you to control task flow execution based on specific conditions. By implementing the `Condition` trait, you can define custom condition logic to determine whether to continue executing subsequent tasks. For example, you can create a validation node that only continues execution when input data meets specific conditions.
49+
50+
```rust
51+
struct MyCondition;
52+
#[async_trait]
53+
impl Condition for MyCondition {
54+
async fn run(&self, _: &mut InChannels, _: &OutChannels, _: Arc<EnvVar>) -> bool {
55+
// Implement condition logic here
56+
true
57+
}
58+
}
59+
```
60+
61+
### Loop DAG
62+
Loop DAG allows you to create task graphs with cyclic dependencies. This is particularly useful when you need to repeatedly execute certain tasks until specific conditions are met. For example, you can create an interactive system where certain tasks need to be executed repeatedly based on user input.
63+
64+
```rust
65+
let mut lop = LoopSubgraph::new("loop".to_string(), &mut node_table);
66+
lop.add_node(processor);
67+
lop.add_node(consumer);
68+
```
69+
70+
### Customized Configuration Parser
71+
Dagrs allows you to design your own parser to define task graphs in custom configuration formats. By implementing the `Parser` trait, you can create parsers for various configuration formats (such as JSON, TOML, or your own custom format) to define task names, dependencies, and execution commands.
72+
73+
For example, in the [dagrs-sklearn](examples/dagrs-sklearn) example project, we provide a custom YAML parser implementation that demonstrates how to create a specialized parser for machine learning workflows. This parser extends the basic YAML format to support additional features specific to machine learning tasks.
74+
75+
```rust
76+
pub struct YamlParser;
77+
78+
impl Parser for YamlParser {
79+
fn parse_tasks(
80+
&self,
81+
file: &str,
82+
specific_actions: HashMap<String, Box<dyn Action>>,
83+
) -> Result<(Graph, EnvVar), ParseError> {
84+
// Custom parsing logic for machine learning workflow
85+
Ok((graph, env_var))
86+
}
87+
}
88+
```
89+
90+
## Examples
91+
92+
### dagrs-sklearn
93+
The example [dagrs-sklearn](examples/dagrs-sklearn) shows how dagrs can help implement machine learning algorithms to train classifiers in a parallel manner. It provides:
94+
1. A custom YAML parser for defining machine learning tasks
95+
2. Integration with scikit-learn for data processing and model training
96+
3. Example workflows for common machine learning tasks
97+
98+
This example shows a typical machine learning workflow with three stages:
99+
- Data preprocessing
100+
- Model training
101+
- Model evaluation
102+
103+
Each stage is defined as a task with its dependencies and execution command. The custom YAML parser handles the configuration and builds the corresponding task graph.
104+
105+
For more detailed info about this example, please see the [notebook.ipynb](examples/dagrs-sklearn/examples/notebook.ipynb) jupyter notebook file.
106+
107+
45108
## Contribution
46109

47110
The `dagrs` project relies on community contributions and aims to simplify getting started. To develop `dagrs`, clone the repository, then install all dependencies, run the test suite and try it out locally. Pick an issue, make changes, and submit a pull request for community review.
@@ -76,7 +139,7 @@ $ git commit -S -s -m 'This is my commit message'
76139

77140
### Rebase the branch
78141

79-
If you have a local git environment and meet the criteria below, one option is to rebase the branch and add your Signed-off-by lines in the new commits. Please note that if others have already begun work based upon the commits in this branch, this solution will rewrite history and may cause serious issues for collaborators (described in the git documentation under The Perils of Rebasing).
142+
If you have a local git environment and meet the criteria below, one option is to rebase the branch and add your Signed-off-by lines in the new commits. Please note that if others have already begun work based upon the commits in this branch, this solution will rewrite history and may cause serious issues for collaborators (described in the git documentation under "The Perils of Rebasing").
80143

81144
You should only do this if:
82145

@@ -104,3 +167,4 @@ Email: Quanyi Ma <[email protected]>, Xiaolong Fu <[email protected]>
104167
### Discord
105168

106169
Welcome to join our discord channel https://discord.gg/4JzaNkRP
170+

0 commit comments

Comments
 (0)