Skip to content

Commit 138d2a4

Browse files
zhangshenghangzhangshenghang
authored
[Feature][Connector-V2] Support typesense connector (#7450)
1 parent aabfc8e commit 138d2a4

File tree

59 files changed

+4025
-8
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

59 files changed

+4025
-8
lines changed

‎.github/workflows/labeler/label-scope-conf.yml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -257,6 +257,12 @@ activemq:
257257
- changed-files:
258258
- any-glob-to-any-file: seatunnel-connectors-v2/connector-activemq/**
259259
- all-globs-to-all-files: '!seatunnel-connectors-v2/connector-!(activemq)/**'
260+
typesense:
261+
- all:
262+
- changed-files:
263+
- any-glob-to-any-file: seatunnel-connectors-v2/connector-typesense/**
264+
- all-globs-to-all-files: '!seatunnel-connectors-v2/connector-!(typesense)/**'
265+
260266
Zeta Rest API:
261267
- changed-files:
262268
- any-glob-to-any-file: seatunnel-engine/**/server/rest/**

‎config/plugin_config

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -88,5 +88,5 @@ connector-web3j
8888
connector-milvus
8989
connector-activemq
9090
connector-sls
91+
connector-typesense
9192
connector-cdc-opengauss
92-
--end--

‎docs/en/connector-v2/sink/Typesense.md

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
# Typesense
2+
3+
## Description
4+
5+
Outputs data to `Typesense`.
6+
7+
## Key Features
8+
9+
- [ ] [Exactly Once](../../concept/connector-v2-features.md)
10+
- [x] [CDC](../../concept/connector-v2-features.md)
11+
12+
## Options
13+
14+
| Name | Type | Required | Default Value |
15+
|------------------|--------|----------|------------------------------|
16+
| hosts | array | Yes | - |
17+
| collection | string | Yes | - |
18+
| schema_save_mode | string | Yes | CREATE_SCHEMA_WHEN_NOT_EXIST |
19+
| data_save_mode | string | Yes | APPEND_DATA |
20+
| primary_keys | array | No | |
21+
| key_delimiter | string | No | `_` |
22+
| api_key | string | No | |
23+
| max_retry_count | int | No | 3 |
24+
| max_batch_size | int | No | 10 |
25+
| common-options | | No | - |
26+
27+
### hosts [array]
28+
29+
The access address for Typesense, formatted as `host:port`, e.g., `["typesense-01:8108"]`.
30+
31+
### collection [string]
32+
33+
The name of the collection to write to, e.g., "seatunnel".
34+
35+
### primary_keys [array]
36+
37+
Primary key fields used to generate the document `id`.
38+
39+
### key_delimiter [string]
40+
41+
Sets the delimiter for composite keys (default is `_`).
42+
43+
### api_key [config]
44+
45+
The `api_key` for secure access to Typesense.
46+
47+
### max_retry_count [int]
48+
49+
The maximum number of retry attempts for batch requests.
50+
51+
### max_batch_size [int]
52+
53+
The maximum size of document batches.
54+
55+
### common options
56+
57+
Common parameters for Sink plugins. Refer to [Common Sink Options](../source-common-options.md) for more details.
58+
59+
### schema_save_mode
60+
61+
Choose how to handle the target-side schema before starting the synchronization task:
62+
- `RECREATE_SCHEMA`: Creates the table if it doesn’t exist, and deletes and recreates it if it does.
63+
- `CREATE_SCHEMA_WHEN_NOT_EXIST`: Creates the table if it doesn’t exist, skips creation if it does.
64+
- `ERROR_WHEN_SCHEMA_NOT_EXIST`: Throws an error if the table doesn’t exist.
65+
66+
### data_save_mode
67+
68+
Choose how to handle existing data on the target side before starting the synchronization task:
69+
- `DROP_DATA`: Retains the database structure but deletes the data.
70+
- `APPEND_DATA`: Retains both the database structure and the data.
71+
- `ERROR_WHEN_DATA_EXISTS`: Throws an error if data exists.
72+
73+
## Example
74+
75+
Simple example:
76+
77+
```bash
78+
sink {
79+
Typesense {
80+
source_table_name = "typesense_test_table"
81+
hosts = ["localhost:8108"]
82+
collection = "typesense_to_typesense_sink_with_query"
83+
max_retry_count = 3
84+
max_batch_size = 10
85+
api_key = "xyz"
86+
primary_keys = ["num_employees","id"]
87+
key_delimiter = "="
88+
schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST"
89+
data_save_mode = "APPEND_DATA"
90+
}
91+
}
92+
```
93+

‎docs/en/connector-v2/source/Typesense.md

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
# Typesense
2+
3+
> Typesense Source Connector
4+
5+
## Description
6+
7+
Reads data from Typesense.
8+
9+
## Key Features
10+
11+
- [x] [Batch Processing](../../concept/connector-v2-features.md)
12+
- [ ] [Stream Processing](../../concept/connector-v2-features.md)
13+
- [ ] [Exactly-Once](../../concept/connector-v2-features.md)
14+
- [x] [Schema](../../concept/connector-v2-features.md)
15+
- [x] [Parallelism](../../concept/connector-v2-features.md)
16+
- [ ] [User-Defined Splits Support](../../concept/connector-v2-features.md)
17+
18+
## Options
19+
20+
| Name | Type | Required | Default |
21+
|------------|--------|----------|---------|
22+
| hosts | array | yes | - |
23+
| collection | string | yes | - |
24+
| schema | config | yes | - |
25+
| api_key | string | no | - |
26+
| query | string | no | - |
27+
| batch_size | int | no | 100 |
28+
29+
### hosts [array]
30+
31+
The access address of Typesense, for example: `["typesense-01:8108"]`.
32+
33+
### collection [string]
34+
35+
The name of the collection to write to, for example: `"seatunnel"`.
36+
37+
### schema [config]
38+
39+
The columns to be read from Typesense. For more information, please refer to the [guide](../../concept/schema-feature.md#how-to-declare-type-supported).
40+
41+
### api_key [config]
42+
43+
The `api_key` for Typesense security authentication.
44+
45+
### batch_size
46+
47+
The number of records to query per batch when reading data.
48+
49+
### Common Options
50+
51+
For common parameters of Source plugins, please refer to [Source Common Options](../source-common-options.md).
52+
53+
## Example
54+
55+
```bash
56+
source {
57+
Typesense {
58+
hosts = ["localhost:8108"]
59+
collection = "companies"
60+
api_key = "xyz"
61+
query = "q=*&filter_by=num_employees:>9000"
62+
schema = {
63+
fields {
64+
company_name_list = array<string>
65+
company_name = string
66+
num_employees = long
67+
country = string
68+
id = string
69+
c_row = {
70+
c_int = int
71+
c_string = string
72+
c_array_int = array<int>
73+
}
74+
}
75+
}
76+
}
77+
}
78+
```
79+

‎docs/zh/connector-v2/sink/Typesense.md

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
# Typesense
2+
3+
## 描述
4+
5+
输出数据到 `Typesense`
6+
7+
## 主要特性
8+
9+
- [ ] [精确一次](../../concept/connector-v2-features.md)
10+
- [x] [cdc](../../concept/connector-v2-features.md)
11+
12+
## 选项
13+
14+
| 名称 | 类型 | 是否必须 | 默认值 |
15+
|------------------|--------|------|------------------------------|
16+
| hosts | array | 是 | - |
17+
| collection | string | 是 | - |
18+
| schema_save_mode | string | 是 | CREATE_SCHEMA_WHEN_NOT_EXIST |
19+
| data_save_mode | string | 是 | APPEND_DATA |
20+
| primary_keys | array | 否 | |
21+
| key_delimiter | string | 否 | `_` |
22+
| api_key | string | 否 | |
23+
| max_retry_count | int | 否 | 3 |
24+
| max_batch_size | int | 否 | 10 |
25+
| common-options | | 否 | - |
26+
27+
### hosts [array]
28+
29+
Typesense的访问地址，格式为 `host:port`，例如：["typesense-01:8108"]
30+
31+
### collection [string]
32+
33+
要写入的集合名，例如：“seatunnel”
34+
35+
### primary_keys [array]
36+
37+
主键字段用于生成文档 `id`。
38+
39+
### key_delimiter [string]
40+
41+
设定复合键的分隔符（默认为 `_`）。
42+
43+
### api_key [config]
44+
45+
typesense 安全认证的 api_key。
46+
47+
### max_retry_count [int]
48+
49+
批次批量请求最大尝试大小
50+
51+
### max_batch_size [int]
52+
53+
批次批量文档最大大小
54+
55+
### common options
56+
57+
Sink插件常用参数，请参考 [Sink常用选项](../sink-common-options.md) 了解详情
58+
59+
### schema_save_mode
60+
61+
在启动同步任务之前，针对目标侧已有的表结构选择不同的处理方案<br/>
62+
选项介绍：<br/>
63+
`RECREATE_SCHEMA` ：当表不存在时会创建，当表已存在时会删除并重建<br/>
64+
`CREATE_SCHEMA_WHEN_NOT_EXIST` ：当表不存在时会创建，当表已存在时则跳过创建<br/>
65+
`ERROR_WHEN_SCHEMA_NOT_EXIST` ：当表不存在时将抛出错误<br/>
66+
67+
### data_save_mode
68+
69+
在启动同步任务之前，针对目标侧已存在的数据选择不同的处理方案<br/>
70+
选项介绍：<br/>
71+
`DROP_DATA`： 保留数据库结构，删除数据<br/>
72+
`APPEND_DATA`：保留数据库结构，保留数据<br/>
73+
`ERROR_WHEN_DATA_EXISTS`：当有数据时抛出错误<br/>
74+
75+
## 示例
76+
77+
简单示例
78+
79+
```bash
80+
sink {
81+
Typesense {
82+
source_table_name = "typesense_test_table"
83+
hosts = ["localhost:8108"]
84+
collection = "typesense_to_typesense_sink_with_query"
85+
max_retry_count = 3
86+
max_batch_size = 10
87+
api_key = "xyz"
88+
primary_keys = ["num_employees","id"]
89+
key_delimiter = "="
90+
schema_save_mode = "CREATE_SCHEMA_WHEN_NOT_EXIST"
91+
data_save_mode = "APPEND_DATA"
92+
}
93+
}
94+
```
95+

‎docs/zh/connector-v2/source/Typesense.md

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
# Typesense
2+
3+
> Typesense 源连接器
4+
5+
## 描述
6+
7+
从 Typesense 读取数据。
8+
9+
## 主要功能
10+
11+
- [x] [批处理](../../concept/connector-v2-features.md)
12+
- [ ] [流处理](../../concept/connector-v2-features.md)
13+
- [ ] [精确一次](../../concept/connector-v2-features.md)
14+
- [x] [Schema](../../concept/connector-v2-features.md)
15+
- [x] [并行度](../../concept/connector-v2-features.md)
16+
- [ ] [支持用户定义的拆分](../../concept/connector-v2-features.md)
17+
18+
## 选项
19+
20+
| 名称 | 类型 | 必填 | 默认值 |
21+
|------------|--------|----|-----|
22+
| hosts | array | 是 | - |
23+
| collection | string | 是 | - |
24+
| schema | config | 是 | - |
25+
| api_key | string | 否 | - |
26+
| query | string | 否 | - |
27+
| batch_size | int | 否 | 100 |
28+
29+
### hosts [array]
30+
31+
Typesense的访问地址，格式为 `host:port`，例如：["typesense-01:8108"]
32+
33+
### collection [string]
34+
35+
要写入的集合名，例如：“seatunnel”
36+
37+
### schema [config]
38+
39+
typesense 需要读取的列。有关更多信息，请参阅：[guide](../../concept/schema-feature.md#how-to-declare-type-supported)。
40+
41+
### api_key [config]
42+
43+
typesense 安全认证的 api_key。
44+
45+
### batch_size
46+
47+
读取数据时，每批次查询数量
48+
49+
### 常用选项
50+
51+
Source 插件常用参数，具体请参考 [Source 常用选项](../source-common-options.md)
52+
53+
## 示例
54+
55+
```bash
56+
source {
57+
Typesense {
58+
hosts = ["localhost:8108"]
59+
collection = "companies"
60+
api_key = "xyz"
61+
query = "q=*&filter_by=num_employees:>9000"
62+
schema = {
63+
fields {
64+
company_name_list = array<string>
65+
company_name = string
66+
num_employees = long
67+
country = string
68+
id = string
69+
c_row = {
70+
c_int = int
71+
c_string = string
72+
c_array_int = array<int>
73+
}
74+
}
75+
}
76+
}
77+
}
78+
```
79+

‎plugin-mapping.properties

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -132,8 +132,9 @@ seatunnel.source.Milvus = connector-milvus
132132
seatunnel.sink.Milvus = connector-milvus
133133
seatunnel.sink.ActiveMQ = connector-activemq
134134
seatunnel.source.Sls = connector-sls
135+
seatunnel.source.Typesense = connector-typesense
136+
seatunnel.sink.Typesense = connector-typesense
135137
seatunnel.source.Opengauss-CDC = connector-cdc-opengauss
136-
137138
seatunnel.transform.Sql = seatunnel-transforms-v2
138139
seatunnel.transform.FieldMapper = seatunnel-transforms-v2
139140
seatunnel.transform.Filter = seatunnel-transforms-v2

0 commit comments

Comments
 (0)