Skip to content

Commit 00dbcba

Browse files
lovromazgonharisomaha-mstfa
authored
[Feature] Better processors (#1378)
* Refactor pkg/plugin and isolate connector plugin (#1302) * refactor pkg/plugin and isolate connector plugin * fix imports * fix tests * fix import * change registry to connectorRegistry --------- Co-authored-by: Haris Osmanagić <haris@meroxa.io> * BP: WASM: Load available WASM processors (#1322) Co-authored-by: Maha Hajja <82542081+maha-hajja@users.noreply.github.com> --------- Co-authored-by: Lovro Mažgon <lovro.mazgon@gmail.com> Co-authored-by: Maha Hajja <82542081+maha-hajja@users.noreply.github.com> * Refactor pkg/plugin and isolate connector plugin (#1302) * refactor pkg/plugin and isolate connector plugin * fix imports * fix tests * fix import * change registry to connectorRegistry --------- Co-authored-by: Haris Osmanagić <haris@meroxa.io> * BP: WASM: Load available WASM processors (#1322) Co-authored-by: Maha Hajja <82542081+maha-hajja@users.noreply.github.com> --------- Co-authored-by: Lovro Mažgon <lovro.mazgon@gmail.com> Co-authored-by: Maha Hajja <82542081+maha-hajja@users.noreply.github.com> * Refactor processor service, processor and node (#1354) * linter * Run processor examples, gather specifications (#1384) * update processor-sdk, use config parameter from conduit-commons * add unified diff lib * add utilities for running processor examples and collecting specifications * update diff readme * fix markdown linter error * fix liter errors * update processor-sdk * fix race condition in test * BP: Display processor example raw data as string (#1397) * display raw data in processor examples as string * ensure examples are formatted the same way in processors.json * BP: add builtin processor part#1 (#1371) * add processors field.set & field.subset.exclude * delete extra test * fix tests * add field.rename processor * add exclusion validation to field.rename processor * refactor + fix error return in Process method * add field.convert processor * add filter processor * only structured data * address reviews part1 * use paramgen, update Configure method, update tests * evaluate value for processor setField * use a slice of referenceResolvers for processors with multiple fields params * do not allow .Position to be set in setField processor * update setField test, add non existent field test * use the new Rename() method from referenceResolver * address reviews * address reviews2 * fix renameField bug + add New() method to each processor * linter fix * add processor examples * processor json * Update pkg/plugin/processor/builtin/convertField.go Co-authored-by: Lovro Mažgon <lovro.mazgon@gmail.com> * address reviews * setFeild example * setFeild example * make generate * fix paramgen regex * generate * fix regex * add paramgen to tools --------- Co-authored-by: Lovro Mažgon <lovro.mazgon@gmail.com> * BP: Add unwrap.debezium (#1393) * BP: Add unwrap.kafkaconnect (#1395) * BP: Add webhook.http (#1382) * BP: Add unwrap.opencdc (#1387) * BP: Add encode.avro (#1401) * updated docs, regenerate * config * first pass * downgrade lib * processing * example * fix config parsing * regenerate * fix config * regenerate, tests * linter, move to opencdc * add test certs * config refactor, tests * more tests * linter * Regenerate * fix linter * git diff * git diff * regenerate * avro examples * run schema registry in examples on port 54321 * fix linter error * rename processor * fix schema registry integration test * don't run avro examples in integration tests --------- Co-authored-by: Lovro Mažgon <lovro.mazgon@gmail.com> * BP: Add custom.javascript (#1374) * add json.decode processor (#1405) * add json.decode processor * make generate * processors * address reviews * make generate * add support for decoding JSON types other than map * make generate * address review comments --------- Co-authored-by: Lovro Mažgon <lovro.mazgon@gmail.com> * BP: List processor plugins in API (#1406) * update proto API definitions * fix proto linter warnings * deprecate ListPlugins * implement plugin list endpoint * processor plugin orchestrator * move plugin endpoints under connector and processor service * BP: add avro.decode (#1410) Co-authored-by: Lovro Mažgon <lovro.mazgon@gmail.com> * avro.decode example, remove base64 decoding --------- Co-authored-by: Lovro Mažgon <lovro.mazgon@gmail.com> * BP: Base64 decode / encode processors (#1412) * encode * base64 decode * organize imports * add encodeJson processor (#1413) * encode json processor * delete forgotten test * address reviews * linter fix * add processor examples * don't encode nil, output string --------- Co-authored-by: Lovro Mažgon <lovro.mazgon@gmail.com> * Builtin processors refactoring, housekeeping (#1402) * refactor builtin processors, generate specs into separate files * move processors into folder impl * go mod tidy * add example summaries * add example summaries * fix name of avro.decode processor * add example summaries * regenerate specs * update default registry * update default registry: fix tests, linter, regenerate * dep upgrades * refactorings * move json encode processor to impl/json * rename field.subset.exclude to field.exclude * rename constructor for unwrap.debezium processor * regenerate specs * dep downgrades --------- Co-authored-by: Haris Osmanagic <haris@meroxa.io> * fix linter * BP: call the conditional execution methods in the processors (#1411) * init * simplify * fix evaluation * add support for batches in runnable processor * batch test in runnable processor * optimize case when all records are passed through in a processor * make sure remaining records are copied over * fix len check * make sure processor doesn't return more records --------- Co-authored-by: Lovro Mažgon <lovro.mazgon@gmail.com> * update examples and specs * disable goconst in processors (docs are repetitive) * upgrade processor sdk * add log when loading wasm processors, more docs updates --------- Co-authored-by: Haris Osmanagić <haris@meroxa.io> Co-authored-by: Maha Hajja <82542081+maha-hajja@users.noreply.github.com>
1 parent 2b08f0d commit 00dbcba

341 files changed

Lines changed: 23647 additions & 11746 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/validate-generated-files.yml‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,6 @@ jobs:
2525
- name: Check generated files
2626
run: |
2727
export PATH=$PATH:$(go env GOPATH)/bin
28-
make install-tools
29-
make generate
30-
make proto-generate
28+
make install-tools generate proto-generate
3129
git diff
3230
git diff --exit-code --numstat

‎.gitignore‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,8 @@ dist/
7575

7676
# Standalone connectors
7777
/connectors
78+
# Standalone processors
79+
/processors
7880

7981
/pipelines
8082

@@ -86,3 +88,6 @@ escape_analysis.txt
8688

8789
# Profiles
8890
*.prof
91+
92+
# Compiled test wasm processors
93+
pkg/plugin/processor/standalone/test/wasm_processors/*/processor.wasm

‎.golangci.yml‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,10 @@
11
run:
22
timeout: 3m
3+
skip-dirs-use-default: false
34
skip-dirs:
4-
- /ui/.*
5+
- ^examples/
6+
- ^ui/
7+
- ^pkg/plugin/processor/builtin/internal/diff # external code
58

69
linters-settings:
710
depguard:
@@ -40,6 +43,9 @@ linters-settings:
4043

4144
issues:
4245
exclude-rules:
46+
- path: 'pkg/plugin/processor/builtin/impl'
47+
linters:
48+
- goconst
4349
- path: '(.+)acceptance_testing\.go'
4450
linters:
4551
- stylecheck

‎Makefile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,7 @@ install-tools: download
6565
@go mod tidy
6666

6767
generate:
68-
go generate ./...
68+
go generate -x ./...
6969

7070
pkg/web/ui/dist:
7171
make ui-dist

‎docs/connectors.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ Once you have chosen a connector to be built-in, you can:
4646

4747
- Download the new package and its dependencies: `go get "github.com/foo/conduit-connector-new"`
4848
- Import the Go module defining the connector
49-
into the [builtin registry](https://github.com/ConduitIO/conduit/blob/main/pkg/plugin/builtin/registry.go)
49+
into the [builtin registry](https://github.com/ConduitIO/conduit/blob/main/pkg/plugin/connector/builtin/registry.go)
5050
and add a new key to `DefaultDispenserFactories`:
5151

5252
```diff

‎go.mod‎

Lines changed: 13 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,14 @@
11
module github.com/conduitio/conduit
22

3-
go 1.21.1
3+
go 1.21.5
44

55
require (
66
buf.build/gen/go/grpc-ecosystem/grpc-gateway/protocolbuffers/go v1.32.0-20231027202514-3f42134f4c56.1
77
github.com/Masterminds/semver/v3 v3.2.1
88
github.com/Masterminds/sprig/v3 v3.2.3
99
github.com/NYTimes/gziphandler v1.1.1
10-
github.com/antchfx/jsonquery v1.3.3
1110
github.com/bufbuild/buf v1.29.0
12-
github.com/conduitio/conduit-commons v0.0.0-20240112191423-58fcb3055cf2
11+
github.com/conduitio/conduit-commons v0.0.1
1312
github.com/conduitio/conduit-connector-file v0.6.0
1413
github.com/conduitio/conduit-connector-generator v0.5.0
1514
github.com/conduitio/conduit-connector-kafka v0.7.1
@@ -18,10 +17,11 @@ require (
1817
github.com/conduitio/conduit-connector-protocol v0.5.1-0.20240104160905-e9e61586fb8d
1918
github.com/conduitio/conduit-connector-s3 v0.5.1
2019
github.com/conduitio/conduit-connector-sdk v0.8.0
20+
github.com/conduitio/conduit-processor-sdk v0.1.0
2121
github.com/conduitio/yaml/v3 v3.3.0
2222
github.com/dgraph-io/badger/v4 v4.2.0
23-
github.com/dop251/goja v0.0.0-20230531210528-d7324b2d74f7
24-
github.com/dop251/goja_nodejs v0.0.0-20230602164024-804a84515562
23+
github.com/dop251/goja v0.0.0-20231027120936-b396bb4c349d
24+
github.com/dop251/goja_nodejs v0.0.0-20231122114759-e84d9a924c5c
2525
github.com/gammazero/deque v0.2.1
2626
github.com/goccy/go-json v0.10.2
2727
github.com/golangci/golangci-lint v1.56.2
@@ -43,6 +43,8 @@ require (
4343
github.com/prometheus/client_model v0.6.0
4444
github.com/prometheus/common v0.47.0
4545
github.com/rs/zerolog v1.32.0
46+
github.com/stealthrocket/wazergo v0.19.1
47+
github.com/tetratelabs/wazero v1.6.0
4648
github.com/twmb/go-cache v1.2.1
4749
go.uber.org/goleak v1.3.0
4850
go.uber.org/mock v0.4.0
@@ -79,7 +81,6 @@ require (
7981
github.com/alexkohler/nakedret/v2 v2.0.2 // indirect
8082
github.com/alexkohler/prealloc v1.0.0 // indirect
8183
github.com/alingse/asasalint v0.0.11 // indirect
82-
github.com/antchfx/xpath v1.2.3 // indirect
8384
github.com/antlr4-go/antlr/v4 v4.13.0 // indirect
8485
github.com/apache/arrow/go/arrow v0.0.0-20211112161151-bc219186db40 // indirect
8586
github.com/apache/thrift v0.19.0 // indirect
@@ -116,7 +117,6 @@ require (
116117
github.com/butuzov/mirror v1.1.0 // indirect
117118
github.com/catenacyber/perfsprint v0.6.0 // indirect
118119
github.com/ccojocar/zxcvbn-go v1.0.2 // indirect
119-
github.com/cenkalti/backoff/v4 v4.2.1 // indirect
120120
github.com/cespare/xxhash/v2 v2.2.0 // indirect
121121
github.com/charithe/durationcheck v0.0.10 // indirect
122122
github.com/chavacava/garif v0.1.0 // indirect
@@ -128,7 +128,7 @@ require (
128128
github.com/denis-tingaikin/go-header v0.4.3 // indirect
129129
github.com/dgraph-io/ristretto v0.1.1 // indirect
130130
github.com/distribution/reference v0.5.0 // indirect
131-
github.com/dlclark/regexp2 v1.8.0 // indirect
131+
github.com/dlclark/regexp2 v1.10.0 // indirect
132132
github.com/docker/cli v24.0.7+incompatible // indirect
133133
github.com/docker/distribution v2.8.3+incompatible // indirect
134134
github.com/docker/docker v25.0.0+incompatible // indirect
@@ -214,7 +214,7 @@ require (
214214
github.com/kisielk/errcheck v1.7.0 // indirect
215215
github.com/kisielk/gotool v1.0.0 // indirect
216216
github.com/kkHAIKE/contextcheck v1.1.4 // indirect
217-
github.com/klauspost/compress v1.17.4 // indirect
217+
github.com/klauspost/compress v1.17.5 // indirect
218218
github.com/klauspost/pgzip v1.2.6 // indirect
219219
github.com/kulti/thelper v0.6.3 // indirect
220220
github.com/kunwardeep/paralleltest v1.0.9 // indirect
@@ -297,7 +297,6 @@ require (
297297
github.com/t-yuki/gocover-cobertura v0.0.0-20180217150009-aaee18c8195c // indirect
298298
github.com/tdakkota/asciicheck v0.2.0 // indirect
299299
github.com/tetafro/godot v1.4.16 // indirect
300-
github.com/tetratelabs/wazero v1.6.0 // indirect
301300
github.com/timakin/bodyclose v0.0.0-20230421092635-574207250966 // indirect
302301
github.com/timonwong/loggercheck v0.9.4 // indirect
303302
github.com/tomarrell/wrapcheck/v2 v2.8.1 // indirect
@@ -319,12 +318,10 @@ require (
319318
go-simpler.org/sloglint v0.4.0 // indirect
320319
go.opencensus.io v0.24.0 // indirect
321320
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.47.0 // indirect
322-
go.opentelemetry.io/otel v1.22.0 // indirect
323-
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.22.0 // indirect
324-
go.opentelemetry.io/otel/metric v1.22.0 // indirect
325-
go.opentelemetry.io/otel/sdk v1.22.0 // indirect
326-
go.opentelemetry.io/otel/trace v1.22.0 // indirect
327-
go.opentelemetry.io/proto/otlp v1.0.0 // indirect
321+
go.opentelemetry.io/otel v1.24.0 // indirect
322+
go.opentelemetry.io/otel/metric v1.24.0 // indirect
323+
go.opentelemetry.io/otel/sdk v1.24.0 // indirect
324+
go.opentelemetry.io/otel/trace v1.24.0 // indirect
328325
go.uber.org/atomic v1.11.0 // indirect
329326
go.uber.org/multierr v1.11.0 // indirect
330327
go.uber.org/zap v1.26.0 // indirect

‎go.sum‎

Lines changed: 26 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -927,10 +927,6 @@ github.com/alexkohler/prealloc v1.0.0/go.mod h1:VetnK3dIgFBBKmg0YnD9F9x6Icjd+9cv
927927
github.com/alingse/asasalint v0.0.11 h1:SFwnQXJ49Kx/1GghOFz1XGqHYKp21Kq1nHad/0WQRnw=
928928
github.com/alingse/asasalint v0.0.11/go.mod h1:nCaoMhw7a9kSJObvQyVzNTPBDbNpdocqrSP7t/cW5+I=
929929
github.com/andybalholm/brotli v1.0.4/go.mod h1:fO7iG3H7G2nSZ7m0zPUDn85XEX2GTukHGRSepvi9Eig=
930-
github.com/antchfx/jsonquery v1.3.3 h1:zjZpbnZhYng3uOAbIfdNq81A9mMEeuDJeYIpeKpZ4es=
931-
github.com/antchfx/jsonquery v1.3.3/go.mod h1:1JG4DqRlRCHgVYDPY1ioYFAGSXGfWHzNgrbiGQHsWck=
932-
github.com/antchfx/xpath v1.2.3 h1:CCZWOzv5bAqjVv0offZ2LVgVYFbeldKQVuLNbViZdes=
933-
github.com/antchfx/xpath v1.2.3/go.mod h1:i54GszH55fYfBmoZXapTHN8T8tkcHfRgLyVwwqzXNcs=
934930
github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY=
935931
github.com/antlr4-go/antlr/v4 v4.13.0 h1:lxCg3LAv+EUK6t1i0y1V6/SLeUi0eKEKdhQAlS8TVTI=
936932
github.com/antlr4-go/antlr/v4 v4.13.0/go.mod h1:pfChB/xh/Unjila75QW7+VU4TSnWnnk9UTnmpPaOR2g=
@@ -1084,8 +1080,8 @@ github.com/cncf/xds/go v0.0.0-20230607035331-e9ce68804cb4/go.mod h1:eXthEFrGJvWH
10841080
github.com/cockroachdb/apd v1.1.0 h1:3LFP3629v+1aKXU5Q37mxmRxX/pIu1nijXydLShEq5I=
10851081
github.com/cockroachdb/apd v1.1.0/go.mod h1:8Sl8LxpKi29FqWXR16WEFZRNSz3SoPzUzeMeY4+DwBQ=
10861082
github.com/colinmarc/hdfs/v2 v2.1.1/go.mod h1:M3x+k8UKKmxtFu++uAZ0OtDU8jR3jnaZIAc6yK4Ue0c=
1087-
github.com/conduitio/conduit-commons v0.0.0-20240112191423-58fcb3055cf2 h1:KWVS2fHWB1NI3hdUg1xxzTB0TT7iNPUt35O8IoW1p5o=
1088-
github.com/conduitio/conduit-commons v0.0.0-20240112191423-58fcb3055cf2/go.mod h1:43mPxHKxsZbmqf1Gw+hpx3ByXAmp8doWTQNNfCoyXp8=
1083+
github.com/conduitio/conduit-commons v0.0.1 h1:UjY/dsfr88N0l8SdJ3hBOYq38qYmV6DEe5taxvdYjds=
1084+
github.com/conduitio/conduit-commons v0.0.1/go.mod h1:WjmUZpazjAolYz8SB++xvJlN47ro3AStDYhwRCK/yRc=
10891085
github.com/conduitio/conduit-connector-file v0.6.0 h1:8tsGeGhKvFwYQZztOOL5/tmOhVShsfo9lQ3b/0fX8kQ=
10901086
github.com/conduitio/conduit-connector-file v0.6.0/go.mod h1:ju7PiB4kTJgqng4KVXDt/Gvw/53kFwSzi5Ez9EDXxNI=
10911087
github.com/conduitio/conduit-connector-generator v0.5.0 h1:zpXHif89DCJ13nftKLv31uI2AJGicpY5H1V7SwldRNo=
@@ -1102,6 +1098,8 @@ github.com/conduitio/conduit-connector-s3 v0.5.1 h1:yRo8004ryCIZc/S3iWQ1rN6pm6bj
11021098
github.com/conduitio/conduit-connector-s3 v0.5.1/go.mod h1:nbxzsyS95gbFJ28Job9vFFB+byRFINSv70/13Yi4mKQ=
11031099
github.com/conduitio/conduit-connector-sdk v0.8.0 h1:gvchqoj5d3AQsBoIosx4i32L8Ex9+5BuAyHi/IM9VD4=
11041100
github.com/conduitio/conduit-connector-sdk v0.8.0/go.mod h1:nOz4K3X6fD8YMe5CPbULwSEE18Eu02ZrpT6o6KwQfxs=
1101+
github.com/conduitio/conduit-processor-sdk v0.1.0 h1:DpCDFZGd9skoAEizZ+5B58X4oyPO0saTsoka+N/Awek=
1102+
github.com/conduitio/conduit-processor-sdk v0.1.0/go.mod h1:9sjasEukN9HAXj1xZ6kLNzz8mpn2TGrbD9mmPYpYNv8=
11051103
github.com/conduitio/yaml/v3 v3.3.0 h1:kbbaOSHcuH39gP4+rgbJGl6DSbLZcJgEaBvkEXJlCsI=
11061104
github.com/conduitio/yaml/v3 v3.3.0/go.mod h1:JNgFMOX1t8W4YJuRZOh6GggVtSMsgP9XgTw+7dIenpc=
11071105
github.com/containerd/log v0.1.0 h1:TCJt7ioM2cr/tfR8GPbGf9/VRAX8D2B4PjzCpfX540I=
@@ -1141,8 +1139,8 @@ github.com/distribution/reference v0.5.0 h1:/FUIFXtfc/x2gpa5/VGfiGLuOIdYa1t65IKK
11411139
github.com/distribution/reference v0.5.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E=
11421140
github.com/dlclark/regexp2 v1.4.1-0.20201116162257-a2a8dda75c91/go.mod h1:2pZnwuY/m+8K6iRw6wQdMtk+rH5tNGR1i55kozfMjCc=
11431141
github.com/dlclark/regexp2 v1.7.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
1144-
github.com/dlclark/regexp2 v1.8.0 h1:rJD5HeGIT/2b5CDk63FVCwZA3qgYElfg+oQK7uH5pfE=
1145-
github.com/dlclark/regexp2 v1.8.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
1142+
github.com/dlclark/regexp2 v1.10.0 h1:+/GIL799phkJqYW+3YbOd8LCcbHzT0Pbo8zl70MHsq0=
1143+
github.com/dlclark/regexp2 v1.10.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
11461144
github.com/dnaeon/go-vcr v1.1.0/go.mod h1:M7tiix8f0r6mKKJ3Yq/kqU1OYf3MnfmBWVbPx/yU9ko=
11471145
github.com/dnaeon/go-vcr v1.2.0/go.mod h1:R4UdLID7HZT3taECzJs4YgbbH6PIGXB6W/sc5OLb6RQ=
11481146
github.com/docker/cli v24.0.7+incompatible h1:wa/nIwYFW7BVTGa7SWPVyyXU9lgORqUb1xfI36MSkFg=
@@ -1159,12 +1157,12 @@ github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4
11591157
github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
11601158
github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE=
11611159
github.com/dop251/goja v0.0.0-20211022113120-dc8c55024d06/go.mod h1:R9ET47fwRVRPZnOGvHxxhuZcbrMCuiqOz3Rlrh4KSnk=
1162-
github.com/dop251/goja v0.0.0-20230531210528-d7324b2d74f7 h1:cVGkvrdHgyBkYeB6kMCaF5j2d9Bg4trgbIpcUrKrvk4=
1163-
github.com/dop251/goja v0.0.0-20230531210528-d7324b2d74f7/go.mod h1:QMWlm50DNe14hD7t24KEqZuUdC9sOTy8W6XbCU1mlw4=
1160+
github.com/dop251/goja v0.0.0-20231027120936-b396bb4c349d h1:wi6jN5LVt/ljaBG4ue79Ekzb12QfJ52L9Q98tl8SWhw=
1161+
github.com/dop251/goja v0.0.0-20231027120936-b396bb4c349d/go.mod h1:QMWlm50DNe14hD7t24KEqZuUdC9sOTy8W6XbCU1mlw4=
11641162
github.com/dop251/goja_nodejs v0.0.0-20210225215109-d91c329300e7/go.mod h1:hn7BA7c8pLvoGndExHudxTDKZ84Pyvv+90pbBjbTz0Y=
11651163
github.com/dop251/goja_nodejs v0.0.0-20211022123610-8dd9abb0616d/go.mod h1:DngW8aVqWbuLRMHItjPUyqdj+HWPvnQe8V8y1nDpIbM=
1166-
github.com/dop251/goja_nodejs v0.0.0-20230602164024-804a84515562 h1:0gomDSJiLLlpfKxQAHt5zj+9toIcyLMPgkI/Mgv7FAU=
1167-
github.com/dop251/goja_nodejs v0.0.0-20230602164024-804a84515562/go.mod h1:X2TOTJ+Uamd454RFp7ig2tmP3hQg0Z2Qk8gbVQmU0mk=
1164+
github.com/dop251/goja_nodejs v0.0.0-20231122114759-e84d9a924c5c h1:hLoodLRD4KLWIH8eyAQCLcH8EqIrjac7fCkp/fHnvuQ=
1165+
github.com/dop251/goja_nodejs v0.0.0-20231122114759-e84d9a924c5c/go.mod h1:bhGPmCgCCTSRfiMYWjpS46IDo9EUZXlsuUaPXSWGbv0=
11681166
github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
11691167
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
11701168
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
@@ -1640,8 +1638,8 @@ github.com/klauspost/compress v1.10.3/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYs
16401638
github.com/klauspost/compress v1.13.1/go.mod h1:8dP1Hq4DHOhN9w426knH3Rhby4rFm6D8eO+e+Dq5Gzg=
16411639
github.com/klauspost/compress v1.15.1/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk=
16421640
github.com/klauspost/compress v1.15.9/go.mod h1:PhcZ0MbTNciWF3rruxRgKxI5NkcHHrHUDtV4Yw2GlzU=
1643-
github.com/klauspost/compress v1.17.4 h1:Ej5ixsIri7BrIjBkRZLTo6ghwrEtHFk7ijlczPW4fZ4=
1644-
github.com/klauspost/compress v1.17.4/go.mod h1:/dCuZOvVtNoHsyb+cuJD3itjs3NbnF6KH9zAO4BDxPM=
1641+
github.com/klauspost/compress v1.17.5 h1:d4vBd+7CHydUqpFBgUEKkSdtSugf9YFmSkvUYPquI5E=
1642+
github.com/klauspost/compress v1.17.5/go.mod h1:/dCuZOvVtNoHsyb+cuJD3itjs3NbnF6KH9zAO4BDxPM=
16451643
github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
16461644
github.com/klauspost/cpuid/v2 v2.0.4/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
16471645
github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
@@ -1966,6 +1964,8 @@ github.com/ssgreg/nlreturn/v2 v2.2.1 h1:X4XDI7jstt3ySqGU86YGAURbxw3oTDPK9sPEi6YE
19661964
github.com/ssgreg/nlreturn/v2 v2.2.1/go.mod h1:E/iiPB78hV7Szg2YfRgyIrk1AD6JVMTRkkxBiELzh2I=
19671965
github.com/stbenjam/no-sprintf-host-port v0.1.1 h1:tYugd/yrm1O0dV+ThCbaKZh195Dfm07ysF0U6JQXczc=
19681966
github.com/stbenjam/no-sprintf-host-port v0.1.1/go.mod h1:TLhvtIvONRzdmkFiio4O8LHsN9N74I+PhRquPsxpL0I=
1967+
github.com/stealthrocket/wazergo v0.19.1 h1:BPrITETPgSFwiytwmToO0MbUC/+RGC39JScz1JmmG6c=
1968+
github.com/stealthrocket/wazergo v0.19.1/go.mod h1:riI0hxw4ndZA5e6z7PesHg2BtTftcZaMxRcoiGGipTs=
19691969
github.com/stoewer/go-strcase v1.3.0 h1:g0eASXYtp+yvN9fK8sH94oCIk0fau9uV1/ZdJ0AVEzs=
19701970
github.com/stoewer/go-strcase v1.3.0/go.mod h1:fAH5hQ5pehh+j3nZfvwdk2RgEgQjAoM8wodgtPmh1xo=
19711971
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
@@ -2076,25 +2076,25 @@ go.opencensus.io v0.24.0 h1:y73uSU6J157QMP2kn2r30vwW1A2W2WFwSCGnAVxeaD0=
20762076
go.opencensus.io v0.24.0/go.mod h1:vNK8G9p7aAivkbmorf4v+7Hgx+Zs0yY+0fOtgBfjQKo=
20772077
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.47.0 h1:sv9kVfal0MK0wBMCOGr+HeJm9v803BkJxGrk2au7j08=
20782078
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.47.0/go.mod h1:SK2UL73Zy1quvRPonmOmRDiWk1KBV3LyIeeIxcEApWw=
2079-
go.opentelemetry.io/otel v1.22.0 h1:xS7Ku+7yTFvDfDraDIJVpw7XPyuHlB9MCiqqX5mcJ6Y=
2080-
go.opentelemetry.io/otel v1.22.0/go.mod h1:eoV4iAi3Ea8LkAEI9+GFT44O6T/D0GWAVFyZVCC6pMI=
2081-
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.22.0 h1:9M3+rhx7kZCIQQhQRYaZCdNu1V73tm4TvXs2ntl98C4=
2082-
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.22.0/go.mod h1:noq80iT8rrHP1SfybmPiRGc9dc5M8RPmGvtwo7Oo7tc=
2079+
go.opentelemetry.io/otel v1.24.0 h1:0LAOdjNmQeSTzGBzduGe/rU4tZhMwL5rWgtp9Ku5Jfo=
2080+
go.opentelemetry.io/otel v1.24.0/go.mod h1:W7b9Ozg4nkF5tWI5zsXkaKKDjdVjpD4oAt9Qi/MArHo=
2081+
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.24.0 h1:t6wl9SPayj+c7lEIFgm4ooDBZVb01IhLB4InpomhRw8=
2082+
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.24.0/go.mod h1:iSDOcsnSA5INXzZtwaBPrKp/lWu/V14Dd+llD0oI2EA=
20832083
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.22.0 h1:FyjCyI9jVEfqhUh2MoSkmolPjfh5fp2hnV0b0irxH4Q=
20842084
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.22.0/go.mod h1:hYwym2nDEeZfG/motx0p7L7J1N1vyzIThemQsb4g2qY=
2085-
go.opentelemetry.io/otel/metric v1.22.0 h1:lypMQnGyJYeuYPhOM/bgjbFM6WE44W1/T45er4d8Hhg=
2086-
go.opentelemetry.io/otel/metric v1.22.0/go.mod h1:evJGjVpZv0mQ5QBRJoBF64yMuOf4xCWdXjK8pzFvliY=
2087-
go.opentelemetry.io/otel/sdk v1.22.0 h1:6coWHw9xw7EfClIC/+O31R8IY3/+EiRFHevmHafB2Gw=
2088-
go.opentelemetry.io/otel/sdk v1.22.0/go.mod h1:iu7luyVGYovrRpe2fmj3CVKouQNdTOkxtLzPvPz1DOc=
2085+
go.opentelemetry.io/otel/metric v1.24.0 h1:6EhoGWWK28x1fbpA4tYTOWBkPefTDQnb8WSGXlc88kI=
2086+
go.opentelemetry.io/otel/metric v1.24.0/go.mod h1:VYhLe1rFfxuTXLgj4CBiyz+9WYBA8pNGJgDcSFRKBco=
2087+
go.opentelemetry.io/otel/sdk v1.24.0 h1:YMPPDNymmQN3ZgczicBY3B6sf9n62Dlj9pWD3ucgoDw=
2088+
go.opentelemetry.io/otel/sdk v1.24.0/go.mod h1:KVrIYw6tEubO9E96HQpcmpTKDVn9gdv35HoYiQWGDFg=
20892089
go.opentelemetry.io/otel/sdk/metric v1.19.0 h1:EJoTO5qysMsYCa+w4UghwFV/ptQgqSL/8Ni+hx+8i1k=
20902090
go.opentelemetry.io/otel/sdk/metric v1.19.0/go.mod h1:XjG0jQyFJrv2PbMvwND7LwCEhsJzCzV5210euduKcKY=
2091-
go.opentelemetry.io/otel/trace v1.22.0 h1:Hg6pPujv0XG9QaVbGOBVHunyuLcCC3jN7WEhPx83XD0=
2092-
go.opentelemetry.io/otel/trace v1.22.0/go.mod h1:RbbHXVqKES9QhzZq/fE5UnOSILqRt40a21sPw2He1xo=
2091+
go.opentelemetry.io/otel/trace v1.24.0 h1:CsKnnL4dUAr/0llH9FKuc698G04IrpWV0MQA/Y1YELI=
2092+
go.opentelemetry.io/otel/trace v1.24.0/go.mod h1:HPc3Xr/cOApsBI154IU0OI0HJexz+aw5uPdbs3UCjNU=
20932093
go.opentelemetry.io/proto/otlp v0.7.0/go.mod h1:PqfVotwruBrMGOCsRd/89rSnXhoiJIqeYNgFYFoEGnI=
20942094
go.opentelemetry.io/proto/otlp v0.15.0/go.mod h1:H7XAot3MsfNsj7EXtrA2q5xSNQ10UqI405h3+duxN4U=
20952095
go.opentelemetry.io/proto/otlp v0.19.0/go.mod h1:H7XAot3MsfNsj7EXtrA2q5xSNQ10UqI405h3+duxN4U=
2096-
go.opentelemetry.io/proto/otlp v1.0.0 h1:T0TX0tmXU8a3CbNXzEKGeU5mIVOdf0oykP+u2lIVU/I=
2097-
go.opentelemetry.io/proto/otlp v1.0.0/go.mod h1:Sy6pihPLfYHkr3NkUbEhGHFhINUSI/v80hjKIs5JXpM=
2096+
go.opentelemetry.io/proto/otlp v1.1.0 h1:2Di21piLrCqJ3U3eXGCTPHE9R8Nh+0uglSnOyxikMeI=
2097+
go.opentelemetry.io/proto/otlp v1.1.0/go.mod h1:GpBHCBWiqvVLDqmHZsoMM3C5ySeKTC7ej/RNTae6MdY=
20982098
go.uber.org/atomic v1.3.2/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
20992099
go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
21002100
go.uber.org/atomic v1.5.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ=

‎pkg/conduit/config.go‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,7 @@ import (
2020
"github.com/conduitio/conduit/pkg/foundation/cerrors"
2121
"github.com/conduitio/conduit/pkg/foundation/database"
2222
"github.com/conduitio/conduit/pkg/foundation/log"
23-
"github.com/conduitio/conduit/pkg/plugin/builtin"
24-
"github.com/conduitio/conduit/pkg/processor"
23+
"github.com/conduitio/conduit/pkg/plugin/connector/builtin"
2524
"github.com/rs/zerolog"
2625
)
2726

@@ -68,13 +67,16 @@ type Config struct {
6867
Path string
6968
}
7069

70+
Processors struct {
71+
Path string
72+
}
73+
7174
Pipelines struct {
7275
Path string
7376
ExitOnError bool
7477
}
7578

7679
PluginDispenserFactories map[string]builtin.DispenserFactory
77-
ProcessorBuilderRegistry *processor.BuilderRegistry
7880

7981
dev struct {
8082
cpuprofile string
@@ -94,10 +96,10 @@ func DefaultConfig() Config {
9496
cfg.Log.Level = "info"
9597
cfg.Log.Format = "cli"
9698
cfg.Connectors.Path = "./connectors"
99+
cfg.Processors.Path = "./processors"
97100
cfg.Pipelines.Path = "./pipelines"
98101

99102
cfg.PluginDispenserFactories = builtin.DefaultDispenserFactories
100-
cfg.ProcessorBuilderRegistry = processor.GlobalBuilderRegistry
101103
return cfg
102104
}
103105

‎pkg/conduit/entrypoint.go‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,7 +89,8 @@ func (*Entrypoint) Flags(cfg *Config) *flag.FlagSet {
8989
flags.StringVar(&cfg.Log.Level, "log.level", cfg.Log.Level, "sets logging level; accepts debug, info, warn, error, trace")
9090
flags.StringVar(&cfg.Log.Format, "log.format", cfg.Log.Format, "sets the format of the logging; accepts json, cli")
9191

92-
flags.StringVar(&cfg.Connectors.Path, "connectors.path", cfg.Connectors.Path, "path to standalone connectors directory")
92+
flags.StringVar(&cfg.Connectors.Path, "connectors.path", cfg.Connectors.Path, "path to standalone connectors' directory")
93+
flags.StringVar(&cfg.Processors.Path, "processors.path", cfg.Processors.Path, "path to standalone processors' directory")
9394

9495
flags.StringVar(&cfg.Pipelines.Path, "pipelines.path", cfg.Pipelines.Path, "path to the directory that has the yaml pipeline configuration files, or a single pipeline configuration file")
9596
flags.BoolVar(&cfg.Pipelines.ExitOnError, "pipelines.exit-on-error", cfg.Pipelines.ExitOnError, "exit Conduit if a pipeline experiences an error while running")

0 commit comments

Comments
 (0)