Destination connector implementation. #1
Conversation
destination/destination.go
Outdated
|
|
||
| filename, ok := record.Metadata["filename"] | ||
| if !ok { | ||
| filename = string(record.Key.Bytes()) |
There was a problem hiding this comment.
What if the key contains some characters that would make an invalid file name? E.g. the key could be a structured ket.
There was a problem hiding this comment.
This(filename missing from metadata) should never happen, record should have the filename metadata, but just to handle this we can use hash for filename?
There was a problem hiding this comment.
That's true, the filename should not be missing, if the record came from an SFTP source. But Conduit should be able to combine different sources, and so, for example, it should be possible to sync files from S3 to an SFTP destination.
The key could actually be the file name?
There was a problem hiding this comment.
Okay, so do you mean if record.Key is of type opencdc.StructuredData which is basically a map, then there should be a key "filename" which could be used here?
There was a problem hiding this comment.
I meant that the key can be a string (i.e. opencdc.RawData), similar to what we have in the S3 source.
There was a problem hiding this comment.
Thanks! A question though: shouldn't we change the source too, to return string keys?
There was a problem hiding this comment.
Agree!, changed.
…ctor-sftp into feat/destination
…ctor-sftp into feat/destination
|
LGTM, thanks @parikshitg! |
Description
Destination implementation for SFTP connector.
Quick checks: