Feature description
We already had a PR for this but it had many problems {all these are just things specific for acceptance tests to work on S3, they are not S3 related bugs}:
#44
-
The acceptance tests expect that the destination writes a record, and the source will retrieve the same record containing the same key and payload.. but this is not the case here because the key would be a random number.. and the payload would be the whole record {S3 destination has a batch size.. so the file name at the destination bucket can't have the same key.. so it's a random number.json}
-
the createdAt timestamp in the record is an int timestamp: ex: 1655131862580631000, and when we try to parse the file body into sdk.Record we get this error:
"CreatedAt":1655131862580631000
err: parsing time "1655131862580631000" as "\"2006-01-02T15:04:05Z07:00\"": cannot parse "1655131862580631000" as "\""
- if the payload has any special characters, a back dash will be added to the string, which will make it not match, ex:
want: "Payload":{"'8B2\u0026U
got : "Payload":"{\"'8B2\\u0026U
- the
isEqualRecords method in acceptance tests assumes that payloads are unique, but in our case, the payload is the whole record.
Feature description
We already had a PR for this but it had many problems {all these are just things specific for acceptance tests to work on S3, they are not S3 related bugs}:
#44
The acceptance tests expect that the destination writes a record, and the source will retrieve the same record containing the same key and payload.. but this is not the case here because the key would be a random number.. and the payload would be the whole record {S3 destination has a batch size.. so the file name at the destination bucket can't have the same key.. so it's a random number.json}
the
createdAttimestamp in the record is an int timestamp: ex: 1655131862580631000, and when we try to parse the file body intosdk.Recordwe get this error:isEqualRecordsmethod in acceptance tests assumes that payloads are unique, but in our case, the payload is the whole record.