Handle different data types #8
Conversation
| REQUIRE_NE(arrow_array_deserialized_ar, nullptr); | ||
| compare_arrow_arrays(*arrow_array_ar, *arrow_array_deserialized_ar); | ||
|
|
||
| // compare_values<T>(ar, deserialized_ar); |
There was a problem hiding this comment.
This is failing on windows, as CHECK_EQ(ar, deserialized_ar);.
cf. error here.
We should either fix this upstream in sparrow (if indeed something is missing) or at least try and use comparison functions from sparrow (to be exposed if not already).
Note that it wasn't the case before this PR, i.e CHECK_EQ(ar, deserialized_ar); was not failing on windows when building (tests are commented out there for now), but maybe this was some latent issue that was exposed when adding more tests and changing the structure?...
There was a problem hiding this comment.
This looks like a compiler issue, this operator is defined ins sparrow
There was a problem hiding this comment.
NOTE:
We should test again - with this PR merged - using sparrow build from source (and then directly from conda-forge after releasing next version and confirming this fixes the issue).
src/serialize.cpp
Outdated
| // - Correctly populating the Flatbuffer-defined metadata for both messages. | ||
|
|
||
| // Create a mutable copy of the input array to allow moving its internal structures | ||
| sparrow::primitive_array<T> mutable_arr = arr; |
There was a problem hiding this comment.
Should we also provide a destructive version of serialize to avoid this copy? I.e. something like
template <typename T>
std::vector<uint8_t> serialize_primitive_array(sparrow::primitive_array<T>&& arr)There was a problem hiding this comment.
Actually I decided to use extract_arrow_structures before handling the metadata (which we need to get using the primitive_array input argument through arr.metadata() and not through arrow_schema to get the right format).
If we add the overload you suggested, we would need to deep copy the metadata, and that's not ideal.
Instead I'm using now get_arrow_structures to avoid copying...
But I had to remove the const from the primitive_array input argument because of errors due to get_arrow_structures not taking a const.
=> Should we provide an overload for const in sparrow for get_arrow_structures knowing that it's calling get_arrow_proxy, and that can take either a const or a non-const ARRAY?
No description provided.