Skip to content

Commit 8204552

Browse files
authored
apacheGH-41654: [MATLAB] Add new arrow.c.Schema MATLAB class which wraps a C Data Interface format ArrowSchema C struct (apache#41674)
### Rationale for this change Now that the MATLAB interface has support for `arrow.tabular.RecordBatch` and `arrow.array.Array`, we should add support for the [C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html) format. The C Data Interface is based around two C struct definitions: (1) `ArrowArray` and (2) `ArrowSchema`. Now that apache#41653 (add support for `arrow.c.Array`) has been addressed, we should add another new MATLAB class (e.g. `arrow.c.Schema`) which wraps the underlying `ArrowSchema` C struct. Once we have added these two MATLAB classes, we can then add import and export functionality to share the Arrow memory between multiple language runtimes running in the same process. This would help enable workflows like sharing Arrow data between the MATLAB Interface to Arrow and `pyarrow` running within the MATLAB process via the [MATLAB interface to Python](https://www.mathworks.com/help/matlab/call-python-libraries.html)). ### What changes are included in this PR? 1. Added a new C++ proxy class called `arrow::matlab::c::proxy::Schema` which wraps an `ArrowSchema` struct pointer. This class is registered as the proxy `arrow.c.proxy.Schema` in order to make it accessible to MATLAB. 2. Added a new MATLAB class called `arrow.c.Schema` that has an `arrow.c.proxy.Schema` instance. It has one public property named `Address`, which is a scalar `uint64`. This property is the memory address of the `ArrowSchema` struct pointer owned by `arrow.c.proxy.Schema`. ### Are these changes tested? Yes. 1. Added a new test class called `test/arrow/c/tSchema.m`. 2. @ sgilmore10 and I created a prototype for importing and exporting arrow `Array`s via the C Data Interface format [here](https://github.com/mathworks/arrow/tree/arrow-array-address). We were able to share arrow `Array`s and `RecordBatch`s between `mlarrow` and `pyarrow`. Our plan now is to submit the necessary MATLAB code incrementally. ### Are there any user-facing changes? Yes. 1. The `arrow.c.Schema` class is user-facing. However, it's only intended for "advanced" use-cases. In the future, we may add higher-level functionality on top of the C Data Interface so that users don't need to interact with it directly. 2. **NOTE**: On destruction, `arrow.c.proxy.Schema` will check to see if the `ArrowSchema` has already been consumed by an importer. If not, `arrow.c.proxy.Schema`'s destructor will call the release callback on the `ArrowSchema` to avoid memory leaks. To the best of our knowledge, this is similar to the how the [Arrow PyCapsule Interface](https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html) works. ### Future Directions 1. apache#41656 2. We should probably follow up with a PR to create shared infrastructure for `arrow.c.Array` and `arrow.c.Schema`, since they are almost identical in design and implementation. ### Notes 1. Thank you @ sgilmore10 for your help with this pull request! * GitHub Issue: apache#41654 Authored-by: Kevin Gurney <[email protected]> Signed-off-by: Kevin Gurney <[email protected]>
1 parent 2ca9ad2 commit 8204552

File tree

7 files changed

+178
-4
lines changed

7 files changed

+178
-4
lines changed

matlab/src/cpp/arrow/matlab/c/proxy/array.h

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -34,8 +34,6 @@ class Array : public libmexclass::proxy::Proxy {
3434
void getAddress(libmexclass::proxy::method::Context& context);
3535

3636
struct ArrowArray arrowArray;
37-
38-
// struct ArrowArray* arrowArray;
3937
};
4038

41-
} // namespace arrow::matlab::c::proxy
39+
} // namespace arrow::matlab::c::proxy
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
#include <cstddef>
19+
#include "arrow/c/abi.h"
20+
21+
#include "arrow/matlab/c/proxy/schema.h"
22+
23+
#include "libmexclass/proxy/Proxy.h"
24+
25+
namespace arrow::matlab::c::proxy {
26+
27+
Schema::Schema() : arrowSchema{} { REGISTER_METHOD(Schema, getAddress); }
28+
29+
Schema::~Schema() {
30+
if (arrowSchema.release != NULL) {
31+
arrowSchema.release(&arrowSchema);
32+
arrowSchema.release = NULL;
33+
}
34+
}
35+
36+
libmexclass::proxy::MakeResult Schema::make(
37+
const libmexclass::proxy::FunctionArguments& constructor_arguments) {
38+
return std::make_shared<Schema>();
39+
}
40+
41+
void Schema::getAddress(libmexclass::proxy::method::Context& context) {
42+
namespace mda = ::matlab::data;
43+
44+
mda::ArrayFactory factory;
45+
auto address = reinterpret_cast<uint64_t>(&arrowSchema);
46+
context.outputs[0] = factory.createScalar(address);
47+
}
48+
49+
} // namespace arrow::matlab::c::proxy
Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
#include "arrow/c/abi.h"
19+
20+
#include "libmexclass/proxy/Proxy.h"
21+
22+
namespace arrow::matlab::c::proxy {
23+
24+
class Schema : public libmexclass::proxy::Proxy {
25+
public:
26+
Schema();
27+
28+
~Schema();
29+
30+
static libmexclass::proxy::MakeResult make(
31+
const libmexclass::proxy::FunctionArguments& constructor_arguments);
32+
33+
protected:
34+
void getAddress(libmexclass::proxy::method::Context& context);
35+
36+
struct ArrowSchema arrowSchema;
37+
};
38+
39+
} // namespace arrow::matlab::c::proxy

matlab/src/cpp/arrow/matlab/proxy/factory.cc

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626
#include "arrow/matlab/array/proxy/timestamp_array.h"
2727
#include "arrow/matlab/buffer/proxy/buffer.h"
2828
#include "arrow/matlab/c/proxy/array.h"
29+
#include "arrow/matlab/c/proxy/schema.h"
2930
#include "arrow/matlab/error/error.h"
3031
#include "arrow/matlab/io/csv/proxy/table_reader.h"
3132
#include "arrow/matlab/io/csv/proxy/table_writer.h"
@@ -101,6 +102,7 @@ libmexclass::proxy::MakeResult Factory::make_proxy(
101102
REGISTER_PROXY(arrow.io.csv.proxy.TableWriter , arrow::matlab::io::csv::proxy::TableWriter);
102103
REGISTER_PROXY(arrow.io.csv.proxy.TableReader , arrow::matlab::io::csv::proxy::TableReader);
103104
REGISTER_PROXY(arrow.c.proxy.Array , arrow::matlab::c::proxy::Array);
105+
REGISTER_PROXY(arrow.c.proxy.Schema , arrow::matlab::c::proxy::Schema);
104106
// clang-format on
105107

106108
return libmexclass::error::Error{error::UNKNOWN_PROXY_ERROR_ID,

matlab/src/matlab/+arrow/+c/Schema.m

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
%SCHEMA Wrapper for an Arrow C Data Interface format ArrowSchema C struct pointer.
2+
3+
% Licensed to the Apache Software Foundation (ASF) under one or more
4+
% contributor license agreements. See the NOTICE file distributed with
5+
% this work for additional information regarding copyright ownership.
6+
% The ASF licenses this file to you under the Apache License, Version
7+
% 2.0 (the "License"); you may not use this file except in compliance
8+
% with the License. You may obtain a copy of the License at
9+
%
10+
% http://www.apache.org/licenses/LICENSE-2.0
11+
%
12+
% Unless required by applicable law or agreed to in writing, software
13+
% distributed under the License is distributed on an "AS IS" BASIS,
14+
% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
15+
% implied. See the License for the specific language governing
16+
% permissions and limitations under the License.
17+
classdef Schema < matlab.mixin.Scalar
18+
19+
properties (Hidden, SetAccess=private, GetAccess=public)
20+
Proxy
21+
end
22+
23+
properties(Dependent, GetAccess=public, SetAccess=private)
24+
Address(1, 1) uint64
25+
end
26+
27+
methods
28+
function obj = Schema()
29+
proxyName = "arrow.c.proxy.Schema";
30+
obj.Proxy = arrow.internal.proxy.create(proxyName);
31+
end
32+
33+
function address = get.Address(obj)
34+
address = obj.Proxy.getAddress();
35+
end
36+
end
37+
end

matlab/test/arrow/c/tSchema.m

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
%TSCHEMA Defines unit tests for arrow.c.Schema.
2+
3+
% Licensed to the Apache Software Foundation (ASF) under one or more
4+
% contributor license agreements. See the NOTICE file distributed with
5+
% this work for additional information regarding copyright ownership.
6+
% The ASF licenses this file to you under the Apache License, Version
7+
% 2.0 (the "License"); you may not use this file except in compliance
8+
% with the License. You may obtain a copy of the License at
9+
%
10+
% http://www.apache.org/licenses/LICENSE-2.0
11+
%
12+
% Unless required by applicable law or agreed to in writing, software
13+
% distributed under the License is distributed on an "AS IS" BASIS,
14+
% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
15+
% implied. See the License for the specific language governing
16+
% permissions and limitations under the License.
17+
classdef tSchema < matlab.unittest.TestCase
18+
19+
methods (Test)
20+
function TestClassStructure(testCase)
21+
schema = arrow.c.Schema();
22+
23+
% Verify schema is an instance of arrow.c.Schema.
24+
testCase.verifyInstanceOf(schema, "arrow.c.Schema");
25+
26+
% Verify schema has one public property named Address.
27+
props = properties(schema);
28+
testCase.verifyEqual(props, {'Address'});
29+
end
30+
31+
function TestAddressProperty(testCase)
32+
schema = arrow.c.Schema();
33+
34+
% It's impossible to know what the value of Address will be.
35+
% Just verify Address is a scalar uint64.
36+
address = schema.Address;
37+
testCase.verifyInstanceOf(address, "uint64");
38+
testCase.verifyTrue(isscalar(address));
39+
end
40+
41+
function TestAddressNoSetter(testCase)
42+
% Verify the Address property is read-only.
43+
schema = arrow.c.Schema();
44+
fcn = @() setfield(schema, "Address", uint64(10));
45+
testCase.verifyError(fcn, "MATLAB:class:SetProhibited");
46+
end
47+
end
48+
end

matlab/tools/cmake/BuildMatlabArrowInterface.cmake

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,8 @@ set(MATLAB_ARROW_LIBMEXCLASS_CLIENT_PROXY_SOURCES "${CMAKE_SOURCE_DIR}/src/cpp/a
7676
"${CMAKE_SOURCE_DIR}/src/cpp/arrow/matlab/io/csv/proxy/table_reader.cc"
7777
"${CMAKE_SOURCE_DIR}/src/cpp/arrow/matlab/index/validate.cc"
7878
"${CMAKE_SOURCE_DIR}/src/cpp/arrow/matlab/buffer/proxy/buffer.cc"
79-
"${CMAKE_SOURCE_DIR}/src/cpp/arrow/matlab/c/proxy/array.cc")
79+
"${CMAKE_SOURCE_DIR}/src/cpp/arrow/matlab/c/proxy/array.cc"
80+
"${CMAKE_SOURCE_DIR}/src/cpp/arrow/matlab/c/proxy/schema.cc")
8081

8182

8283
set(MATLAB_ARROW_LIBMEXCLASS_CLIENT_PROXY_FACTORY_INCLUDE_DIR "${CMAKE_SOURCE_DIR}/src/cpp/arrow/matlab/proxy")

0 commit comments

Comments
 (0)