Skip to content

Commit 23b1fdc

Browse files
houjunhoujun
authored
SQLite and RocksDB support for KVtags (#165)
SQLite and RocksDB support for KVtags
1 parent 1e03014 commit 23b1fdc

13 files changed

+1164
-259
lines changed

‎CMakeLists.txt

+14
Original file line numberDiff line numberDiff line change
@@ -467,7 +467,21 @@ if(PDC_ENABLE_FASTBIT)
467467
set(ENABLE_FASTBIT 1)
468468
endif()
469469

470+
471+
# Metadata with RocksDB
472+
#-----------------------------------------------------------------------------
473+
option(PDC_ENABLE_ROCKSDB "Enable RocksDB (experimental)." OFF)
474+
if(PDC_ENABLE_ROCKSDB)
475+
set(ENABLE_ROCKSDB 1)
476+
endif()
477+
478+
# Metadata with SQLite
470479
#-----------------------------------------------------------------------------
480+
option(PDC_ENABLE_SQLITE3 "Enable SQLite3 (experimental)." OFF)
481+
if(PDC_ENABLE_SQLITE3)
482+
set(ENABLE_SQLITE3 1)
483+
endif()
484+
471485
# Check availability of symbols
472486
#-----------------------------------------------------------------------------
473487
check_symbol_exists(malloc_usable_size "malloc.h" HAVE_MALLOC_USABLE_SIZE)

‎docs/source/api.rst

+29-12
Original file line numberDiff line numberDiff line change
@@ -253,6 +253,33 @@ PDC object APIs
253253
* Delete data from an object.
254254
* For developers: see pdc_client_connect.c. Use PDC_obj_get_info to retrieve name. Then forward name to servers to fulfill requests.
255255

256+
---------------------------
257+
PDC region APIs
258+
---------------------------
259+
260+
261+
---------------------------
262+
PDC property APIs
263+
---------------------------
264+
265+
266+
---------------------------
267+
PDC metadata APIs
268+
---------------------------
269+
PDC maintains object metadata (obj name, dimension, create time, etc.) in a distributed hash table. Each object's metadata can be
270+
accessed with its object ID. Users can also issue metadata queries to retrieve the object IDs that meet the query constraints.
271+
272+
PDC allows users to add key-value tags to each object, where key is a string and value can be a binary array of any datatype and length.
273+
The key-value tags are stored in an in-memory linked list by default.
274+
275+
PDC has metadata indexing and querying support when DART is enabled. See ``DART`` section in the Developer Notes.
276+
277+
PDC additionally supports managing the key-value tags with RocksDB and SQLite, both are considered experimental at the moment.
278+
Either RocksDB or SQLite can be enabled by turning on the ``PDC_ENABLE_ROCKSDB`` or ``PDC_USE_SQLITE3`` flag in CMake, setting the
279+
``ROCKSDB_DIR`` or ``SQLITE3_DIR`` and setting the environment variable ``PDC_USE_ROCKSDB`` or ``PDC_USE_SQLITE3`` to 1 before launching the server.
280+
Users can use the same PDC query APIs when RocksDB or SQLite is enabled.
281+
282+
256283
* perr_t PDCobj_put_tag(pdcid_t obj_id, char *tag_name, void *tag_value, psize_t value_size)
257284
* Input:
258285
* obj_id: Local object ID
@@ -285,17 +312,7 @@ PDC object APIs
285312
* For developers: see pdc_client_connect.c. Need to use PDCtag_delete to submit RPCs to the servers for metadata update.
286313
287314
---------------------------
288-
PDC region APIs
289-
---------------------------
290-
291-
292-
---------------------------
293-
PDC property APIs
294-
---------------------------
295-
296-
297-
---------------------------
298-
PDC query APIs
315+
PDC Data query APIs
299316
---------------------------
300317

301318
* pdc_query_t *PDCquery_create(pdcid_t obj_id, pdc_query_op_t op, pdc_var_type_t type, void *value)
@@ -883,4 +900,4 @@ Developers notes
883900
* Object
884901

885902
* Object property See `Object Property <file:///Users/kenneth/Documents/Berkeley%20Lab/pdc/docs/build/html/pdcapis.html#object-property>`_
886-
* Object structure (pdc_obj_pkg.h and pdc_obj.h) See `Object Structure <file:///Users/kenneth/Documents/Berkeley%20Lab/pdc/docs/build/html/pdcapis.html#object-structure>`_
903+
* Object structure (pdc_obj_pkg.h and pdc_obj.h) See `Object Structure <file:///Users/kenneth/Documents/Berkeley%20Lab/pdc/docs/build/html/pdcapis.html#object-structure>`_

‎docs/source/developer-notes.rst

+7-1
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,13 @@ For No-index approach, here are the APIs you can call for different communicatio
138138
* PDC_Client_query_kvtag (point-to-point)
139139
* PDC_Client_query_kvtag_mpi (collective)
140140

141+
The default PDC kvtags are stored within each object's metadata as a linked list, and any query involves traversing the list in memory.
142+
143+
We have additional support to manage the kvtags with RocksDB and SQLite. With this approach, each PDC server creates and accesses its own RocksDB and SQLite database file, which is stored as an in-memory file in /tmp directory. When RocksDB or SQLite is enabled with setting the environment variable ``PDC_USE_ROCKSDB=1`` or ``PDC_USE_SQLITE3=1``.
144+
With the RocksDB implementation, each kvtag is stored as a RocksDB key-value pair. To differenciate the kvtags for different objects, we encode the object ID to the key string used for the RocksDB, and store the value as the RocksDB value. As a result, the value can be retrieved directly when its object ID and key string is known. Otherwise we must iterate over the entire DB to search for an kvtag.
145+
With the SQLite3 implementation, each kvtag is inserted as a row in a SQLite3 table. Currently, the table has the following columns and SQLite3 datatypes: objid (INTEGER), name (TEXT), value_text(TEXT), value_int(INTEGER), value_float(REAL), value_double(REAL), value_blob(BLOB). We create a SQL SELECT statement automatically on the server when receiving a query request from the PDC client. Currently this implementation is focused on supporting string/text affix search and integer/float (single) value match search.
146+
Currently, both the RocksDB and the SQLite implementation are developed for benchmarking purpose, the database files are removed at server finalization time, and restart is not supported.
147+
141148
Index-facilitated Approach
142149
---------------------------------------------
143150

@@ -398,7 +405,6 @@ Also, to make sure your code with Julia function calls doesn't get compiled when
398405

399406
For more info on embedded Julia support, please visit: `Embedded Julia https://docs.julialang.org/en/v1/manual/embedding/`_.
400407

401-
402408
---------------------------------------------
403409
Docker Support
404410
---------------------------------------------

‎src/api/pdc_client_connect.c

+10-2
Original file line numberDiff line numberDiff line change
@@ -9020,8 +9020,16 @@ _standard_all_gather_result(int query_sent, int *n_res, uint64_t **pdc_ids, MPI_
90209020
uint64_t *all_ids = (uint64_t *)malloc(ntotal * sizeof(uint64_t));
90219021
MPI_Allgatherv(*pdc_ids, *n_res, MPI_UINT64_T, all_ids, all_nmeta_array, disp, MPI_UINT64_T, world_comm);
90229022

9023+
if (*pdc_ids)
9024+
free(*pdc_ids);
9025+
90239026
*n_res = ntotal;
90249027
*pdc_ids = all_ids;
9028+
9029+
free(all_nmeta_array);
9030+
free(disp);
9031+
9032+
return;
90259033
}
90269034

90279035
void
@@ -9127,7 +9135,7 @@ PDC_Client_query_kvtag_mpi(const pdc_kvtag_t *kvtag, int *n_res, uint64_t **pdc_
91279135

91289136
if (*n_res <= 0) {
91299137
*n_res = 0;
9130-
*pdc_ids = (uint64_t *)malloc(0);
9138+
*pdc_ids = NULL;
91319139
}
91329140
else {
91339141
// print the pdc ids returned by this client, along with the client id
@@ -9349,4 +9357,4 @@ PDC_Client_search_obj_ref_through_dart_mpi(dart_hash_algo_t hash_algo, char *que
93499357
}
93509358
#endif
93519359

9352-
/******************** Collective Object Selection Query Ends *******************************/
9360+
/******************** Collective Object Selection Query Ends *******************************/

‎src/server/CMakeLists.txt

+26-7
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,17 @@ if(PDC_ENABLE_FASTBIT)
66
find_library(FASTBIT_LIBRARY fastbit $ENV{HOME}/cori/fastbit-2.0.3/install)
77
endif()
88

9+
if(PDC_ENABLE_ROCKSDB)
10+
add_definitions(-DENABLE_ROCKSDB=1)
11+
find_path(ROCKSDB_INCLUDE_DIR include/db.h)
12+
find_library(ROCKSDB_LIBRARY rocksdb 8.1.1< REQUIRED)
13+
endif()
14+
15+
if(PDC_ENABLE_SQLITE3)
16+
add_definitions(-DENABLE_SQLITE3=1)
17+
find_package(SQLite3 3.31.0 REQUIRED)
18+
endif()
19+
920
include_directories(
1021
${PDC_COMMON_INCLUDE_DIRS}
1122
${PDC_INCLUDES_BUILD_TIME}
@@ -28,6 +39,7 @@ include_directories(
2839
${PDC_SOURCE_DIR}/src/utils/include
2940
${MERCURY_INCLUDE_DIR}
3041
${FASTBIT_INCLUDE_DIR}
42+
${ROCKSDB_INCLUDE_DIR}
3143
)
3244

3345
add_definitions( -DIS_PDC_SERVER=1 )
@@ -57,9 +69,17 @@ add_library(pdc_server_lib
5769
)
5870
if(PDC_ENABLE_FASTBIT)
5971
message(STATUS "Enabled fastbit")
60-
target_link_libraries(pdc_server_lib mercury ${PDC_COMMONS_LIBRARIES} -lm -ldl ${PDC_EXT_LIB_DEPENDENCIES} ${FASTBIT_LIBRARY}/libfastbit.so)
72+
target_link_libraries(pdc_server_lib ${MERCURY_LIBRARY} ${PDC_COMMONS_LIBRARIES} -lm -ldl ${PDC_EXT_LIB_DEPENDENCIES} ${FASTBIT_LIBRARY}/libfastbit.so)
73+
elseif(PDC_ENABLE_ROCKSDB)
74+
if(PDC_ENABLE_SQLITE3)
75+
target_link_libraries(pdc_server_lib ${MERCURY_LIBRARY} ${PDC_COMMONS_LIBRARIES} -lm -ldl ${PDC_EXT_LIB_DEPENDENCIES} ${ROCKSDB_LIBRARY} SQLite::SQLite3)
76+
else()
77+
target_link_libraries(pdc_server_lib ${MERCURY_LIBRARY} ${PDC_COMMONS_LIBRARIES} -lm -ldl ${PDC_EXT_LIB_DEPENDENCIES} ${ROCKSDB_LIBRARY})
78+
endif()
79+
elseif(PDC_ENABLE_SQLITE3)
80+
target_link_libraries(pdc_server_lib ${MERCURY_LIBRARY} ${PDC_COMMONS_LIBRARIES} -lm -ldl ${PDC_EXT_LIB_DEPENDENCIES} SQLite::SQLite3)
6181
else()
62-
target_link_libraries(pdc_server_lib mercury ${PDC_COMMONS_LIBRARIES} -lm -ldl ${PDC_EXT_LIB_DEPENDENCIES})
82+
target_link_libraries(pdc_server_lib ${MERCURY_LIBRARY} ${PDC_COMMONS_LIBRARIES} -lm -ldl ${PDC_EXT_LIB_DEPENDENCIES})
6383
endif()
6484

6585
add_executable(pdc_server.exe
@@ -78,10 +98,9 @@ if(NOT ${PDC_INSTALL_BIN_DIR} MATCHES ${PROJECT_BINARY_DIR}/bin)
7898
install(
7999
TARGETS
80100
pdc_server.exe
81-
DESTINATION ${PDC_INSTALL_BIN_DIR}
101+
pdc_server_lib
102+
LIBRARY DESTINATION ${PDC_INSTALL_LIB_DIR}
103+
ARCHIVE DESTINATION ${PDC_INSTALL_LIB_DIR}
104+
RUNTIME DESTINATION ${PDC_INSTALL_BIN_DIR}
82105
)
83106
endif()
84-
85-
86-
87-

‎src/server/include/pdc_server.h

+12
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,18 @@
4949
#include "iapi.h"
5050
#endif
5151

52+
#ifdef ENABLE_ROCKSDB
53+
#include "rocksdb/c.h"
54+
extern rocksdb_t *rocksdb_g;
55+
extern int use_rocksdb_g;
56+
#endif
57+
58+
#ifdef ENABLE_SQLITE3
59+
#include "sqlite3.h"
60+
extern sqlite3 *sqlite3_db_g;
61+
extern int use_sqlite3_g;
62+
#endif
63+
5264
#ifdef ENABLE_MULTITHREAD
5365
// Mercury multithread
5466
#include "mercury_thread.h"

‎src/server/include/pdc_server_metadata.h

+10
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,8 @@ extern pdc_remote_server_info_t *pdc_remote_server_info_g;
6262
extern double total_mem_usage_g;
6363
extern int is_hash_table_init_g;
6464
extern int is_restart_g;
65+
extern int use_rocksdb_g;
66+
extern int use_sqlite3_g;
6567

6668
/****************************/
6769
/* Library Private Typedefs */
@@ -83,6 +85,14 @@ typedef struct pdc_cont_hash_table_entry_t {
8385
pdc_kvtag_list_t *kvtag_list_head;
8486
} pdc_cont_hash_table_entry_t;
8587

88+
#ifdef ENABLE_SQLITE3
89+
typedef struct pdc_sqlite3_query_t {
90+
pdcid_t **obj_ids;
91+
int nobj;
92+
int nalloc;
93+
} pdc_sqlite3_query_t;
94+
#endif
95+
8696
/***************************************/
8797
/* Library-private Function Prototypes */
8898
/***************************************/

0 commit comments

Comments
 (0)