Skip to content

Commit

Permalink
uNVMe v19.01
Browse files Browse the repository at this point in the history 
        - support uNVMe-Blobfs for LBA SSD (refer to doc/uNVMe2.0_Blobfs.pdf)
        - bug fix for KV SSD related iterate error code
	- refactorings

=============================================================================================
commit c820b9f67844476a0927c2897da4eb4184e59da6
Author: Wan Heo <[email protected]>
Date:   Tue Jan 8 20:03:18 2019 +0900

    [MOD] test_iterate - fix to proper error code

commit 8684726b925b63d9f7e4b94ca3b3106bce59eb1e
Author: Sungjun Park <[email protected]>
Date:   Wed Jan 9 01:10:17 2019 +0900

    updated documents for uNVMe-Blobfs

commit 06231c5847bf88668e05f3c022a5a3041418d324
Author: ks0204.kim <[email protected]>
Date:   Tue Jan 8 03:15:41 2019 +0900

    [MOD] fix build.sh to properly delete app/ dirs

commit 450f827745cdf1a33de805cff7e3e8c26df58289
Author: ks0204.kim <[email protected]>
Date:   Wed Dec 19 13:30:22 2018 +0900

    [MOD] fix _kv_nvme_iterate_open() to properly return error code

commit 14ed34da79cdc42d3e93999df04876d1821f3821
Author: Im, Junhyeok <[email protected]>
Date:   Wed Dec 12 17:41:55 2018 +0900

    [MOD] blobfs_perf: core_mask doesn't need to be continous

commit e8a1f3942fbf08f4dc1deef0cefc10b739207ebd
Author: Im, Junhyeok <[email protected]>
Date:   Mon Dec 10 16:58:55 2018 +0900

    [MOD] updates for http://107.108.222.42:8080/projects/M18HDOJ/issues/M18HDOJ-78 and blobfs_perf

    1) mod README to contain correct source tree
    2) blobfs_perf: exit normally after showing 'help' msg

commit 41dd2fa7b55e4ee68d9b924288924cdad9b9d30c
Author: Im, Junhyeok <[email protected]>
Date:   Thu Dec 6 19:04:20 2018 +0900

    [MOD] blobfs_perf: fix verify option and help msg, add preventing codes for wrong args

    1) fix verify option (-v) to work properly
    2) fix README and help msg to show default values of options properly
    3) add preventing codes for user input error

commit fa113de54cce0499d4ef114957179cda441b49b8
Author: Im, Junhyeok <[email protected]>
Date:   Thu Dec 6 15:19:45 2018 +0900

    [MOD] kv_driver.c: fix to preventing invalid num_cq_threads when cq_thread_mask=0x0

commit 711262cb0f43ab1261f3626ee09306f591fddb84
Author: Im, Junhyeok <[email protected]>
Date:   Thu Dec 6 13:46:18 2018 +0900

    [MOD] blobfs_perf(uNVMe & vanilla SPDK) mod (shows default value of options, etc)

    1) shows default values of each option at help (--help, -h)
    2) read test shows 'open/create fail' msg when users try to read 'not existing' files
    3) (uNVMe blobfs_perf) fix 'option -f(prefetch threshold)' to work properly

commit bc0088c059c43999541136424c21529592c3caf4
Author: ks0204.kim <[email protected]>
Date:   Fri Nov 30 09:52:25 2018 -0800

    [MOD] fix app build order

commit 95d31658b0581ccb45432f6af89376ca90659557
Author: Im, Junhyeok <[email protected]>
Date:   Fri Nov 30 16:43:23 2018 +0900

    uNVMe Blobfs

commit ed50af7cf2e35d559634f2f526a193bb9684f243
Author: Im, Junhyeok <[email protected]>
Date:   Tue Nov 6 16:02:18 2018 +0900

    [MOD] updates for http://107.108.222.42:8080/browse/M18HDOJ-53, set core_mask inside the UTs

commit 061362d1e961b05ed34e43e1c87ddf557d41285b
Author: ks0204.kim <[email protected]>
Date:   Tue Nov 6 14:58:50 2018 +0900

    [MOD] updates for http://107.108.222.42:8080/browse/M18HDOJ-48,  fix typos and potential defect

commit 2395b55aca39646bbda27c06d91074fdc8a40c03
Author: ks0204.kim <[email protected]>
Date:   Fri Nov 2 17:00:47 2018 +0900

    [MOD] remove deprecated files, change some source files permission (755->644)

commit a4fa63ebf5029460166a454ebf0dc5fb893e394e
Author: ks0204.kim <[email protected]>
Date:   Fri Nov 2 13:59:32 2018 +0900

    [MOD] fix iterate TC bug to call huge memory free API (free -> kv_free)

commit cbcbfa098dd01cac34b8df33998366acdcf14ae1
Author: ks0204.kim <[email protected]>
Date:   Fri Nov 2 13:52:35 2018 +0900

    [MOD] fix typos

commit ce0b6d1b0b0488b61bd44d47e17fdcd9b2c4202f
Author: ks0204.kim <[email protected]>
Date:   Fri Nov 2 13:11:48 2018 +0900

    [MOD] update doc uNVMe2.0_SDK_Evaluation_Guide_v1.1.pdf / uNVMe2.0_SDK_Programming_Guide_v1.1.pdf

commit f321819207f972de8180e3c4d0294ad5174e581b
Author: ks0204.kim <[email protected]>
Date:   Wed Oct 31 14:28:52 2018 +0900

    [MOD] fix compatibility issue : library ordering issue on SQA machine

commit 31b257dd1b8faa68f26fc252b8fb67992fcdcdc4
Author: ks0204.kim <[email protected]>
Date:   Tue Oct 30 11:22:43 2018 +0900

    [MOD] make kv_nvme built by make.sh app

commit a9b89fbf7e1a400227591020cfd67a0c2d28698e
Author: Im, Junhyeok <[email protected]>
Date:   Wed Oct 24 10:39:21 2018 +0900

    [MOD] fix iterate TCs to keep up with new iterate response format

commit d457574c6d972cdf30cafccdfa31f5d57639bdf4
Author: ks0204.kim <[email protected]>
Date:   Tue Oct 23 04:32:10 2018 +0900

    [MOD] fix host prp buffer layout : 1) fallback for slower spdk_dma_allocate() api  2) not to see invalid prp offset on more SSDs

commit f9471ea23e2864480c136f579feed217f0ac09d6
Author: Wan Heo <[email protected]>
Date:   Thu Oct 18 19:08:48 2018 +0900

    [MOD] kv_nvme - remove auto-reset after format command

commit 82926fb5cbd77b4f13c0e2484ba742799f8dc329
Author: Im, Junhyeok <[email protected]>
Date:   Thu Sep 20 15:33:43 2018 +0900

    [MOD] kv_nvme: remove fbarray files when kv_nvme (as secondary process) terminated
  • Loading branch information
@KyungsanKim
KyungsanKim committed Jan 9, 2019
1 parent dfb875c commit a7a344c
Show file tree
Hide file tree
Showing 71 changed files with 797 additions and 288 deletions.
8 changes: 5 additions & 3 deletions README
View file
Original file line number Diff line number Diff line change
Expand Up @@ -62,27 +62,29 @@ Source tree

uNVMe/
├── app
│   ├── blobfs_perf
│   ├── external
│   │   └── fio
│   ├── fio_plugin
│   ├── fuse
│   ├── mkfs
│   └── unvme_rocksdb
├── bin
├── doc
├── driver
│   ├── core
│   ├── external
│   │   ├── common
│   │   ├── dpdk
│   │   └── spdk
│   │   ├── dpdk-18.05
│   │   └── spdk-18.04.1
│   ├── include
│   ├── mk
│   └── test
├── include
├── io
│   ├── src
│   │   ├── common
│   │   └── slab
│   └── tests
├── mk
└── script

37 changes: 23 additions & 14 deletions app/fio_plugin/unvme2_fio_plugin.c
View file
Original file line number Diff line number Diff line change
Expand Up @@ -288,20 +288,29 @@ static int kv_fio_setup(struct thread_data *td)
fprintf(stderr, "unvme2_fio_plugin is built with fio version=%d.%d\n",FIO_MAJOR_VERSION, FIO_MINOR_VERSION);
kv_sdk_info();

assert(engine_option->json_path != NULL);
if (engine_option->json_path == NULL) {
fprintf(stderr, "Need to specify the path of uNVMe configuration file to option 'json_path'\n");
goto err;
}

if (engine_option->key_size < KV_MIN_KEY_LEN || engine_option->key_size > KV_MAX_KEY_LEN) {
fprintf(stderr, "keysize(%u) should be between %u and %u.\n", \
engine_option->key_size, KV_MIN_KEY_LEN, KV_MAX_KEY_LEN);
goto err;
}
assert((engine_option->key_size >= KV_MIN_KEY_LEN) && (engine_option->key_size <= KV_MAX_KEY_LEN));

ret = kv_fio_parse_config_file(engine_option->json_path, td);
assert(ret == KV_SUCCESS);
if (ret != KV_SUCCESS) {
fprintf(stderr, "Failed to parse configuration file. Please check the configuration is described properly\n");
goto err;
}

g_sdk_opt.app_hugemem_size = kv_fio_calc_hugemem_size(td);
ret = kv_sdk_init(KV_SDK_INIT_FROM_STR, &g_sdk_opt);
assert(ret == KV_SUCCESS);
if (ret != KV_SUCCESS) {
fprintf(stderr, "Failed to initialize SDK(ret: 0x%x). Please check the configuration(%s)\n", ret, engine_option->json_path);
goto err;
}

kv_fio_set_io_function(td);

Expand Down Expand Up @@ -351,15 +360,15 @@ static int kv_fio_setup(struct thread_data *td)

ret = kv_fio_get_dev_info_on_bdf(f->file_name, dev_info);
if (ret != KV_SUCCESS) {
fprintf(stderr, "Can't find SSD(\"%s\"). Please Check the fio option 'filename' is set properly.\n", f->file_name);
fprintf(stderr, "Failed to find the device %s. Please check the option 'filename' is set properly\n", f->file_name);
goto err;
}
assert(ret == KV_SUCCESS);

f->real_file_size = kv_nvme_get_total_size(dev_info->dev_handle);
if (f->real_file_size == KV_ERR_INVALID_VALUE) {
fprintf(stderr, "Can't identify the total size of SSD(\"%s\").\n", f->file_name);
fprintf(stderr, "Failed to get total size of the device %s\n", f->file_name);
goto err;
}
assert(f->real_file_size != KV_ERR_INVALID_VALUE);

#if (CHECK_FIO_VERSION(FIO_MAJOR_VERSION, FIO_MINOR_VERSION) >= CHECK_FIO_VERSION(2,18))
f->filetype = FIO_TYPE_BLOCK;
Expand All @@ -377,8 +386,8 @@ static int kv_fio_setup(struct thread_data *td)
fio_thread->core_id = kv_fio_get_core_and_qid_for_io_thread(fio_thread, arr_qid, &num_possible_qid);
if (fio_thread->core_id < 0) {
fprintf(stderr, "This job can't generate IO workload due to the wrong configuration. \
Please check the core_mask options of your device(s)\n");
assert(0);
Please check the option 'core_mask' of the device(s)\n");
goto err;
}
fio_thread->qid = arr_qid[td->subjob_number % num_possible_qid];
//fprintf(stderr, "thread #%u: subjob #%u, core #%d, queue #%d\n", td->thread_number, td->subjob_number, fio_thread->core_id, fio_thread->qid);
Expand All @@ -405,6 +414,8 @@ static int kv_fio_setup(struct thread_data *td)
pthread_mutex_unlock(&mutex);

return 0;
err:
return 1;
}

static int kv_fio_open(struct thread_data *td, struct fio_file *f)
Expand Down Expand Up @@ -467,16 +478,14 @@ static int kv_fio_io_u_init(struct thread_data *td, struct io_u *io_u)
orig_buffer = td->orig_buffer;
}
io_u->buf = orig_buffer + (io_u->index * aligned_max_bs);
io_u->engine_data = fio_req;

fio_req->io = io_u;
fio_req->fio_thread = fio_thread;
fio_req->key_size = fio_thread->key_size;
fio_req->key = kv_zalloc(MEM_ALIGN(fio_req->key_size, 4)); //for long key support
assert(fio_req->key != NULL);

io_u->engine_data = fio_req;

return 0;
return fio_req->key == NULL;
}

static void kv_fio_io_u_free(struct thread_data *td, struct io_u *io_u)
Expand Down
42 changes: 31 additions & 11 deletions app/fuse/fuse.c
View file
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,15 @@ __call_fn(void *arg1, void *arg2)
fn(arg2);
}

static void
__send_request_mq(fs_request_fn fn, void *arg, int qid)
{
struct spdk_event *event;

event = spdk_event_allocate(qid, __call_fn, (void *)fn, arg);
spdk_event_call(event);
}

static void
__send_request(fs_request_fn fn, void *arg)
{
Expand Down Expand Up @@ -293,8 +302,15 @@ init_cb(void *ctx, struct spdk_filesystem *fs, int fserrno)
{
struct spdk_event *event;

if ((fs == NULL) || (fserrno != 0)) {
SPDK_ERRLOG("Failed to initialize fuse.\n");
spdk_app_stop(fserrno);
kv_sdk_finalize();
exit(EXIT_FAILURE);
}
g_fs = fs;
g_channel = spdk_fs_alloc_io_channel_sync(g_fs);
set_fs_set_send_request_mq_fn(fs, __send_request_mq);
g_channel = spdk_fs_alloc_io_channel_sync(g_fs, 0);
event = spdk_event_allocate(1, start_fuse_fn, NULL, NULL);
spdk_event_call(event);
}
Expand Down Expand Up @@ -325,7 +341,6 @@ int main(int argc, char **argv)
{
kv_sdk sdk_opt;
struct spdk_app_opts opts = {};
struct spdk_env_opts env_opts;
char* config_json_path;
uint64_t app_hugemem_size_mb;
uint64_t fs_cache_size_mb;
Expand All @@ -343,25 +358,30 @@ int main(int argc, char **argv)
g_fuse_argc = argc - 2;
g_fuse_argv = &argv[1];

kv_sdk_load_option(&sdk_opt, config_json_path);
memset(&sdk_opt, 0, sizeof(kv_sdk));
rc = kv_sdk_load_option(&sdk_opt, config_json_path);
if (rc != KV_SUCCESS) {
fprintf(stderr, "Error while loading JSON configuration.\n");
exit(EXIT_FAILURE);
}

if (sdk_opt.ssd_type != LBA_TYPE_SSD) {
fprintf(stderr, "This application does not support KV SSD.\n");
exit(EXIT_FAILURE);
}

app_hugemem_size_mb = sdk_opt.app_hugemem_size / ((uint64_t)(1024 * 1024));
if(MIN_APP_HUGEMEM_SIZE_MB > app_hugemem_size_mb) {
fprintf(stderr, "app_hugemem_size must be larger than MIN_APP_HUGEMEM_SIZE_MB(%llu)\n", MIN_APP_HUGEMEM_SIZE_MB);
exit(1);
exit(EXIT_FAILURE);
}
else{
fs_cache_size_mb = app_hugemem_size_mb - MIN_APP_HUGEMEM_SIZE_MB;
}

env_opts.name = "KV_Interface";
env_opts.mem_size = app_hugemem_size_mb;
env_opts.core_mask = "0x3";
env_opts.shm_id = getpid();

rc = kv_sdk_init_with_spdk_opts(KV_SDK_INIT_FROM_STR, &sdk_opt, &env_opts);
rc = kv_sdk_init(KV_SDK_INIT_FROM_STR, &sdk_opt);
if (rc != KV_SUCCESS) {
SPDK_ERRLOG("Error while doing sdk init.\n");
fprintf(stderr, "Error while doing sdk init.\n");
exit(EXIT_FAILURE);
}

Expand Down
2 changes: 1 addition & 1 deletion app/fuse/lba_sdk_config.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"cache": "off",
"cache_algorithm": "radix",
"cache_reclaim_policy" : "lru",
"slab_size" : 24,
"slab_size" : 44,
"app_hugemem_size" : 3072,
"slab_alloc_policy" : "huge",
"ssd_type" : "lba",
Expand Down
2 changes: 1 addition & 1 deletion app/mkfs/lba_sdk_config.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"cache": "off",
"cache_algorithm": "radix",
"cache_reclaim_policy" : "lru",
"slab_size" : 24,
"slab_size" : 44,
"app_hugemem_size" : 512,
"slab_alloc_policy" : "huge",
"ssd_type" : "lba",
Expand Down
22 changes: 19 additions & 3 deletions app/mkfs/mkfs.c
View file
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,12 @@ init_cb(void *ctx, struct spdk_filesystem *fs, int fserrno)
{
struct spdk_event *event;

if ((fs == NULL) || (fserrno != 0)) {
SPDK_ERRLOG("Failed to initialize blobfs.\n");
spdk_app_stop(fserrno);
kv_sdk_finalize();
exit(EXIT_FAILURE);
}
event = spdk_event_allocate(0, shutdown_cb, fs, NULL);
spdk_event_call(event);
}
Expand Down Expand Up @@ -144,19 +150,29 @@ int main(int argc, char **argv)
config_json_path = argv[2];
}

kv_sdk_load_option(&sdk_opt, config_json_path);
memset(&sdk_opt, 0, sizeof(kv_sdk));
rc = kv_sdk_load_option(&sdk_opt, config_json_path);
if (rc != KV_SUCCESS) {
fprintf(stderr, "Error while loading JSON configuration.\n");
exit(EXIT_FAILURE);
}

if (sdk_opt.ssd_type != LBA_TYPE_SSD) {
fprintf(stderr, "This application does not support KV SSD.\n");
exit(EXIT_FAILURE);
}

app_hugemem_size_mb = sdk_opt.app_hugemem_size / ((uint64_t)(1024 * 1024));
if(MIN_APP_HUGEMEM_SIZE_MB > app_hugemem_size_mb) {
fprintf(stderr, "app_hugemem_size must be larger than MIN_APP_HUGEMEM_SIZE_MB(%llu)\n", MIN_APP_HUGEMEM_SIZE_MB);
exit(1);
exit(EXIT_FAILURE);
} else {
fs_cache_size_mb = app_hugemem_size_mb - MIN_APP_HUGEMEM_SIZE_MB;
}

rc = kv_sdk_init(KV_SDK_INIT_FROM_STR, &sdk_opt);
if (rc != KV_SUCCESS) {
SPDK_ERRLOG("Error while doing sdk init.\n");
fprintf(stderr, "Error while doing sdk init.\n");
exit(EXIT_FAILURE);
}

Expand Down
Loading

0 comments on commit a7a344c

Please sign in to comment.