Error: failed to start tikv

[20211202na]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do?

When running $ tiup cluster start tidb-test, I received an error

Starting component pd
	Starting instance node4:2379
	Starting instance node3:2379
	Starting instance node5:2379
	Start instance node5:2379 success
	Start instance node3:2379 success
	Start instance node4:2379 success
Starting component tikv
	Starting instance node8:20162
	Starting instance node6:20160
	Starting instance node7:20161

Error: failed to start tikv: failed to start: node7 tikv-20161.service, please check the instance's log(/local/tidb-deploy/tikv-20161/log) for more detail.: timed out waiting for port 20161 to be started after 2m0s

Verbose debug logs has been written to /users/nobi/.tiup/logs/tiup-cluster-debug-2022-02-16-02-18-50.log.

This is the output of $ tiup cluster display tidb-test

And here is the error message in /local/tidb-deploy/tikv-20161/log/tikv.log:

[2022/02/16 02:23:18.076 -07:00] [WARN] [server.rs:435] ["failed to remove space holder on starting: No such file or directory (os error 2)"]
[2022/02/16 02:23:20.938 -07:00] [FATAL] [lib.rs:465] ["Failed to reserve space for recovery: Operation not supported (os error 95)."] [backtrace="   0: tikv_util::set_panic_hook::{{closure}}\n             at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tikv_util/src/lib.rs:464:18\n   1: std::panicking::rust_panic_with_hook\n             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/panicking.rs:626:17\n   2: std::panicking::begin_panic_handler::{{closure}}\n             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/panicking.rs:519:13\n   3: std::sys_common::backtrace::__rust_end_short_backtrace\n             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/sys_common/backtrace.rs:141:18\n   4: rust_begin_unwind\n             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/panicking.rs:515:5\n   5: std::panicking::begin_panic_fmt\n             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/panicking.rs:457:5\n   6: server::server::TiKVServer<ER>::init_fs::{{closure}}\n             at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/server/src/server.rs:445:26\n   7: core::result::Result<T,E>::map_err\n             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/core/src/result.rs:835:27\n      server::server::TiKVServer<ER>::init_fs\n             at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/server/src/server.rs:441:13\n      server::server::run_tikv\n             at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/server/src/server.rs:155:9\n   8: tikv_server::main\n             at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/cmd/tikv-server/src/main.rs:190:5\n   9: core::ops::function::FnOnce::call_once\n             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/core/src/ops/function.rs:227:5\n      std::sys_common::backtrace::__rust_begin_short_backtrace\n             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/sys_common/backtrace.rs:125:18\n  10: main\n  11: __libc_start_main\n             at /build/glibc-2ORdQG/glibc-2.27/csu/../csu/libc-start.c:310\n  12: <unknown>\n"] [location=components/server/src/server.rs:445] [thread_name=main]

4. What version of TiUP are you using (tiup --version)?
   tiup version: 1.9.0
   tidb version: 5.4.0
   The cluster is set up on Emulab, and here is the config:
   https://github.com/20211202na/tidb-data/blob/main/topology.yaml

[chengshihard]

我也碰到了这个问题 v5.4.0 请教一下 最后怎么解决的~~

[tabokie]

Duplicate with tikv/tikv#12543. Workaround is to set tikv config storage.reserve-space to "0".

[Anutrix]

Same issue is causing part of pingcap/tidb-docker-compose#111.