Worker crashes with 'DataObject is not finished'

[vojtechcima]

Rain worker crashes with thread 'main' panicked at 'DataObject is not finished', src/worker/graph/dataobj.rs:81:18.

The deployment consisted of 1 local and 1 remote worker running on Exoscale infrastructure connecting from a remote client.

Worker log including stack backtrace:

ubuntu@test-0:~$ rain worker <server-ip>:7210 --listen=7211
🌧  INFO 2018-03-26T10:22:40Z Starting Rain 0.1.1 as worker
🌧  INFO 2018-03-26T10:22:40Z Resources: 1 cpus
🌧  INFO 2018-03-26T10:22:40Z Working directory: "/tmp/rain-work/worker-test-0-1261"
🌧  INFO 2018-03-26T10:22:40Z Server address <server-ip>:7210 was resolved as <server-ip>:7210
🌧  INFO 2018-03-26T10:22:40Z Start listening on port=7211
🌧  INFO 2018-03-26T10:22:40Z Connecting to server addr=<server-ip>:7210
🌧  INFO 2018-03-26T10:22:40Z Connected to server; registering as worker
🌧  INFO 2018-03-26T10:22:43Z New connection from <server-ip>:40118
🌧  INFO 2018-03-26T10:23:33Z New connection from 127.0.0.1:54122
thread 'main' panicked at 'DataObject is not finished', src/worker/graph/dataobj.rs:81:18
note: Run with `RUST_BACKTRACE=1` for a backtrace.
ubuntu@test-0:~$ RUST_BACKTRACE=1 rain worker 185.150.9.32:7210 --listen=7211
🌧  INFO 2018-03-26T10:24:19Z Starting Rain 0.1.1 as worker
🌧  INFO 2018-03-26T10:24:19Z Resources: 1 cpus
🌧  INFO 2018-03-26T10:24:19Z Working directory: "/tmp/rain-work/worker-test-0-1317"
🌧  INFO 2018-03-26T10:24:19Z Server address 185.150.9.32:7210 was resolved as 185.150.9.32:7210
🌧  INFO 2018-03-26T10:24:19Z Start listening on port=7211
🌧  INFO 2018-03-26T10:24:19Z Connecting to server addr=185.150.9.32:7210
🌧  INFO 2018-03-26T10:24:19Z Connected to server; registering as worker
🌧  INFO 2018-03-26T10:24:23Z New connection from 185.150.9.32:40124
🌧  INFO 2018-03-26T10:24:24Z New connection from 127.0.0.1:54126
thread 'main' panicked at 'DataObject is not finished', src/worker/graph/dataobj.rs:81:18
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at /checkout/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::print
             at /checkout/src/libstd/sys_common/backtrace.rs:68
             at /checkout/src/libstd/sys_common/backtrace.rs:57
   2: std::panicking::default_hook::{{closure}}
             at /checkout/src/libstd/panicking.rs:381
   3: std::panicking::default_hook
             at /checkout/src/libstd/panicking.rs:397
   4: std::panicking::rust_panic_with_hook
             at /checkout/src/libstd/panicking.rs:577
   5: std::panicking::begin_panic
   6: <librain::worker::rpc::datastore::DataStoreImpl as librain::datastore_capnp::data_store::Server>::create_reader
   7: <librain::datastore_capnp::data_store::ServerDispatch<_T> as capnp::capability::Server>::dispatch_call
   8: <futures::future::lazy::Lazy<F, R> as futures::future::Future>::poll
   9: <capnp_rpc::attach::AttachFuture<F, T> as futures::future::Future>::poll
  10: <futures::future::chain::Chain<A, B, C>>::poll
  11: <futures::future::chain::Chain<A, B, C>>::poll
  12: futures::task_impl::std::set
  13: <capnp_rpc::forked_promise::ForkedPromise<F> as futures::future::Future>::poll
  14: <futures::future::chain::Chain<A, B, C>>::poll
  15: <futures::future::select::Select<A, B> as futures::future::Future>::poll
  16: <futures::future::map::Map<A, F> as futures::future::Future>::poll
  17: <futures::future::map_err::MapErr<A, F> as futures::future::Future>::poll
  18: <futures::future::chain::Chain<A, B, C>>::poll
  19: futures::task_impl::std::set
  20: <futures::stream::futures_unordered::FuturesUnordered<T> as futures::stream::Stream>::poll
  21: <capnp_rpc::task_set::TaskSet<T, E> as futures::future::Future>::poll
  22: <futures::future::chain::Chain<A, B, C>>::poll
  23: futures::task_impl::std::set
  24: <futures::stream::futures_unordered::FuturesUnordered<T> as futures::stream::Stream>::poll
  25: <capnp_rpc::task_set::TaskSet<T, E> as futures::future::Future>::poll
  26: <futures::future::map_err::MapErr<A, F> as futures::future::Future>::poll
  27: <futures::task_impl::Spawn<T>>::poll_future_notify
  28: <std::thread::local::LocalKey<T>>::with
  29: <tokio::executor::current_thread::scheduler::Scheduler<U>>::tick
  30: <scoped_tls::ScopedKey<T>>::set
  31: <std::thread::local::LocalKey<T>>::with
  32: <std::thread::local::LocalKey<T>>::with
  33: tokio_core::reactor::Core::poll
  34: tokio_core::reactor::Core::turn
  35: rain::main
  36: std::rt::lang_start::{{closure}}
  37: std::panicking::try::do_call
             at /checkout/src/libstd/rt.rs:59
             at /checkout/src/libstd/panicking.rs:480
  38: __rust_maybe_catch_panic
             at /checkout/src/libpanic_unwind/lib.rs:101
  39: std::rt::lang_start_internal
             at /checkout/src/libstd/panicking.rs:459
             at /checkout/src/libstd/panic.rs:365
             at /checkout/src/libstd/rt.rs:58
  40: main
  41: __libc_start_main
  42: _start

Server log:

🌧  INFO 2018-03-26T10:24:05Z Starting Rain 0.1.1 server at port 0.0.0.0:7210
🌧  INFO 2018-03-26T10:24:05Z Start listening on address=0.0.0.0:7210
🌧  INFO 2018-03-26T10:24:05Z Dashboard is running at http://test-1:8080/
🌧  INFO 2018-03-26T10:24:05Z Lite dashboard is running at http://test-1:8080/lite/
🌧  INFO 2018-03-26T10:24:16Z New connection from 127.0.0.1:54004
🌧  INFO 2018-03-26T10:24:16Z Connection 127.0.0.1:54004 registered as worker 127.0.0.1:7211 with Resources { cpus: 1 }
🌧  INFO 2018-03-26T10:24:20Z New connection from 185.150.8.101:42994
🌧  INFO 2018-03-26T10:24:20Z Connection 185.150.8.101:42994 registered as worker 185.150.8.101:7211 with Resources { cpus: 1 }
🌧  INFO 2018-03-26T10:24:24Z New connection from 195.113.113.226:55798
🌧  INFO 2018-03-26T10:24:24Z Connection 195.113.113.226:55798 registered as client
🌧  INFO 2018-03-26T10:24:24Z New task submission (11 tasks, 11 data objects) from client 195.113.113.226:55798
🌧  INFO 2018-03-26T10:24:24Z New get_state request (0 tasks, 1 data objects) from client
🌧  INFO 2018-03-26T10:24:24Z Client 195.113.113.226:55798 disconnected
🌧  INFO 2018-03-26T10:24:25Z New connection from 195.113.113.226:55800
🌧  INFO 2018-03-26T10:24:25Z Connection 195.113.113.226:55800 registered as client
🌧  INFO 2018-03-26T10:24:25Z New task submission (11 tasks, 11 data objects) from client 195.113.113.226:55800
🌧 ERROR 2018-03-26T10:24:26Z Connection to worker 185.150.8.101:7211 lost
thread 'main' panicked at 'not yet implemented', src/server/state.rs:94:9
note: Run with `RUST_BACKTRACE=1` for a backtrace.