Uncaught ACTOR_NOT_ACTIVATED error emitted from ActorManager.deactivateActor (triggered by automatic cleanup?)

[radicaled]

Expected Behavior

No error to be thrown if the ActorManager cannot deactivate an unactivated actor as part of whatever background work it does by default.

Actual Behavior

Occasionally an error message in the form of {"error":"ACTOR_NOT_ACTIVATED","errorMsg":"The actor 3010219526 was not activated"} is thrown by the Dapr JS SDK and caught by our error tracking services. We do not deactivate actors ourselves, so this must be in accordance with some kind of automatic management on the Dapr SDK's part.

The stacktrace is as follows (actor id = <ACTOR_ID>, <PROJECT_DIRECTORY> = our project directory):

Error: {"error":"ACTOR_NOT_ACTIVATED","errorMsg":"The actor <ACTOR_ID> was not activated"}
ActorManager.deactivateActor(/opt/<PROJECT_DIRECTORY>/node_modules/@dapr/dapr/actors/runtime/ActorManager.js:60)
ActorRuntime.deactivate(/opt/<PROJECT_DIRECTORY>/node_modules/@dapr/dapr/actors/runtime/ActorRuntime.js:124)        
HTTPServerActor.handlerDeactivate(/opt/<PROJECT_DIRECTORY>/node_modules/@dapr/dapr/implementation/Server/HTTPServer/actor.js:70)        
<unknown>(/opt/<PROJECT_DIRECTORY>/node_modules/dd-trace/packages/datadog-instrumentations/src/router.js:50)        
bound handlerDeactivate(/opt/<PROJECT_DIRECTORY>/node_modules/dd-trace/packages/datadog-shimmer/src/shimmer.js:26)    
Layer.handle [as handle_request](/opt/<PROJECT_DIRECTORY>/node_modules/express/lib/router/layer.js:95)    
next(/opt/<PROJECT_DIRECTORY>/node_modules/express/lib/router/route.js:144)      
Route.dispatch(/opt/<PROJECT_DIRECTORY>/node_modules/express/lib/router/route.js:114)  
<unknown>(/opt/<PROJECT_DIRECTORY>/node_modules/dd-trace/packages/datadog-instrumentations/src/router.js:50)        
bound dispatch(/opt/<PROJECT_DIRECTORY>/node_modules/dd-trace/packages/datadog-shimmer/src/shimmer.js:26)    
Layer.handle [as handle_request](/opt/<PROJECT_DIRECTORY>/node_modules/express/lib/router/layer.js:95)    
<unknown>(/opt/<PROJECT_DIRECTORY>/node_modules/express/lib/router/index.js:284)        
param(/opt/<PROJECT_DIRECTORY>/node_modules/express/lib/router/index.js:365)      
param(/opt/<PROJECT_DIRECTORY>/node_modules/express/lib/router/index.js:376)      
param(/opt/<PROJECT_DIRECTORY>/node_modules/express/lib/router/index.js:376)      
Function.process_params(/opt/<PROJECT_DIRECTORY>/node_modules/express/lib/router/index.js:421)

Steps to Reproduce the Problem

Unsure. We are currently evaluating Dapr in our development environment which is low traffic, so presumably we are triggering some kind of idle timeout. The last known activity these actors have before seeing this error is thrown is approximately ~1hr prior. No errors from our code are thrown inside these actors AFAIK.

[radicaled]

I believe I tracked this down to a bug in HTTPServer.handlerDeactivate:

js-sdk/src/implementation/Server/HTTPServer/actor.ts

Line 90 in fc52f99

const { actorTypeName, actorId } = req.params;

When we invoke an actor locally, we are using an ID like "123" (a string representing a number) because due to another bug[1] in the SDK where can't use our full IDs ("posts/123"). Because of that bug[1], we're forced to stripe out some of the data ("posts/") and are just left with the remainder ("123"). Locally that's fine.

But when a message comes through HTTPServer.handlerDeactivate, "123" gets parsed as 123 (integer) due through the usage of JSON format. Internally, the actor is registered with the ID "123" and not 123 so this causes ActorManager.deactivateActor to throw.

Does that sound right? This is just from manual code tracing; we're not really set up to debug the JS SDK.

The reason we're using HTTP is purely because that's what the JS SDK tutorials set the communication method to. I can see that if we had used gRPC this wouldn't have been an issue. Is it advised to use gRPC with the JS SDK to avoid these kinds of issues, including the one below[1]?

[1]: The JS SDK directly injects your actor ID into a URL without any safe encoding, so trying to use an ID like posts/123 ends up with an invalid URL like actors/MyActorName/posts/123/myActorFunction. I have not reported this bug.

[MrMint]

Similar to what we ran into here:

#627
#628

[radicaled]

Good catch. I somehow missed both of those when searching up this issue before posting.

Hopefully it gets fixed soon -- we backed out of Dapr due to this + several other bugs. Our ultimate impression was that while the .NET and Java SDKs seem to be in good shape, the JS SDKs aren't really fit for purpose yet.