Implementation of StubDataClayObject

[marcmonfort]

Implementation of StubDataClayObject

[alexbarcelo]

Feels good! Some inline comments that should be addressed.

Finally, we will need some tests. Off the top of my hat it should be enough to do: checking make_persistent with alias, checking if the client can put a Stub Dog instance in a Person, checking get/set of attributes, checking the call of activemethods.

[marcmonfort]

There is still a bug at retrieving remote instances of DataClayObject sub-classes that are not accessible for the client.

When trying to deserialize the pickle binary, it tries to automatically loads the subclass module, and it raises a ModuleNotFoundError. This happens even when the reducer_override is set to use only the parent DataClayObject:

def reducer_override(self, obj):
    if isinstance(obj, DataClayObject):
        if not obj._dc_is_registered:
            obj.make_persistent()
        return DataClayObject.get_by_id, (obj._dc_meta.id,)
    else:
        return NotImplemented

This problem happens because if get_by_id is a class (or class‐) method, then when it’s looked up on an instance of a subclass (say, model.family.Person), it becomes a bound method whose self is that subclass. That means the pickle machinery records that it needs to fetch get_by_id from the subclass (hence the reference to “model.family” and “Person”) instead of from DataClayObject.

The error can be replicated by running the following script:

from dataclay import Client, StubDataClayObject

client = Client(host="127.0.0.1")
client.start()

# Instantiate the stubs of remote classes
PersonStub = StubDataClayObject["model.family.Person"]
DogStub = StubDataClayObject["model.family.Dog"]

# Create a Person and a Dog from the stubs
person = PersonStub(name="Alice", age=30)
dog = DogStub(name="Rex", age=5)
person.dog = dog

# BUG: The access to the `dog` attribute will raise a `ModuleNotFoundError`
person.dog

[alexbarcelo]

That bug (and similar ones) where things that I expected to stumble upon.

Same happens if we have a function that returns a persistent object (e.g. a function Person.get_spouse_dog() -> Dog or, maybe trickier, Person.get_family_dogs() -> list[Dog].

I was thinking on some ways to address this.

1. Support a flag for the client to fallback to Stubs.

Given that the server ultimately sends object ID and information on the class, it is conceivable that when "DATACLAY_STUB_FALLBACK_FOR_UNKNOWN_CLASSES=True" (flag name TBD) if there is an ImportError in the process of creating the class, the dataClay getter and dataClay active method return transparently fallback to return a Stub

2. Have a context manager

This is the same as the previous one but with something like with client.use_stubs(): and then, everything inside the contextmanager uses Stub instances. So returns and getters are transparently converted to Stub.

3. Have a helper function

This is cumbersome to the programmer, but maybe works as a workaround or a stopgag measure (assuming it is much easier than the previous one, which I am not sure).

dataclay.attribute_to_stub(person, "dog")

or

dataclay.return_to_stub(person.get_spouse_dog, *args, **kwargs)

I am not sure how that would work with a list[Dog]... supporting any nested structure would mean falling back to solutions 1&2, if I'm not mistaken.

---

@marcmonfort think a little bit on that, have some time estimate, and if it is not trivial (I think it is NOT trivial) then we can add some "known limitations" section in the documentation and leave it as is right now.

[marcmonfort]

All detected corner cases should have been resolved.

Note that when using stubs, the retrieval of class information is directed to a random backend. In deployments where certain backends lack the class module while others have it, this may lead to failures.

[alexbarcelo]

LGTM