[23503] String field of message truncated to 4 characters on receiving side???

[wcneill]

I have created a message that is successfully being written by a DataWriter. That message is represented in the below IDL:

module vehicle_state
{
    struct MessageInfo 
    {
        int64 timestamp_us;         // Epoch timestamp in microseconds
        string<64> msg_uid;         // Unique identifier for the message
        string<64> source_id;       // Identifier for the source node of the data item
    };

    struct SystemConfigMsg {
        MessageInfo info;
        string<2048> json_config;  // (key, value):  subsystem_id, system_config
    };
};

On inspection of the outgoing message just before the DataWriter->write() call, the message contains the expected json_config payload. That payload is the very short and sweet text below:

{
    "System1" : {
        "STATE_A" : "open",
        "STATE_B" : "close",
        "STATE_C" : "kill"
    },

    "System2" : {
        "STATE_A" : "close",
        "STATE_B" : "kill",
        "STATE_C" : "open"
    }
}

My DataReaderListener detects the incoming message, but the json_config field of the incoming message is only this: {"Sy

The code that sends the message out is this:

using namespace eprosima::fastdds::dds;
        SystemConfigMsg msg;

        auto now = std::chrono::system_clock::now();
        auto timestamp = std::chrono::duration_cast<std::chrono::microseconds>(now.time_since_epoch());
        std::string msg_uid = "vsm-" + std::to_string(timestamp.count());

        msg.info().timestamp_us(timestamp.count());
        msg.info().source_id("vsm");
        msg.info().msg_uid(msg_uid);
        msg.json_config(JsonUtils::serialize(m_configuration));

        DDSDataWrapper envelope(msg);
        std::cout << "THE OUTGOING MESSAGE IS " << envelope.data().json_config() << std::endl;
        std::cout << "THE OUTGOING MESSAGE IS " << static_cast<SystemConfigMsg*>(envelope.ptr())->json_config() << std::endl;

        if(!mp_dds_node->write("ConfigurationTopic", envelope))
        {
            LOG_ERROR("State Manager - Failed to send system configuration.");
            throw std::runtime_error("State Manager - Failed to send out configuration");
        }
    }

The DataReaderListener that is receiving and processing the incoming message is below:

ReaderCallbackListener *conf_listener = new ReaderCallbackListener();
    conf_listener->set_callback(
        [this](DataReader *dr)
        {
            // get Json string from state manager's message, parse it and call
            vehicle_state::SystemConfigMsg msg;
            SampleInfo info;

            while (dr->take_next_sample(&msg, &info) == RETCODE_OK)
            {
                if (info.valid_data)
                {
                    // std::string config_string = msg.json_config().c_str();
                    std::cout << "THE CONFIG IS: " << msg.json_config().c_str() << std::endl;
                    // configure_states(msg.json_config().to_string());
                }
            }
            LOG_INFO("Configuration received from state manager");
        });

I have tried removing all whitespace and newline characters from the JSON file, but that did nothing.

Why is my data successfully being sent, but truncated on the receiving side?

[Javgilavi]

Hello @wcneill,

Thank you very much for providing the details.
We will reproduce the issue in our environment and let you know as soon as we have any updates.

[Mario-DL]

Hi @wcneill,

I have tried reproducing the issue with the provided IDL and json (nlohman) in Fast DDS master but I could not reproduce the issue.

    // Publisher code
    std::ifstream config("json_configuration.json");
    std::stringstream buffer;
    buffer << config.rdbuf();
    auto json_cfg = nlohmann::json::parse(buffer.str());
    auto now = std::chrono::system_clock::now();
    auto timestamp = std::chrono::duration_cast<std::chrono::microseconds>(now.time_since_epoch());
    std::string msg_uid = "vsm-" + std::to_string(timestamp.count());

    std::cout << "Publishing message with configuration: " << json_cfg.dump() << std::endl;

    msg_.info().timestamp_us(timestamp.count());
    msg_.info().source_id("vsm");
    msg_.info().msg_uid(msg_uid);
    msg_.json_config(json_cfg.dump());

    ret = (RETCODE_OK == writer_->write(&msg_));

    // Subscriber code
    while ((!is_stopped()) && (RETCODE_OK == reader->take_next_sample(&msg_, &info)))
    {
        if ((info.instance_state == ALIVE_INSTANCE_STATE) && info.valid_data)
        {
            received_samples_++;
            std::cout << "Message: '" << msg_.json_config() << "' RECEIVED" << std::endl;
            if (samples_ > 0 && (received_samples_ >= samples_))
            {
                stop();
            }
        }
    }

I am suspecting that JsonUtils::serialize(m_configuration) may be introducing some character that make the string being escaped. Could you share more details on the serialization ?
Also, which version of Fast CDR are you using ?

[wcneill]

Good morning!

The serialization happens via the nlohmann JSON for C++ library... specifically, that method is just a wrapper around Json.dump() (edit: I see now you are using the same).

Below I have included most of the non-FastDDS code in play, which are just convenience wrappers for the most part. But, maybe I have done something for the sake of convenience that is the problem. Sorry for the wall of code, I just want to be helpful.

If you would like to see any other code definitions, just say the word. This is just a generic prototype and I have gotten permission to share code.

    std::string serialize(const Json &json_obj)
    {
        return json_obj.dump();
    }

This method reads the file:

    std::optional<JsonUtils::Json> read_json_file(std::string path, std::size_t max_bytes)
    {

        std::ifstream file(path);
        if (!file)
        {
            throw std::runtime_error("Failed to read configuration file");
        }

        std::size_t maxbytes = 1024;
        char buffer[max_bytes + 1] = {};
        file.read(buffer, max_bytes);

        std::size_t count = file.gcount();

        std::string json_str(buffer, max_bytes);

        file.close();
        std::optional<JsonUtils::Json> json = JsonUtils::parse(json_str);

        return json;
    }

Which is used here, simply storing the resulting Json object as a member variable:

    void StateManager::set_configuration(const std::string &config_path)
    {
        try
        {
            m_configuration = JsonUtils::read_json_file(config_path, 1024).value();
        }
        catch (std::bad_optional_access e)
        {
            LOG_ERROR("State Manager failed to read or parse configuration json from file");
            throw e;
        }
    }

And here is the full code for the function that sends out the message using that member variable:

void StateManager::send_configuration()
    {
        using namespace eprosima::fastdds::dds;
        SystemConfigMsg msg;

        auto now = std::chrono::system_clock::now();
        auto timestamp = std::chrono::duration_cast<std::chrono::microseconds>(now.time_since_epoch());
        std::string msg_uid = "vsm-" + std::to_string(timestamp.count());

        msg.info().timestamp_us(timestamp.count());
        msg.info().source_id("vsm");
        msg.info().msg_uid(msg_uid);
        msg.json_config(JsonUtils::serialize(m_configuration));

        DDSDataWrapper envelope(msg);
        std::cout << "THE OUTGOING MESSAGE IS " << envelope.data().json_config() << std::endl;
        std::cout << "THE OUTGOING MESSAGE IS " << static_cast<SystemConfigMsg*>(envelope.ptr())->json_config() << std::endl;

        if(!mp_dds_node->write("ConfigurationTopic", envelope))
        {
            LOG_ERROR("State Manager - Failed to send system configuration.");
            throw std::runtime_error("State Manager - Failed to send out configuration");
        }
    }

If you are curious about the wrapper around write, it is this:

/**
     * @brief Publishes a sample to the DDS topic.
     * @param data The topic instance to publish.
     * @return True on success, false otherwise.
     */
    bool write(const std::string& topic_name, DDSDataWrapperBase &msg) override
    {

        using namespace eprosima::fastdds::dds;

        auto it = writer_map_.find(topic_name);

        if (it == writer_map_.end()) {
            LOG_ERROR("No data writer found for " + topic_name);
            return false; // Writer not found for the topic
        }

        DataWriter* writer = it->second;

        if(!writer) 
        {
            LOG_ERROR("Null pointer found in place of data writer for topic " + topic_name);
            return false;
        }

        ReturnCode_t code = writer->write(msg.ptr());
        if (code != RETCODE_OK)
        {
            LOG_ERROR("DataWriter failed with code: " + code);
            return false;
        }
        
        LOG_INFO("Message successfully written");
        return true;
    }

And finally, the type erasure class which is simply a wrapper around any message type:

// Type erasure games, so a DDS node can handle any data type
class DDSDataWrapperBase 
{
    public:
        virtual ~DDSDataWrapperBase() = default;
        virtual void* ptr() = 0;
        virtual const std::type_index type() const = 0;
};

template <typename T>
class DDSDataWrapper : public DDSDataWrapperBase 
{

public:
    DDSDataWrapper(T& data) : data_(data) {}

    /**
     * @brief Returns a pointer to the internal data. 
     */
    void* ptr() override {
        return static_cast<void*>(&data_);
    }

    const std::type_index type() const override {
        return typeid(T);
    }

    const T& data() const {
        return data_;
    }

private:
    T data_;
};

template <typename T>
T& unwrap(DDSDataWrapperBase& base) 
{
    if (base.type() != typeid(T)) {
        throw std::bad_cast();
    }
    return *static_cast<T*>(base.ptr());
}

I thought there might be extra characters added as well, so I removed all white space from the json file before I parsed it, and verified that the outgoing string contained nothing but the characters of the JSON content followed by \000.

As far as the version of fastcdr, I'm not sure. Both FastDDS and Fast CDR were installed via a dev environment setup script which just pulls and builds the latest version. Here are the .so files:

I am working on Rocky Linux 9, if that helps at all.

[Mario-DL]

Thanks for the completeness of the reply and sharing the files. I've been doing several tests but unfortunately I could not reproduce the issue yet. I tested with

• Fast DDS v3.2.2
• Fast CDR v2.3.0
• Generating the types with Fast DDS Gen: v4.0.0, v4.0.1 and v4.1.0
• Under Rocky Linux 9 GCC 11.5
• Also, checked with nlohman v3.11.3, v3.12.0 and v3.10.5

Which version of Fast DDS Gen are you using ? fastddsgen -version
I'm afraid I would need the exact reproducer, would it be possible for you to send it to [email protected] ?

[wcneill]

Hi Mario,

I wrote a minimal reproducible example that removes all wrapper and convenience code I wrote and just uses vanilla FastDDS. I am emailing it to you now.

I think that the disconnect between your attempts and mine (my fault) is that I'm creating two "DDS nodes" within the same application rather than writing two applications. This was for the sake of a quick prototype. If that doesn't make sense now, I think it will when you see the code

**Edit: ** I separated out the publishing and subscribing nodes into two separate compiled executables, and the issue remains. I didn't think it should matter, and it appears I was right.

[wcneill]

Mario was able to solve the problem. I was using the wrong type support, which was masked by the fact that part of the payload was coming through despite the incorrect type support.

Thank you Mario!