Using Bonsai ZeroMQ to parse multipart messages containing imaging data

[vruetten]

Hello Bonsai Crew!

I’d like to use the Bonsai ZeroMQ library to let Bonsai receive imaging data. I’ve installed the ZeroMQ package, instantiated a ZeroMQ subscriber node, and set the address/port number, and I can see that bytes are successfully streaming; however, I’m stuck on the next part.

Each message received is a multipart message consisting of three parts:

• an envelop/topic (in my case, this would be ‘IMAGE’ for now)
• a header composed of 40 bytes, which contains the frame metadata (pixels per line, number of lines, number of channels, timestamp, current frame number)
• the frame data

Could you please advise on the best way to parse each message into its constituent parts, reshaping the data to ultimately enable downstream image processing? I’ve looked at the ZeroMQ docs/examples, but couldn’t find a close enough example.

Here’s what I have so far:
test_zeromq.bonsai.zip

Thanks in advance for any help!

best,

Ginny

[RoboDoig]

Hi @vruetten - I'm assuming you don't have any control over how the header is composed? Is this enforced by the camera data output? Normally I would suggest putting each variable of the metadata into an individual NetMQFrame and then stacking all those into a multipart message. That way the receiving Subscriber can just index the message to get each piece of metadata.

If you need to send the byte array I've attached an example below that may do what you want. On the Publisher side of this workflow I've just recreated what I believe your message looks like from your description, which is a message with 3 frames (topic, data header, image data). For speed here the data header is just the image height and width.

On the Subscriber side I'm getting the topic from the First frame and the image data from the Last frame. I use the Index node to get the middle frame with the header. NetMQMessage exposes the First and Last frame by default but you need to use Index to get anything in between. To parse the data header I've written an extensions that uses the C# BitConverter to recover the Width and Height integers from the byte array sent over the network. It's convenient to use an extension here but it is possible to do array parsing directly in Bonsai for certain message formats.

Hope this helps, do let me know if anything is unclear!

ZeroMQExample.zip

[vruetten]

Thanks a lot for the quick response. I now have it working! 🥳 🥳 🥳