plotting online trajectory

[ReemyAliNasser]

Hi,
I am trying to plot the real x,y coordinates of the animal during its behavior
I only succeed plotting one coordinate a time, but not the whole trajectory, within each frame i can't keep the previous circles
(also tried using histogram2D but it plots on a black background and cannot control the parameters of color, thickness..)

could you please help? I am attaching my flow where a circle is plotted in each frame representing the center mass of the animal

thanks
pulse upon entering ROI and home_8.zip

[chiyu1203]

This request is harder than it sounds like in reactive programming. How about dragging the centroid back to the video nodes so that you can visualise its location in real time. Is that something you were hoping for?

Hi,
I am trying to plot the real x,y coordinates of the animal during its behavior
I only succeed plotting one coordinate a time, but not the whole trajectory, within each frame i can't keep the previous circles
(also tried using histogram2D but it plots on a black background and cannot control the parameters of color, thickness..)

could you please help? I am attaching my flow where a circle is plotted in each frame representing the center mass of the animal

thanks
pulse upon entering ROI and home_8.zip

[ReemyAliNasser1]

I am able to visualize the centroid in real time, I am looking forward to keeping it on the video as the animal moves so I keep following the trajectory.

But also would be happy to get to know how to plot the trajectory at the end at the video image, this will help me.

(My problem is that I keep receiving errors regarding the type of the parameters I am trying to introduce to the nodes, so for example thought that I can use polyLine but I cannot connect to it the accumulation of the centroid node as they are not the same type (array of point2f and Mat.Net) )

[chiyu1203]

I meant to drag the LargestBinaryRegion to the window of video, like this video. However, it does not give you the trajectory, but only the LargestBinaryRegion to visualise in real time, so I agree ZedGraph or BonZeb will be a better method.

[ablot]

Have you tried the Bonsai.Gui.ZedGraph library? (https://bonsai-rx.org/gui/) I managed to do something similar using the LineGraph node.

[ReemyAliNasser]

it is working! did you tried to plot the trajectory on top of the video image?

[banchan86]

Hi @ReemyAliNasser thought this would be a fun exercise for me to try, and it turns out you can create a trajectory with the PolyLine.

You almost got there with your own try, all that was missing was a WindowCount to have a history of a N number of elements (I use a Skip factor of 1 frame and N count of elements to create a sliding window), and ToArray operators nestled within the SelectMany to shape the output into a format that PolyLine accepts (a Point[][] or an array of arrays). Technically we only need one line array, but PolyLine can draw multiple lines, hence the array of arrays.

If you want a longer trajectory, you can adjust the Count in WindowCount. You might need to adjust the threshold (I tested it with a phone flashlight against a dark background on my webcam).

Here is what it looks like

Attached zip file of the example:

#2434_Drawing_trajectory.zip

[ReemyAliNasser]

It is working! Is there a way to plot the whole trajectory from the start?

[banchan86]

@ReemyAliNasser Ah, I missed out that most important bit! I have something for you, but I am not sure if it's the most straightforward or efficient way to do it, as I just kinda mashed a few things together. If anybody knows of a simpler/more efficient/more bonsaic way, please let me know!

The basic principle I used to modify the workflow is that, in this case, we want each point that is generated to get appended to a list or an array, and then to send the updated list or array every time a point is created as a Point [][]input toPolyLines`.

Unfortunately, I could not figure out how to do something with that with the core bonsai operators, but I found an AccumulateToImmutableList operator in the AllenNeuralDynamics.Core.Design (thanks Bruno :P) package that seemed to do the trick. To convert that ImmutableList to a normal array, I used an ExpressionTransform from the Bonsai.Scripting.Expressions package, and then packaged it once more with a ToArray in the SelectMany.

Important note: the array conversion method for this type seems to only be supported in the new pre-release version of the Bonsai.Scripting.Expressions. In the Bonsai Package Manager, you need to tick the box that says Include prerelease and look for the 2.10.0-preview2 version.

This essentially redraws the trajectory every frame; you might run into performance issues, as I only tested it for a bit online. Let me know how it goes (I imagine you can also save the points and draw it offline on a loaded video and not in realtime, or draw the trajectory on a single end frame).

Workflow attached:
#2434_Drawing_all_trajectory.zip

[ReemyAliNasser]

Yes! This is really cool! Thanks!
now if I want to only plot the trajectory on the video image, how I am supposed to do so?

[banchan86]

Hi @ReemyAliNasser! If you want to trace the trajectory on the whole video offline, you can use this script, just replace the VideoCaptureDevice with a FileCapture from the Bonsai.Vision package to load a video.

If you want to trace the trajectory on a single image, it will require creating another script. I can outline what you will need, and then you can see if you can figure it out. If you can't figure it out, get back to me.

1. In this script you can save the points as a .csv using a CsvWriter from the Bonsai.IO package
2. In a separate script, you would first load an image using the LoadImage and add it as an Image to a Canvas. You can then use a CsvReader to read the points, once they have loaded, then package it with two ToArrays and then pass it to an add PolyLine on top of the image Canvas.

[ReemyAliNasser]

Hi!
I could not use the FileCapture because I am using SpinnakerCapture so went on the second suggestion.
The problem was that the type of the x,y coordinates saved in the file are not acceptable for the points property of the PolyLine
I am attaching a flow that I tried but it is not working because the x,y coords are not transferred to the further nodes
offline tajectory.zip

[banchan86]

@ReemyAliNasser there were a few missing pieces that needed to be added together, I have uploaded a modified workflow that is working on my computer. Let me know if anything is unclear

1. You need to add a Parse operator to convert each row from CsvReader that is read as a string into an tuple of Int that can be passed into the X, Y coords for CreatePoint.
2. You do not need to the AccumulateToList operator anymore, that is only needed for updating each frame, for a single frame you can just pass a final Point [][] which you can achieve with the double ToArray operators.

I had to modify the parameters to test on my computer, so adjust them back to your original parameters!

offline tajectory.zip