[WIP] Extract sync event timestamps from SpikeGLX streams #1657
Conversation
|
Hi @guptadivyansh |
|
Hi Manish,
For now I'm testing with some data that I recorded myself. It's about
2-3GB, but still too big to add to the testing suite. Once I have something
working, we can think about how to add a worthwhile test case that includes
at least a few seconds of data.
…On Mon, Mar 17, 2025, 7:54 PM Manish Mohapatra ***@***.***> wrote:
Hi @guptadivyansh <https://github.com/guptadivyansh>
Thank you so much for working on this issue!
This is a real-world (although really big) dataset
<https://datasets.datalad.org/?dir=/labs/mvdm/OdorSequence/sourcedata/raw/M541/M541-2024-08-31_g0>
that has simultaneously recorded sync signals from 2 imec device and 1 nidq
device. If you think it is too cumbersome to handle, I can try clipping it
and trying to submit a pull-request for much more sensibly sized dataset on
here
<https://gin.g-node.org/NeuralEnsemble/ephy_testing_data/src/master/spikeglx>
—
Reply to this email directly, view it on GitHub
<#1657 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACFXJPVC774KKR2EWX6JZHL2U4K7TAVCNFSM6AAAAABZEF63MOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMZQGUZTMMZWGI>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
[image: manimoh]*manimoh* left a comment (NeuralEnsemble/python-neo#1657)
<#1657 (comment)>
Hi @guptadivyansh <https://github.com/guptadivyansh>
Thank you so much for working on this issue!
This is a real-world (although really big) dataset
<https://datasets.datalad.org/?dir=/labs/mvdm/OdorSequence/sourcedata/raw/M541/M541-2024-08-31_g0>
that has simultaneously recorded sync signals from 2 imec device and 1 nidq
device. If you think it is too cumbersome to handle, I can try clipping it
and trying to submit a pull-request for much more sensibly sized dataset on
here
<https://gin.g-node.org/NeuralEnsemble/ephy_testing_data/src/master/spikeglx>
—
Reply to this email directly, view it on GitHub
<#1657 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACFXJPVC774KKR2EWX6JZHL2U4K7TAVCNFSM6AAAAABZEF63MOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMZQGUZTMMZWGI>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Thanks so much for working on this. Please ping Alessio and Sam to review when you're ready! |
guptadivyansh
changed the title
|
I think this should expose all that we need from neo to solve SpikeInterface/spikeinterface#2234 that is, the timestamps of sync edges of the correct channels in any given streams. Actually aligning the edges across streams can/should(?) be done in spikeinterface, likely here. I also noticed that the nidq digital channel processing introduced in #1383 is hardcoded for the first segment. I can fix it in a separate PR. Once that is done, we should be able to get sync events from all segments too. |
| @@ -292,33 +293,113 @@ def _event_count(self, event_channel_idx, block_index=None, seg_index=None): | |||
| return timestamps.size | |||
|
|
|||
| def _get_event_timestamps(self, block_index, seg_index, event_channel_index, t_start=None, t_stop=None): | |||
| #TODO: fix seg_index usage, currently hardcoded for first segment | |||
| timestamps, durations, labels = [], None, [] | |||
| info = self.signals_info_dict[0, "nidq"] # There are no events that are not in the nidq stream | |||
There was a problem hiding this comment.
currently the event detector for nidq digital channels ignores the seg_index. This needs fixing for detecting sync events across segments.
Should still work for multi-segment imec streams though
| sync_line = self.get_analogsignal_chunk(channel_names = [f'XA{niChan}'], | ||
| stream_index = stream_index, | ||
| seg_index = seg_index) | ||
| #Does this need to be scaled by channel gain before threshold? |
| @@ -204,6 +204,7 @@ def _parse_header(self): | |||
| # This is true only in case of 'nidq' stream | |||
| for stream_name in stream_names: | |||
| if "nidq" in stream_name: | |||
| #TODO: loop over all segments to add nidq events to _events_memmap | |||
There was a problem hiding this comment.
I think the np.unpackbits below would load the nidq digital data into memory?
|
@alejoe91 , @samuelgarcia , would be grateful if you could have a look! |
|
There's a messy testing notebook here that uses some of my data but I still need to look into which (if any) of the current testing datasets have simultaneously recorded sync lines for more than a couple of seconds. Ideally, we want a good mix of multiple probes, and sync signal in nidq digital and analog channels. Most of the neural channels in the test imec streams can be dropped to keep a managable filesize. (otherwise we'll need a ~23 MB test file per second per probe) |
| timestamps, durations, labels = [], None, [] | ||
| info = self.signals_info_dict[0, "nidq"] # There are no events that are not in the nidq stream | ||
| dig_ch = info["digital_channels"] | ||
| if len(dig_ch) > 0: | ||
| event_data = self._events_memmap | ||
| channel = dig_ch[event_channel_index] | ||
| channel = dig_ch[event_channel_index] # 'XD0', 'XD1', etc. | ||
| ch_idx = 7 - int(channel[2:]) # They are in the reverse order |
There was a problem hiding this comment.
This calculation of channel index is also something that I had hard-coded and will work only when there are exactly 8 digital channels saved (which was the cases for our setup, but isn't generally true); but not when you save fewer channels or have I/O devices that have more than 8 digital channels.
There was a problem hiding this comment.
Thanks for pointing this out, this is something to fix.
|
I can take a look at this but first we should determine if there is available data on the testing set that contains the cases that you want to cover in your scope: https://gin.g-node.org/NeuralEnsemble/ephy_testing_data/src/master/spikeglx |
Slowly working towards SpikeInterface/spikeinterface#2234
Trying to reuse the nidq digital channel event processing for handling sync edges. Will then align edges and change sampling rate/time vector as necessary
Also, I'm not sure if any of the current test files have simultaneously recorded sync signals in both imec and nidq streams.