Skip to content

Commit ef169fb

Browse files
authored
Simplify npx data-loading scripts & fix parameters for loading data (#216)
* Simplify npx data-loading scripts & fix incorrect parameters - sample rate 3e5 --> 30e3 - Use default gains for np1 (1000 for AP & 50 for LFP) - Load data with spikeinterface, but use matplotlib to plot - Reduce dependencies - Reduce plot customizations - Other refactorings: - Unify related numpy arrays into a single dict - Add code cell comments - Add start_t & dur parameters to all scripts - Change uV to mV for rhs2116 dc data
1 parent 3ce06ff commit ef169fb

File tree

9 files changed

+389
-337
lines changed

9 files changed

+389
-337
lines changed

articles/hardware/hs64/load-data.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,8 @@ uid: hs64_load-data
33
title: Load Data
44
---
55

6-
The following python script can be used to load and plot the data produced by the Headstage 64 [example workflow](xref:hs64_workflow).
6+
The following python script can be used to load and plot the data produced by the Headstage 64
7+
[example workflow](xref:hs64_workflow).
78

89
[!code-python[](../../../workflows/hardware/hs64/load-hs64.py)]
910

articles/hardware/np1e/load-data.md

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -3,15 +3,15 @@ uid: np1e_load-data
33
title: Load Data
44
---
55

6-
The following python script can be used to load and plot the data produced by the NeuropixelsV1e Headstage [example workflow](xref:np1e).
6+
The following python script can be used to load and plot the data produced by the NeuropixelsV1e
7+
Headstage [example workflow](xref:np1e).
78

89
[!code-python[](../../../workflows/hardware/np1e/load-np1e.py)]
910

1011
> [!NOTE]
11-
> To plot probeinterface data, [save the probe configuration file](xref:np1e_gui#save-probeinterface-file) into the same directory of your data.
12-
13-
> [!NOTE]
14-
> This script will attempt to load entire files into arrays. For long recordings, data will need to
15-
> be split into more manageable chunks by:
16-
> - Modifying this script to partially load files
17-
> - Modifying the workflow to cyclically create new files after a certain duration
12+
> - To plot probeinterface data, [save the probe configuration file](xref:np1e_gui#save-probeinterface-file)
13+
> into the same directory of your data.
14+
> - This script will attempt to load entire files into arrays. For long recordings, data will need to
15+
> be split into more manageable chunks by:
16+
> - Modifying this script to partially load files
17+
> - Modifying the workflow to cyclically create new files after a certain duration

articles/hardware/np2e/load-data.md

Lines changed: 11 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -3,15 +3,18 @@ uid: np2e_load-data
33
title: Load Data
44
---
55

6-
The following python script can be used to load and plot the data produced by the NeuropixelsV1e Headstage [example workflow](xref:np2e).
6+
The following python script can be used to load and plot the data produced by the NeuropixelsV2e
7+
Headstage [example workflow](xref:np2e).
78

89
[!code-python[](../../../workflows/hardware/np2e/load-np2e.py)]
910

1011
> [!NOTE]
11-
> To plot probeinterface data, [save the probe configuration file](xref:np2e_gui#save-probeinterface-file) into the same directory of your data.
12-
13-
> [!NOTE]
14-
> This script will attempt to load entire files into arrays. For long recordings, data will need to
15-
> be split into more manageable chunks by:
16-
> - Modifying this script to partially load files
17-
> - Modifying the workflow to cyclically create new files after a certain duration
12+
> - To plot probeinterface data, [save the probe configuration file](xref:np2e_gui#save-probeinterface-file)
13+
> into the same directory of your data.
14+
> - By default, this script only populates plots for the Neuropixels 2.0 probe A and leaves plots
15+
> for Neuropixels 2.0 probe B empty. If you are recording from both probes and want to plot probe
16+
> B ephys and probe data, uncomment the relevant lines in the script.
17+
> - This script will attempt to load entire files into arrays. For long recordings, data will need to
18+
> be split into more manageable chunks by:
19+
> - Modifying this script to partially load files
20+
> - Modifying the workflow to cyclically create new files after a certain duration

articles/hardware/rhs2116/load-data.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,8 @@ uid: rhs2116_load-data
33
title: Load Data
44
---
55

6-
The following python script can be used to load and plot the data produced by the Headstage Rhs2116 [example workflow](xref:rhs2116).
6+
The following python script can be used to load and plot the data produced by the Headstage Rhs2116
7+
[example workflow](xref:rhs2116).
78

89
[!code-python[](../../../workflows/hardware/rhs2116/load-rhs2116.py)]
910

workflows/hardware/hs64/load-hs64.py

Lines changed: 92 additions & 95 deletions
Original file line numberDiff line numberDiff line change
@@ -1,127 +1,124 @@
1+
# Import necessary packages
12
import os
23
import numpy as np
34
import matplotlib.pyplot as plt
45

5-
# Load data from headstage64 tutorial workflow
6-
suffix = '0'; # Change to match file names' suffix
7-
# Change this to the directory of your data. In this example, data's in the same directory as this data loading Python script
8-
data_directory = os.path.dirname(os.path.realpath(__file__))
9-
start_t = 1.0 # when to start plotting data (seconds)
10-
dur = 1.0 # duration of data to plot
6+
#%% Set parameters for loading data
117

12-
plt.close('all')
8+
suffix = 0 # Change to match filenames' suffix
9+
data_directory = 'C:/Users/open-ephys/Documents/data/hs64' # Change to match files' directory
10+
plot_num_channels = 10 # Number of channels to plot
11+
start_t = 3.0 # Plot start time (seconds)
12+
dur = 2.0 # Plot time duration (seconds)
13+
14+
# RHD2164 constants
15+
ephys_uV_multiplier = 0.195
16+
aux_uV_multiplier = 37.4
17+
offset = 32768
18+
num_channels = 64
19+
20+
#%% Load acquisition session data
1321

14-
#%% Metadata
1522
dt = {'names': ('time', 'acq_clk_hz', 'block_read_sz', 'block_write_sz'),
1623
'formats': ('datetime64[us]', 'u4', 'u4', 'u4')}
1724
meta = np.genfromtxt(os.path.join(data_directory, f'start-time_{suffix}.csv'), delimiter=',', dtype=dt, skip_header=1)
18-
print(f"Recording was started at {meta['time']} GMT")
19-
20-
#%% RHD2164 ephys data
21-
start_t = 1.0 # when to start plotting data (seconds)
22-
dur = 1.0 # duration of data to plot
23-
plot_channel_offset_uV = 1000 # Vertical offset between each channel in the time series
24-
25-
hs64 = {}
26-
hs64['time'] = np.fromfile(os.path.join(data_directory, f'rhd2164-clock_{suffix}.raw'), dtype=np.uint64) / meta['acq_clk_hz']
27-
hs64['ephys'] = np.reshape(np.fromfile(os.path.join(data_directory, f'rhd2164-ephys_{suffix}.raw'), dtype=np.uint16), (-1, 64))
28-
29-
# Make arrays for plotting
30-
b = np.bitwise_and(hs64['time'] >= start_t, hs64['time'] < start_t + dur)
31-
time = hs64['time'][b]
32-
rhd2164_ephys = hs64['ephys'][b, :].astype(np.double)
33-
34-
# Convert to uV and offset each channel by some plot_channel_offset_uV
35-
bit_depth = 16
36-
scalar = 0.195
37-
offset = (2 ** (bit_depth - 1)) * scalar
38-
rhd2164_ephys = rhd2164_ephys * scalar - offset + np.arange(rhd2164_ephys.shape[1])[None, :] * offset / 4
39-
40-
fig = plt.figure()
41-
plt.plot(time, rhd2164_ephys, 'k', linewidth=0.25)
42-
plt.tick_params(axis='y', which='both', left=False, right=False, labelleft=False)
43-
plt.xlabel("time (sec)")
44-
plt.ylabel("channel")
45-
plt.title('RHD2164 Ephys Data')
46-
fig.set_size_inches(5,8)
47-
plt.tight_layout()
25+
print(f'Recording was started at {meta["time"]} GMT')
26+
print(f'Acquisition clock rate was {meta["acq_clk_hz"] / 1e6 } MHz')
27+
28+
#%% Load RHD2164 data
4829

49-
#%% RHD2164 aux data
50-
hs64['aux'] = np.reshape(np.fromfile(os.path.join(data_directory, f'rhd2164-aux_{suffix}.raw'), dtype=np.uint16), (-1, 3))
30+
rhd2164 = {}
5131

52-
# Make arrays for plotting
53-
b = np.bitwise_and(hs64['time'] >= start_t, hs64['time'] < start_t + dur)
54-
time = hs64['time'][b]
55-
rhd2164_aux = hs64['aux'][b, :].astype(np.double)
32+
# Load RHD2164 clock data and convert clock cycles to seconds
33+
rhd2164['time'] = np.fromfile(os.path.join(data_directory, f'rhd2164-clock_{suffix}.raw'), dtype=np.uint64) / meta['acq_clk_hz']
5634

57-
# Convert to uV and offset each channel by some plot_channel_offset_uV
58-
scalar = 37.4
59-
rhd2164_aux_wave = (rhd2164_aux - 2 **(bit_depth - 1)) * scalar
35+
# Load and scale RHD2164 ephys data
36+
ephys = np.reshape(np.fromfile(os.path.join(data_directory, f'rhd2164-ephys_{suffix}.raw'), dtype=np.uint16), (-1, num_channels))
37+
rhd2164['ephys_uV'] = (ephys.astype(np.float32) - offset) * ephys_uV_multiplier
6038

61-
plt.figure()
62-
plt.plot(time, rhd2164_aux_wave)
63-
plt.xlabel("time (sec)")
64-
plt.ylabel("channel")
65-
plt.title('RHD2164 Auxiliary Data')
39+
# Load and scale RHD2164 aux data
40+
aux = np.reshape(np.fromfile(os.path.join(data_directory, f'rhd2164-aux_{suffix}.raw'), dtype=np.uint16), (-1, 3))
41+
rhd2164['aux_uV'] = (aux.astype(np.float32) - offset) * aux_uV_multiplier
42+
43+
rhd2164_time_mask = np.bitwise_and(rhd2164['time'] >= start_t, rhd2164['time'] < start_t + dur)
44+
45+
#%% Load BNO055 data
6646

67-
#%% Bno055
6847
dt = {'names': ('clock', 'euler', 'quat', 'is_quat_id', 'accel', 'grav', 'temp'),
6948
'formats': ('u8', '(1,3)f8', '(1,4)f8', '?', '(1,3)f8', '(1,3)f8', 'f8')}
7049
bno055 = np.genfromtxt(os.path.join(data_directory, f'bno055_{suffix}.csv'), delimiter=',', dtype=dt)
7150

51+
# Convert clock cycles to seconds
7252
bno055_time = bno055['clock'] / meta['acq_clk_hz']
7353

74-
plt.figure()
54+
bno055_time_mask = np.bitwise_and(bno055_time >= start_t, bno055_time < start_t + dur)
7555

76-
plt.subplot(231)
77-
plt.plot(bno055_time, bno055['euler'].squeeze())
78-
plt.xlabel("time (sec)")
79-
plt.ylabel("angle (deg.)")
80-
plt.ylim(-185, 365)
81-
plt.legend(['yaw', 'pitch', 'roll'])
82-
plt.title('Euler')
56+
#%% Load TS4231 data
8357

84-
plt.subplot(232)
85-
plt.plot(bno055_time, bno055['quat'].squeeze())
86-
plt.xlabel("time (sec)")
87-
plt.ylim(-1.1, 1.1)
88-
plt.legend(['X', 'Y', 'Z', 'W'])
89-
plt.title('Quaternion')
58+
# Load TS4231 data
59+
dt = {'names': ('clock', 'position'),
60+
'formats': ('u8', '(1,3)f8')}
61+
ts4231 = np.genfromtxt(os.path.join(data_directory, f'ts4231_{suffix}.csv'), delimiter=',', dtype=dt)
9062

91-
plt.subplot(233)
92-
plt.plot(bno055_time, bno055['accel'].squeeze())
93-
plt.xlabel("time (sec)")
94-
plt.ylabel("accel. (m/s^2)")
95-
plt.legend(['X', 'Y', 'Z'])
96-
plt.title('Lin. Accel.')
63+
# Convert clock cycles to seconds
64+
ts4231_time = ts4231['clock'] / meta['acq_clk_hz']
9765

98-
plt.subplot(234)
99-
plt.plot(bno055_time, bno055['grav'].squeeze())
100-
plt.xlabel("time (sec)")
101-
plt.ylabel("accel. (m/s^2)")
102-
plt.legend(['X', 'Y', 'Z'])
103-
plt.title('Gravity Vec.')
66+
ts4231_time_mask = np.bitwise_and(ts4231_time >= start_t, ts4231_time < start_t + dur)
10467

105-
plt.subplot(235)
106-
plt.plot(bno055_time, bno055['temp'].squeeze())
107-
plt.xlabel("time (sec)")
108-
plt.ylabel("temp. (°C)")
109-
plt.title('Headstage Temp.')
68+
#%% Plot time series
11069

111-
plt.tight_layout()
70+
fig = plt.figure(figsize=(12, 10))
11271

113-
#%% TS4231
114-
dt = {'names': ('clock', 'position'),
115-
'formats': ('u8', '(1,3)f8')}
116-
ts4231 = np.genfromtxt(os.path.join(data_directory, f'ts4231_{suffix}.csv'), delimiter=',', dtype=dt)
72+
# Plot RHD2164 ephys data
73+
plt.subplot(711)
74+
plt.plot(rhd2164['time'][rhd2164_time_mask], rhd2164['ephys_uV'][:,0:plot_num_channels][rhd2164_time_mask])
75+
plt.xlabel('Time (seconds)')
76+
plt.ylabel('Voltage (µV)')
77+
plt.title('RHD2164 Ephys Data')
11778

118-
ts4231_time = ts4231['clock'] / meta['acq_clk_hz']
119-
plt.figure()
79+
# Plot RHD2164 aux data
80+
plt.subplot(712)
81+
plt.plot(rhd2164['time'][rhd2164_time_mask], rhd2164['aux_uV'][rhd2164_time_mask])
82+
plt.xlabel('Time (seconds)')
83+
plt.ylabel('Voltage (µV)')
84+
plt.title('RHD2164 Aux Data')
85+
86+
# Plot BNO055 data
87+
plt.subplot(713)
88+
plt.plot(bno055_time[bno055_time_mask], bno055['euler'].squeeze()[bno055_time_mask])
89+
plt.xlabel('Time (seconds)')
90+
plt.ylabel('degrees')
91+
plt.title('Euler Angles')
92+
plt.legend(['Yaw', 'Pitch', 'Roll'])
93+
94+
plt.subplot(714)
95+
plt.plot(bno055_time[bno055_time_mask], bno055['quat'].squeeze()[bno055_time_mask])
96+
plt.xlabel('Time (seconds)')
97+
plt.title('Quaternion')
98+
plt.legend(['X', 'Y', 'Z', 'W'])
12099

121-
plt.plot(ts4231_time, ts4231['position'].squeeze())
122-
plt.xlabel("time (sec)")
123-
plt.ylabel("position (units)")
124-
plt.legend(['x', 'y', 'z'])
100+
plt.subplot(715)
101+
plt.plot(bno055_time[bno055_time_mask], bno055['accel'].squeeze()[bno055_time_mask])
102+
plt.xlabel('Time (seconds)')
103+
plt.ylabel('m/s\u00b2')
104+
plt.title('Linear Acceleration')
105+
plt.legend(['X', 'Y', 'Z'])
106+
107+
plt.subplot(716)
108+
plt.plot(bno055_time[bno055_time_mask], bno055['grav'].squeeze()[bno055_time_mask])
109+
plt.xlabel('Time (seconds)')
110+
plt.ylabel('m/s\u00b2')
111+
plt.title('Gravity Vector')
112+
plt.legend(['X', 'Y', 'Z'])
113+
114+
# Plot TS4231 data
115+
plt.subplot(717)
116+
plt.plot(ts4231_time[ts4231_time_mask], ts4231['position'].squeeze()[ts4231_time_mask])
117+
plt.xlabel('Time (seconds)')
118+
plt.ylabel('Position (units)')
125119
plt.title('Position Data')
120+
plt.legend(['X', 'Y', 'Z'])
121+
122+
fig.tight_layout()
126123

127124
plt.show()

0 commit comments

Comments
 (0)