Deployed fd517b8 with MkDocs version: 1.5.3

Author: janluxemburk

dataloaders/index.html

@@ -566,26 +566,30 @@ <h1 id="using-dataloaders">Using dataloaders</h1>
 <p>Apart from loading data into dataframes, the <code>cesnet-datazoo</code> package provides dataloaders for processing data in smaller batches.</p>
 <p>An example of how dataloaders can be used is in <code>cesnet_datazoo.datasets.loaders</code> or in the following snippet:</p>
 <div class="highlight"><pre><span></span><code><span class="k">def</span> <span class="nf">load_from_dataloader</span><span class="p">(</span><span class="n">dataloader</span><span class="p">:</span> <span class="n">DataLoader</span><span class="p">):</span>
+    <span class="n">other_fields</span> <span class="o">=</span> <span class="p">[]</span>
     <span class="n">data_ppi</span> <span class="o">=</span> <span class="p">[]</span>
     <span class="n">data_flowstats</span> <span class="o">=</span> <span class="p">[]</span>
     <span class="n">labels</span> <span class="o">=</span> <span class="p">[]</span>
-    <span class="k">for</span> <span class="n">batch_ppi</span><span class="p">,</span> <span class="n">batch_flowstats</span><span class="p">,</span> <span class="n">batch_labels</span> <span class="ow">in</span> <span class="n">dataloader</span><span class="p">:</span>
+    <span class="k">for</span> <span class="n">batch_other_fields</span><span class="p">,</span> <span class="n">batch_ppi</span><span class="p">,</span> <span class="n">batch_flowstats</span><span class="p">,</span> <span class="n">batch_labels</span> <span class="ow">in</span> <span class="n">dataloader</span><span class="p">:</span>
+        <span class="n">other_fields</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">batch_other_fields</span><span class="p">)</span>
         <span class="n">data_ppi</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">batch_ppi</span><span class="p">)</span>
         <span class="n">data_flowstats</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">batch_flowstats</span><span class="p">)</span>
         <span class="n">labels</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">batch_labels</span><span class="p">)</span>
+    <span class="n">df_other_fields</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">concat</span><span class="p">(</span><span class="n">other_fields</span><span class="p">,</span> <span class="n">ignore_index</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
     <span class="n">data_ppi</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">concatenate</span><span class="p">(</span><span class="n">data_ppi</span><span class="p">)</span>
     <span class="n">data_flowstats</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">concatenate</span><span class="p">(</span><span class="n">data_flowstats</span><span class="p">)</span>
     <span class="n">labels</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">concatenate</span><span class="p">(</span><span class="n">labels</span><span class="p">)</span>
-    <span class="k">return</span> <span class="n">data_ppi</span><span class="p">,</span> <span class="n">data_flowstats</span><span class="p">,</span> <span class="n">labels</span>
+    <span class="k">return</span> <span class="n">df_other_fields</span><span class="p">,</span> <span class="n">data_ppi</span><span class="p">,</span> <span class="n">data_flowstats</span><span class="p">,</span> <span class="n">labels</span>
 </code></pre></div>
-<p>When a dataloader is iterated, the returned data are in the format <code>tuple(batch_ppi, batch_flowstats, batch_labels)</code>. The batch size <em>B</em> is configured with <code>batch_size</code> and <code>test_batch_size</code> config options.
+<p>When a dataloader is iterated, the returned data are in the format <code>tuple(batch_other_fields,  batch_ppi, batch_flowstats, batch_labels)</code>. Batch size <em>B</em> is configured with <code>batch_size</code> and <code>test_batch_size</code> config options.
 The shapes are:</p>
 <ul>
-<li>batch_ppi - <code>(B, [3, 4], 30)</code> - the middle dimension is either 4 when TCP push flags are used (<code>use_push_flags</code>) or 3 otherwise.</li>
-<li>batch_flowstats <code>(B, F)</code> - where F is the number of flowstats features computed with <a class="autorefs autorefs-internal" href="../reference_dataset_config/#config.DatasetConfig.get_flowstats_features_len">DatasetConfig.get_flowstats_features_len</a>. To get the order and names of flowstats features, call <a class="autorefs autorefs-internal" href="../reference_dataset_config/#config.DatasetConfig.get_flowstats_feature_names_expanded">DatasetConfig.get_flowstats_feature_names_expanded</a>. The batch_flowstats array includes flow statistics, TCP features (if available and configured), and bins of packet histograms (if available and configured). See the <a class="autorefs autorefs-internal" href="../features/#features">data features</a> page for more information about features.</li>
-<li>batch_labels <code>(B)</code> - integer labels encoded with <code>LabelEncoder</code> available at <code>dataset.encoder</code>.</li>
+<li>batch_other_fields <code>pd.DataFrame (B, C)</code> - a Pandas DataFrame with <a class="autorefs autorefs-internal" href="../features/#other-fields">auxiliary fields</a>, such as communicating hosts, flow times, and more fields extracted from the ClientHello message. If the <code>return_other_fields</code> config option is false, this will be an empty DataFrame. DataFrame columns C depend on the used dataset and are available at <code>dataset_config.other_fields</code>.</li>
+<li>batch_ppi - <code>np.ndarray (B, [3, 4], 30)</code> - the middle dimension is either 4 when TCP push flags are used (<code>use_push_flags</code>) or 3 otherwise.</li>
+<li>batch_flowstats <code>np.ndarray (B, F)</code> - where F is the number of flowstats features computed with <a class="autorefs autorefs-internal" href="../reference_dataset_config/#config.DatasetConfig.get_flowstats_features_len">DatasetConfig.get_flowstats_features_len</a>. To get the order and names of flowstats features, call <a class="autorefs autorefs-internal" href="../reference_dataset_config/#config.DatasetConfig.get_flowstats_feature_names_expanded">DatasetConfig.get_flowstats_feature_names_expanded</a>. The batch_flowstats array includes flow statistics, TCP features (if available and configured), and bins of packet histograms (if available and configured). See the <a class="autorefs autorefs-internal" href="../features/#features">data features</a> page for more information about features.</li>
+<li>batch_labels <code>np.ndarray (B)</code> - integer labels encoded with a <code>LabelEncoder</code> instance available at <code>dataset.encoder</code>.</li>
 </ul>
-<p>Data returned from dataloaders are scaled depending on the selected configuration; see <a class="autorefs autorefs-internal" href="../reference_dataset_config/#config.DatasetConfig"><code>DatasetConfig</code></a> for options.</p>
+<p>PPI and flow statistics features returned from dataloaders are scaled depending on the selected configuration; see <a class="autorefs autorefs-internal" href="../reference_dataset_config/#config.DatasetConfig"><code>DatasetConfig</code></a> for options.</p>
 
 
 

features/index.html

@@ -847,7 +847,7 @@ <h2 id="tcp-features">TCP features</h2>
 </tbody>
 </table>
 <h2 id="other-fields">Other fields</h2>
-<p>Most of those fields are not yet available in the <a class="autorefs autorefs-internal" href="../reference_cesnet_dataset/#datasets.cesnet_dataset.CesnetDataset"><code>CesnetDataset</code></a> class. To access them, create an instance of <code>cesnet_datazoo.pytables_data.pytables_dataset.PyTablesDataset</code> and set <code>return_all_fields</code> to true.</p>
+<p>Datasets contain auxiliary information about samples, such as communicating hosts, flow times, and more fields extracted from the ClientHello message. The <a class="autorefs autorefs-internal" href="../dataset_metadata/#metadata">dataset metadata</a> page lists available fields in individual datasets. </p>
 <table>
 <thead>
 <tr>
@@ -860,80 +860,75 @@ <h2 id="other-fields">Other fields</h2>
 <tr>
 <td>ID</td>
 <td>Per-dataset unique flow identifier</td>
-<td></td>
+<td><code>return_other_fields</code></td>
 </tr>
 <tr>
 <td>TIME_FIRST</td>
-<td>Timestamp of the first packet in format <em>YYYY-MM-DDTHH-MM-SS.ffffff</em></td>
-<td></td>
+<td>Timestamp of the first packet</td>
+<td><code>return_other_fields</code></td>
 </tr>
 <tr>
 <td>TIME_LAST</td>
-<td>Timestamp of the last packet in format <em>YYYY-MM-DDTHH-MM-SS.ffffff</em></td>
-<td></td>
+<td>Timestamp of the last packet</td>
+<td><code>return_other_fields</code></td>
 </tr>
 <tr>
 <td>SRC_IP</td>
 <td>Source IP address</td>
-<td><code>return_ips</code></td>
+<td><code>return_other_fields</code></td>
 </tr>
 <tr>
 <td>DST_IP</td>
 <td>Destination IP address</td>
-<td><code>return_ips</code></td>
+<td><code>return_other_fields</code></td>
 </tr>
 <tr>
 <td>DST_ASN</td>
 <td>Destination Autonomous System number</td>
-<td></td>
+<td><code>return_other_fields</code></td>
 </tr>
 <tr>
 <td>SRC_PORT</td>
 <td>Source port</td>
-<td><code>return_ips</code></td>
+<td><code>return_other_fields</code></td>
 </tr>
 <tr>
 <td>DST_PORT</td>
 <td>Destination port</td>
-<td><code>return_ips</code></td>
+<td><code>return_other_fields</code></td>
 </tr>
 <tr>
 <td>PROTOCOL</td>
 <td>Transport protocol</td>
-<td></td>
+<td><code>return_other_fields</code></td>
 </tr>
 <tr>
 <td>TLS_SNI / QUIC_SNI</td>
 <td>Server Name Indication domain</td>
-<td></td>
+<td><code>return_other_fields</code></td>
 </tr>
 <tr>
 <td>TLS_JA3</td>
 <td>JA3 fingerprint</td>
-<td></td>
+<td><code>return_other_fields</code></td>
 </tr>
 <tr>
 <td>QUIC_VERSION</td>
 <td>QUIC protocol version</td>
-<td></td>
+<td><code>return_other_fields</code></td>
 </tr>
 <tr>
 <td>QUIC_USER_AGENT</td>
 <td>User agent string if available in the QUIC Initial Packet</td>
-<td></td>
-</tr>
-<tr>
-<td>APP</td>
-<td>Web service label</td>
-<td></td>
-</tr>
-<tr>
-<td>CATEGORY</td>
-<td>Service category label</td>
-<td></td>
+<td><code>return_other_fields</code></td>
 </tr>
 </tbody>
 </table>
+<!-- 
+| APP                     | Web service label                                                                   |                                 |
+| CATEGORY                | Service category label                                                              |                                 | 
+-->
+
 <h2 id="details-about-packet-histograms-and-ppi">Details about packet histograms and PPI</h2>
 <p>Due to differences in implementation between packet sequences (<a href="https://github.com/CESNET/ipfixprobe/blob/master/process/pstats.cpp">pstats.cpp</a>) and packet histogram (<a href="https://github.com/CESNET/ipfixprobe/blob/master/process/phists.cpp">phist.cpp</a>) plugins of the ipfixprobe exporter, the number of packets in PPI and histograms can differ (even for flows shorter than 30 packets). The differences are summarized in the following table.
 Note that this is related to TLS over TCP datasets.</p>