thesis.tex

\documentclass[twoside,11pt,nocover,
  printed, %% This option enables the default options for the
           %% digital version of a document. Replace with `printed`
           %% to enable the default options for the printed version
           %% of a document.
  notable,   %% Causes the coloring of tables. Replace with `notable`
           %% to restore plain tables.
  nolof,     %% Prints the List of Figures. Replace with `nolof` to
           %% hide the List of Figures.
  nolot,     %% Prints the List of Tables. Replace with `nolot` to
           %% hide the List of Tables.
  nopalatino, %% Disable palatino globally, enable later using packages
              %% for normal text, not mathmode
  %% More options are listed in the user guide at
  %% <http://mirrors.ctan.org/macros/latex/contrib/fithesis/guide/mu/fi.pdf>.
]{fithesis3}

%% The following section sets up the locales used in the thesis.
\usepackage[resetfonts]{cmap} %% cmap makes the PDF searchable and copyable

% Following packages are enabled by fithesis palatino option
\usepackage{lmodern}
% \usepackage{mathpazo} % Palatino font for mathematics, breaks nice \mathbb
\usepackage{tgpagella} % Palatino font for normal text

\usepackage[T1]{fontenc}
\usepackage[
  main=english, %% By using `czech` or `slovak` as the main locale
                %% instead of `english`, you can typeset the thesis
                %% in either Czech or Slovak, respectively.
  czech,british %% The additional keys allow
]{babel}        %% foreign texts to be typeset as follows:
%%
%%   \begin{otherlanguage}{czech}   ... \end{otherlanguage}

%% The following section sets up the metadata of the thesis.
\thesissetup{
    date          = \the\year/\the\month/\the\day,
    university    = mu,
    faculty       = fi,
    type          = d,
    author        = Petr Velan,
    gender        = m,
    advisor       = {doc. Ing. Pavel Čeleda, Ph.D.},
    title         = {Application-Aware Flow Monitoring},
    TeXtitle      = {Application-Aware Flow Monitoring},
    keywords      = {network, monitoring, measurement, flow, application flow, NetFlow, IPFIX, encryption, performance, 100Gbps},
    TeXkeywords   = {network, monitoring, measurement, flow, application flow, NetFlow, IPFIX, encryption, performance, 100\,Gbps},
    bib           = {bibliography.bib,publications.bib,rfcused.bib},
}
\thesislong{abstract}{%

\noindent The Internet has become a crucial medium for business, entertainment, communication, learning, access to information, and other human endeavours. As the society becomes increasingly dependent upon the availability and security of online services, the revenue from disrupting the normal operation of these services increases as well. Gaining unauthorised access, disrupting functionality, stealing and selling confidential data, and impersonation are only a few examples of these disruptions. Various systems, such as firewalls and anti-virus software, are intended to thwart these attacks and reduce the associated risks. To protect users on their networks, administrators often deploy network monitoring systems to detect and mitigate the threats.

Network flow monitoring is an instrument which allows observing traffic passing through given point in the network and gathering aggregated information about the observed network connections. The aggregated nature of provided information allows to scale the flow monitoring to high-speed networks and monitor traffic rates up to and including hundreds of gigabits per second. Flow data is mainly used for security purposes. However, due to its versatility, it is often used for network management tasks, such as capacity planning and accounting, as well. As both network communication protocols and attack vectors are becoming increasingly sophisticated, the flow monitoring needs to evolve as well. The current trends for flow monitoring are the extraction of supplementary information from application protocols, identification of encrypted traffic, and monitoring of high-speed networks.

% generic contribution summary
This thesis contributes to the progression of flow monitoring by exploring the possibilities unlocked by extending the flow data with application-specific information. We show how the construction of flows is affected by the addition, present the benefits to traffic analysis and assess the inevitable performance loss. To compensate for the lost performance several novel optimisation techniques are proposed for the flow monitoring process. Recognizing that the increasing deployment of encryption is going to limit the benefits of application flow monitoring, we perform a survey of methods for measurement of encrypted traffic. The thesis is concluded by an outlook towards future possibilities for flow monitoring advancement.

% specific contributions
The first contribution of this thesis is a revised definition of flow that attempts to improve and update currently used definitions so that it better matches current flow monitoring practices. A formal definition of flow, together with algorithms for flow construction based on this definition is provided as well. We demonstrate that the revised flow definition allows for use cases that were not covered by NetFlow v9 and IPFIX definitions, such as monitoring of traffic containing fragmented packets.

Using the revised terminology and definitions, we focus on application flow monitoring, which is one of the current trends in flow monitoring. Firstly, an overview of the state of application flow monitoring is provided. Secondly, practical definitions of application flow and application flow record are given to circumscribe application flow monitoring. Thirdly, an experimental study on the design of an HTTP application protocol parser is presented. The study quantifies how application flow monitoring increases the demand for computational resources and decreases the performance of the flow monitoring system.

The application flow monitoring provides new data for traffic analysis. We study several use cases for which the application flow monitoring can be applied in detail. The first one uses information from HTTP headers to detect new classes of attacks on the application layer. The second one shows how the additional information can be used to analyse utilisation of IPv6 transition mechanisms. Then, we show that adding geolocation information to flow records can be used for advanced traffic analysis. And last, a method for characterising network traffic is described. It allows comparing multiple network traces and searching for similarities.

The monitoring of high-speed networks is one of the current trends in flow monitoring. Firstly, we describe the state-of-the-art in this area. Secondly, we identify and propose multiple optimisations including those that rely on programmable network interface cards. Some of these optimisations are then used to build and demonstrate a high-density flow monitoring system capable of processing sixteen 10\,Gb links in a single box.

Classification of encrypted traffic and identification of applications using encryption has become a widely researched topic in the last decade. A survey of methods for measurement of encrypted traffic is, therefore, one of the contributions of this thesis. We show that surprisingly detailed information can be obtained using these methods. In specific cases, even the content of the encrypted connection can be established.

This work concludes with a vision for the future of the flow monitoring. We identify several directions of future research of flow monitoring and a novel approach to monitoring of tunnelled traffic and application layer is proposed. We believe that the perception of the whole application flow monitoring should be revised to facilitate future demands for more complete and better-structured data.
}

\thesislong{thanks}{
I would like to thank my advisor, colleagues, and fellow researchers for their input and insights that helped to shape my research and this thesis. Furthermore, I could have hardly finished this work without the relentless support of my friends and family. They have my eternal gratitude.
}

\usepackage{makeidx}      %% The `makeidx` package contains
\makeindex                %% helper commands for index typesetting.
%% These additional packages are used within the document:
% \usepackage{paralist}     %% Compact list environments
\usepackage{amsmath}      %% Mathematics
\usepackage{amsthm}
\usepackage{amsfonts}
\usepackage[hyphens]{url} %% Hyperlinks
% \usepackage{markdown}     %% Lightweight Markup

% override the default iso-numeric bibtex style by loading the package ourselves
\usepackage[backend=biber,
%         style=,
%         citestyle=numeric-comp,
        sorting=none,
        autolang=other,
        sortlocale=auto,
        defernumbers=true,
        backref=true,
        backrefstyle=three,
        dateabbrev=false, % do not shorten visited on dates
        urldate=long      % use full dates, no 20/10/2018
        ]{biblatex}

\DefineBibliographyStrings{english}{%
  urlseen = {Accessed on},
  backrefpage = {page},% originally "cited on page"
  backrefpages = {pages},% originally "cited on pages"
}
        
% clear unnecessary fields that might be filled in the bib files.
% \AtEveryBibitem affects only bibliography, \AtEveryCitekey to affect the \fullcite command as well
% example: \AtEveryCitekey{\ifentrytype{incollection}{\clearfield{doi}}{}}

% Using \DeclareSourcemap is preferred as it prevents the entries from getting into the .bbl file in the first place
% keep editors only for books
\DeclareSourcemap{
  \maps[datatype=bibtex]{
    \map{ % remove url related field from article, inproceedings and incollectionz
      \pertype{article}
      \pertype{inproceedings}
      \pertype{incollection}
      \step[fieldset=url, null]
      \step[fieldset=urlday, null]
      \step[fieldset=urlmonth, null]
      \step[fieldset=urlyear, null]
      \step[fieldset=urldateera, null]
    }
    \map{ % remove doi from inproceedings and incollection
      \pertype{inproceedings}
      \pertype{incollection}
      \pertype{misc}
      \step[fieldset=doi, null]
    }
    \map{ % remove editors from all types except inbook
      \pernottype{inbook}
      \step[fieldset=editor, null]
    }
    \map[overwrite=true]{ % select authors publications
      \pertype{article}
      \pertype{inproceedings}
      \step[fieldsource=author,
            match={Velan},
            final]
      \step[fieldset=keywords, fieldvalue=velan]
    }
  }
}


% TO-DO notes
% \usepackage{xargs}
% \usepackage[dvipsnames]{xcolor}
% \usepackage[colorinlistoftodos,prependcaption,textsize=small,textwidth=35mm]{todonotes}
% \newcommandx{\unsure}[2][1=]{\todo[linecolor=red,backgroundcolor=red!25,bordercolor=red,#1]{#2}}
% \newcommandx{\change}[2][1=]{\todo[linecolor=blue,backgroundcolor=blue!25,bordercolor=blue,#1]{#2}}
% \newcommandx{\info}[2][1=]{\todo[linecolor=OliveGreen,backgroundcolor=OliveGreen!25,bordercolor=OliveGreen,#1]{#2}}
% \newcommandx{\improve}[2][1=]{\todo[linecolor=Plum,backgroundcolor=Plum!25,bordercolor=Plum,#1]{#2}}
% \newcommandx{\thiswillnotshow}[2][1=]{\todo[disable,#1]{#2}}
% \newcommandx{\itodo}[2][1=]{\todo[inline,#1]{#2}}
% \newcommandx{\iunsure}[2][1=]{\unsure[inline,#1]{#2}}
% \newcommandx{\ichange}[2][1=]{\change[inline,#1]{#2}}
% \newcommandx{\iinfo}[2][1=]{\info[inline,#1]{#2}}
% \newcommandx{\iimprove}[2][1=]{\improve[inline,#1]{#2}}

% Following fixes use of cline in tables, which is broken by 
% czech babel (- is made into active character)
\makeatletter
\begingroup
\toks0=\expandafter{\@cline{#1}-{#2}\@nil}
\@ifpackageloaded{booktabs}{%
  \toks2=\expandafter{\@@@cmidrule[{#1}-{#2}]{#3}{#4}}%
}{}
\catcode`-=\active
\edef\x{\gdef\unexpanded{\@cline#1-#2\@nil}{\the\toks0}}\x
\@ifpackageloaded{booktabs}{%
  \edef\x{\gdef\unexpanded{\@@@cmidrule[#1-#2]#3#4}{\the\toks2}}\x
}{}
\endgroup
\makeatother

% Definitions style using amsthm package
\newtheoremstyle{defnstyle} % name
    {1em}                        % Space above
    {1em}                        % Space below
    {\itshape}                   % Body font
    {}                           % Indent amount
    {\bfseries}                  % Theorem head font
    {\newline}                   % Punctuation after theorem head
    {.5em}                       % Space after theorem head
    {}  % Theorem head spec (can be left empty, meaning ‘normal’)

\theoremstyle{defnstyle}
\newtheorem{defn}{Definition}[chapter]
\newtheorem*{defnn}{Definition}

% \usepackage{graphicx}
% \usepackage{a4wide}
\usepackage{tabularx}
% \usepackage{amssymb}
% \usepackage[titletoc]{appendix}
% \usepackage{color}
\usepackage{textcomp} % itemize bullet symbol font
% \usepackage[hidelinks]{hyperref}
\usepackage{pifont}
% \usepackage[table]{xcolor}
% \usepackage{subfig}
\usepackage{caption} % needed by subcaption
\usepackage{subcaption} % subfloats and subtables
% \usepackage{floatrow}
% \floatsetup[figure]{style=plain,subcapbesideposition=center}
% \usepackage[ruled, lined, linesnumbered,norelsize]{algorithm2e}
\usepackage{multirow} % Complex tables
\usepackage{booktabs} % \toprule, \midrule, \bottomrule
\usepackage{enumitem} % Allows to use [noitemsep] in itemize and enumerate enviroment definitions
\usepackage[boxed, chapter]{algorithm}
\usepackage{algorithmic}
\usepackage{rotating}
\usepackage{varwidth} % Rotated table with caption
\usepackage{tabu}
\usepackage{bigstrut}

% Environment for chapter introductions (use \setlength{\parindent}{15pt} to indent paragraphs)
\newenvironment{chapintro}
{\begin{displayquote}\setlist[itemize]{leftmargin=15mm,rightmargin=10mm,after=\vspace*{-\baselineskip}}\itshape\leftskip=5mm\rightskip=5mm}
{\end{displayquote}}

% Define our own compactitem list
\newlist{compactitem}{itemize}{3} % 3 is max-depth
\setlist[compactitem]{label=\textbullet,leftmargin=*,nosep,after=\vspace*{-\baselineskip}}

% Define indexmacro that does repeat the keyword
\newcommand\Index[1]{#1\index{#1}}

% Checkmark deffinition
\newcommand{\cmark}{\raisebox{-1pt}{\ding{51}}}%
\newcommand{\xmark}{}%

% Put words in equations in \mli to correct spacing
\newcommand{\mli}[1]{\mathit{#1}}

% Load packages required by fithesis. We can setup those packages after this point
\thesisload

% Change geometry, needs to be placed after \thesisload
\usepackage[paper=a4paper,top=2.5cm,bottom=2.5cm,left=2.0cm,right=2.0cm,foot=1cm,bindingoffset=1.0cm]{geometry} % for print
% \usepackage[paper=a4paper,top=2.5cm,bottom=2.5cm,left=2.5cm,right=2.5cm,foot=1cm]{geometry} % digital

% Chapter titles
\usepackage[noindentafter]{titlesec}
% Numbered chapters
\titleformat{name=\chapter}[display]
{\normalfont\huge\bfseries}{\chaptertitlename\ \thechapter}{20pt}{\Huge}

% Numberless chapters
\titleformat{name=\chapter,numberless}[display]
{\normalfont\Large\bfseries}{}{0pt}{}
\titlespacing{name=\chapter,numberless}{0pt}{-40pt}{20pt}

% Redefine thesisheadings style to have section names on left pages
\makeatletter
\def\ps@thesisheadings{%
  \def\chaptermark##1{%
    \markboth{%
      \ifnum\c@secnumdepth >\m@ne
        \thechapter.\ %
      \fi ##1}{}}
  \def\sectionmark##1{%
    \markright{%
      \thesection.\ ##1%
    }
  }
  \let\@oddfoot\@empty
  \let\@oddhead\@empty
  \def\@oddhead{%
    \vbox{%
      \hbox to \textwidth{%
      \hfil{\sc\rightmark}}%
      \vskip 4pt\hrule}}
  \if@twoside
    \def\@evenhead{%
      \vbox{%
        \hbox to \textwidth{%
          {\sc\leftmark}%
          \hfil}
        \vskip 4pt\hrule}}
  \else
    \let\@evenhead\@oddhead
  \fi 
  \def\@oddfoot{\hfil\PageFont\thepage}
  \if@twoside
    \def\@evenfoot{\PageFont\thepage\hfil}%
  \else
    \let\@evenfoot\@oddfoot
  \fi 
  \let\@mkboth\markboth}
\makeatother

% Quotes in italics with centered reference (package csquotes)
\renewcommand{\mkbegdispquote}[2]{\itshape}
\renewcommand{\mkenddispquote}[2]{\par\centering #1#2}

% Environment for definitions using csquotes for centering
\newenvironment{definition}{\begin{displayquote}\vspace{-2em}\begin{defn}}
{\end{defn}\end{displayquote}}

% Make multiple authors be printed as first et al. (package biblatex)
\ExecuteBibliographyOptions{maxnames=999, maxcitenames=2}

% This removes a warning about missing font shape (sl used in \printbibliography)
\makeatletter
\input{ts1qpl.fd}
\makeatother
\DeclareFontShape{TS1}{qpl}{m}{sl}{<->ssub * qpl/m/it}{}

% This removes a warning about missing font shape (ls used in toc)
\makeatletter
\input{t1qpl.fd}
\makeatother
\DeclareFontShape{T1}{qpl}{m}{sl}{<->ssub * qpl/m/it}{}

% supress warning about multiple page groups in included pdfs
\pdfsuppresswarningpagegroup=1

% We want to use the AMSb symbol for mathbb
\DeclareSymbolFontAlphabet{\mathbb}{AMSb}

% Highlight overfull hboxes
% \overfullrule=10mm

% Build only specific chapters
% \includeonly{chapters/07-next-generation-flow}
% \hypersetup{draft}

\begin{document}

%------------------------------------------------------------------------------
% Include all chapters

\include{chapters/01-introduction}

\include{chapters/02-network-flow-monitoring}

\include{chapters/03-application-flow-monitoring}

\include{chapters/04-flow-monitoring-use-cases}

\include{chapters/05-flow-monitoring-performance}

\include{chapters/06-measurement-of-encrypted-traffic}

\include{chapters/07-next-generation-flow}

\include{chapters/08-conclusions}

%------------------------------------------------------------------------------
% Bibliography

% Allow more breaks in URLs in bibliography
\setcounter{biburlnumpenalty}{6000}
\setcounter{biburllcpenalty}{7000}
\setcounter{biburlucpenalty}{8000}
% Allow more white space per line
\emergencystretch=1em

% Custom bibheading with unnumbered section
\defbibheading{simplebibheading}{\section*{#1}}

% Print bibliography heading and include it in ToC
\printbibheading[heading=bibintoc]

% Print authored publications
\newrefcontext[labelprefix={A}]
\printbibliography[keyword=velan,heading=simplebibheading,title={Authored publications referenced in the thesis}]
\endrefcontext

% Print the rest of the publications
\begin{otherlanguage}{british}
\printbibliography[notkeyword=velan,heading=simplebibheading,title={Other publications referenced in the thesis},resetnumbers=true]
\end{otherlanguage}

% Print bibliography in british format, it is nicer than american
% \begin{otherlanguage}{british}
% \printbibliography[heading=bibintoc] %% Print the bibliography.
% \end{otherlanguage}

%------------------------------------------------------------------------------

% Print index
% \makeatletter\thesis@blocks@clear\makeatother
% \phantomsection %% Print the index and insert it into the
% \addcontentsline{toc}{chapter}{\indexname} %% table of contents.
% \printindex

%------------------------------------------------------------------------------

\appendix %% Start the appendices.
% List of authored publications

% Redefine fullcite to print all names, not just maxcitenames
\preto\fullcite{\AtNextCite{\defcounter{maxnames}{99}}}
\include{chapters/10-publication-list}

\end{document}