From 9d7fe0f744cc32c03cabfb817f8fef8cd3292200 Mon Sep 17 00:00:00 2001 From: taiyeba Date: Thu, 4 Apr 2019 16:53:54 +0200 Subject: [PATCH 01/15] Documentation updates for v0.15 --- Conan-configs.rst | 78 ++++++++++++++++++++ Conan-package-build.rst | 106 +++++++++++++++++++++++++++ Conanigans.rst | 24 +++++++ Contributing-to-IncludeOS.rst | 13 ++-- Deeper-understanding.rst | 5 +- Editable-mode.rst | 119 +++++++++++++++++++++++++++++++ Examples-and-libraries.rst | 86 ---------------------- Examples.rst | 51 +++++++++++++ Getting-started.rst | 130 +++++++++++++--------------------- Howto-Kernel-dev.rst | 50 +++++++++++++ Howto-Service-dev.rst | 117 ++++++++++++++++++++++++++++++ Introduction.rst | 26 +++++++ Jenkins-CI.rst | 39 ++++++---- Linux-guide.rst | 53 ++++++++++++++ Macos-guide.rst | 53 ++++++++++++++ More-information.rst | 25 ++++++- README.rst | 2 +- Tools.rst | 101 ++++++++++++++++++++++++++ conf.py | 10 ++- index.rst | 12 ++-- 20 files changed, 897 insertions(+), 203 deletions(-) create mode 100644 Conan-configs.rst create mode 100644 Conan-package-build.rst create mode 100644 Conanigans.rst create mode 100644 Editable-mode.rst delete mode 100644 Examples-and-libraries.rst create mode 100644 Examples.rst create mode 100644 Howto-Kernel-dev.rst create mode 100644 Howto-Service-dev.rst create mode 100644 Introduction.rst create mode 100644 Linux-guide.rst create mode 100644 Macos-guide.rst create mode 100644 Tools.rst diff --git a/Conan-configs.rst b/Conan-configs.rst new file mode 100644 index 0000000..b26129b --- /dev/null +++ b/Conan-configs.rst @@ -0,0 +1,78 @@ +.. _Conan configs: + +Conan Configuration for IncludeOS +================================= + +We have set up a repository `includeos/conan_config `__ that helps IncludeOS users configure all the necessary conan settings. + +.. warning:: + This overwrites any existing conan configuration. + Thus you should set your ``CONAN_USER_HOME`` to create a separate conan home folder for testing. + +To configure using this repo just do: + +:: + + $ conan config install https://github.com/includeos/conan_config.git + + +This adds IncludeOS bintray remote in your conan remotes and also installs all the +profiles we have in the repository for you. + +Profiles +~~~~~~~~ + +Conan uses `profiles `__ +to build packages. By default IncludeOS will build with ``clang 6.0`` if +``CONAN_PROFILE`` is not defined. Passing ``-DCONAN_DISABLE_CHECK_COMPILER`` +during build disables this check. + +Profiles we are developing with can be found in `includeos/conan_config `__ repository under ``conan_config/profiles/`` . +If you have not used the ``conan config install`` command above, then to install the profiles, copy them over to your user ``./conan/profiles`` folder for browse through our `config repo `__ . + +The target profiles we have verified are the following: + +- `clang-6.0-linux-x86 `__ +- `clang-6.0-linux-x86_64 `__ +- `gcc-7.3.0-linux-x86_64 `__ +- `clang-6.0-macos-x86_64 `__ + +To ensure the profile/s has been installed do: + +:: + + $ conan profile list + + +Verify the content of your profile by: + +:: + + $ conan profile show + + +If your profile is on the list and contents are verified, you are set to use the +profile for building. + +IncludeOS Bintray Repository +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The bintray repository is where all the packages used to build IncludeOS +are uploaded. Adding the repo to your conan remotes will give you access to all +our packages developed for IncludeOS. + +You can check your repository remotes with: + +:: + + $ conan remote list + + +If the includeos remote is not added you have to add it to be able to download +and use packages. Once they are downloaded they are stored in the local cache. + +To add the IncludeOS bintray repository to your conan remotes: + +:: + + $ conan remote add includeos-test https://api.bintray.com/conan/includeos diff --git a/Conan-package-build.rst b/Conan-package-build.rst new file mode 100644 index 0000000..d9acd46 --- /dev/null +++ b/Conan-package-build.rst @@ -0,0 +1,106 @@ +.. _Conan package build: + +Playing with Packages +===================== + +If you are interested in editing/building our dependency packages on your own, +you can checkout our repositories at `includeos/ `__ . +You can find the external dependency recipes at `includeos/conan `__ . +Currently we build with ``clang-6`` and ``gcc-7.3.0`` compiler toolchains. +It is expected that these are already installed on your system. + +.. note:: + Uploaded packages with their recipes can be found on `IncludeOS Bintray `__ + +Building Tools +~~~~~~~~~~~~~~ + +The **GNU Binutils** are a collection of binary tools we have added to the profiles +so that the binaries are always executable inside the conan environment. +They are not an actual part of the final binary. + +The binutils package must be built for the Host it's intended to run on. Therefore +the binutils package is built using a special *toolchain profile* that doesn't have +a requirement on binutils itself. + +To build ``bintuils`` using our `includeos/conan `__ recipes: + +- Clone the repository +- Do ``conan create`` as follows: + +:: + + conan create /binutils/2.31 -pr -toolchain includeos/stable + + +For MacOS users, we currently have a `apple-clang-10-macos-toolchain `__ for building the binutils package. + +Building External Dependencies +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To build our other dependencies you may use the conan recipes we have in the +`conan `__ repository. + +.. note:: + If you plan to build dependencies you might need to ensure you have other missing libraries installed. + +**Requirements** + +- GNU C++ compiler - ``g++-multilib`` +- Secure Sockets Layer toolkit ``libssl-dev`` + +Building musl +------------- + +:: + + conan create /musl/1.1.18 -pr includeos/stable + + + +Building llvm stdc++ stdc++abi and libunwind +-------------------------------------------- + +If these recipes do not have a fixed version in the conan recipe then you have +to specify it alongside the ``user/channel`` as ``package/version@user/channel`` +otherwise you can use the same format at musl above. + +:: + + conan create /llvm/libunwind -pr libunwind/7.0.1@includeos/stable + conan create /llvm/libcxxabi -pr libcxxabi/7.0.1@includeos/stable + conan create /llvm/libcxx -pr libcxx/7.0.1@includeos/stable + + +Building IncludeOS libraries and tools +-------------------------------------- + +We have moved the libraries and tools created by IncludeOS outside the IncludeOS +repository. You can now find them all in their own repositories inside the IncludeOS +organization. + +To build the libraries and tools, + +:: + + $ git clone https://github.com/includeos/mana.git + $ cd mana + $ conan create . includeos/latest -pr clang-6.0-linux-x86_64 + + +Below is a list of some of our Libraries/Tools: + +* `Vmbuild `__ + +* `Vmrunner `__ + +* `Mana `__ + +* `Microlb `__ + +* `Diskbuilder `__ + +* `NaCl `__ + +.. note:: + To get your packages in Editable mode, look at the page on editable package mode. diff --git a/Conanigans.rst b/Conanigans.rst new file mode 100644 index 0000000..85eaf94 --- /dev/null +++ b/Conanigans.rst @@ -0,0 +1,24 @@ +.. _Conanigans : + +Conan-igans for IncludeOS +========================== + +We have recently changed the way we build IncludeOS. We are now using the C/C++ +package manager - Conan to build IncludeOS. We have developed conan packages for +IncludeOS and all it's dependency libraries which we have currently made available +on `Bintray `__ . + +.. toctree:: + :maxdepth: 2 + :caption: All things development + + Conan-configs + Conan-package-build + Editable-mode + +**Contributing to our package recipes** + +If you are interested in contributing to our package recipes, +check out conan recipes in our repositories at `github/includeos `__. +Most of them are developed and tested on our `jenkins server `__ before uploading to bintray. +To know more about our rules on contributions please refer to :ref:`Contributing to IncludeOS `. If you have any questions, feel free to ask. we are usually present in our `Slack channel `__ for any kind of questions. diff --git a/Contributing-to-IncludeOS.rst b/Contributing-to-IncludeOS.rst index 673d9a6..40a11b2 100644 --- a/Contributing-to-IncludeOS.rst +++ b/Contributing-to-IncludeOS.rst @@ -6,9 +6,7 @@ Contributing to IncludeOS Clone, edit and send us a pull request on GitHub ------------------------------------------------ -IncludeOS is being developed on GitHub. Clone the `repository `__, send us a `pull request `__ and `chat with us on Slack `__. - -Send any and all pull requests to the `dev-branch `__. It's ok if it comes from your master branch. +IncludeOS is being developed on GitHub. Clone the `repository `__, send us a `pull request `__ and `chat with us on Slack `__. Guidelines ~~~~~~~~~~ @@ -116,7 +114,7 @@ Code formatting /////////////////////////////////////////////////////////////////////////////// struct Dirent { - + /////////////////////////////////////////////////////////////////////////////// explicit Dirent(File_system* fs, const Enttype t = INVALID_ENTITY, const std::string& n = "", const uint64_t blk = 0, const uint64_t pr = 0, @@ -146,12 +144,13 @@ Code formatting - If your editor supports :code:`.editorconfig`, use it. + Issue tracker ------------- Post any issues not already mentioned, in the `issue tracker on GitHub `__. You can also post questions not answered by editing the :ref:`FAQ` on GitHub. -Gitter chat ------------ +Chat on Slack +------------- -We are usually present in our `public gitter channel `__ for any kinds of questions. +We are usually present in our `Slack channel `__ for any kind of questions. diff --git a/Deeper-understanding.rst b/Deeper-understanding.rst index 255c06b..59fb8ac 100644 --- a/Deeper-understanding.rst +++ b/Deeper-understanding.rst @@ -1,15 +1,14 @@ .. _Deeper understanding: -Deeper understanding +More about IncludeOS ==================== .. Own content for this section .. toctree:: :maxdepth: 2 - :caption: Contents: + :caption: Contents The-build-process The-boot-process Fun-with-Guns-and-Knives - Jenkins-CI diff --git a/Editable-mode.rst b/Editable-mode.rst new file mode 100644 index 0000000..bc5b87d --- /dev/null +++ b/Editable-mode.rst @@ -0,0 +1,119 @@ +.. _Editable mode: + +Editable package Mode +===================== + +If you are a kernel developer or are simple interested in fiddling with our +kernel code, you can use the includeos package in ``editable`` mode. This is useful +when working with several interconnected components and you would like to test +your changes across several libraries or functionalities. + +Below we have written down a few steps to get you started with editable packages. +You can read more about it at `packages in editable mode `__. + +.. note:: + Currently this is an experimental feature on conan version 1.13 and they + have mentioned that for future releases the feature is subject to breaking changes. + +To get started with getting the conan package in editable mode, + +* Create a ``build`` folder to build IncludeOS + +* Edit the ``etc/layout.txt`` : ( **uses jinja2 syntax** ) needed to parse the build folder correctly. + +Do some local adaptions for where your build folder is relative to the includeos source folder by setting build_dir in the third line as follows : + +:: + + {% set simple=true%} + + {% set build_dir='build' %} + + {% if simple==false %} + {% set arch=settings.arch|string %} + {% set platform=options.platform|string %} + {% set build_dir=build_dir+'/'+arch+'/'+platform %} + {% endif %} + + [bindirs] + {#This is so that we can find os.cmake after including conanbuildinfo.cmake #} + cmake + + [includedirs] + {#This is to ensure the include directory in conanbuildinfo.cmake includes our API#} + api + + [libdirs] + {#This is so that we can find our libraries #} + {{ build_dir }}/plugins + {{ build_dir }}/drivers + {{ build_dir }}/lib + {{ build_dir }}/platform + + [resdirs] + {#This is so that we can find ldscript and search for drivers plugins etc#} + {{ build_dir }} + +.. note:: + In the non simple form it is possible to have multiple build folders from the same source which allows multiple architectures and configurations to be tested from the same source however the complexity increases + + +Set package into editable mode +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Make sure to adjust the version to whatever is appropriate. + +:: + + conan editable add . includeos/$(conan inspect -a version . | cut -d " " -f 2)@includeos/latest --layout=etc/layout.txt + + +Check Status +~~~~~~~~~~~~ + +The package is now in **editable mode** and any dependencies of IncludeOS will +pick this IncludeOS package from your local cache. If you make any changes to the +code a simple `make` should be enough. However if your dependencies have changed +you need to redo the `conan install` step. + +Here is an example on how it looks when its pulled into cache as editable: + +:: + + $ conan editable list + includeos/0.14.1-1052@includeos/latest + Path: ~/git/IncludeOS + Layout: ~/git/IncludeOS/etc/layout.txt + + +Build IncludeOS +~~~~~~~~~~~~~~~ + +Asuming the buildfolder is build under the includeos source directory the following is enough. +you can also manually perform the build step for the editable package however doing the step below ensures all parameters are transfered correctly from your conan profile and options into the build. + +:: + + conan install -if build . -pr (-o options like platform=nano etc) + conan build -bf build . + +*Building small changes once the first build is done* + +Once IncludeOS is built by the conan build command you simply have to make sure to issue a make command in the build location + +:: + + cd build && make + or + cmake build --build + +Finalizing Changes +~~~~~~~~~~~~~~~~~~ + +Once the code is *finalized* and you want to verify that the conan package +still builds remove the editable and do a conan create on the package: + +:: + + $ conan editable remove includeos/0.15.0@includeos/stable + $ conan create includeos/Latest -pr diff --git a/Examples-and-libraries.rst b/Examples-and-libraries.rst deleted file mode 100644 index 03274ca..0000000 --- a/Examples-and-libraries.rst +++ /dev/null @@ -1,86 +0,0 @@ -.. _Examples and libraries: - -Examples and libraries -====================== - -Examples -~~~~~~~~ - -`256 Color VGA `__ - -`IRCd `__ - -`LiveUpdate `__ - -`SQlite `__ - -`STREAM `__ - -`TCP perf `__ - -`TLS server `__ - -`Acorn `__ - -`Demo Linux `__ - -`Demo Service `__ - -`Mender `__ - -`MicroLB `__ - -`Router `__ - -`Dualnic `__ - -`Scoped Profiler `__ - -`SMP `__ - -`Snake `__ - -`Syslog `__ - -`TCP `__ - -`Vlan `__ - -`Websocket `__ - -`UniK Test Service `__ - -Libraries -~~~~~~~~~ - -`LiveUpdate `__ - -`MicroLB `__ - -`Uplink `__ - -`NaCl `__ - -`Mana `__ - -`Protobuf `__ - -`Mender `__ - -`URI `__ - -`HTTP `__ - -`Bucket `__ - -`Dashboard `__ - -`Cookie `__ - -`Path to Regex `__ - -`JSON `__ - -`Director `__ - -`Butler `__ diff --git a/Examples.rst b/Examples.rst new file mode 100644 index 0000000..81c11ff --- /dev/null +++ b/Examples.rst @@ -0,0 +1,51 @@ +.. _Examples: + +Examples +======== + +Here we have a list of all the examples you will find in the `Examples Repository `__ . + +To build an example, follow the steps in the readme. Note that your +build folder can be anywhere, as long as you point to the source recipe during +`conan install -pr `. + + +`256 Color VGA `__ + +`IRCd `__ + +`LiveUpdate `__ + +`STREAM `__ + +`TCP perf `__ + +`TLS server `__ + +`Acorn `__ + +`Demo Service `__ + +`Http Client `__ + +`MicroLB `__ + +`Protpbuf `__ + +`Router `__ + +`Scoped Profiler `__ + +`Snake `__ + +`Starlight `__ + +`Syslog `__ + +`TCP `__ + +`Transfer `__ + +`Vlan `__ + +`Websocket `__ diff --git a/Getting-started.rst b/Getting-started.rst index 970a4f0..0d3539d 100644 --- a/Getting-started.rst +++ b/Getting-started.rst @@ -1,107 +1,77 @@ .. _Getting started: -Getting started -=============== +Getting started with IncludeOS +============================== -Set custom location and compiler -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +IncludeOS is an open source project and can be found on `Github `__ . +Whether you are a kernel or a service developer using MacOS or Linux, below is a list of sections that will guide you through the process of building and starting an IncludeOS service. -By default the project is installed to /usr/local/includeos. +.. toctree:: + :maxdepth: 1 -However, it is recommended to choose a custom location as well as select the compiler we want clang to find. + Linux-guide + Macos-guide + Howto-Kernel-dev + Howto-Service-dev -To do this we can edit ~/.bashrc, adding these lines at the end of the file: - -:: - - export INCLUDEOS_PREFIX=/includeos - export PATH=$PATH:$INCLUDEOS_PREFIX/bin - -This will also crucially make the boot program visible globally, so that you can simply run :code:`boot ` inside any service folder. - -Install libraries -~~~~~~~~~~~~~~~~~ - -**NOTE:** The script will install packages and create a network bridge. - -:: - - $ git clone https://github.com/hioa-cs/IncludeOS - $ cd IncludeOS - $ ./install.sh - -**The script will:** - -- Install the required dependencies: :code:`curl make clang-5.0 nasm bridge-utils qemu`. -- Build IncludeOS with CMake: - + Download the latest binary release bundle from github together with the required git submodules. - + Unzip the bundle to the current build directory. - + Build several tools used with IncludeOS, including vmbuilder, which turns your service into a bootable image. - + Install everything in :code:`$INCLUDEOS_PREFIX/includeos` (defaults to :code:`/usr/local`). - -Configuration of your IncludeOS installation can be done inside :code:`build/` with :code:`ccmake ..`. - -Testing the installation -~~~~~~~~~~~~~~~~~~~~~~~~ +Hello World with IncludeOS +~~~~~~~~~~~~~~~~~~~~~~~~~~ -A successful setup enables you to build and run a virtual machine. Running: +To start of we welcome you to trying a simple traditional Hello World with IncludeOS. +You don't need to install IncludeOS to run a service, so just clone the service +repository. :: - $ ./test.sh + $ git clone https://github.com/includeos/hello_world + $ cd hello_world -will build and run `this example service `__. +Dependencies +------------ -.. Testing the example service is further down on the page +To build the service you will need: -Writing your first service -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -1. Create a blank directory. -2. Create a minimal ```service.cpp``` -3. Running "boot ." will add a CMakeList.txt based on the `./seed `__. -4. Update the `CMakeLists.txt `__ to specify the name of your project, enable any needed drivers or plugins, etc. +* Cmake, make, nasm +* Clang or GCC +* `Conan `__ -**Example:** +To boot VMs locally you will need: -:: - - $ mkdir ~/my_service - $ cd ~/my_service - $ emacs service.cpp - ... add your code - $ boot . - -Take a look at the `examples `__ and the `tests `__ on GitHub. These all started out as copies of the same seed. - -Testing the demo service -~~~~~~~~~~~~~~~~~~~~~~~~ - -A suitable service to test your installation is the Demo Service, found in ```examples/demo_service```. It contains a simplistic web server that will serve out a single, static page. +* qemu +* python3 +* python packages: psutil, jsonschema +The `IncludeOS `__ conan packages are available on +bintray. Therefore you will need to add the remote to your list. Conan requires +the use of a profile during building a package, which can also be installed using +the command below. :: - $ cd examples/demo_service - $ boot --create-bridge . + $ conan config install https://github.com/includeos/conan_config -Now the service should be running. In another shell session you can try to ping the service to see if responds. +.. note:: + For Mac OS ensure that you have a working installation of `brew `__ to be able to install all dependencies. For more specific installation instructions for mac or linux checkout the linux guide or mac os guide listed above. To get help with your conan setup follow the instruction at :ref:`Conan Configuration `. -:: - $ ping -c 3 10.0.0.42 - PING 10.0.0.42 (10.0.0.42): 56 data bytes - 64 bytes from 10.0.0.42: icmp_seq=0 ttl=64 time=0.379 ms - 64 bytes from 10.0.0.42: icmp_seq=1 ttl=64 time=0.370 ms - 64 bytes from 10.0.0.42: icmp_seq=2 ttl=64 time=0.639 ms - $ +Running ``conan profile list`` will show the profiles installed. +- Linux users can typically use ``clang-6.0-linux-x86_64`` +- MacOS users can use ``clang-6.0-macos-x86_64``. +You can also develop your own profiles. -Great. The final step is to see if we get a web page from the service. +The following steps let you build and boot the IncludeOS hello world example. :: - $ links -dump http://10.0.0.42/ - IncludeOS The C++ Unikernel + $ git clone https://github.com/includeos/hello_world.git + $ mkdir your_build_dir && cd "$_" + $ conan install ../hello_world -pr + $ source activate.sh + $ cmake ../hello_world + $ cmake --build . + $ boot hello - You have successfully booted an IncludeOS TCP service with simple http. - For a more sophisticated example, take a look at Acorn. +.. warning:: + Once you're done ``$ source deactivate.sh`` will reset your environment to + its previous state. +For more advanced examples see the `examples repo `__. diff --git a/Howto-Kernel-dev.rst b/Howto-Kernel-dev.rst new file mode 100644 index 0000000..eb80854 --- /dev/null +++ b/Howto-Kernel-dev.rst @@ -0,0 +1,50 @@ +.. _Howto Kernel dev: + +Kernel Developers guide +======================= + +To build IncludeOS with our conan packages you will need: + +- A `Profile `__ +- Access to our built `conan packages `__ + +To get access to the packages add our bintray remote to your conan remote list: + +:: + + $ conan remote add includeos https://api.bintray.com/conan/includeos/includeos + +.. note:: + We have two channels for our packages in our remote: + + * ``latest`` the latest built package + * ``stable`` the last stable built package + + +Compilers +--------- + +Currently we are building IncludeOS on Linux with ``clang-6.0`` and ``gcc-7.3``. +All our dependencies, libraries and tools found on Bintray are built and tested +with these two compilers. + + +Cross-Platform Building +^^^^^^^^^^^^^^^^^^^^^^^ + +To build IncludeOS for MacOS on Linux, you can use the ``clang-6.0-macos-x86_64`` +profile. + + +Kernel Development +------------------ + + +Using our Packages +------------------ + + + + +Uploading your Packages +----------------------- diff --git a/Howto-Service-dev.rst b/Howto-Service-dev.rst new file mode 100644 index 0000000..bcdea69 --- /dev/null +++ b/Howto-Service-dev.rst @@ -0,0 +1,117 @@ +.. _Howto Service dev: + +Service Developers guide +======================== + +Examples are now built with conan packages. The IncludeOS demo examples have now +been moved to `includeos/demo-examples `__. + +To start build your first service clone the repository: + +:: + + $ git clone https://github.com/includeos/demo-examples.git + $ cd demo-examples && ls + + +You will see a list of all the example services you can try building. + +Building a Service +------------------ + +To build the demo service, create a ``build`` folder inside the ``demo_service`` folder +and install with the profile you would like to use. + +:: + + $ cd demo_service + $ mkdir build + $ cd build + $ conan install .. -pr + + +Installing with the chosen profile, fetches the profile configurations and Installs +the requirements in the ``conanfile.txt``. If the required packages are not in the +local conan cache they are downloaded and installed. If all the packages required +are already in the local conan cache then it moves on to apply build requirements +and generates the required virtualenv scripts and cmake information. + +Next to build the service do: + +:: + + $ source activate.sh + $ cmake .. + $ cmake --build . + + +Doing ``cmake`` configures and generates the build files and they are written to +the build folder. Then doing ``cmake --build .`` builds the target service. You +should see the last line as: + +:: + + [100%] Built target demo + +``demo`` is the name of this service for the demo_service. That's the name you will +use when starting the service. Do ``ls`` to see all the files created. There is also +an executable file named `demo` which is now the service you will start. + + +Starting an IncludeOS Instance +------------------------------ + +Now that you have all your build requirements ready, you can start the service. +If you skipped the step of activating the virtual environment in the previous step remeber to do: + +:: + + $ source activate.sh + + +Now you can boot the demo service with ``boot ``. + +:: + + $ boot demo + + +This should start your demo service and show something along the following lines: + +:: + + ================================================================================ + IncludeOS (x86_64 / 64-bit) + +--> Running [ IncludeOS Demo Service ] + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + Service started + Made device_ptr, adding to machine + [ Network ] Creating stack for VirtioNet on eth0 (MTU=1500) + [ Inet ] Bringing up eth0 on CPU 0 + *** Basic demo service started *** + + +To build more examples follow the `README `__ in the examples repository. + + +Creating your own service +------------------------- + +Writing your first service + +To start writing your first service, +- Copy the `demo_service/service.cpp `__ example. +- Then start implementing the ``Service::start`` function in the ``Service`` class, located in ``/service.cpp`` (very simple example provided). This function will be called once the OS is up and running. +- Update the ``CMakeLists.txt`` to specify the name of your project, enable any needed drivers or plugins, etc. +- Update the ``conanfile.txt`` to specify any build requirements and generators. + +You should then be able to run your service in the same way as the demo_service. + +To add a new service to our demo examples, make a PR to our `demo-examples `__ repo. Make sure to +add a README in your example folder with description of your service. + + +.. warning:: + Remember to deactivate your service environment after your work with: + + ``$ source deactivate.sh`` diff --git a/Introduction.rst b/Introduction.rst new file mode 100644 index 0000000..7f3022c --- /dev/null +++ b/Introduction.rst @@ -0,0 +1,26 @@ +.. _Introduction: + + +Introduction +~~~~~~~~~~~~ + +IncludeOS is an open source unikernel operating system project project designed for cloud services. +It is a minimal tasking library written in C++ which allows users to run applications written in C++ without an operating system. IncludeOS adds operating system functionality to an application by deploying a minimal virtual machine to run the application. Adding ``#include `` to ``c++`` code like this: + +:: + + #include + + int main() { + printf("Hello World! \n"); + } + +This literally includes a minimal operating system into the service during link-time. The way this works is that during link time the build system extracts service requirements from the pre-compiled OS-library and forms a single binary. To learn more about our open source project and it's potential uses visit the `technology `__ page over at our website. + +.. toctree:: + :hidden: + + Features + Tools + Examples + Using-memdisks diff --git a/Jenkins-CI.rst b/Jenkins-CI.rst index cc9b32c..23e7dd4 100644 --- a/Jenkins-CI.rst +++ b/Jenkins-CI.rst @@ -1,28 +1,36 @@ .. _Jenkins CI: -Jenkins CI -========== +CI Pipeline for Developers +========================== -.. Update +We use continuous integration to build and test our projects on a regular basis. +You will find all the repositories currently using CI under the +IncludeOS Github Organization on our Jenkins server. -`Jenkins.includeos.org `__ ---------------------------------------------------------- +Jenkins CI +~~~~~~~~~~ -Getting your personal build to build on the jenkins server ----------------------------------------------------------- +`Jenkins-public.includeos.org `__ +-------------------------------------------------------------------------------------- -If you want to get your personal fork of IncludeOS to build with every commit this procedure will show you what steps to go through. -Things to take note off: +Personal builds on the Jenkins server +------------------------------------- -- Will by default build on your dev branch. This will be easier to change at a later date. +If you want to get your personal fork of IncludeOS to build with every commit +this procedure will show you what steps to go through. + +**Things to take note off:** - Will look for the repo: ``https://github.com//IncludeOS`` -- Does not merge with upstream dev automatically as of this date. +- Does not merge with upstream dev automatically unless approved by an IncludeOS developer. + + +Setting up the Webhook +---------------------- Follow these steps to get it to work: -------------------------------------- 1. Go to the settings page for your **personal fork** @@ -36,8 +44,13 @@ Follow these steps to get it to work: |Checkmark| -Then when I create the tests results will be available on `Jenkins.includeos.org `__ +Then when the tests are complete, the results will be available on `jenkins-public.includeos.org `__ .. |Settings menu| image:: http://i.imgur.com/wfoYcaD.png .. |Payload URL| image:: http://i.imgur.com/g0gEcBq.png .. |Checkmark| image:: http://i.imgur.com/yUTIwZ1.png?1 + +Travis CI +~~~~~~~~~ + +We have recently started using `Travis CI `__ to build and upload some of our dependencies for macos. diff --git a/Linux-guide.rst b/Linux-guide.rst new file mode 100644 index 0000000..2dcac88 --- /dev/null +++ b/Linux-guide.rst @@ -0,0 +1,53 @@ +.. _Linux guide: + +Linux Guide +=========== + +We are currently developing and testing on ``Ubuntu-18.04``. To get started with +building on Linux, install the following dependencies. + + +Dependencies +------------ + +For building the kernel you will need: + +* `C/C++ package manager `__ (Conan 1.13.1 or newer) +* cmake, make, nasm +* clang, or alternatively gcc on linux. Prebuilt packages are available for clang 6.0 and gcc 7.3. + +To boot VMs locally you will also need: + +* qemu +* python3 +* python packages: psutil, jsonschema + +You can use the recipe below to get all the dependencies in place. + +:: + + $ apt-get install python3-pip python3-dev git cmake clang-6.0 gcc nasm make qemu + $ pip3 install setuptools wheel conan psutil jsonschema + + +*Getting Conan remotes and profiles* + +We have set up a repository that will configure your local conan to use our build profiles and remote repositories. + +.. warning:: + This overwrites any existing conan configuration. + Thus you should set your ``CONAN_USER_HOME`` to create a separate conan home folder for testing. + +:: + + $ conan config install https://github.com/includeos/conan_config.git + +If you prefer to set up conan manually the full configuration can be found in the [conan_config](https://github.com/includeos/conan_config.git)-repository. + +.. _Build linux: + +Building the kernel on Linux +---------------------------- + +To build IncludeOS on Linux for Linux you can use the profile named +``clang-6.0-linux-x86_64`` or ``gcc-7.3.0-linux-x86_64`` : diff --git a/Macos-guide.rst b/Macos-guide.rst new file mode 100644 index 0000000..7097a21 --- /dev/null +++ b/Macos-guide.rst @@ -0,0 +1,53 @@ +.. _Macos guide: + +MacOS Guide +=========== + +To start building on MacOS make sure to have a working installation of [brew](https://brew.sh/) to be able to install all dependencies. + +Dependencies +------------ + +For building the kernel you will need: + +* `C/C++ package manager `__ (Conan 1.13.1 or newer) +* cmake, make, nasm +* clang (Prebuilt packages are available for clang 6.0) + +To boot VMs locally you will also need: + +* qemu +* python3 +* python packages: psutil, jsonschema + +Once you have [homebrew](https://brew.sh/) you can use our ``brew tap`` to install the dependencies as follows. + +:: + + $ brew tap includeos/includeos + $ brew install includeos + +Checkout our `homebrew formula for IncludeOS `__ + + +*Getting Conan remotes and profiles* + +We have set up a repository that will configure your local conan to use our build profiles and remote repositories. + +.. warning:: + This overwrites any existing conan configuration. + Thus you should set your ``CONAN_USER_HOME`` to create a separate conan home folder for testing. + +:: + + $ conan config install https://github.com/includeos/conan_config.git + +If you prefer to set up conan manually the full configuration can be found in the [conan_config](https://github.com/includeos/conan_config.git)-repository. + +.. _Build mac: + +Building the kernel on Mac +-------------------------- + +To build IncludeOS on Linux for Linux with clang-6.0 you can use the profile named +``clang-6.0-macos-x86_64``: diff --git a/More-information.rst b/More-information.rst index 08ad3d6..5869f57 100644 --- a/More-information.rst +++ b/More-information.rst @@ -11,6 +11,29 @@ More information `IncludeOS on Facebook `__ +FLOSSUK +~~~~~~~ + +IncludeOS - Per Buer +-------------------- +.. raw:: html + +
+ +
+ +NDC Conference 2018 +~~~~~~~~~~~~~~~~~~~ + +IncludeOS - running C++ applications without an operating system - Per Buer +--------------------------------------------------------------------------- +.. raw:: html + +
+ +
+ + CppCon September 2017 ~~~~~~~~~~~~~~~~~~~~~ @@ -72,4 +95,4 @@ CppCast July 14, 2016 ~~~~~~~~~~~~~~~~~~~~~ `IncludeOS with Alfred Bratterud `__ ----------------------------------------------------------------------------------- \ No newline at end of file +---------------------------------------------------------------------------------- diff --git a/README.rst b/README.rst index 7fea5e1..e80ab57 100644 --- a/README.rst +++ b/README.rst @@ -7,7 +7,7 @@ Hosted by `Read the Docs `__ $ git clone git@github.com:includeos/documentation.git $ cd documentation - $ pip install sphinx sphinx-autobuild + $ pip install sphinx sphinx-autobuild sphinx_rtd_theme $ $ make html $ open _build/html/.html // to test your changes diff --git a/Tools.rst b/Tools.rst new file mode 100644 index 0000000..e5732e5 --- /dev/null +++ b/Tools.rst @@ -0,0 +1,101 @@ +.. _Tools: + +Tools, Libraries and Dependencies +================================= + +Here we have a list of tools, libraries and other external dependencies used by +IncludeOS. Some are developed by IncludeOS and others are third party dependencies. +Note that for some third party dependencies, IncludeOS has their own fork to make +custom changes needed for the development of IncludeOS. + +Tools +~~~~~ + +The tools and libraries developed by IncludeOS is listed here. + +`Diskbuilder `__ +----------------------------------------------------------- + +`Vmbuild `__ +--------------------------------------------------- + +`Vmrunner `__ +----------------------------------------------------- + +.. tools_NaCl: + +:ref:`NaCl ` +------------------ + +Libraries +~~~~~~~~~ + +Libraries developed by IncludeOS are listed here. +You will also find a list of achieved libraries below. + +`MicroLB `__ +-------------------------------------------------- + +`Mana `__ +-------------------------------------------- + +`Mender `__ +------------------------------------------------ + +`Bucket `__ +------------------------------------------------ + +Archived Libraries +------------------ + +We have a number of libraries here that were developed by us, but are no longer +maintained. Feel free to browse if you are interested in any of them. + +`URI `__ + +`HTTP `__ + +`Dashboard `__ + +`Cookie `__ + +`Path to Regex `__ + +`JSON `__ + +`Director `__ + +`Butler `__ + +Dependencies +~~~~~~~~~~~~ + +Below are a list of external Dependencies used by IncludeOS kernel. + +- Binutils + +- GSL + +- `Botan `__ + +- Http-parser + +- Lest + +- Libgcc + +- Llvm + +- `Musl `__ + +- Openssl + +- Protobuf + +- Rapidjson + +- S2n + +- `Solo5 `_ + +- Uzlib diff --git a/conf.py b/conf.py index 8e9682f..a741487 100644 --- a/conf.py +++ b/conf.py @@ -54,9 +54,9 @@ # built documents. # # The short X.Y version. -version = u'0.9.4' +version = u'0.15.0' # The full version, including alpha/beta/rc tags. -release = u'0.9.4' +release = u'0.15.0' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. @@ -82,7 +82,8 @@ # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. # -html_theme = "default" +html_theme = "sphinx_rtd_theme" +#html_theme = "default" #html_theme = "classic" #html_theme_options = { #"rightsidebar": "true", @@ -160,6 +161,3 @@ def setup(app): author, 'IncludeOS', 'One line description of project.', 'Miscellaneous'), ] - - - diff --git a/index.rst b/index.rst index 2cb0963..8064f44 100644 --- a/index.rst +++ b/index.rst @@ -6,19 +6,19 @@ IncludeOS For more information: Visit our `website `__, check out the project on `GitHub `__ or come chat with us on `Slack `__! .. toctree:: + :hidden: :maxdepth: 2 - :caption: Contents: + Introduction + Deeper-understanding Getting-started - Contributing-to-IncludeOS - Features - Examples-and-libraries + Conanigans NaCl - Using-memdisks Security + Contributing-to-IncludeOS + Jenkins-CI More-information FAQ - Deeper-understanding Publications Roadmap Acknowledgements From cfd7044155ace09c702210a8e0b519ed8d321476 Mon Sep 17 00:00:00 2001 From: taiyeba Date: Fri, 5 Apr 2019 08:03:54 +0200 Subject: [PATCH 02/15] More-info: fixing embedded links to youtube videos --- More-information.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/More-information.rst b/More-information.rst index 5869f57..80deabe 100644 --- a/More-information.rst +++ b/More-information.rst @@ -19,7 +19,7 @@ IncludeOS - Per Buer .. raw:: html
- +
NDC Conference 2018 @@ -30,7 +30,7 @@ IncludeOS - running C++ applications without an operating system - Per Buer .. raw:: html
- +
From cecd854d6a279a423b47ae31e5b249975df7c406 Mon Sep 17 00:00:00 2001 From: Taiyeba Date: Thu, 9 May 2019 15:37:35 +0200 Subject: [PATCH 03/15] Cleaning: cleared out incomplete parts --- Howto-Kernel-dev.rst | 14 -------------- Linux-guide.rst | 8 -------- Macos-guide.rst | 8 -------- More-information.rst | 2 +- 4 files changed, 1 insertion(+), 31 deletions(-) diff --git a/Howto-Kernel-dev.rst b/Howto-Kernel-dev.rst index eb80854..27891a9 100644 --- a/Howto-Kernel-dev.rst +++ b/Howto-Kernel-dev.rst @@ -34,17 +34,3 @@ Cross-Platform Building To build IncludeOS for MacOS on Linux, you can use the ``clang-6.0-macos-x86_64`` profile. - - -Kernel Development ------------------- - - -Using our Packages ------------------- - - - - -Uploading your Packages ------------------------ diff --git a/Linux-guide.rst b/Linux-guide.rst index 2dcac88..9bb4bc6 100644 --- a/Linux-guide.rst +++ b/Linux-guide.rst @@ -43,11 +43,3 @@ We have set up a repository that will configure your local conan to use our buil $ conan config install https://github.com/includeos/conan_config.git If you prefer to set up conan manually the full configuration can be found in the [conan_config](https://github.com/includeos/conan_config.git)-repository. - -.. _Build linux: - -Building the kernel on Linux ----------------------------- - -To build IncludeOS on Linux for Linux you can use the profile named -``clang-6.0-linux-x86_64`` or ``gcc-7.3.0-linux-x86_64`` : diff --git a/Macos-guide.rst b/Macos-guide.rst index 7097a21..ab5441c 100644 --- a/Macos-guide.rst +++ b/Macos-guide.rst @@ -43,11 +43,3 @@ We have set up a repository that will configure your local conan to use our buil $ conan config install https://github.com/includeos/conan_config.git If you prefer to set up conan manually the full configuration can be found in the [conan_config](https://github.com/includeos/conan_config.git)-repository. - -.. _Build mac: - -Building the kernel on Mac --------------------------- - -To build IncludeOS on Linux for Linux with clang-6.0 you can use the profile named -``clang-6.0-macos-x86_64``: diff --git a/More-information.rst b/More-information.rst index 80deabe..39fd864 100644 --- a/More-information.rst +++ b/More-information.rst @@ -30,7 +30,7 @@ IncludeOS - running C++ applications without an operating system - Per Buer .. raw:: html
- +
From c4da25b2909352fa0c8b0d4c7cf6b24200a87761 Mon Sep 17 00:00:00 2001 From: Taiyeba Date: Thu, 9 May 2019 16:08:14 +0200 Subject: [PATCH 04/15] Formatting and includeos repo update: hioa-cs -> includeos --- Acknowledgements.rst | 2 +- Conan-configs.rst | 2 +- Contributing-to-IncludeOS.rst | 4 ++-- Howto-Service-dev.rst | 8 +++++--- Jenkins-CI.rst | 9 +++++---- Linux-guide.rst | 4 ++-- Macos-guide.rst | 8 ++++---- Publications.rst | 2 +- The-boot-process.rst | 10 +++++----- The-build-process.rst | 12 ++++++------ Using-memdisks.rst | 2 +- index.rst | 2 +- 12 files changed, 34 insertions(+), 31 deletions(-) diff --git a/Acknowledgements.rst b/Acknowledgements.rst index e983043..727bb30 100644 --- a/Acknowledgements.rst +++ b/Acknowledgements.rst @@ -26,7 +26,7 @@ We've also received continuous support from the Dept. of Computer Science, which Contributors ~~~~~~~~~~~~ -The `GitHub contributors list `__ speaks for itself. Alf André Walla (fwsGonzo) probably deserves a medal. +The `GitHub contributors list `__ speaks for itself. Alf André Walla (fwsGonzo) probably deserves a medal. Some contributions are not directly visible in code. The `ASN research group `__ has been supporting IncludeOS form the beginning, with everything from equipment, to system administration, to statistical advice and paper reviews. A very special thanks to these guys: diff --git a/Conan-configs.rst b/Conan-configs.rst index b26129b..01c1897 100644 --- a/Conan-configs.rst +++ b/Conan-configs.rst @@ -75,4 +75,4 @@ To add the IncludeOS bintray repository to your conan remotes: :: - $ conan remote add includeos-test https://api.bintray.com/conan/includeos + $ conan remote add includeos https://api.bintray.com/conan/includeos diff --git a/Contributing-to-IncludeOS.rst b/Contributing-to-IncludeOS.rst index 40a11b2..da909ee 100644 --- a/Contributing-to-IncludeOS.rst +++ b/Contributing-to-IncludeOS.rst @@ -33,7 +33,7 @@ If you're making drastic changes to a file, but mostly adding comments, reformat **5. Please don't redo the folder-structure** -If you have suggestions for this, just post an `issue `__ explaining the benefits of your suggested structure. +If you have suggestions for this, just post an `issue `__ explaining the benefits of your suggested structure. Code formatting ~~~~~~~~~~~~~~~ @@ -148,7 +148,7 @@ Code formatting Issue tracker ------------- -Post any issues not already mentioned, in the `issue tracker on GitHub `__. You can also post questions not answered by editing the :ref:`FAQ` on GitHub. +Post any issues not already mentioned, in the `issue tracker on GitHub `__. You can also post questions not answered by editing the :ref:`FAQ` on GitHub. Chat on Slack ------------- diff --git a/Howto-Service-dev.rst b/Howto-Service-dev.rst index bcdea69..433cb0c 100644 --- a/Howto-Service-dev.rst +++ b/Howto-Service-dev.rst @@ -97,17 +97,19 @@ To build more examples follow the `README `__ example. + - Then start implementing the ``Service::start`` function in the ``Service`` class, located in ``/service.cpp`` (very simple example provided). This function will be called once the OS is up and running. + - Update the ``CMakeLists.txt`` to specify the name of your project, enable any needed drivers or plugins, etc. + - Update the ``conanfile.txt`` to specify any build requirements and generators. You should then be able to run your service in the same way as the demo_service. -To add a new service to our demo examples, make a PR to our `demo-examples `__ repo. Make sure to +To add a new service to our demo examples, make a PR to our `examples `__ repo. Make sure to add a README in your example folder with description of your service. diff --git a/Jenkins-CI.rst b/Jenkins-CI.rst index 23e7dd4..70edfa1 100644 --- a/Jenkins-CI.rst +++ b/Jenkins-CI.rst @@ -10,8 +10,8 @@ IncludeOS Github Organization on our Jenkins server. Jenkins CI ~~~~~~~~~~ -`Jenkins-public.includeos.org `__ --------------------------------------------------------------------------------------- +Build Node: `Jenkins-public.includeos.org `__ +--------------------------------------------------------------------------------------------------- Personal builds on the Jenkins server @@ -24,7 +24,7 @@ this procedure will show you what steps to go through. - Will look for the repo: ``https://github.com//IncludeOS`` -- Does not merge with upstream dev automatically unless approved by an IncludeOS developer. +- Does not merge with upstream automatically unless approved by an IncludeOS developer. Setting up the Webhook @@ -53,4 +53,5 @@ Then when the tests are complete, the results will be available on `jenkins-publ Travis CI ~~~~~~~~~ -We have recently started using `Travis CI `__ to build and upload some of our dependencies for macos. +We have recently started using `Travis CI `__ to build and upload some of our dependencies for macos. We are also building and +testing deploy for our `homebrew-includeos` project. diff --git a/Linux-guide.rst b/Linux-guide.rst index 9bb4bc6..708d827 100644 --- a/Linux-guide.rst +++ b/Linux-guide.rst @@ -30,7 +30,7 @@ You can use the recipe below to get all the dependencies in place. $ pip3 install setuptools wheel conan psutil jsonschema -*Getting Conan remotes and profiles* +**Getting Conan remotes and profiles** We have set up a repository that will configure your local conan to use our build profiles and remote repositories. @@ -42,4 +42,4 @@ We have set up a repository that will configure your local conan to use our buil $ conan config install https://github.com/includeos/conan_config.git -If you prefer to set up conan manually the full configuration can be found in the [conan_config](https://github.com/includeos/conan_config.git)-repository. +If you prefer to set up conan manually the full configuration can be found in the `conan_config `__ repository. diff --git a/Macos-guide.rst b/Macos-guide.rst index ab5441c..463e816 100644 --- a/Macos-guide.rst +++ b/Macos-guide.rst @@ -3,7 +3,7 @@ MacOS Guide =========== -To start building on MacOS make sure to have a working installation of [brew](https://brew.sh/) to be able to install all dependencies. +To start building on MacOS make sure to have a working installation of `brew `__ to be able to install all dependencies. Dependencies ------------ @@ -20,7 +20,7 @@ To boot VMs locally you will also need: * python3 * python packages: psutil, jsonschema -Once you have [homebrew](https://brew.sh/) you can use our ``brew tap`` to install the dependencies as follows. +Once you have `homebrew `__ you can use our ``brew tap`` to install the dependencies as follows. :: @@ -30,7 +30,7 @@ Once you have [homebrew](https://brew.sh/) you can use our ``brew tap`` to insta Checkout our `homebrew formula for IncludeOS `__ -*Getting Conan remotes and profiles* +**Getting Conan remotes and profiles** We have set up a repository that will configure your local conan to use our build profiles and remote repositories. @@ -42,4 +42,4 @@ We have set up a repository that will configure your local conan to use our buil $ conan config install https://github.com/includeos/conan_config.git -If you prefer to set up conan manually the full configuration can be found in the [conan_config](https://github.com/includeos/conan_config.git)-repository. +If you prefer to set up conan manually the full configuration can be found in the `conan_config `__ repository. diff --git a/Publications.rst b/Publications.rst index a31a363..053cee1 100644 --- a/Publications.rst +++ b/Publications.rst @@ -6,7 +6,7 @@ Publications Initial IncludeOS paper, 2015 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- Bratterud, A.; Walla, A.; Haugerud, H.; Engelstad; P.E., Begnum, K., "`IncludeOS: a resource efficient unikernel for cloud services `__" in *Cloud Computing Technology and Science (CloudCom), 2015 IEEE 7th International Conference on* +- Bratterud, A.; Walla, A.; Haugerud, H.; Engelstad; P.E., Begnum, K., "`IncludeOS: a resource efficient unikernel for cloud services `__" in *Cloud Computing Technology and Science (CloudCom), 2015 IEEE 7th International Conference on* Motivating paper, 2013 ~~~~~~~~~~~~~~~~~~~~~~ diff --git a/The-boot-process.rst b/The-boot-process.rst index fb5833e..180a8d1 100644 --- a/The-boot-process.rst +++ b/The-boot-process.rst @@ -3,10 +3,10 @@ The x86 boot process ==================== -1. When booting from a "hard drive", BIOS loads the first stage bootloader, either grub or `bootloader.asm `__, starting at ``_start``. +1. When booting from a "hard drive", BIOS loads the first stage bootloader, either grub or `bootloader.asm `__, starting at ``_start``. 2. The bootloader - or Qemu with ``-kernel`` - sets up segments, switches to 32-bit protected mode, loads the service (an elf-binary ``your_service`` consisting of the OS classes, libraries and your service) from disk. For a multiboot compliant boot system (grub or qemu -kernel) the machine is now in the state `specified by multiboot `__. -3. The bootloader hands over control to the OS, by jumping to the ``_start`` symbol inside `start.asm `__. From there it will call architecture specific initialization and eventually `kernel_start.cpp `__. Note that this can be overridden to make custom kernels, such as the minimal `x86_nano `__ platform used for the chainloader. -4. The OS initializes ``.bss``, calls global constructors, and then calls ``OS::start`` in the `OS class `__. +3. The bootloader hands over control to the OS, by jumping to the ``_start`` symbol inside `start.asm `__. From there it will call architecture specific initialization and eventually `kernel_start.cpp `__. Note that this can be overridden to make custom kernels, such as the minimal `x86_nano `__ platform used for the chainloader. +4. The OS initializes ``.bss``, calls global constructors, and then calls ``OS::start`` in the `OS class `__. 5. The OS class sets up interrupts, initializes devices, plugins, drivers etc. etc. -6. Finally the OS class (still ``OS::start``) calls ``Service::start()`` (as for instance `here `__) or ``main()`` if you prefer that (such as `here `__), either of which must be provided by your service. -7. Once your service is done initializing, e.g. having indirectly subscribed to certain events like incoming network packets by setting up a HTTP server, the OS resumes the `OS::event_loop() `__ which again drives your service. +6. Finally the OS class (still ``OS::start``) calls ``Service::start()`` (as for instance `here `__) or ``main()`` if you prefer that (such as `here `__), either of which must be provided by your service. +7. Once your service is done initializing, e.g. having indirectly subscribed to certain events like incoming network packets by setting up a HTTP server, the OS resumes the `OS::event_loop() `__ which again drives your service. diff --git a/The-build-process.rst b/The-build-process.rst index cb87e02..4333819 100644 --- a/The-build-process.rst +++ b/The-build-process.rst @@ -3,18 +3,18 @@ The build process ================= -1. Installing IncludeOS means building all the OS components, such as `IRQ manager `__, `PCI manager `__, the OS class etc., combining them into a static library ``os.a`` using GNU ``ar``, and putting it in an architecture specific directory under ``$INCLUDEOS_PREFIX`` along with all the public os-headers (the "IncludeOS API"). This is what you'll be including parts of, into the service. Device drivers are built as their own libraries, and must be `explicitly added `__ in the CMakeLists.txt of your service. This makes it possible to only include the drivers you want, while still not having to explicitly mention a particular driver in your code. +1. Installing IncludeOS means building all the OS components, such as `IRQ manager `__, `PCI manager `__, the OS class etc., combining them into a static library ``os.a`` using GNU ``ar``, and putting it in an architecture specific directory under ``$INCLUDEOS_PREFIX`` along with all the public os-headers (the "IncludeOS API"). This is what you'll be including parts of, into the service. Device drivers are built as their own libraries, and must be `explicitly added `__ in the CMakeLists.txt of your service. This makes it possible to only include the drivers you want, while still not having to explicitly mention a particular driver in your code. -2. When the service gets built it will turn into object files, which eventually gets statically linked with the os-library, drivers, plugins etc. It will also get linked with the pre-built standard libraries (:code:`libc.a`, :code:`libc++.a` etc.) which we provide as a downloadable bundle, pre-built using `this script `__. Only the objects actually needed by the service will be linked, turning it all into one minimal elf-binary, :code:`your_service`, with OS included. +2. When the service gets built it will turn into object files, which eventually gets statically linked with the os-library, drivers, plugins etc. It will also get linked with the pre-built standard libraries (:code:`libc.a`, :code:`libc++.a` etc.) which we provide as a downloadable bundle, pre-built using `this script `__. Only the objects actually needed by the service will be linked, turning it all into one minimal elf-binary, :code:`your_service`, with OS included. 3. This binary contains a multiboot header, which has all the information the bootloader needs to boot it. This gives you a few options for booting, all available through the simple :code:`boot` tool that comes with IncludeOS: - - **Qemu kernel option**: For 32-bit ELF binaries qemu can load it directly without a bootloader, provided a correct multiboot header. This is what ``boot `` will do by default. The boot tool will generate something like ``$ qemu_system_x86_64 -kernel your_service ...``, which will boot your service directly. Adding ``-nographic`` will make the serial port output appear in your terminal. For 64-bit ELF binaries Qemu has a paranoid check that prevents this, so we're using a 32-bit IncludeOS as `chainloader `__ for that. If ``boot `` detects a 64-bit ELF it will use the 32-bit chainloader as ``-kernel``, and add the 64 bit binary as a "kernel module", e.g. ``-initrd ``. The chainloader will copy the 64-bit binary to the appropriate location in memory, modify the multiboot info provided by the bootloader to the kernel, and jump to the new kernel, which boots as if loaded directly by e.g. GRUB. + - **Qemu kernel option**: For 32-bit ELF binaries qemu can load it directly without a bootloader, provided a correct multiboot header. This is what ``boot `` will do by default. The boot tool will generate something like ``$ qemu_system_x86_64 -kernel your_service ...``, which will boot your service directly. Adding ``-nographic`` will make the serial port output appear in your terminal. For 64-bit ELF binaries Qemu has a paranoid check that prevents this, so we're using a 32-bit IncludeOS as `chainloader `__ for that. If ``boot `` detects a 64-bit ELF it will use the 32-bit chainloader as ``-kernel``, and add the 64 bit binary as a "kernel module", e.g. ``-initrd ``. The chainloader will copy the 64-bit binary to the appropriate location in memory, modify the multiboot info provided by the bootloader to the kernel, and jump to the new kernel, which boots as if loaded directly by e.g. GRUB. - - **Legacy**: Attach our own minimal bootloader, using the utility `vmbuild `__. It combines our minimal bootloader and ``your_service`` binary into a disk image called ``your_service.img``. At this point the bootloader gets the size and location of the service hardcoded into it. The major drawback of using this bootloader is that it doesn't fetch information about system memory from the BIOS so you can't know exactly how much memory you have, above 65Mb. (Which CMOS can provide) + - **Legacy**: Attach our own minimal bootloader, using the utility `vmbuild `__. It combines our minimal bootloader and ``your_service`` binary into a disk image called ``your_service.img``. At this point the bootloader gets the size and location of the service hardcoded into it. The major drawback of using this bootloader is that it doesn't fetch information about system memory from the BIOS so you can't know exactly how much memory you have, above 65Mb. (Which CMOS can provide) - **Grub**: Embed the binary into a GRUB filesystem, and have the Grub chainloader boot it for you. This is what we're doing when `booting on Google Compute Engine `__. You can do this on Linux using ``boot -g ``, which will produce a bootable ``your_service.grub.img``. Note that GRUB is larger than IncludeOS itself, so expect a few megabytes added to the image size. -4. To run with vmware or virtualbox, the image has to be converted into a supported format, such as ``vdi`` or ``vmdk``. This is easily done in one command with the ``qemu-img``-tool, that comes with Qemu. We have a `script for that too `__. Detailed information about booting in vmware, which is as easy as ``boot``, is `provided here `__. +4. To run with vmware or virtualbox, the image has to be converted into a supported format, such as ``vdi`` or ``vmdk``. This is easily done in one command with the ``qemu-img``-tool, that comes with Qemu. We have a `script for that too `__. Detailed information about booting in vmware, which is as easy as ``boot``, is `provided here `__. -Inspect the main `CMakeLists.txt `__ and then follow the trail of cmake scripts in the added subfolders for information about how the OS build happens. For more information about building individual services, check out the `CMakeLists.txt `__ of one of the example services, plus the linker script, `linker.ld `__ for the layout of the final binary. Note that most of the CMake magic for link- and include paths, adding drivers, plugins etc. is tucked away in the `post.service.cmake `__. +Inspect the main `CMakeLists.txt `__ and then follow the trail of cmake scripts in the added subfolders for information about how the OS build happens. For more information about building individual services, check out the `CMakeLists.txt `__ of one of the example services, plus the linker script, `linker.ld `__ for the layout of the final binary. Note that most of the CMake magic for link- and include paths, adding drivers, plugins etc. is tucked away in the `post.service.cmake `__. diff --git a/Using-memdisks.rst b/Using-memdisks.rst index c940737..19e6796 100644 --- a/Using-memdisks.rst +++ b/Using-memdisks.rst @@ -42,7 +42,7 @@ Now, the files and folders from your memdisk are available using normal C/C++ fi Traditional C-style file functions (``fopen()``, ``fgets()``, ``fread()`` and friends) as well as POSIX functions (``open()``, ``read()`` etc.) are also available. -It's also possible to use memdisks without mounting them in the Virtual File Systems and to use the native IncludeOS file system functionality, see `this example `__ for more information. +It's also possible to use memdisks without mounting them in the Virtual File Systems and to use the native IncludeOS file system functionality, see `this example `__ for more information. Addendum: ``memdisk()`` helper function: diff --git a/index.rst b/index.rst index 8064f44..0fd697a 100644 --- a/index.rst +++ b/index.rst @@ -3,7 +3,7 @@ IncludeOS **IncludeOS** is an includeable, minimal library operating system for C++ services running in the cloud or on devices. Starting a program with ``#include ``, will literally include a whole little operating system into your service during link-time. -For more information: Visit our `website `__, check out the project on `GitHub `__ or come chat with us on `Slack `__! +For more information: Visit our `website `__, check out the project on `GitHub `__ or come chat with us on `Slack `__! .. toctree:: :hidden: From e5ba6ba9f73b5261095dcfa2a65fef36a296d068 Mon Sep 17 00:00:00 2001 From: Martin Nordsletten Date: Mon, 13 May 2019 15:20:12 +0200 Subject: [PATCH 05/15] Cleaning: Removed unused css function --- _templates/layout.html | 2 -- conf.py | 2 +- 2 files changed, 1 insertion(+), 3 deletions(-) delete mode 100644 _templates/layout.html diff --git a/_templates/layout.html b/_templates/layout.html deleted file mode 100644 index 1e4b607..0000000 --- a/_templates/layout.html +++ /dev/null @@ -1,2 +0,0 @@ -{% extends "!layout.html" %} -{% set css_files = css_files + [ "_static/css/custom_style.css" ] %} \ No newline at end of file diff --git a/conf.py b/conf.py index a741487..f44662e 100644 --- a/conf.py +++ b/conf.py @@ -68,7 +68,7 @@ # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. # This patterns also effect to html_static_path and html_extra_path -exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] +exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store', 'README.rst'] # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'sphinx' From 62256d487c13b8f9b3181e5c2072027dfd812537 Mon Sep 17 00:00:00 2001 From: Martin Nordsletten Date: Mon, 13 May 2019 15:20:39 +0200 Subject: [PATCH 06/15] Restructure: Moved files into folders + 3 main categories --- Jenkins-CI.rst | 57 ------------------- Examples.rst => getting_started/Examples.rst | 0 Features.rst => getting_started/Features.rst | 0 .../Getting-started.rst | 2 +- .../Howto-Kernel-dev.rst | 0 .../Howto-Service-dev.rst | 0 getting_started/Install.rst | 4 ++ .../Introduction.rst | 0 .../Linux-guide.rst | 0 .../Macos-guide.rst | 0 Tools.rst => getting_started/Tools.rst | 0 .../Using-memdisks.rst | 0 .../Conan-configs.rst | 0 .../Conan-package-build.rst | 0 Conanigans.rst => in_depth/Conanigans.rst | 0 .../Contributing-to-IncludeOS.rst | 0 .../Deeper-understanding.rst | 0 .../Editable-mode.rst | 0 .../Fun-with-Guns-and-Knives.rst | 0 NaCl.rst => in_depth/NaCl.rst | 0 .../The-boot-process.rst | 0 .../The-build-process.rst | 0 index.rst | 47 +++++++++------ .../Acknowledgements.rst | 0 FAQ.rst => more_info/FAQ.rst | 0 .../More-information.rst | 0 .../Publications.rst | 0 Roadmap.rst => more_info/Roadmap.rst | 0 Security.rst => more_info/Security.rst | 2 +- 29 files changed, 35 insertions(+), 77 deletions(-) delete mode 100644 Jenkins-CI.rst rename Examples.rst => getting_started/Examples.rst (100%) rename Features.rst => getting_started/Features.rst (100%) rename Getting-started.rst => getting_started/Getting-started.rst (99%) rename Howto-Kernel-dev.rst => getting_started/Howto-Kernel-dev.rst (100%) rename Howto-Service-dev.rst => getting_started/Howto-Service-dev.rst (100%) create mode 100644 getting_started/Install.rst rename Introduction.rst => getting_started/Introduction.rst (100%) rename Linux-guide.rst => getting_started/Linux-guide.rst (100%) rename Macos-guide.rst => getting_started/Macos-guide.rst (100%) rename Tools.rst => getting_started/Tools.rst (100%) rename Using-memdisks.rst => getting_started/Using-memdisks.rst (100%) rename Conan-configs.rst => in_depth/Conan-configs.rst (100%) rename Conan-package-build.rst => in_depth/Conan-package-build.rst (100%) rename Conanigans.rst => in_depth/Conanigans.rst (100%) rename Contributing-to-IncludeOS.rst => in_depth/Contributing-to-IncludeOS.rst (100%) rename Deeper-understanding.rst => in_depth/Deeper-understanding.rst (100%) rename Editable-mode.rst => in_depth/Editable-mode.rst (100%) rename Fun-with-Guns-and-Knives.rst => in_depth/Fun-with-Guns-and-Knives.rst (100%) rename NaCl.rst => in_depth/NaCl.rst (100%) rename The-boot-process.rst => in_depth/The-boot-process.rst (100%) rename The-build-process.rst => in_depth/The-build-process.rst (100%) rename Acknowledgements.rst => more_info/Acknowledgements.rst (100%) rename FAQ.rst => more_info/FAQ.rst (100%) rename More-information.rst => more_info/More-information.rst (100%) rename Publications.rst => more_info/Publications.rst (100%) rename Roadmap.rst => more_info/Roadmap.rst (100%) rename Security.rst => more_info/Security.rst (51%) diff --git a/Jenkins-CI.rst b/Jenkins-CI.rst deleted file mode 100644 index 70edfa1..0000000 --- a/Jenkins-CI.rst +++ /dev/null @@ -1,57 +0,0 @@ -.. _Jenkins CI: - -CI Pipeline for Developers -========================== - -We use continuous integration to build and test our projects on a regular basis. -You will find all the repositories currently using CI under the -IncludeOS Github Organization on our Jenkins server. - -Jenkins CI -~~~~~~~~~~ - -Build Node: `Jenkins-public.includeos.org `__ ---------------------------------------------------------------------------------------------------- - - -Personal builds on the Jenkins server -------------------------------------- - -If you want to get your personal fork of IncludeOS to build with every commit -this procedure will show you what steps to go through. - -**Things to take note off:** - -- Will look for the repo: ``https://github.com//IncludeOS`` - -- Does not merge with upstream automatically unless approved by an IncludeOS developer. - - -Setting up the Webhook ----------------------- - -Follow these steps to get it to work: - -1. Go to the settings page for your **personal fork** - -|Settings menu| - -2. Navigate to the *Webhooks & Services* section and press the *Add webhook* button. Then enter the following url into the *Payload URL* section ``https://jenkins.includeos.org/github-webhook/``. Then press *Add webhook* - -|Payload URL| - -3. To make sure this works, go back to the webhooks page and make sure you see the green checkmark next to the url. This might take a few seconds, so refresh the page. - -|Checkmark| - -Then when the tests are complete, the results will be available on `jenkins-public.includeos.org `__ - -.. |Settings menu| image:: http://i.imgur.com/wfoYcaD.png -.. |Payload URL| image:: http://i.imgur.com/g0gEcBq.png -.. |Checkmark| image:: http://i.imgur.com/yUTIwZ1.png?1 - -Travis CI -~~~~~~~~~ - -We have recently started using `Travis CI `__ to build and upload some of our dependencies for macos. We are also building and -testing deploy for our `homebrew-includeos` project. diff --git a/Examples.rst b/getting_started/Examples.rst similarity index 100% rename from Examples.rst rename to getting_started/Examples.rst diff --git a/Features.rst b/getting_started/Features.rst similarity index 100% rename from Features.rst rename to getting_started/Features.rst diff --git a/Getting-started.rst b/getting_started/Getting-started.rst similarity index 99% rename from Getting-started.rst rename to getting_started/Getting-started.rst index 0d3539d..53936bc 100644 --- a/Getting-started.rst +++ b/getting_started/Getting-started.rst @@ -1,4 +1,4 @@ -.. _Getting started: +.. _in depth install: Getting started with IncludeOS ============================== diff --git a/Howto-Kernel-dev.rst b/getting_started/Howto-Kernel-dev.rst similarity index 100% rename from Howto-Kernel-dev.rst rename to getting_started/Howto-Kernel-dev.rst diff --git a/Howto-Service-dev.rst b/getting_started/Howto-Service-dev.rst similarity index 100% rename from Howto-Service-dev.rst rename to getting_started/Howto-Service-dev.rst diff --git a/getting_started/Install.rst b/getting_started/Install.rst new file mode 100644 index 0000000..ab56f4f --- /dev/null +++ b/getting_started/Install.rst @@ -0,0 +1,4 @@ +.. _Install: + +Install +======= diff --git a/Introduction.rst b/getting_started/Introduction.rst similarity index 100% rename from Introduction.rst rename to getting_started/Introduction.rst diff --git a/Linux-guide.rst b/getting_started/Linux-guide.rst similarity index 100% rename from Linux-guide.rst rename to getting_started/Linux-guide.rst diff --git a/Macos-guide.rst b/getting_started/Macos-guide.rst similarity index 100% rename from Macos-guide.rst rename to getting_started/Macos-guide.rst diff --git a/Tools.rst b/getting_started/Tools.rst similarity index 100% rename from Tools.rst rename to getting_started/Tools.rst diff --git a/Using-memdisks.rst b/getting_started/Using-memdisks.rst similarity index 100% rename from Using-memdisks.rst rename to getting_started/Using-memdisks.rst diff --git a/Conan-configs.rst b/in_depth/Conan-configs.rst similarity index 100% rename from Conan-configs.rst rename to in_depth/Conan-configs.rst diff --git a/Conan-package-build.rst b/in_depth/Conan-package-build.rst similarity index 100% rename from Conan-package-build.rst rename to in_depth/Conan-package-build.rst diff --git a/Conanigans.rst b/in_depth/Conanigans.rst similarity index 100% rename from Conanigans.rst rename to in_depth/Conanigans.rst diff --git a/Contributing-to-IncludeOS.rst b/in_depth/Contributing-to-IncludeOS.rst similarity index 100% rename from Contributing-to-IncludeOS.rst rename to in_depth/Contributing-to-IncludeOS.rst diff --git a/Deeper-understanding.rst b/in_depth/Deeper-understanding.rst similarity index 100% rename from Deeper-understanding.rst rename to in_depth/Deeper-understanding.rst diff --git a/Editable-mode.rst b/in_depth/Editable-mode.rst similarity index 100% rename from Editable-mode.rst rename to in_depth/Editable-mode.rst diff --git a/Fun-with-Guns-and-Knives.rst b/in_depth/Fun-with-Guns-and-Knives.rst similarity index 100% rename from Fun-with-Guns-and-Knives.rst rename to in_depth/Fun-with-Guns-and-Knives.rst diff --git a/NaCl.rst b/in_depth/NaCl.rst similarity index 100% rename from NaCl.rst rename to in_depth/NaCl.rst diff --git a/The-boot-process.rst b/in_depth/The-boot-process.rst similarity index 100% rename from The-boot-process.rst rename to in_depth/The-boot-process.rst diff --git a/The-build-process.rst b/in_depth/The-build-process.rst similarity index 100% rename from The-build-process.rst rename to in_depth/The-build-process.rst diff --git a/index.rst b/index.rst index 0fd697a..133c696 100644 --- a/index.rst +++ b/index.rst @@ -1,24 +1,35 @@ IncludeOS ========= -**IncludeOS** is an includeable, minimal library operating system for C++ services running in the cloud or on devices. Starting a program with ``#include ``, will literally include a whole little operating system into your service during link-time. +**IncludeOS** is an includable, minimal library operating system for C++ services running in the cloud or on devices. Starting a program with ``#include ``, will literally include a whole operating system into your service during link-time. -For more information: Visit our `website `__, check out the project on `GitHub `__ or come chat with us on `Slack `__! +For more information: Visit our `website `__, check out the project on `GitHub `__ or come chat with us on `Slack `__! .. toctree:: - :hidden: - :maxdepth: 2 - - Introduction - Deeper-understanding - Getting-started - Conanigans - NaCl - Security - Contributing-to-IncludeOS - Jenkins-CI - More-information - FAQ - Publications - Roadmap - Acknowledgements + :maxdepth: 2 + :caption: Getting started + + getting_started/Introduction + getting_started/Install + getting_started/Getting-started + + +.. toctree:: + :maxdepth: 2 + :caption: In depth + + in_depth/Deeper-understanding + in_depth/Conanigans + in_depth/NaCl + in_depth/Contributing-to-IncludeOS + +.. toctree:: + :maxdepth: 2 + :caption: More information + + more_info/Security + more_info/FAQ + more_info/More-information + more_info/Publications + more_info/Roadmap + more_info/Acknowledgements diff --git a/Acknowledgements.rst b/more_info/Acknowledgements.rst similarity index 100% rename from Acknowledgements.rst rename to more_info/Acknowledgements.rst diff --git a/FAQ.rst b/more_info/FAQ.rst similarity index 100% rename from FAQ.rst rename to more_info/FAQ.rst diff --git a/More-information.rst b/more_info/More-information.rst similarity index 100% rename from More-information.rst rename to more_info/More-information.rst diff --git a/Publications.rst b/more_info/Publications.rst similarity index 100% rename from Publications.rst rename to more_info/Publications.rst diff --git a/Roadmap.rst b/more_info/Roadmap.rst similarity index 100% rename from Roadmap.rst rename to more_info/Roadmap.rst diff --git a/Security.rst b/more_info/Security.rst similarity index 51% rename from Security.rst rename to more_info/Security.rst index 906b662..b022706 100644 --- a/Security.rst +++ b/more_info/Security.rst @@ -3,4 +3,4 @@ Security ======== -If you have found a security issue in IncludeOS please avoid the public issue tracker. Instead send an email to security@includeos.org. If you would like to encrypt your mail please use `this key `__ also available on most public `PGP key servers `__. +If you have found a security issue in IncludeOS please avoid the public issue tracker. Instead send an email to security@includeos.org. If you would like to encrypt your mail please use `this key <_static/IncludeOS_pgp_public_key.asc>`__ also available on most public `PGP key servers `__. From fc27861ddb5e91f4f211eb2e6f895c6a064d7927 Mon Sep 17 00:00:00 2001 From: Martin Nordsletten Date: Mon, 13 May 2019 16:32:37 +0200 Subject: [PATCH 07/15] Rewrote the getting started section --- getting_started/Developing_includeos.rst | 11 +++ getting_started/Examples.rst | 51 ------------ getting_started/Features.rst | 48 ------------ getting_started/Getting-started.rst | 77 ------------------- getting_started/Hello_world.rst | 38 +++++++++ getting_started/Install.rst | 54 +++++++++++++ getting_started/Introduction.rst | 50 ++++++++++-- getting_started/Linux-guide.rst | 45 ----------- getting_started/Macos-guide.rst | 45 ----------- in_depth/Deeper-understanding.rst | 2 + {getting_started => in_depth}/Tools.rst | 0 .../Using-memdisks.rst | 0 index.rst | 3 +- 13 files changed, 151 insertions(+), 273 deletions(-) create mode 100644 getting_started/Developing_includeos.rst delete mode 100644 getting_started/Examples.rst delete mode 100644 getting_started/Features.rst delete mode 100644 getting_started/Getting-started.rst create mode 100644 getting_started/Hello_world.rst delete mode 100644 getting_started/Linux-guide.rst delete mode 100644 getting_started/Macos-guide.rst rename {getting_started => in_depth}/Tools.rst (100%) rename {getting_started => in_depth}/Using-memdisks.rst (100%) diff --git a/getting_started/Developing_includeos.rst b/getting_started/Developing_includeos.rst new file mode 100644 index 0000000..28ca92c --- /dev/null +++ b/getting_started/Developing_includeos.rst @@ -0,0 +1,11 @@ +.. _developing includeos: + +Developing with IncludeOS +============================== + +Whether you are a kernel or a service developer, below is a list of sections that will guide you through the process of building and starting an IncludeOS service. + +.. toctree:: + + Howto-Kernel-dev + Howto-Service-dev diff --git a/getting_started/Examples.rst b/getting_started/Examples.rst deleted file mode 100644 index 81c11ff..0000000 --- a/getting_started/Examples.rst +++ /dev/null @@ -1,51 +0,0 @@ -.. _Examples: - -Examples -======== - -Here we have a list of all the examples you will find in the `Examples Repository `__ . - -To build an example, follow the steps in the readme. Note that your -build folder can be anywhere, as long as you point to the source recipe during -`conan install -pr `. - - -`256 Color VGA `__ - -`IRCd `__ - -`LiveUpdate `__ - -`STREAM `__ - -`TCP perf `__ - -`TLS server `__ - -`Acorn `__ - -`Demo Service `__ - -`Http Client `__ - -`MicroLB `__ - -`Protpbuf `__ - -`Router `__ - -`Scoped Profiler `__ - -`Snake `__ - -`Starlight `__ - -`Syslog `__ - -`TCP `__ - -`Transfer `__ - -`Vlan `__ - -`Websocket `__ diff --git a/getting_started/Features.rst b/getting_started/Features.rst deleted file mode 100644 index f2f374e..0000000 --- a/getting_started/Features.rst +++ /dev/null @@ -1,48 +0,0 @@ -.. _Features: - -Features -======== - -.. Needs an update - -A non-exhaustive, possibly outdated feature list - -- Low memory footprint -- Support for the following hypervisors: - + Qemu / KVM - + Solo5/hvt - + VMWare ESXi -- **C++11/14 support** - + Full C++11/14/17 language support with `clang `__ v5 and later. - + Standard C++ library\*\* (STL) `libc++ `__ from `LLVM `__ - + Exceptions and stack unwinding (currently using `libgcc `__) -- **Standard C library** using `musl `__ -- **Virtio Network driver** with DMA. `Virtio `__ provides a highly efficient and widely supported I/O virtualization. Like most implementations IncludeOS currently uses "legacy mode", but we're working towards the new `Virtio 1.0 OASIS standard `__ -- **A highly modular TCP/IP-stack** written from scratch - + TCP with a few extensions (SAck, TSVal) - + UDP module - + DHCP and DNS clients that (as far as we know) work on the most common cloud platforms - + ICMP: Send/receive ping and some error handling code - + ARP cache - + An IP <-> Link layer/driver separation layer that will allow future link layers, such as WiFi - + Minimal beginnings on IPv6 support - -- **Completely silent while idling**. As we documented in our `IEEE CloudCom 2013 paper `__, running a regular interval timer for concurrency inside a virtual machine will impose a significant CPU-load on hypervisors running many virtual machines. IncludeOS disables the timer interrupts completely when idle, making it use no CPU at all. This makes IncludeOS services well suited for resource saving through overbooking schemes. - -- **Node.js-style callback-based programming** - everything happens in one efficient thread with no I/O blocking or unnecessary guest-side context switching. - -- **No race conditions**. Delegated IRQ handling makes race conditions in "userspace" "impossible". ...unless you implement threads yourself (you have the access) or we do. - -- **All the guns and all the knives:** - - + IncludeOS services run in ring 0, in a single address space without protection. That's a lot of power to play with. For example: Try ``asm("hlt")`` in a normal userspace program - then try it in IncludeOS. Explain to the duck exactly what's going on ... and it will tell you why Intel made VT-x (Yes IBM was way behind Alan Turing). That's a virtualization gold nugget, in reward of your mischief. If you believe in these kinds of lessons, there's always more :ref:`Fun with Guns and Knives`. - - + *Hold your forces! I and James Gosling strongly object to guns and knives!* - - - For good advice on how not to use these powers, look to the `Wisdom of the Jedi Council `__. - - If you found the gold nugget above, you'll know that the physical CPU protects you from others - and others from you. And that's a pretty solid protection compared to, say, `openssl `__. If you need protection from yourself, that too can be gained by aquiring the 10 000 lines of `Wisdom from the Jedi Council `__, or also from our friends at `Mirage `__ ;-) - - *Are the extra guns and knives really features?* For explorers, yes. For a Joint Strike Fighter autopilot? Noooo. You need `even more wisdom `__ for that. - -.. Limitations: - -If it's not listed under features, chances are that we don't have it yet. \ No newline at end of file diff --git a/getting_started/Getting-started.rst b/getting_started/Getting-started.rst deleted file mode 100644 index 53936bc..0000000 --- a/getting_started/Getting-started.rst +++ /dev/null @@ -1,77 +0,0 @@ -.. _in depth install: - -Getting started with IncludeOS -============================== - -IncludeOS is an open source project and can be found on `Github `__ . -Whether you are a kernel or a service developer using MacOS or Linux, below is a list of sections that will guide you through the process of building and starting an IncludeOS service. - -.. toctree:: - :maxdepth: 1 - - Linux-guide - Macos-guide - Howto-Kernel-dev - Howto-Service-dev - -Hello World with IncludeOS -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -To start of we welcome you to trying a simple traditional Hello World with IncludeOS. -You don't need to install IncludeOS to run a service, so just clone the service -repository. - -:: - - $ git clone https://github.com/includeos/hello_world - $ cd hello_world - -Dependencies ------------- - -To build the service you will need: - -* Cmake, make, nasm -* Clang or GCC -* `Conan `__ - -To boot VMs locally you will need: - -* qemu -* python3 -* python packages: psutil, jsonschema - -The `IncludeOS `__ conan packages are available on -bintray. Therefore you will need to add the remote to your list. Conan requires -the use of a profile during building a package, which can also be installed using -the command below. - -:: - - $ conan config install https://github.com/includeos/conan_config - -.. note:: - For Mac OS ensure that you have a working installation of `brew `__ to be able to install all dependencies. For more specific installation instructions for mac or linux checkout the linux guide or mac os guide listed above. To get help with your conan setup follow the instruction at :ref:`Conan Configuration `. - -Running ``conan profile list`` will show the profiles installed. -- Linux users can typically use ``clang-6.0-linux-x86_64`` -- MacOS users can use ``clang-6.0-macos-x86_64``. -You can also develop your own profiles. - -The following steps let you build and boot the IncludeOS hello world example. - -:: - - $ git clone https://github.com/includeos/hello_world.git - $ mkdir your_build_dir && cd "$_" - $ conan install ../hello_world -pr - $ source activate.sh - $ cmake ../hello_world - $ cmake --build . - $ boot hello - -.. warning:: - Once you're done ``$ source deactivate.sh`` will reset your environment to - its previous state. - -For more advanced examples see the `examples repo `__. diff --git a/getting_started/Hello_world.rst b/getting_started/Hello_world.rst new file mode 100644 index 0000000..0a60d10 --- /dev/null +++ b/getting_started/Hello_world.rst @@ -0,0 +1,38 @@ +.. _hello world: + +Hello World with IncludeOS +========================== + +You don't need to manually install IncludeOS to build a service. Our pipeline builds IncludeOS conan packages and uploads them to `bintray `__. +A service then specifies which version of IncludeOS to use when building. +To get started with the `hello_world` example just clone the service repository: + +:: + + $ git clone https://github.com/includeos/hello_world + $ cd hello_world + $ mkdir your_build_dir && cd "$_" + +Then install all the requirements of the service. You will have to select a conan profile to use. Running ``conan profile list`` will show the profiles installed. +- Linux users can typically use ``clang-6.0-linux-x86_64`` +- MacOS users can use ``clang-6.0-macos-x86_64``. +You can also develop your own profiles. + +:: + + $ conan install ../hello_world -pr + +Then we activate a virtual environment which sets up the build job with the correct paths. Finally we build using cmake and make. This produces an elf binary that is bootable. The included utility ``boot`` can be used to launch in qemu. + +:: + + $ source activate.sh + $ cmake .. + $ cmake --build . + $ boot hello + +.. note:: + Once you're done ``$ source deactivate.sh`` will reset your environment to + its previous state. + +For more advanced examples see the `examples repo `__. diff --git a/getting_started/Install.rst b/getting_started/Install.rst index ab56f4f..0d8e77a 100644 --- a/getting_started/Install.rst +++ b/getting_started/Install.rst @@ -2,3 +2,57 @@ Install ======= + +*Introduction* + +Dependencies +------------ + +For building the kernel you will need: + +* `C/C++ package manager `__ (Conan 1.13.1 or newer) +* cmake, make, nasm +* clang, or alternatively gcc on linux. Prebuilt packages are available for clang 6.0 and gcc 7.3. + +To boot VMs locally you will also need: + +* qemu +* python3 +* python packages: psutil, jsonschema + +Ubuntu installation +------------------- +We are currently developing and testing on ``Ubuntu-18.04``. To get started with +building on Linux, install the following dependencies. + +:: + + $ apt-get install python3-pip python3-dev git cmake clang-6.0 gcc nasm make qemu + $ pip3 install setuptools wheel conan psutil jsonschema + +macOS installation +------------------ +If you have `homebrew `__ you can use our ``brew tap`` to install the dependencies as follows. + +:: + + $ brew tap includeos/includeos + $ brew install includeos + +Checkout our `homebrew formula for IncludeOS `__ + +Setting up Conan +---------------- +**Getting Conan remotes and profiles** + +IncludeOS uses the conan package manager to handle dependencies. To use the profiles and remotes required to use IncludeOS there is an existing conan config repository that can be used. + +.. warning:: + This will overwrite any existing conan configuration. To test without touching existing conan configurations set the environment variable ``CONAN_USER_HOME`` to create a separate conan home folder. + +To install the IncludeOS conan config issue the following command: +:: + + $ conan config install https://github.com/includeos/conan_config.git + +If you prefer to set up conan manually the full configuration can be found in the `conan_config `__ repository. diff --git a/getting_started/Introduction.rst b/getting_started/Introduction.rst index 7f3022c..7dc000e 100644 --- a/getting_started/Introduction.rst +++ b/getting_started/Introduction.rst @@ -17,10 +17,48 @@ It is a minimal tasking library written in C++ which allows users to run applica This literally includes a minimal operating system into the service during link-time. The way this works is that during link time the build system extracts service requirements from the pre-compiled OS-library and forms a single binary. To learn more about our open source project and it's potential uses visit the `technology `__ page over at our website. -.. toctree:: - :hidden: +Features +^^^^^^^^^^ +.. Needs an update - Features - Tools - Examples - Using-memdisks +A non-exhaustive, possibly outdated feature list + +- Low memory footprint +- Support for the following hypervisors: + + Qemu / KVM + + Solo5/hvt + + VMWare ESXi +- **C++11/14 support** + + Full C++11/14/17 language support with `clang `__ v5 and later. + + Standard C++ library\*\* (STL) `libc++ `__ from `LLVM `__ + + Exceptions and stack unwinding (currently using `libgcc `__) +- **Standard C library** using `musl `__ +- **Virtio Network driver** with DMA. `Virtio `__ provides a highly efficient and widely supported I/O virtualization. Like most implementations IncludeOS currently uses "legacy mode", but we're working towards the new `Virtio 1.0 OASIS standard `__ +- **A highly modular TCP/IP-stack** written from scratch + + TCP with a few extensions (SAck, TSVal) + + UDP module + + DHCP and DNS clients that (as far as we know) work on the most common cloud platforms + + ICMP: Send/receive ping and some error handling code + + ARP cache + + An IP <-> Link layer/driver separation layer that will allow future link layers, such as WiFi + + Minimal beginnings on IPv6 support + +- **Completely silent while idling**. As we documented in our `IEEE CloudCom 2013 paper `__, running a regular interval timer for concurrency inside a virtual machine will impose a significant CPU-load on hypervisors running many virtual machines. IncludeOS disables the timer interrupts completely when idle, making it use no CPU at all. This makes IncludeOS services well suited for resource saving through overbooking schemes. + +- **Node.js-style callback-based programming** - everything happens in one efficient thread with no I/O blocking or unnecessary guest-side context switching. + +- **No race conditions**. Delegated IRQ handling makes race conditions in "userspace" "impossible". ...unless you implement threads yourself (you have the access) or we do. + +- **All the guns and all the knives:** + + + IncludeOS services run in ring 0, in a single address space without protection. That's a lot of power to play with. For example: Try ``asm("hlt")`` in a normal userspace program - then try it in IncludeOS. Explain to the duck exactly what's going on ... and it will tell you why Intel made VT-x (Yes IBM was way behind Alan Turing). That's a virtualization gold nugget, in reward of your mischief. If you believe in these kinds of lessons, there's always more :ref:`Fun with Guns and Knives`. + + + *Hold your forces! I and James Gosling strongly object to guns and knives!* + + - For good advice on how not to use these powers, look to the `Wisdom of the Jedi Council `__. + - If you found the gold nugget above, you'll know that the physical CPU protects you from others - and others from you. And that's a pretty solid protection compared to, say, `openssl `__. If you need protection from yourself, that too can be gained by aquiring the 10 000 lines of `Wisdom from the Jedi Council `__, or also from our friends at `Mirage `__ ;-) + - *Are the extra guns and knives really features?* For explorers, yes. For a Joint Strike Fighter autopilot? Noooo. You need `even more wisdom `__ for that. + +.. Limitations: + +If it's not listed under features, chances are that we don't have it yet. diff --git a/getting_started/Linux-guide.rst b/getting_started/Linux-guide.rst deleted file mode 100644 index 708d827..0000000 --- a/getting_started/Linux-guide.rst +++ /dev/null @@ -1,45 +0,0 @@ -.. _Linux guide: - -Linux Guide -=========== - -We are currently developing and testing on ``Ubuntu-18.04``. To get started with -building on Linux, install the following dependencies. - - -Dependencies ------------- - -For building the kernel you will need: - -* `C/C++ package manager `__ (Conan 1.13.1 or newer) -* cmake, make, nasm -* clang, or alternatively gcc on linux. Prebuilt packages are available for clang 6.0 and gcc 7.3. - -To boot VMs locally you will also need: - -* qemu -* python3 -* python packages: psutil, jsonschema - -You can use the recipe below to get all the dependencies in place. - -:: - - $ apt-get install python3-pip python3-dev git cmake clang-6.0 gcc nasm make qemu - $ pip3 install setuptools wheel conan psutil jsonschema - - -**Getting Conan remotes and profiles** - -We have set up a repository that will configure your local conan to use our build profiles and remote repositories. - -.. warning:: - This overwrites any existing conan configuration. - Thus you should set your ``CONAN_USER_HOME`` to create a separate conan home folder for testing. - -:: - - $ conan config install https://github.com/includeos/conan_config.git - -If you prefer to set up conan manually the full configuration can be found in the `conan_config `__ repository. diff --git a/getting_started/Macos-guide.rst b/getting_started/Macos-guide.rst deleted file mode 100644 index 463e816..0000000 --- a/getting_started/Macos-guide.rst +++ /dev/null @@ -1,45 +0,0 @@ -.. _Macos guide: - -MacOS Guide -=========== - -To start building on MacOS make sure to have a working installation of `brew `__ to be able to install all dependencies. - -Dependencies ------------- - -For building the kernel you will need: - -* `C/C++ package manager `__ (Conan 1.13.1 or newer) -* cmake, make, nasm -* clang (Prebuilt packages are available for clang 6.0) - -To boot VMs locally you will also need: - -* qemu -* python3 -* python packages: psutil, jsonschema - -Once you have `homebrew `__ you can use our ``brew tap`` to install the dependencies as follows. - -:: - - $ brew tap includeos/includeos - $ brew install includeos - -Checkout our `homebrew formula for IncludeOS `__ - - -**Getting Conan remotes and profiles** - -We have set up a repository that will configure your local conan to use our build profiles and remote repositories. - -.. warning:: - This overwrites any existing conan configuration. - Thus you should set your ``CONAN_USER_HOME`` to create a separate conan home folder for testing. - -:: - - $ conan config install https://github.com/includeos/conan_config.git - -If you prefer to set up conan manually the full configuration can be found in the `conan_config `__ repository. diff --git a/in_depth/Deeper-understanding.rst b/in_depth/Deeper-understanding.rst index 59fb8ac..684da7a 100644 --- a/in_depth/Deeper-understanding.rst +++ b/in_depth/Deeper-understanding.rst @@ -11,4 +11,6 @@ More about IncludeOS The-build-process The-boot-process + Using-memdisks + Tools Fun-with-Guns-and-Knives diff --git a/getting_started/Tools.rst b/in_depth/Tools.rst similarity index 100% rename from getting_started/Tools.rst rename to in_depth/Tools.rst diff --git a/getting_started/Using-memdisks.rst b/in_depth/Using-memdisks.rst similarity index 100% rename from getting_started/Using-memdisks.rst rename to in_depth/Using-memdisks.rst diff --git a/index.rst b/index.rst index 133c696..4ad8f6d 100644 --- a/index.rst +++ b/index.rst @@ -11,7 +11,8 @@ For more information: Visit our `website `__, check o getting_started/Introduction getting_started/Install - getting_started/Getting-started + getting_started/Hello_world + getting_started/Developing_includeos .. toctree:: From cf670d57e7072f899366921dfe4bef866bbe68a9 Mon Sep 17 00:00:00 2001 From: Martin Nordsletten Date: Tue, 14 May 2019 10:17:36 +0200 Subject: [PATCH 08/15] Rewrote and restructured the more information section --- index.rst | 4 +- more_info/More-information.rst | 98 ---------------------------------- more_info/Publications.rst | 14 ----- more_info/Resources.rst | 68 +++++++++++++++++++++++ more_info/Security.rst | 2 +- 5 files changed, 70 insertions(+), 116 deletions(-) delete mode 100644 more_info/More-information.rst delete mode 100644 more_info/Publications.rst create mode 100644 more_info/Resources.rst diff --git a/index.rst b/index.rst index 4ad8f6d..e22bd1e 100644 --- a/index.rst +++ b/index.rst @@ -14,7 +14,6 @@ For more information: Visit our `website `__, check o getting_started/Hello_world getting_started/Developing_includeos - .. toctree:: :maxdepth: 2 :caption: In depth @@ -28,9 +27,8 @@ For more information: Visit our `website `__, check o :maxdepth: 2 :caption: More information + more_info/Resources more_info/Security more_info/FAQ - more_info/More-information - more_info/Publications more_info/Roadmap more_info/Acknowledgements diff --git a/more_info/More-information.rst b/more_info/More-information.rst deleted file mode 100644 index 39fd864..0000000 --- a/more_info/More-information.rst +++ /dev/null @@ -1,98 +0,0 @@ -.. _More information: - -More information -================ - -`IncludeOS website `__ - -`IncludeOS blog `__ - -`IncludeOS on Twitter `__ - -`IncludeOS on Facebook `__ - -FLOSSUK -~~~~~~~ - -IncludeOS - Per Buer --------------------- -.. raw:: html - -
- -
- -NDC Conference 2018 -~~~~~~~~~~~~~~~~~~~ - -IncludeOS - running C++ applications without an operating system - Per Buer ---------------------------------------------------------------------------- -.. raw:: html - -
- -
- - -CppCon September 2017 -~~~~~~~~~~~~~~~~~~~~~ - -Deconstructing the OS: The devil's in the side effects ------------------------------------------------------- - -.. raw:: html - -
- -
- -Delegate this! Designing with delegates in modern C++ ------------------------------------------------------ - -.. raw:: html - -
- -
- -Official YouTube channel for IncludeOS, 2017 -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Installing includeos on macos ------------------------------ - -.. raw:: html - -
- -
- -C++ Weekly October 3, 2016 -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Episode 31: IncludeOS ---------------------- - -.. raw:: html - -
- -
- -CppCon September 19, 2016 -~~~~~~~~~~~~~~~~~~~~~~~~~ - -#include : From bootloader to REST API with the new C++ ------------------------------------------------------------ - -.. raw:: html - -
- -
- -CppCast July 14, 2016 -~~~~~~~~~~~~~~~~~~~~~ - -`IncludeOS with Alfred Bratterud `__ ----------------------------------------------------------------------------------- diff --git a/more_info/Publications.rst b/more_info/Publications.rst deleted file mode 100644 index 053cee1..0000000 --- a/more_info/Publications.rst +++ /dev/null @@ -1,14 +0,0 @@ -.. _Publications: - -Publications -============ - -Initial IncludeOS paper, 2015 -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -- Bratterud, A.; Walla, A.; Haugerud, H.; Engelstad; P.E., Begnum, K., "`IncludeOS: a resource efficient unikernel for cloud services `__" in *Cloud Computing Technology and Science (CloudCom), 2015 IEEE 7th International Conference on* - -Motivating paper, 2013 -~~~~~~~~~~~~~~~~~~~~~~ - -- Bratterud, A.; Haugerud, H., "Maximizing Hypervisor Scalability using Minimal Virtual Machines": See `IEEE Explore for abstract and citation details `__ diff --git a/more_info/Resources.rst b/more_info/Resources.rst new file mode 100644 index 0000000..0625eb0 --- /dev/null +++ b/more_info/Resources.rst @@ -0,0 +1,68 @@ +.. _Videos resources: + +Resources +========= + +Links +----- + +* `Website `__ +* `Slack `__ +* `Blog `__ +* `IncludeOS on Twitter `__ +* `IncludeOS on Facebook `__ + +Publications +------------ + +Initial IncludeOS paper, 2015 + Bratterud, A.; Walla, A.; Haugerud, H.; Engelstad; P.E., Begnum, K., "`IncludeOS: a resource efficient unikernel for cloud services `__" in *Cloud Computing Technology and Science (CloudCom), 2015 IEEE 7th International Conference on* + +Motivating paper, 2013 + Bratterud, A.; Haugerud, H., "Maximizing Hypervisor Scalability using Minimal Virtual Machines": See `IEEE Explore for abstract and citation details `__ + +Videos +------ + +* FLOSSUK 2018: IncludeOS by Per Buer + +.. raw:: html + +
+ +
+ +* NDC Conference 2018: IncludeOS - running C++ applications without an operating system by Per Buer + +.. raw:: html + +
+ +
+ + +* CppCon 2017: Deconstructing the OS: The devil's in the side effects by Alfred Bratterud + +.. raw:: html + +
+ +
+ +* CppCon 2017: Delegate this! Designing with delegates in modern C++ by Alfred Bratterud + +.. raw:: html + +
+ +
+ +* CppCon 2016: #include : From bootloader to REST API with the new C++ by Alfred Bratterud + +.. raw:: html + +
+ +
+ +* CppCast 2016: `IncludeOS with Alfred Bratterud `__ diff --git a/more_info/Security.rst b/more_info/Security.rst index b022706..b7cc036 100644 --- a/more_info/Security.rst +++ b/more_info/Security.rst @@ -3,4 +3,4 @@ Security ======== -If you have found a security issue in IncludeOS please avoid the public issue tracker. Instead send an email to security@includeos.org. If you would like to encrypt your mail please use `this key <_static/IncludeOS_pgp_public_key.asc>`__ also available on most public `PGP key servers `__. +If you have found a security issue in IncludeOS please avoid the public issue tracker. Instead send an email to security@includeos.org. If you would like to encrypt your mail please use `this key <../_static/IncludeOS_pgp_public_key.asc>`__ also available on most public `PGP key servers `__. From 42d9ecd2c52eed15079d2c2b89d6e19826d59e08 Mon Sep 17 00:00:00 2001 From: Martin Nordsletten Date: Tue, 14 May 2019 10:22:56 +0200 Subject: [PATCH 09/15] Rewrote the tools overview --- in_depth/Tools.rst | 96 ++++++++++------------------------------------ 1 file changed, 21 insertions(+), 75 deletions(-) diff --git a/in_depth/Tools.rst b/in_depth/Tools.rst index e5732e5..2a311e0 100644 --- a/in_depth/Tools.rst +++ b/in_depth/Tools.rst @@ -9,41 +9,21 @@ Note that for some third party dependencies, IncludeOS has their own fork to mak custom changes needed for the development of IncludeOS. Tools -~~~~~ +----- The tools and libraries developed by IncludeOS is listed here. -`Diskbuilder `__ ------------------------------------------------------------ - -`Vmbuild `__ ---------------------------------------------------- - -`Vmrunner `__ ------------------------------------------------------ - -.. tools_NaCl: - -:ref:`NaCl ` ------------------- +* `Vmrunner `__ - Easily boot binaries in qemu +* :ref:`NaCl ` - Configuration language for network services +* `Diskbuilder `__ +* `Vmbuild `__ Libraries -~~~~~~~~~ - -Libraries developed by IncludeOS are listed here. -You will also find a list of achieved libraries below. - -`MicroLB `__ --------------------------------------------------- - -`Mana `__ --------------------------------------------- +--------- -`Mender `__ ------------------------------------------------- - -`Bucket `__ ------------------------------------------------- +* `MicroLB `__ - Load balancer +* `Mana `__ - Web application framework +* `Bucket `__ - Simplified in-memory database Archived Libraries ------------------ @@ -51,51 +31,17 @@ Archived Libraries We have a number of libraries here that were developed by us, but are no longer maintained. Feel free to browse if you are interested in any of them. -`URI `__ - -`HTTP `__ - -`Dashboard `__ - -`Cookie `__ - -`Path to Regex `__ - -`JSON `__ - -`Director `__ - -`Butler `__ - -Dependencies -~~~~~~~~~~~~ - -Below are a list of external Dependencies used by IncludeOS kernel. - -- Binutils - -- GSL - -- `Botan `__ - -- Http-parser - -- Lest - -- Libgcc - -- Llvm - -- `Musl `__ - -- Openssl - -- Protobuf - -- Rapidjson - -- S2n +* `URI `__ +* `HTTP `__ +* `Dashboard `__ +* `Cookie `__ +* `Path to Regex `__ +* `JSON `__ +* `Director `__ +* `Butler `__ +* `Mender `__ -- `Solo5 `_ +External dependencies +--------------------- -- Uzlib +All of the external dependencies used by IncludeOS are handled by conan. The recipes describing how they are built are kept in a separate repository: `includeos/conan `__ From 9efd85800e85fc2b432b1c21d6d63ff0d6df0ce6 Mon Sep 17 00:00:00 2001 From: Taiyeba Date: Tue, 14 May 2019 10:31:42 +0200 Subject: [PATCH 10/15] examples: added link to example repo and hightlight conan install --- Examples.rst | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/Examples.rst b/Examples.rst index 81c11ff..5ebcc64 100644 --- a/Examples.rst +++ b/Examples.rst @@ -5,9 +5,8 @@ Examples Here we have a list of all the examples you will find in the `Examples Repository `__ . -To build an example, follow the steps in the readme. Note that your -build folder can be anywhere, as long as you point to the source recipe during -`conan install -pr `. +To build an example, follow the steps in the `Readme `__ . Note that your +build folder can be anywhere, as long as you point to the source recipe during conan install - ``conan install -pr `` . `256 Color VGA `__ From aae902a23a3779b64e06e4f2678211cfe92542dc Mon Sep 17 00:00:00 2001 From: Taiyeba Date: Tue, 14 May 2019 12:31:08 +0200 Subject: [PATCH 11/15] more_info: adding more to FAQ --- more_info/FAQ.rst | 85 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) diff --git a/more_info/FAQ.rst b/more_info/FAQ.rst index acfccf5..066af45 100644 --- a/more_info/FAQ.rst +++ b/more_info/FAQ.rst @@ -3,8 +3,93 @@ FAQ === +General +------- + +What compilers do you support? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +IncludeOS is an open source project developed in C++. We support `Clang `__ and +`GCC C `__ compilers. + + +What build systems are supported? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +We mostly develop on linux ``x86_64`` and macos ``x86_64`` systems. But we have support for linux ``x86`` as well. We are currently developing for linux ``aarch64`` systems. + Will IncludeOS run on ARM? ~~~~~~~~~~~~~~~~~~~~~~~~~~ Not yet, but we're working towards that. Please let us know if you've got experience with ARM architecture and time to spare. + + +Can I compile IncludeOS offline? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Yes, it is possible to compile offline if you have all the required library +packages in your local conan cache. + + +Packages +-------- + +How do I install IncludeOS? +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You do not need to install IncludeOS to run a service. We have pre-built +conan packages for `IncludeOS kernel `__ that you can use to build your services. + +What versions of IncludeOS are available in packages? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +At the moment we store up to 10 latest packages of a library or IncludeOS kernel +unless there are security vulnerabilities found in them. + + +Can I build an older version of IncludeOS that is no longer available? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Yes, it is possible. The oldest package we +have at the moment on bintray is ``includeos/0.14.1-1088@includeos/latest``. +If you want to build an older version of IncludeOS, checkout to that tag or commit +and see if a `conanfile.py` is available. If it's available you can build IncludeOS +with conan. Otherwise, look at the `README` for that ``tag`` and you should be able +build with the ``install.sh`` script, which was our old way of building the kernel. Please Note that we no longer support those versions. + + +Can I run several versions of IncludeOS instances? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Yes, you can run applications with several versions of IncludeOS instances. +By default, all our service examples are built with the latest version of +IncludeOS. To build a service with a different version, you can edit the +`conanfile.txt` and under ``[requires]`` you can specify the IncludeOS version +you want to build like this: ``includeos/0.14.0@includeos/latest``. + + +Development and Contributions +----------------------------- + +What if I want to make changes to a package? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can edit a package using ``conan editable``. Have a look at our guide on +how to get :ref:`Editable mode`. + +Can I contribute to IncludeOS libraries? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Yes, you may contribute to our libraries, but they need to be verified by our +kernel developers and pass our Jenkins tests before the libraries are updated +and packaged. + +Can I use my own external libraries to build IncludeOS? +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Yes, you can. You can use your own external libraries or different versions of +a library to build IncludeOS by editing the `conanfile.py` with your package names +and versions you can achieve this. Note that conan follows semantic versioning by +default, thus to mark your own versioning you will need to edit the ``package_id()`` +method in the `conanfile.py` to your own versioning choice. From e5470005a5942d7c1fdbd952f0cbe04af9936aac Mon Sep 17 00:00:00 2001 From: Martin Nordsletten Date: Tue, 14 May 2019 14:09:30 +0200 Subject: [PATCH 12/15] Added documentation for building with the docker container --- getting_started/Hello_world.rst | 7 ++++++- getting_started/Hello_world_docker.rst | 28 ++++++++++++++++++++++++++ 2 files changed, 34 insertions(+), 1 deletion(-) create mode 100644 getting_started/Hello_world_docker.rst diff --git a/getting_started/Hello_world.rst b/getting_started/Hello_world.rst index 0a60d10..0debd5a 100644 --- a/getting_started/Hello_world.rst +++ b/getting_started/Hello_world.rst @@ -3,7 +3,12 @@ Hello World with IncludeOS ========================== -You don't need to manually install IncludeOS to build a service. Our pipeline builds IncludeOS conan packages and uploads them to `bintray `__. +If you would rather build the service using Docker please see: + +.. toctree:: + Hello_world_docker + +You don't need to manually build IncludeOS to build a service. Our pipeline builds IncludeOS conan packages and uploads them to `bintray `__. A service then specifies which version of IncludeOS to use when building. To get started with the `hello_world` example just clone the service repository: diff --git a/getting_started/Hello_world_docker.rst b/getting_started/Hello_world_docker.rst new file mode 100644 index 0000000..7242c16 --- /dev/null +++ b/getting_started/Hello_world_docker.rst @@ -0,0 +1,28 @@ +.. _hello world docker: + +Using Docker +============ + +To build the hello world service with Docker you must first build the docker container from the IncludeOS repo:: + + $ git clone https://github.com/includeos/IncludeOS.git + $ docker build -t includeos IncludeOS + +This creates a Docker container that contains the correct build tools and conan configuration for building a service. It still requires to download the actual conan packages, but that will be completed when building a specific service. + +To use the docker container for building the hello world service:: + + $ git clone https://github.com/includeos/hello_world.git + $ cd hello_world + $ docker run --rm -v $PWD:/service includeos + +This will perform the following: + +* Create a bind mount of the service files to ``/service`` inside the container +* Run the ``CMD`` from the docker container, which performs: + + * ``conan install`` to download all required conan packages + * ``cmake ..`` to configure the service + * ``cmake --build .`` to build the service + +The finished service will be available as an elf binary in ``build/hello``. From 0bd3ffb9477a10b0ff8e75ad7499458f58884b4c Mon Sep 17 00:00:00 2001 From: Taiyeba Date: Tue, 14 May 2019 14:33:33 +0200 Subject: [PATCH 13/15] in_depth: new file on Testing added --- in_depth/Testing.rst | 72 ++++++++++++++++++++++++++++++++++++++++++++ index.rst | 1 + 2 files changed, 73 insertions(+) create mode 100644 in_depth/Testing.rst diff --git a/in_depth/Testing.rst b/in_depth/Testing.rst new file mode 100644 index 0000000..bf7f3e5 --- /dev/null +++ b/in_depth/Testing.rst @@ -0,0 +1,72 @@ +.. _Testing : + +Testing IncludeOS +================= + +To test IncludeOS as a kernel and it's features, we have written a number of tests +that can be reused by anyone to test there work on IncludeOS. You will find a list +of our tests in the `IncludeOS/test `__ folder. + +.. note:: The following testing instructions only work on Linux at the moment. + +Unit Tests +~~~~~~~~~~ + +All the tests are grouped by categories as listed in the folder. The unit tests for each of the categories of tests are located under ``IncludeOS/test//unit/.cpp``. + +To run all the unit tests simultaneously do: + +:: + + $ conan install IncludeOS/test -pr gcc-7.3.0-linux-x86_64 -s build_type=Debug + $ . ./activate.sh + +If you want to get the coverage report for the unit tests as well do: + +:: + + # cmake coverage + $ . ./activate.sh && cmake -DCOVERAGE=ON -DCODECOV_HTMLOUTPUTDIR= IncludeOS/test + # make coverage + $ . ./activate.sh && make -j nproc coverage + +After running the test, you can then view the coverage report in html format +in the path you specified. + +Integration Testing +~~~~~~~~~~~~~~~~~~~ + +To do integration testing you will need to have all the necessary tools installed. +We use our `puppet file `__ to configure our test instances. +You can use the puppet manifest to install on your local machine. + + +To install and build the integration tests: + +:: + + $ conan install test/integration -pr clang-6.0-linux-x86_64 + $ . ./activate.sh + $ cmake $SRC/test/integration -DSTRESS=ON -DCMAKE_BUILD_TYPE=Debug + $ make -j nproc + +If you would like to build the tests without setting up the network tools, you can use ``-DFOR_PRODUCTION=OFF`` with the ``cmake`` command. This is however not recommended, as keeping the production setting to ON gives proper random. + +We use ``ctest`` to run the integration tests. Along with integration tests we +also have stress tests. To skip the stress tests and run only the integration +tests at random, do: + +:: + + $ ctest -E stress --output-on-failure --schedule-random + +To run only the stress tests and skip all other tests, do: + +:: + + $ ctest -R stress -E integration --output-on-failure + +You can have a look at our `Jenkinsfile `__ to get an overview of how we run the tests. You will notice we run our unit tests and coverage with +the gcc profile, currently thats the one we are focusing on. + +.. warning:: Remember to run ``. ./deactivate.sh`` to reset your environment to previous state after testing is complete. diff --git a/index.rst b/index.rst index e22bd1e..2c6a95e 100644 --- a/index.rst +++ b/index.rst @@ -21,6 +21,7 @@ For more information: Visit our `website `__, check o in_depth/Deeper-understanding in_depth/Conanigans in_depth/NaCl + in_depth/Testing in_depth/Contributing-to-IncludeOS .. toctree:: From 2eaec319873dcfffcac3c8b2b64db67b75241521 Mon Sep 17 00:00:00 2001 From: Taiyeba Date: Tue, 14 May 2019 15:00:54 +0200 Subject: [PATCH 14/15] in_depth: Minor fixes in Testing --- in_depth/Testing.rst | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/in_depth/Testing.rst b/in_depth/Testing.rst index bf7f3e5..00b485b 100644 --- a/in_depth/Testing.rst +++ b/in_depth/Testing.rst @@ -3,8 +3,8 @@ Testing IncludeOS ================= -To test IncludeOS as a kernel and it's features, we have written a number of tests -that can be reused by anyone to test there work on IncludeOS. You will find a list +To test IncludeOS and it's features, we have written a number of tests +that can be reused by anyone to test their work on IncludeOS. You will find a list of our tests in the `IncludeOS/test `__ folder. .. note:: The following testing instructions only work on Linux at the moment. @@ -14,23 +14,30 @@ Unit Tests All the tests are grouped by categories as listed in the folder. The unit tests for each of the categories of tests are located under ``IncludeOS/test//unit/.cpp``. -To run all the unit tests simultaneously do: +To install and build the unit tests do: :: $ conan install IncludeOS/test -pr gcc-7.3.0-linux-x86_64 -s build_type=Debug $ . ./activate.sh +Then to run the unit tests in random order do; + +:: + + $ ctest --schedule-random + + If you want to get the coverage report for the unit tests as well do: :: - # cmake coverage + # build with coverage $ . ./activate.sh && cmake -DCOVERAGE=ON -DCODECOV_HTMLOUTPUTDIR= IncludeOS/test - # make coverage + # build and test with coverage $ . ./activate.sh && make -j nproc coverage -After running the test, you can then view the coverage report in html format +After running the unit tests, you can then view the coverage report in html format in the path you specified. Integration Testing @@ -50,7 +57,7 @@ To install and build the integration tests: $ cmake $SRC/test/integration -DSTRESS=ON -DCMAKE_BUILD_TYPE=Debug $ make -j nproc -If you would like to build the tests without setting up the network tools, you can use ``-DFOR_PRODUCTION=OFF`` with the ``cmake`` command. This is however not recommended, as keeping the production setting to ON gives proper random. +If you are running on a virtual machine without direct access to hardware, or you are lacking the RDRAND/RDSEED cpu instructions, you can use ``-DFOR_PRODUCTION=OFF`` with the ``cmake`` command. This is however not recommended, as keeping the production setting to ``ON`` gives proper random. We use ``ctest`` to run the integration tests. Along with integration tests we also have stress tests. To skip the stress tests and run only the integration @@ -66,7 +73,6 @@ To run only the stress tests and skip all other tests, do: $ ctest -R stress -E integration --output-on-failure -You can have a look at our `Jenkinsfile `__ to get an overview of how we run the tests. You will notice we run our unit tests and coverage with -the gcc profile, currently thats the one we are focusing on. +You can have a look at our `Jenkinsfile `__ to get an overview of how we run the tests. You will notice we build our unit tests and coverage with the gcc profile. .. warning:: Remember to run ``. ./deactivate.sh`` to reset your environment to previous state after testing is complete. From 03613a5a233e5dd85e5dbc26f970d9e72f98ce67 Mon Sep 17 00:00:00 2001 From: Taiyeba Date: Tue, 14 May 2019 15:12:46 +0200 Subject: [PATCH 15/15] in_depth: Testing updated with proper unit testing instructions --- in_depth/Testing.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/in_depth/Testing.rst b/in_depth/Testing.rst index 00b485b..cf2363e 100644 --- a/in_depth/Testing.rst +++ b/in_depth/Testing.rst @@ -20,6 +20,8 @@ To install and build the unit tests do: $ conan install IncludeOS/test -pr gcc-7.3.0-linux-x86_64 -s build_type=Debug $ . ./activate.sh + $ cmake + $ make -j nproc Then to run the unit tests in random order do; @@ -28,7 +30,7 @@ Then to run the unit tests in random order do; $ ctest --schedule-random -If you want to get the coverage report for the unit tests as well do: +If you want to get the coverage report for your unit tests as well do: ::