You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
## What IS a software container anyway? (And what's it good for?)
4
+
5
+
A container allows you to stick an application and all of its dependencies into a single package. This makes your application portable, shareable, and reproducible.
6
+
7
+
Containers foster portability and reproducibility because they package **ALL** of an applications dependencies... including its own tiny operating system!
8
+
9
+
This means your application won't break when you port it to a new environment. Your app brings its environment with it.
10
+
11
+
Here are some examples of things you can do with containers:
12
+
13
+
- Package an analysis pipeline so that it runs on your laptop, in the cloud, and in a high performance computing (HPC) environment to produce the same result.
14
+
- Publish a paper and include a link to a container with all of the data and software that you used so that others can easily reproduce your results.
15
+
- Install and run an application that requires a complicated stack of dependencies with a few keystrokes.
16
+
- Create a pipeline or complex workflow where each individual program is meant to run on a different operating system.
17
+
18
+
## How do containers differ from virtual machines (VMs)
19
+
20
+
Containers and VMs are both types of virtualization. But it's important to understand the differences between the two and know when to use each.
21
+
22
+
**Virtual Machines** install every last bit of an operating system (OS) right down to the core software that allows the OS to control the hardware (called the _kernel_). This means that VMs:
23
+
- Are complete in the sense that you can use a VM to interact with your computer via a different OS.
24
+
- Are extremely flexible. For instance you an install a Windows VM on a Mac using software like [VirtualBox](https://www.virtualbox.org/wiki/VirtualBox).
25
+
- Are slow and resource hungry. Every time you start a VM it has to bring up an entirely new OS.
26
+
27
+
**Containers** share a kernel with the host OS. This means that Containers:
28
+
- Are less flexible than VMs. For example, a Linux container must be run on a Linux host OS. (Although you can mix and match distributions.) In practice, containers are only extensively developed on Linux.
29
+
- Are much faster and lighter weight than VMs. A container may be just a few MB.
30
+
- Start and stop quickly and are suitable for running single apps.
31
+
32
+
Because of their differences, VMs and containers serve different purposes and should be favored under different circumstances.
33
+
- VMs are good for long running interactive sessions where you may want to use several different applications. (Checking email on Outlook and using Microsoft Word and Excel).
34
+
- Containers are better suited to running one or two applications, often non-interactively, in their own custom environments.
35
+
36
+
## Docker
37
+
38
+
[Docker](https://www.docker.com/) is currently the most widely used container software. It has several strengths and weaknesses that make it a good choice for some projects but not for others.
39
+
40
+
**philosophy**
41
+
42
+
Docker is built for running multiple containers on a single system and it allows containers to share common software features for efficiency. It also seeks to fully isolate each container from all other containers and from the host system.
43
+
44
+
Docker assumes that you will be a root user. Or that it will be OK for you to elevate your privileges if you are not a root user.
45
+
See https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface for details.
46
+
47
+
**strengths**
48
+
49
+
- Mature software with a large user community
50
+
-[Docker Hub](https://hub.docker.com/)!
51
+
- A place to build and host your containers
52
+
- Fully integrated into core Docker
53
+
- Over 100,000 pre-built containers
54
+
- Provides an ecosystem for container orchestration
55
+
- Rich feature set
56
+
57
+
**weaknesses**
58
+
59
+
- Difficult to learn
60
+
- Hidden innards
61
+
- Complex container model (layers)
62
+
- Not architected with security in mind
63
+
- Not built for HPC (but good for cloud)
64
+
65
+
Docker shines for DevOPs teams providing cloud-native micro-services to users.
66
+
67
+
## Singularity
68
+
69
+
[Singularity](http://singularity.lbl.gov/) is a relatively new container software invented by Greg Kurtzer while at Lawrence Berkley National labs and now developed by his company [Sylabs](sylabs.io). It was developed with security, scientific software, and HPC systems in mind.
70
+
71
+
**philosophy**
72
+
73
+
Singularity assumes ([more or less](http://containers-ftw.org/SCI-F/)) that each application will have its own container. It does not seek to fully isolate containers from one another or the host system.
74
+
75
+
Singularity assumes that you will have a build system where you are the root user, but that you will also have a production system where you may or may not be the root user.
76
+
77
+
**strengths**
78
+
- Easy to learn and use (relatively speaking)
79
+
- Approved for HPC ([installed on some of the biggest HPC systems in the world](http://singularity.lbl.gov/citation-registration#clusters))
80
+
- Can convert Docker containers to Singularity and run containers directly from Docker Hub
Copy file name to clipboardExpand all lines: 02-basic-usage/README.md
+1-1
Original file line number
Diff line number
Diff line change
@@ -1,4 +1,4 @@
1
-
# Downloading and interacting with containers
1
+
# Downloading and Interacting with Containers
2
2
3
3
This section will be useful for container consumers. (i.e. those who really just want to use containers somebody else built.) The next chapter will explore topics more geared toward container producers (i.e. those who want/need to build containers from scratch).
Copy file name to clipboardExpand all lines: 04-the-ecosystem/README.md
+1-1
Original file line number
Diff line number
Diff line change
@@ -1,4 +1,4 @@
1
-
# The Singularity ecosystem
1
+
# The Singularity Ecosystem
2
2
3
3
We've spent a lot of time on building and using your own containers so that you understand how Singularity works. Now let's talk more about the [Singularity Container Services](https://cloud.sylabs.io/home) and [Docker Hub](https://hub.docker.com/).
Copy file name to clipboardExpand all lines: 05-bind-mounts/README.md
+1-1
Original file line number
Diff line number
Diff line change
@@ -1,4 +1,4 @@
1
-
# Bind mounting host system directories into a container
1
+
# Accessing Host Files with Bind Mounts
2
2
3
3
It's possible to create and modify files on the host system from within the container. In fact, that's exactly what we did in the previous example when we created output files in our home directory.
Copy file name to clipboardExpand all lines: 06-runscript/README.md
+1-1
Original file line number
Diff line number
Diff line change
@@ -1,4 +1,4 @@
1
-
# The Runscript: Making containerized apps behave like normal apps
1
+
# The Runscript: Making Containerized Apps Behave Like Normal Apps
2
2
3
3
Consider an application that takes one file as input, analyzes the data in the file, and produces another file as output. This is obviously a very common situation.
Copy file name to clipboardExpand all lines: 07-fake-installation/README.md
+1-1
Original file line number
Diff line number
Diff line change
@@ -1,4 +1,4 @@
1
-
# Faking a native app installation within a Singularity container
1
+
# Faking a Native Installation within a Singularity Container
2
2
3
3
How would you like to install an app with one more more commands inside of a Singularity container and then just forget it's in a container and use it like any other app?
Copy file name to clipboardExpand all lines: 08-misc/README.md
+28-3
Original file line number
Diff line number
Diff line change
@@ -1,8 +1,33 @@
1
-
# Singularity Instances
1
+
# Miscellaneous Topics and FAQs
2
+
3
+
## X11 and OpenGL
4
+
5
+
You can use Singularity containers to display graphics through common protocols. To do this, you need to install the proper graphics stack within the Singularity container. For instance if you want to display X11 graphics you must install `xorg` within your container. In an Ubuntu container the command would look like this.
6
+
7
+
```
8
+
$ apt-get install xorg
9
+
```
10
+
11
+
## GPU computing
12
+
13
+
In Singularity v2.3+ the experimental `--nv` option will look for NVIDIA libraries on the host system and automatically bind mount them to the container so that GPUs work seamlessly.
14
+
15
+
## Using the network on the host system
16
+
17
+
Network ports on the host system are accessible from within the container and work seamlessly. For example, you could install ipython within a container, start a jupyter notebook instance, and then connect to that instance using a browser running outside of the container on the host system or from another host.
18
+
19
+
## A note on SUID programs and daemons
20
+
21
+
Some programs need root privileges to run. These often include services or daemons that start via the `init.d` or `system.d` systems and run in the background. For instance, `sshd` the ssh daemon that listens on port 22 and allows another user to connect to your computer requires root privileges. You will not be able to run it in a container unless you start the container as root.
22
+
23
+
Other programs may set the SUID bit or capabilities to run as root or with elevated privileges without your knowledge. For instance, the well-known `ping` program actually runs with elevated privileges (and needs to since it sets up a raw network socket). This program will not run in a container unless you are root in the container.
24
+
25
+
26
+
## Long-running Instances
2
27
3
28
Up to now all of our examples have run Singularity containers in the foreground. But what if you want to run a service like a web server or a database in a Singularity container in the background?
4
29
5
-
## lolcow (useless) example
30
+
###lolcow (useless) example
6
31
In Singularity v2.4+, you can use the [`instance` command group](http://singularity.lbl.gov/docs-instances) to start and control container instances that run in the background. To demonstrate, let's start an instance of our `lolcow.simg` container running in the background.
7
32
8
33
```
@@ -59,7 +84,7 @@ Stopping cow2 instance of /home/dave/lolcow.simg (PID=10855)
59
84
Stopping cow3 instance of /home/dave/lolcow.simg (PID=10885)
60
85
```
61
86
62
-
## nginx (useful) example
87
+
###nginx (useful) example
63
88
64
89
These examples are not very useful because `lolcow.simg` doesn't run any services. Let's extend the example to something useful by running a local nginx web server in the background. This command will download the official nginx image from Docker Hub and start it in a background instance called "web". (The commands need to be executed as root so that nginx can run with the privileges it needs.)
# <b>Creating and running software containers with Singularity</b>
2
-
### <i>How to use [Singularity](https://sylabs.io/guides/latest/user-guide/)! </i>
3
2
4
-
##Table of Contents
3
+
### <i>How to use [Singularity](https://sylabs.io/guides/latest/user-guide/)! </i>
5
4
6
-
-[Installation](/00-installation)
7
5
8
-
This is an introductory workshop on Singularity. It was originally taught by David Godlove at the [NIH HPC](https://hpc.nih.gov/), but the content has since been adapted to a general audience. For more information about the topics covered here, see the following:
6
+
This is an introductory workshop on Singularity. It was originally taught by [David Godlove](https://github.com/GodloveD) at the [NIH HPC](https://hpc.nih.gov/), but the content has since been adapted to a general audience. For more information about the topics covered here, see the following:
## What IS a software container anyway? (And what's it good for?)
19
-
20
-
A container allows you to stick an application and all of its dependencies into a single package. This makes your application portable, shareable, and reproducible.
21
-
22
-
Containers foster portability and reproducibility because they package **ALL** of an applications dependencies... including its own tiny operating system!
23
-
24
-
This means your application won't break when you port it to a new environment. Your app brings its environment with it.
25
-
26
-
Here are some examples of things you can do with containers:
27
-
28
-
- Package an analysis pipeline so that it runs on your laptop, in the cloud, and in a high performance computing (HPC) environment to produce the same result.
29
-
- Publish a paper and include a link to a container with all of the data and software that you used so that others can easily reproduce your results.
30
-
- Install and run an application that requires a complicated stack of dependencies with a few keystrokes.
31
-
- Create a pipeline or complex workflow where each individual program is meant to run on a different operating system.
32
-
33
-
## How do containers differ from virtual machines (VMs)
34
-
35
-
Containers and VMs are both types of virtualization. But it's important to understand the differences between the two and know when to use each.
36
-
37
-
**Virtual Machines** install every last bit of an operating system (OS) right down to the core software that allows the OS to control the hardware (called the _kernel_). This means that VMs:
38
-
- Are complete in the sense that you can use a VM to interact with your computer via a different OS.
39
-
- Are extremely flexible. For instance you an install a Windows VM on a Mac using software like [VirtualBox](https://www.virtualbox.org/wiki/VirtualBox).
40
-
- Are slow and resource hungry. Every time you start a VM it has to bring up an entirely new OS.
41
-
42
-
**Containers** share a kernel with the host OS. This means that Containers:
43
-
- Are less flexible than VMs. For example, a Linux container must be run on a Linux host OS. (Although you can mix and match distributions.) In practice, containers are only extensively developed on Linux.
44
-
- Are much faster and lighter weight than VMs. A container may be just a few MB.
45
-
- Start and stop quickly and are suitable for running single apps.
46
-
47
-
Because of their differences, VMs and containers serve different purposes and should be favored under different circumstances.
48
-
- VMs are good for long running interactive sessions where you may want to use several different applications. (Checking email on Outlook and using Microsoft Word and Excel).
49
-
- Containers are better suited to running one or two applications, often non-interactively, in their own custom environments.
50
-
51
-
## Docker
52
-
53
-
[Docker](https://www.docker.com/) is currently the most widely used container software. It has several strengths and weaknesses that make it a good choice for some projects but not for others.
54
-
55
-
**philosophy**
56
-
57
-
Docker is built for running multiple containers on a single system and it allows containers to share common software features for efficiency. It also seeks to fully isolate each container from all other containers and from the host system.
58
-
59
-
Docker assumes that you will be a root user. Or that it will be OK for you to elevate your privileges if you are not a root user.
60
-
See https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface for details.
61
-
62
-
**strengths**
63
-
64
-
- Mature software with a large user community
65
-
-[Docker Hub](https://hub.docker.com/)!
66
-
- A place to build and host your containers
67
-
- Fully integrated into core Docker
68
-
- Over 100,000 pre-built containers
69
-
- Provides an ecosystem for container orchestration
70
-
- Rich feature set
71
-
72
-
**weaknesses**
73
-
74
-
- Difficult to learn
75
-
- Hidden innards
76
-
- Complex container model (layers)
77
-
- Not architected with security in mind
78
-
- Not built for HPC (but good for cloud)
79
-
80
-
Docker shines for DevOPs teams providing cloud-native micro-services to users.
81
-
82
-
## Singularity
83
-
84
-
[Singularity](http://singularity.lbl.gov/) is a relatively new container software invented by Greg Kurtzer while at Lawrence Berkley National labs and now developed by his company [Sylabs](sylabs.io). It was developed with security, scientific software, and HPC systems in mind.
85
-
86
-
**philosophy**
87
-
88
-
Singularity assumes ([more or less](http://containers-ftw.org/SCI-F/)) that each application will have its own container. It does not seek to fully isolate containers from one another or the host system.
89
-
90
-
Singularity assumes that you will have a build system where you are the root user, but that you will also have a production system where you may or may not be the root user.
91
-
92
-
**strengths**
93
-
- Easy to learn and use (relatively speaking)
94
-
- Approved for HPC ([installed on some of the biggest HPC systems in the world](http://singularity.lbl.gov/citation-registration#clusters))
95
-
- Can convert Docker containers to Singularity and run containers directly from Docker Hub
0 commit comments