@@ -5,13 +5,13 @@ title: Documentation
5
5
6
6
### Cloud Haskell Platform
7
7
8
- This is the [ * Cloud Haskell Platform* ] [ 1 ] . Cloud Haskell is a set of libraries
8
+ This is the [ * Cloud Haskell Platform* ] [ cloud-haskell ] . Cloud Haskell is a set of libraries
9
9
that bring Erlang-style concurrency and distribution to Haskell programs. This
10
10
project is an implementation of that distributed computing interface, where
11
11
processes communicate with one another through explicit message passing rather
12
12
than shared memory.
13
13
14
- Originally described by the joint [ Towards Haskell in the Cloud] [ 12 ] paper,
14
+ Originally described by the joint [ Towards Haskell in the Cloud] [ haskell11-ch ] paper,
15
15
Cloud Haskell has be re-written from the ground up and supports a rich and
16
16
growing number of features for
17
17
@@ -22,40 +22,37 @@ growing number of features for
22
22
* working with several network transport implementations (and more in the pipeline)
23
23
* supporting * static* values (required for remote communication)
24
24
25
- There is a recent
26
- [ presentation] ( http://sneezy.cs.nott.ac.uk/fun/2012-02/ coutts-2012-02-28.pdf )
25
+ There is a
26
+ [ presentation] [ fun201202- coutts]
27
27
on Cloud Haskell and this reimplementation, which is worth reading in conjunction
28
28
with the documentation and wiki pages on this website..
29
29
30
30
Cloud Haskell comprises the following components, some of which are complete,
31
31
others experimental.
32
32
33
- * [ distributed-process] [ 2 ] : Base concurrency and distribution support
34
- * [ distributed-process-platform] [ 3 ] : The Cloud Haskell Platform - APIs
35
- * [ distributed-static] [ 4 ] : Support for static values
36
- * [ rank1dynamic] [ 5 ] : Like ` Data.Dynamic ` and ` Data.Typeable ` but supporting polymorphic values
37
- * [ network-transport] [ 6 ] : Generic ` Network.Transport ` API
38
- * [ network-transport-tcp] [ 7 ] : TCP realisation of ` Network.Transport `
39
- * [ network-transport-inmemory] [ 8 ] : In-memory realisation of ` Network.Transport ` (incomplete)
40
- * [ network-transport-composed] [ 9 ] : Compose two transports (very preliminary)
41
- * [ distributed-process-simplelocalnet] [ 10 ] : Simple backend for local networks
42
- * [ distributed-process-azure] [ 11 ] : Azure backend for Cloud Haskell (proof of concept)
33
+ * [ distributed-process] [ distributed-process ] : Base concurrency and distribution support
34
+ * [ distributed-process-platform] [ distributed-process-platform ] : The Cloud Haskell Platform - APIs
35
+ * [ distributed-static] [ distributed-static ] : Support for static values
36
+ * [ rank1dynamic] [ rank1dynamic ] : Like ` Data.Dynamic ` and ` Data.Typeable ` but supporting polymorphic values
37
+ * [ network-transport] [ network-transport ] : Generic ` Network.Transport ` API
38
+ * [ network-transport-tcp] [ network-transport-tcp ] : TCP realisation of ` Network.Transport `
39
+ * [ network-transport-inmemory] [ network-transport-inmemory ] : In-memory realisation of ` Network.Transport ` (incomplete)
40
+ * [ network-transport-composed] [ network-transport-composed ] : Compose two transports (very preliminary)
41
+ * [ distributed-process-simplelocalnet] [ distributed-process-simplelocalnet ] : Simple backend for local networks
42
+ * [ distributed-process-azure] [ distributed-process-azure ] : Azure backend for Cloud Haskell (proof of concept)
43
43
44
44
One of Cloud Haskell's goals is to separate the transport layer from the
45
- * process layer* , so that the transport backend is entirely independent:
46
- it is envisaged that this interface might later be used by models
47
- other than the Cloud Haskell paradigm, and that applications built
48
- using Cloud Haskell might be easily configured to work with different
49
- backend transports.
45
+ * process layer* , so that the transport backend is entirely independent. In fact
46
+ other projects can and do reuse the transport layer, even if they don't use or
47
+ have their own process layer (see e.g. [ HdpH] [ hdph ] ).
50
48
51
49
Abstracting over the transport layer allows different protocols for
52
50
message passing, including TCP/IP, UDP,
53
- [ MPI] ( http://en.wikipedia.org/wiki/Message_Passing_Interface ) ,
51
+ [ MPI] ( http://en.wikipedia.org/wiki/Message_Passing_Interface ) ,
54
52
[ CCI] ( http://www.olcf.ornl.gov/center-projects/common-communication-interface/ ) ,
55
- ZeroMQ, SSH, MVars, Unix pipes, and more. Each of these transports would provide
56
- its own implementation of the ` Network.Transport ` and provide a means of creating
57
- new connections for use within ` Control.Distributed.Process ` . This separation means
58
- that transports might be used for other purposes than Cloud Haskell.
53
+ [ ZeroMQ] ( http://zeromq.org ) , [ SSH] ( http://openssh.com ) , MVars, Unix pipes, and more. Each of these transports provides
54
+ its own implementation of the ` Network.Transport ` API and provide a means of creating
55
+ new connections for use within ` Control.Distributed.Process ` .
59
56
60
57
The following diagram shows dependencies between the various subsystems,
61
58
in an application using Cloud Haskell, where arrows represent explicit
@@ -94,8 +91,8 @@ In this diagram, the various nodes roughly correspond to specific modules:
94
91
Transport Implementation : Network.Transport.*
95
92
96
93
An application is built using the primitives provided by the Cloud
97
- Haskell layer, provided by ` Control.Distributed.Process ` module, which
98
- provides abstractions such as nodes and processes.
94
+ Haskell layer, provided by the ` Control.Distributed.Process ` module, which
95
+ defines abstractions such as nodes and processes.
99
96
100
97
The application also depends on a Cloud Haskell Backend, which
101
98
provides functions to allow the initialisation of the transport layer
@@ -105,21 +102,21 @@ It is, of course, possible to create new Cloud Haskell nodes by
105
102
using a Network Transport Backend such as ` Network.Transport.TCP `
106
103
directly.
107
104
108
- The Cloud Haskell interface and backend, make use of the Transport
105
+ The Cloud Haskell interface and backend make use of the Transport
109
106
interface provided by the ` Network.Transport ` module.
110
107
This also serves as an interface for the ` Network.Transport.* `
111
108
module, which provides a specific implementation for this transport,
112
- and may, for example, be based on some external library written in
109
+ and may, for example, be based on some external library written in
113
110
Haskell or C.
114
111
115
112
### Network Transport Abstraction Layer
116
113
117
- Cloud Haskell's generic [ network-transport] [ 6 ] API is entirely independent of
114
+ Cloud Haskell's generic [ network-transport] [ network-transport ] API is entirely independent of
118
115
the concurrency and messaging passing capabilities of the * process layer* .
119
116
Cloud Haskell applications are built using the primitives provided by the
120
- * process layer* (i.e., [ distributed-process] [ 2 ] ), which provides abstractions
117
+ * process layer* (i.e., [ distributed-process] [ distributed-process ] ), which provides abstractions
121
118
such as nodes and processes. Applications must also depend on a Cloud Haskell
122
- Backend , which provides functions to allow the initialisation of the transport
119
+ backend , which provides functions to allow the initialisation of the transport
123
120
layer using whatever topology might be appropriate to the application.
124
121
125
122
` Network.Transport ` is a network abstraction layer geared towards specific
@@ -128,7 +125,7 @@ classes of applications, offering the following high level concepts:
128
125
* Nodes in the network are represented by ` EndPoint ` s. These are heavyweight stateful objects.
129
126
* Each ` EndPoint ` has an ` EndPointAddress ` .
130
127
* Connections can be established from one ` EndPoint ` to another using the ` EndPointAddress ` of the remote end.
131
- * The ` EndPointAddress ` can be serialised and sent over the network, where as ` EndPoint ` s and connections cannot.
128
+ * The ` EndPointAddress ` can be serialised and sent over the network, whereas ` EndPoint ` s and connections cannot.
132
129
* Connections between ` EndPoint ` s are unidirectional and lightweight.
133
130
* Outgoing messages are sent via a ` Connection ` object that represents the sending end of the connection.
134
131
* Incoming messages for ** all** of the incoming connections on an ` EndPoint ` are collected via a shared receive queue.
@@ -160,16 +157,16 @@ of other `Network.Transport` APIs if required, but for the most part this
160
157
is irrelevant and the application will interact with Cloud Haskell through
161
158
the * Process Layer* and * Platform* .
162
159
163
- For more details about ` Network.Transport ` please see the [ wiki page] [ 20 ] .
160
+ For more details about ` Network.Transport ` please see the [ wiki page] ( /wiki/networktransport.html ) .
164
161
165
162
### Concurrency and Distribution
166
163
167
164
The * Process Layer* is where Cloud Haskell's support for concurrency and
168
165
distributed programming are exposed to application developers. This layer
169
- deals explicitly with
166
+ deals explicitly with
170
167
171
168
The core of Cloud Haskell's concurrency and distribution support resides in the
172
- [ distributed-process] [ 2 ] library. As well as the APIs necessary for starting
169
+ [ distributed-process] [ distributed-process ] library. As well as the APIs necessary for starting
173
170
nodes and forking processes on them, we find all the basic primitives required
174
171
to
175
172
@@ -215,7 +212,7 @@ runProcess :: LocalNode -> Process () -> IO ()
215
212
{% endhighlight %}
216
213
217
214
Once we've spawned some processes, they can communicate with one another
218
- using the messaging primitives provided by [ distributed-processes] [ 2 ] ,
215
+ using the messaging primitives provided by [ distributed-processes] [ distributed-processes ] ,
219
216
which are well documented in the haddocks.
220
217
221
218
### What is Serializable
@@ -254,10 +251,10 @@ We create channels with a call to `newChan`, and send/receive on them using the
254
251
channelsDemo :: Process ()
255
252
channelsDemo = do
256
253
(sp, rp) <- newChan :: Process (SendPort String, ReceivePort String)
257
-
254
+
258
255
-- send on a channel
259
256
spawnLocal $ sendChan sp "hello!"
260
-
257
+
261
258
-- receive on a channel
262
259
m <- receiveChan rp
263
260
say $ show m
@@ -272,7 +269,7 @@ need to spawn a process and send a bunch a messages to it, then wait for
272
269
replies however; we can’t send a ` ReceivePort ` since it is not ` Serializable ` .
273
270
274
271
` ReceivePort ` s can be merged, so we can listen on several simultaneously. In the
275
- latest version of [ distributed-process] [ 2 ] , we can listen for * regular* messages
272
+ latest version of [ distributed-process] [ distributed-process ] , we can listen for * regular* messages
276
273
and multiple channels at the same time, using ` matchChan ` in the list of
277
274
allowed matches passed ` receiveWait ` and ` receiveTimeout ` .
278
275
@@ -313,7 +310,7 @@ and decide whether to oblige or not.
313
310
314
311
### Rethinking the Task Layer
315
312
316
- [ Towards Haskell in the Cloud] [ 12 ] describes a multi-layered architecture, in
313
+ [ Towards Haskell in the Cloud] [ haskell11-ch ] describes a multi-layered architecture, in
317
314
which manipulation of concurrent processes and message passing between them
318
315
is managed in the * process layer* , whilst a higher level API described as the
319
316
* task layer* provides additional features such as
@@ -322,19 +319,19 @@ is managed in the *process layer*, whilst a higher level API described as the
322
319
* data centric processing model
323
320
* a promise (or * future* ) abstraction, representing the result of a calculation that may or may not have yet completed
324
321
325
- The [ distributed-process-platform] [ 18 ] library implements parts of the
322
+ The [ distributed-process-platform] [ distributed-process-platform ] library implements parts of the
326
323
* task layer* , but takes a very different approach to that described
327
- in the original paper and implemented by the [ remote] [ 14 ] package. In particular,
324
+ in the original paper and implemented by the [ remote] [ remote ] package. In particular,
328
325
we diverge from the original design and defer to many of the principles
329
- defined by Erlang's [ Open Telecom Platform] [ 13 ] , taking in some well established
326
+ defined by Erlang's [ Open Telecom Platform] [ OTP ] , taking in some well established
330
327
Haskell concurrency design patterns along the way.
331
328
332
- In fact, [ distributed-process-platform] [ 18 ] does not really consider the
329
+ In fact, [ distributed-process-platform] [ distributed-process-platform ] does not really consider the
333
330
* task layer* in great detail. We provide an API comparable to remote's
334
- ` Promise ` in Control.Distributed.Process.Platform.Async. This API however,
335
- is derived from Simon Marlow's [ Control.Concurrent.Async] [ 19 ] package, and is not
331
+ ` Promise ` in ` Control.Distributed.Process.Platform.Async ` . This API however,
332
+ is derived from Simon Marlow's [ Control.Concurrent.Async] [ async ] package, and is not
336
333
limited to blocking queries on ` Async ` handles in the same way. Instead our
337
- [ API] [ 17 ] handles both blocking and non-blocking queries, polling
334
+ [ API] [ d-p-platform-async ] handles both blocking and non-blocking queries, polling
338
335
and working with lists of ` Async ` handles. We also eschew throwing exceptions
339
336
to indicate asynchronous task failures, instead handling * task* and connectivity
340
337
failures using monitors. Users of the API need only concern themselves with the
@@ -356,13 +353,13 @@ demoAsync = do
356
353
357
354
-- we can cancel the task if we want to
358
355
-- cancel hAsync
359
-
356
+
360
357
-- or cancel it and wait until it has exited
361
358
-- cancelWait hAsync
362
-
359
+
363
360
-- we can wait on the task and timeout if it's still busy
364
361
Nothing <- waitTimeout (within 3 Seconds) hAsync
365
-
362
+
366
363
-- or finally, we can block until the task is finished!
367
364
asyncResult <- wait hAsync
368
365
case asyncResult of
@@ -379,7 +376,7 @@ around `Async` that disallows side effects is relatively simple, and we
379
376
do not consider the presence of side effects a barrier to fault tolerance
380
377
and automated process restarts. Erlang does not forbid * IO* in its processes,
381
378
and yet that doesn't render supervision trees ineffective. They key is to
382
- provide a rich enough API that statefull processes can recognise whether or
379
+ provide a rich enough API that stateful processes can recognise whether or
383
380
not they need to provide idempotent initialisation routines.
384
381
385
382
The utility of preventing side effects using the type system is, however, not
@@ -391,7 +388,7 @@ Work is also underway to provide abstractions for managing asynchronous tasks
391
388
at a higher level, focussing on workload distribution and load regulation.
392
389
393
390
The kinds of task that can be performed by the async implementations in
394
- [ distributed-process-platform] [ 3 ] are limited only by their return type:
391
+ [ distributed-process-platform] [ distributed-process-platform ] are limited only by their return type:
395
392
it ** must** be ` Serializable ` - that much should've been obvious by now.
396
393
The type of asynchronous task definitions comes in two flavours, one for
397
394
local nodes which require no remote-table or static serialisation dictionary,
@@ -400,11 +397,11 @@ and another for tasks you wish to execute on remote nodes.
400
397
{% highlight haskell %}
401
398
-- | A task to be performed asynchronously.
402
399
data AsyncTask a =
403
- AsyncTask
400
+ AsyncTask
404
401
{
405
402
asyncTask :: Process a -- ^ the task to be performed
406
403
}
407
- | AsyncRemoteTask
404
+ | AsyncRemoteTask
408
405
{
409
406
asyncTaskDict :: Static (SerializableDict a)
410
407
-- ^ the serializable dict required to spawn a remote process
@@ -430,14 +427,14 @@ domain was more *haskell-ish* than working with bare send and receive primitives
430
427
The ` Async ` sub-package also provides a type safe interface for receiving data,
431
428
although it is limited to running a computation and waiting for its result.
432
429
433
- The [ Control.Distributed.Processes.Platform.ManagedProcess] [ 21 ] API provides a
430
+ The [ Control.Distributed.Processes.Platform.ManagedProcess] [ d-p-platform-ManagedProcess ] API provides a
434
431
number of different abstractions that can be used to achieve similar benefits
435
432
in your code. It works by introducing a standard protocol between your process
436
433
and the * world outside* , which governs how to handle request/reply processing,
437
434
exit signals, timeouts, sleeping/hibernation with ` threadDelay ` and even provides
438
435
hooks that terminating processes can use to clean up residual state.
439
436
440
- The [ API documentation] [ 21 ] is quite extensive, so here we will simply point
437
+ The [ API documentation] [ d-p-platform-ManagedProcess ] is quite extensive, so here we will simply point
441
438
out the obvious differences. A process implemented with ` ManagedProcess `
442
439
can present a type safe API to its callers (and the server side code too!),
443
440
although that's not its primary benefit. For a very simplified example:
@@ -489,8 +486,8 @@ API, which looks a lot like `Async` but manages exit signals in a single thread
489
486
configurable task pools and task supervision strategy part of its API.
490
487
491
488
More complex examples of the ` ManagedProcess ` API can be seen in the
492
- [ Managed Processes tutorial] [ 22 ] . API documentation for HEAD is available
493
- [ here] [ 21 ] .
489
+ [ Managed Processes tutorial] ( tutorials/tutorial3.html ) . API documentation for HEAD is available
490
+ [ here] [ d-p-platform-ManagedProcess ] .
494
491
495
492
### Supervision Trees
496
493
@@ -500,25 +497,22 @@ TBC
500
497
501
498
TBC
502
499
503
- [ 1 ] : http://www.haskell.org/haskellwiki/Cloud_Haskell
504
- [ 2 ] : https://github.com/haskell-distributed/distributed-process
505
- [ 3 ] : https://github.com/haskell-distributed/distributed-process-platform
506
- [ 4 ] : http://hackage.haskell.org/package/distributed-static
507
- [ 5 ] : http://hackage.haskell.org/package/rank1dynamic
508
- [ 6 ] : http://hackage.haskell.org/package/network-transport
509
- [ 7 ] : http://hackage.haskell.org/package/network-transport-tcp
510
- [ 8 ] : https://github.com/haskell-distributed/network-transport-inmemory
511
- [ 9 ] : https://github.com/haskell-distributed/network-transport-composed
512
- [ 10 ] : http://hackage.haskell.org/package/distributed-process-simplelocalnet
513
- [ 11 ] : http://hackage.haskell.org/package/distributed-process-azure
514
- [ 12 ] : http://research.microsoft.com/en-us/um/people/simonpj/papers/parallel/remote.pdf
515
- [ 13 ] : http://en.wikipedia.org/wiki/Open_Telecom_Platform
516
- [ 14 ] : http://hackage.haskell.org/package/remote
517
- [ 15 ] : http://www.erlang.org/doc/design_principles/sup_princ.html
518
- [ 16 ] : http://www.erlang.org/doc/man/supervisor.html
519
- [ 17 ] : http://hackage.haskell.org/package/distributed-process-platform/Control-Distributed-Process-Platform-Async.html
520
- [ 18 ] : https://github.com/haskell-distributed/distributed-process-platform
521
- [ 19 ] : http://hackage.haskell.org/package/async
522
- [ 20 ] : /wiki/networktransport.html
523
- [ 21 ] : http://hackage.haskell.org/package/distributed-process-platform/Control-Distributed-Process-Platform-ManagedProcess.html
524
- [ 22 ] : /tutorials/tutorial3.html
500
+ [ cloud-haskell ] : http://haskell-distributed.github.io/documentation.html
501
+ [ fun201202-coutts ] : http://sneezy.cs.nott.ac.uk/fun/2012-02/coutts-2012-02-28.pdf
502
+ [ distributed-process ] : https://github.com/haskell-distributed/distributed-process
503
+ [ distributed-process-platform ] : https://github.com/haskell-distributed/distributed-process-platform
504
+ [ distributed-static ] : http://hackage.haskell.org/package/distributed-static
505
+ [ rank1dynamic ] : http://hackage.haskell.org/package/rank1dynamic
506
+ [ network-transport ] : http://hackage.haskell.org/package/network-transport
507
+ [ network-transport-tcp ] : http://hackage.haskell.org/package/network-transport-tcp
508
+ [ network-transport-inmemory ] : https://github.com/haskell-distributed/network-transport-inmemory
509
+ [ network-transport-composed ] : https://github.com/haskell-distributed/network-transport-composed
510
+ [ distributed-process-simplelocalnet ] : http://hackage.haskell.org/package/distributed-process-simplelocalnet
511
+ [ distributed-process-azure ] : http://hackage.haskell.org/package/distributed-process-azure
512
+ [ hdph ] : http://hackage.haskell.org/package/hdph
513
+ [ haskell11-ch ] : http://research.microsoft.com/en-us/um/people/simonpj/papers/parallel/remote.pdf
514
+ [ OTP ] : http://en.wikipedia.org/wiki/Open_Telecom_Platform
515
+ [ remote ] : http://hackage.haskell.org/package/remote
516
+ [ d-p-platform-async ] : http://hackage.haskell.org/package/distributed-process-platform/Control-Distributed-Process-Platform-Async.html
517
+ [ async ] : http://hackage.haskell.org/package/async
518
+ [ d-p-platform-ManagedProcess ] : http://hackage.haskell.org/package/distributed-process-platform/Control-Distributed-Process-Platform-ManagedProcess.html
0 commit comments