tidy up

Author: Tim Watson

Diff for: _layouts/documentation.html

@@ -22,8 +22,7 @@
           <div data-spy="affix" data-offset-bottom="290">
             <ul class="nav nav-list sidenav">
               <li><a href="#cloud_haskell_platform"><i class="icon-chevron-right"></i> Cloud Haskell Platform</a></li>
-              <li><a href="#architecture"><i class="icon-chevron-right"></i> Architecture</a></li>
-              <li><a href="#network_abstraction_layer"><i class="icon-chevron-right"></i> Network Abstraction Layer</a></li>
+              <li><a href="#network_transport_abstraction_layer"><i class="icon-chevron-right"></i> Network Abstraction Layer</a></li>
               <li><a href="#concurrency_and_distribution"><i class="icon-chevron-right"></i> Concurrency and Distribution</a></li>
               <li><a href="#rethinking_the_task_layer"><i class="icon-chevron-right"></i> Rethinking the Task Layer</a></li>
            </ul>

Diff for: documentation.md

@@ -22,12 +22,6 @@ growing number of features for
 * working with several network transport implementations (and more in the pipeline)
 * supporting *static* values (required for remote communication)
 
-API documentation for the latest releases is available on hackage. The latest
-(HEAD) API documentation for the platform can be viewed
-[here](/static/doc/distributed-process-platform/index.html).
-
-### Architecture
-
 Cloud Haskell comprises the following components, some of which are complete,
 others experimental.
 
@@ -42,9 +36,8 @@ others experimental.
 * [distributed-process-simplelocalnet][10]: Simple backend for local networks
 * [distributed-process-azure][11]: Azure backend for Cloud Haskell (proof of concept)
 
-
-One goal of Cloud Haskell is to separate the transport layer from the
-process layer, so that the transport backend is entirely independent:
+One of Cloud Haskell's goals is to separate the transport layer from the
+*process layer*, so that the transport backend is entirely independent:
 it is envisaged that this interface might later be used by models
 other than the Cloud Haskell paradigm, and that applications built
 using Cloud Haskell might be easily configured to work with different
@@ -63,6 +56,30 @@ The following diagram shows dependencies between the various subsystems,
 in an application using Cloud Haskell, where arrows represent explicit
 directional dependencies.
 
+-----
+
+    +------------------------------------------------------------+
+    |                        Application                         |
+    +------------------------------------------------------------+
+                 |                               |
+                 V                               V
+    +-------------------------+   +------------------------------+
+    |      Cloud Haskell      |<--|    Cloud Haskell Backend     |
+    |  (distributed-process)  |   | (distributed-process-...)    |
+    +-------------------------+   +------------------------------+
+                 |           ______/             |
+                 V           V                   V
+    +-------------------------+   +------------------------------+
+    |   Transport Interface   |<--|   Transport Implementation   |
+    |   (network-transport)   |   |   (network-transport-...)    |
+    +-------------------------+   +------------------------------+
+                                                 |
+                                                 V
+                                  +------------------------------+
+                                  | Haskell/C Transport Library  |
+                                  +------------------------------+
+
+-----
 
 In this diagram, the various nodes roughly correspond to specific modules:
 
@@ -184,25 +201,40 @@ pass data between processes using *ordinary* concurrency primitives such as
 types like `TMVar a` just as normal Haskell threads are. Numerous features
 in [distributed-process-platform][3] use this facility, for example the way
 that `Control.Distributed.Processes.Platform.Async.AsyncSTM` handles passing
-the result of its computation back to the caller:
+the result of its computation back to the caller, as the following snippet
+demonstrates:
+
+----
 
 {% highlight haskell %}
-  workerPid <- spawnLocal $ do
-        -- ... some setup
-        r <- proc
-        void $ liftIO $ atomically $ putTMVar result (AsyncDone r)
+    root <- getSelfPid
+    result <- liftIO $ newEmptyTMVarIO
+    sigStart <- liftIO $ newEmptyTMVarIO
+    (sp, rp) <- newChan
+
+    -- listener/response proxy
+    insulator <- spawnLocal $ do
+        worker <- spawnLocal $ do
+            liftIO $ atomically $ takeTMVar sigStart
+            r <- proc
+            void $ liftIO $ atomically $ putTMVar result (AsyncDone r)
+
+        sendChan sp worker  -- let the parent process know the worker pid
+
+        wref <- monitor worker
+        rref <- case shouldLink of
+                    True  -> monitor root >>= return . Just
+                    False -> return Nothing
+        finally (pollUntilExit worker result)
+                (unmonitor wref >>
+                    return (maybe (return ()) unmonitor rref))
+
+    workerPid <- receiveChan rp
+    liftIO $ atomically $ putTMVar sigStart ()
+    -- etc ....
 {% endhighlight %}
 
-For example, we might implement a local process group using *only* message
-passing, and when members enter or leave the group, a *master* process does
-the book keeping to ensure the other members of the group can retain a
-consist view. If we want to introduce a *group level barrier* to facilitate
-mutual exclusion, we have two choices for handling this. If the process
-group allows members to enter and leave on an ad-hoc basis, then a shared
-memory based solution is a poor choice, because there is no *sane* way to
-pass the `MVar` (or whatever) to new joiners. Locking is best achieved using
-a messaging based protocol in this instance, which complicates the implementation
-
+----
 
 Processes reside on nodes, which in our implementation map directly to the
 `Control.Distributed.Processes.Node` module. Given a configured

Diff for: wiki/maintainers.md

@@ -9,32 +9,88 @@ wiki: Maintainers
 This guide is specifically for maintainers, and outlines the
 development process and in particular, the branching strategy.
 
-#### Master == Stable
+#### Branching/Merging Policy
 
 The master branch is the **stable** branch, and should always be
 in a *releasable* state. This means that on the whole, only small
 self contained commits or topic branch merges should be applied
 to master, and tagged releases should always be made against it.
 
-#### Development
+#### Development Branches
 
 Ongoing work can either be merged into master when complete or
 merged into development. Development is effectively an integration
 branch, to make sure ongoing changes and new features play nicely
-with one another.
+with one another. On the other hand, master is a 'stable' branch
+and therefore you should only merge into it if the result will be
+releasable.
 
-#### Releases
+In general, we try to merge changes that belong to a major version
+upgrade into development, whereas changes that will go into the
+next minor version upgrade can be merged into master.
 
-Remember to update the change log for each project when releasing it.
-I forgot to add the changes to the changelog when tagging the recent
-distributed-process-0.4.2 release, but in general they should be added
-*before* tagging the release.
+#### Keeping History
+
+Try to make only clean commits, so that bisect will continue to work.
+At the same time, it's best to avoid making destructive updates. If
+you're planning on doing lots of squashing, then work in a branch
+and don't commit directly to development - and **definitely** not to
+master.
 
 #### Follow the <a href="/wiki/contributing.html">Contributing guidelines</a>
 
 What's good for the goose...
 
-#### After releasing, send out a mail
+#### Making API documentation available on the website
+
+Currently this is a manual process. If you don't sed/awk out the
+reference/link paths, it'll be a mess. We will add a script to
+handle this some time soon.
+
+There is also an open ticket to set up nightly builds, which will
+update the HEAD haddocks (on the website) and produce an 'sdist'
+bundle and add that to the website too.
+
+See https://cloud-haskell.atlassian.net/browse/INFRA-1 for details.
+
+### Release Process
+
+First of all, a few prior warnings. **Do not** tag any projects
+until *after* you've finished the release. If you build and tag
+three projects, only to find that a subsequent dependent package
+needs a bit of last minute surgery, you'll be sorry you didn't
+wait. With that in mind....
+
+Before releasing any source code, make sure that all the jira tickets
+added to the release are either resolved or remove them from the
+release if you've decided to exclude them.
+
+First, make sure all the version numbers and dependencies are aligned.
+
+* bump the version numbers for each project that is being released
+* bump the dependency versions across each project if needed
+* make sure you've run a complete integration build and everything still installs ok
+* bump the dependency version numbers in the cloud-haskell meta package
+
+Now you'll want to go and update the change log and release notes for each
+project. Change logs are stored in the individual project's git repository,
+whilst release notes are stored on the wiki. This is easy to forget, as I
+discovered! Change logs should be more concise than full blown release
+notes.
+
+Generate the packages with `cabal sdist` and test them all locally, then
+upload them to hackage. Don't forget to upload the cloud-haskell meta-package
+too!
+
+#### After the release
+
+**Now** you should tag all the projects with the relevant version number.
+Since moving to individual git repositories, the tagging scheme is now
+`x.y.z` and *not* `<project>-x.y.z`.
+
+Once the release is out, you should go to [JIRA](https://cloud-haskell.atlassian.net)
+and close all the tickets for the release. Jira has a nice 'bulk change'
+feature that makes this very easy.
 
-To the Parallel Haskell Mailing List, and anywhere else that makes
-sense.
+After that, it's time to tweet about the release, post to the parallel-haskell
+mailing list, blog etc. Spread the word.

Diff for: wiki/networktransport.md

@@ -38,7 +38,7 @@ completely stable yet. The design of the transport layer may also still change.
 Feedback and suggestions are most welcome. Email [Duncan](mailto:[email protected]) or [Edsko](mailto:[email protected]) at Well-Typed, find us at #HaskellTransportLayer on
 freenode, or post on the [Parallel Haskell][2] mailing list.
 
-You may also submit issues on the [JIRA issue tracker][3].
+You may also submit issues on the [JIRA issue tracker][8].
 
 ### Hello World
 
@@ -112,3 +112,4 @@ If you are interested in helping out, please add a brief paragraph to
 [5]: /wiki/newtransports.html
 [6]: /wiki/newdesign.html
 [7]: /wiki/protocols.html
+[8]: https://cloud-haskell.atlassian.net/issues/?filter=10002

Diff for: wiki/newdesign.md

@@ -100,6 +100,8 @@ A particular challenge is the per-connection performance parameters. It is vital
 
 The following diagram shows dependencies between the various modules for the initial Cloud Haskell implementation. Arrows represent explicit module dependencies.
 
+----
+
     +------------------------------+
     |        Application           |
     +------------------------------+
@@ -114,11 +116,14 @@ The following diagram shows dependencies between the various modules for the ini
     | Haskell network (IP) library |
     +------------------------------+
 
+----
+
 As the diagram indicates, the initial implementation is monolithic and uses a single specific transport (TCP/IP).
 
 The next diagram shows the various modules that are envisaged in the new design. We partition the system into the Cloud Haskell layer and a separate network transport layer. Each of the two layers has backend packages for different transports.
 
-{% highlight %}
+----
+
     +------------------------------------------------------------+
     |                        Application                         |
     +------------------------------------------------------------+
@@ -139,7 +144,8 @@ The next diagram shows the various modules that are envisaged in the new design.
                                   +------------------------------+
                                   | Haskell/C Transport Library  |
                                   +------------------------------+
-{% endhighlight %}
+
+----
 
 We still expect applications to use the the Cloud Haskell layer directly. Additionally the application also depends on a specific Cloud Haskell backend, which provides functions to allow the initialisation of the transport layer using whatever topology might be appropriate to the application.