Merge pull request #837 from propensive/master

Include blog on Scala LSP meeting

Author: propensive

blog/_posts/2018-02-14-tooling.md

@@ -0,0 +1,203 @@
+---
+layout: blog-detail
+post-type: blog
+by: Jon Pretty
+title: "Towards A Brighter Tooling Future for Scala"
+---
+
+# Towards A Brighter Tooling Future for Scala
+
+Scala has, for its entire existence, been on the back foot with its tooling
+support. This was true when the language first arrived in 2004 as little more
+than a command-line compiler, through 2010 when Typesafe's (now Lightbend's)
+push for commercial adoption took an early focus on improving the Scala IDE for
+Eclipse, through to today, as the tooling experience is very much improved
+thanks in part to the work of JetBrains on IntelliJ, but still lags behind the
+tooling support some other languages enjoy.
+
+People care about language tools, and it says a lot for Scala that its success
+has come without ever having had the best tooling. But a better development
+experience in other languages over the years has undoubtedly limited the
+adoption of Scala.
+
+Now, in 2018, there have been some developments primarily outside of the Scala
+community which present a new opportunity to change that.
+
+## The Language Server Protocol
+
+The language server protocol (LSP) initiative, started by Microsoft, but
+already seeing wide adoption, is a language-agnostic protocol for allowing
+tools which consume information about code ("clients", such as IDEs) to
+communicate with compilers and other tools which can provide that information
+("servers", or as the protocol describes them, "language smartness providers").
+The LSP proposal has had early successes and has collected a lot of momentum
+behind it. So it's no surprise that there's already a lot of enthusiasm within
+the Scala community around the idea.
+
+During discussions in the last Scala Center Advisory Board meeting, one of the
+Scala Center sponsors proposed that the Scala Center could support the
+initiative, so we scheduled a meeting in January to get the ball rolling.
+
+With a many players from both commercial and open-source backgrounds already
+involved in tool development for Scala, we wanted to make sure that all the
+developers who had already contributed significantly to Scala's tooling could
+attend, but without having so many attendees as to make the meeting
+unmanageable. So we privately invited about fifty people suggested by the
+Advisory Board members, more than half of whom attended.
+
+This "closed doors" approach for the initial meeting was a practical
+compromise, made just for this one-off meeting. There are certainly other
+people in the community who have contributed to tooling who didn't get the
+chance to attend. To those we overlooked: sorry! we certainly want to include
+everyone who cares deeply about Scala's tooling, but it wasn't possible to run
+a meeting with more than about thirty attendees.
+
+## The Initial Meeting
+
+The goals of this first meeting were, nevertheless, modest.
+
+Firstly, with a number of different existing Scala tools which could play a
+part in Scala's LSP story, almost nobody in the meeting was familiar with them
+all, so to get everyone to a common understanding of the ecosystem, the largest
+part of the meeting was spent hearing different tool authors speak about their
+tools, and how they might play a part in the jigsaw.
+
+Secondly, we wanted to decide upon what our next steps should be, and how the
+Scala Center could help the initiative. This was a task of setting up a
+framework in which decisions about the Scala LSP initiative could be made in
+future, in a way which the majority of people supported.
+
+During the meeting, we heard presentations on
+[Scalameta](http://scalameta.org/) and
+[SemanticDB](https://github.com/scalameta/scalameta/blob/master/semanticdb/README.md),
+[Metals](https://github.com/scalameta/metals),
+[Bloop](https://github.com/scalacenter/bloop),
+[SBT](https://www.scala-sbt.org/),
+[Pants](https://www.pantsbuild.org/index.html), [Bazel](https://bazel.build/),
+[Ensime](https://github.com/ensime), [IntelliJ
+IDE](https://www.jetbrains.com/idea/), [Scala IDE](http://scala-ide.org/),
+[Dotty IDE](http://guillaume.martres.me/ide_paper.pdf) (Visual Studio Code),
+and [TASTY](http://goo.gl/Mn6EhH).
+
+Another important premise for the meeting was that the tooling should be, as
+much as possible, compiler-agnostic. That is to say, we should not need to use
+a different protocol to work with Scala 2.11 or 2.12 or Dotty, and nor should
+users receive a different front-end experience for any Scala-like language.
+This support for other compilers is important for two reasons, even though the
+vast majority of current users run Scalac.
+
+Firstly, compilers like Dotty and
+[Rsc](https://github.com/twitter/reasonable-scala) want to hit the ground
+running with their tooling support; waiting another decade for a new tooling
+ecosystem to be invented is not an option.  When the time comes for Scala users
+to upgrade to Dotty, nobody wants the additional barrier of having to upgrade
+their build tool, documentation tools or IDE at the same time. They want a
+seamless transition. That's much easier to achieve by considering other
+compilers *now* than would be later.
+
+The second reason is that there is a huge amount of excitement and enthusiasm
+around these new compilers, and we want to channel that into the Scala LSP
+initiative, and take advantage of the commitment from these other developers to
+pull together towards a common goal.
+
+In the meeting we heard representations from the project leads of the Scalac
+compiler, Dotty, Rsc and Hydra about any unique features of their compiler
+implementations that might have any bearing on the LSP initiative.
+
+We didn't have time during the meeting for much detailed discussion, though the
+attendees talked about a number of ideas and concerns around what shape the
+initiative should take and how it should proceed. One distinction that was
+clear throughout the discussion was the need to separate a Scala implementation
+of the LSP (the protocol as described my Microsoft), from a more general "build
+server protocol" which would allow for the exchange of other information
+specific to Scala-like languages, like dependencies and classpaths, and this
+is reflected the main outcome of the discussion.
+
+Minutes for the meeting are available in the
+[scalacenter/tooling-working-groups
+repository](https://github.com/scalacenter/tooling-working-groups/blob/master/meetings/2018-01-17/minutes.md)
+
+## Next Steps
+
+I made a proposal during the meeting that the next steps should include the
+election of two working groups to take forward ideas for an implementation of
+LSP for Scala, and a second "Scala Tooling Protocols" group for defining other
+non-LSP protocols for Scala. These groups should have,
+
+ - 6-9 members in each
+ - some overlap in membership between the two groups
+ - a mix of experts (more experience than free time) and enthusiasts (more free
+   time than experience)
+ - diverse representation of a variety of existing projects
+ - at least one member from the Scala Center, but not overrepresentation
+
+As an open-source project, the dynamics of a disparate collection of developers
+is often better suited to innovation than shipping working software by a
+deadline, and a large part of the challenge will be to keep everyone on track,
+despite the freedom that Open Source grants us. We hope that that leadership
+can come from the Scala Center.
+
+## Election Process
+
+We want to encourage an open process for discussion of tooling for Scala. We
+want to involve many diverse minds, and we want to be inclusive. In the past,
+it has often been most expedient to invite a selection of known experts to take
+part. This has largely led to technical success, but has made it harder for
+many newcomers to engage with the Scala decision-making process. So to try to
+improve upon this, and to give the working groups greater legitimacy, we will
+be holding elections to choose the members of the LSP and STP working groups.
+This will, in part, be an experiment, but we hope it signals greater openness
+in how decisions which affect the entire Scala community are made.
+
+If the Scala community had a list of its members, we could invite each member
+to vote on the makeup of the working groups, but of course, that's not how the
+community works. We also do not want to open an election up to anything that
+could be so trivially compromised as an online poll.
+
+We will therefore take an imperfect but pragmatic approach and open voting only
+to those members of the Scala community who are willing to devote a small
+amount of their time to participating in a series of live votes, conducted
+online, which will take place on Wednesday, 28 February starting at 6pm GMT.
+Full details of the rules governing this process are [described
+here](https://github.com/scalacenter/tooling-working-groups/blob/master/nominations/election.md).
+
+We will announce the composition of the working groups after the voting
+immediately after voting has finished.
+
+## Becoming a Working Group Member
+
+Sitting as a member on either of the working groups is an opportunity to
+participate and influence an exciting area of development for Scala, and will
+be a position of some respect and privilege. But bootstrapping these groups in
+a way that is fair is harder than it might at first seem: the nature of our
+initial meeting meant that not everyone could be involved, but we do not want
+to exclude anyone from the opportunity to sit on these working groups.
+
+We feel that it is very important to include a combination of experience and
+energy in both working groups, so whilst the expertise of their members will be
+essential to their success, a willingness to participate and contribute are
+also extremely valuable selection criteria, and anyone lacking experience but
+having a strong desire to be involved and dedicated to the cause should not
+hesitate to seek candidacy: that commitment will be very valuable to the
+working groups.
+
+Candidates to the working group should be nominated by opening a pull request
+to the [scalacenter/tooling-working-groups
+repository](https://github.com/scalacenter/tooling-working-groups), following
+the [instructions described
+here](https://github.com/scalacenter/tooling-working-groups/blob/master/nominations/README.md),
+and we strongly encourage anyone with an interest in Scala's tooling, and a
+willingness to devote some time to participating in working group discussions
+to be a candidate.
+
+I will be giving a keynote about the future of Scala Tooling at [Scala
+Sphere](http://scala.sphere.it/) in April, a large conference in Kraków
+dedicated to making Scala developer tools better, now in its third year. I hope
+to be making some exciting announcements there about our prospects for vastly
+improved tooling support for Scala! Meanwhile, the working groups will have a
+challenging task ahead of them, but with a their collective experience, the
+energy and enthusiasm Scala's tooling contributors, and the support of the
+entire community, we can be much more optimistic about that future than ever
+before.
+
+