updated site

Author: eed3si9n

2.x/docs/en/changes/migrating-from-sbt-1.x.html

@@ -203,6 +203,11 @@ <h2 id="bare-settings-changes"><a class="header" href="#bare-settings-changes">B
 </code></pre>
 <h3 id="migrating-thisbuild"><a class="header" href="#migrating-thisbuild">Migrating ThisBuild</a></h3>
 <p>In sbt 2.x, bare settings settings should no longer be scoped to <code>ThisBuild</code>. One benefit of the new <em>common settings</em> over <code>ThisBuild</code> is that it would act in a more predictable delegation. These settings are inserted between plugins settings and those defined in <code>settings(...)</code>, meaning they can be used to define settings like <code>Compile / scalacOptions</code>, which was not possible with <code>ThisBuild</code>.</p>
+<h2 id="migrating-to-slash-syntax"><a class="header" href="#migrating-to-slash-syntax">Migrating to slash syntax</a></h2>
+<p>sbt 1.x supported both the sbt 0.13 style syntax and the slash syntax. sbt 2.x removes the support for the sbt 0.13 syntax, so use the slash syntax for both sbt shell and in <code>build.sbt</code>:</p>
+<pre><code class="language-scala">&lt;project-id&gt; / Config / intask / key
+</code></pre>
+<p>For example, <code>test:compile</code> will no longer work on the shell. Use <code>Test/compile</code> instead. See <a href="https://eed3si9n.com/syntactic-scalafix-rule-for-unified-slash-syntax/">syntactic Scalafix rule for unified slash syntax</a> for semi-automated migration of <code>build.sbt</code> files.</p>
 <h2 id="cross-building-sbt-plugins"><a class="header" href="#cross-building-sbt-plugins">Cross building sbt plugins</a></h2>
 <p>In sbt 2.x, if you cross build an sbt plugin with Scala 3.x and 2.12.x, it will automatically cross build against sbt 1.x and sbt 2.x:</p>
 <pre><code class="language-scala">// using sbt 2.x
@@ -233,18 +238,59 @@ <h3 id="cross-building-sbt-plugin-with-sbt-1x"><a class="header" href="#cross-bu
     },
   )
 </code></pre>
-<h2 id="migrating-to-slash-syntax"><a class="header" href="#migrating-to-slash-syntax">Migrating to slash syntax</a></h2>
-<p>sbt 1.x supported both the sbt 0.13 style syntax and the slash syntax. sbt 2.x removes the support for the sbt 0.13 syntax, so use the slash syntax for both sbt shell and in <code>build.sbt</code>:</p>
-<pre><code class="language-scala">&lt;project-id&gt; / Config / intask / key
-</code></pre>
-<p>For example, <code>test:compile</code> will no longer work on the shell. Use <code>Test/compile</code> instead. See <a href="https://eed3si9n.com/syntactic-scalafix-rule-for-unified-slash-syntax/">syntactic Scalafix rule for unified slash syntax</a> for semi-automated migration of <code>build.sbt</code> files.</p>
 <h2 id="changes-to-"><a class="header" href="#changes-to-">Changes to <code>%%</code></a></h2>
 <p>In sbt 2.x, <code>ModuleID</code>'s <code>%%</code> operator has become platform-aware. For JVM subprojects, <code>%%</code> works as before, encoding Scala suffix (for example <code>_3</code>) on Maven repositories.</p>
 <h3 id="migrating--operator"><a class="header" href="#migrating--operator">Migrating <code>%%%</code> operator</a></h3>
 <p>When Scala.JS or Scala Native becomes available on sbt 2.x, <code>%%</code> will encode both the Scala version (such as <code>_3</code>) and the platform suffix (<code>_sjs1</code> etc). As a result, <code>%%%</code> can be replaced with <code>%%</code>:</p>
 <pre><code class="language-scala">libraryDependencies += "org.scala-js" %% "scalajs-dom" % "2.8.0"
 </code></pre>
 <p>Use <code>.platform(Platform.jvm)</code> in case where JVM libraries are needed.</p>
+<h2 id="the-plugincompat-technique"><a class="header" href="#the-plugincompat-technique">The PluginCompat technique</a></h2>
+<p>To use the same <code>*.scala</code> source but target both sbt 1.x and 2.x, we can create a shim, for example an object named <code>PluginCompat</code> in both <code>src/main/scala-2.12/</code> and <code>src/main/scala-3/</code>.</p>
+<h3 id="migrating-classpath-type"><a class="header" href="#migrating-classpath-type">Migrating Classpath type</a></h3>
+<p>sbt 2.x changed the <code>Classpath</code> type to be an alias of the <code>Seq[Attributed[xsbti.HashedVirtualFileRef]]</code> type. The following is a shim created to work with classpaths from both sbt 1.x and 2.x.</p>
+<pre><code class="language-scala">// src/main/scala-3/PluginCompat.scala
+
+package sbtfoo
+
+import java.nio.file.{ Path =&gt; NioPath }
+import sbt.*
+import xsbti.{ FileConverter, HashedVirtualFileRef, VirtualFile }
+
+private[sbtfoo] object PluginCompat:
+  type FileRef = HashedVirtualFileRef
+  type Out = VirtualFile
+
+  def toNioPath(a: Attributed[HashedVirtualFileRef])(using conv: FileConverter): NioPath =
+    conv.toPath(a.data)
+  inline def toFile(a: Attributed[HashedVirtualFileRef])(using conv: FileConverter): File =
+    toNioPath(a).toFile()
+  def toNioPaths(cp: Seq[Attributed[HashedVirtualFileRef]])(using conv: FileConverter): Vector[NioPath] =
+    cp.map(toNioPath).toVector
+  inline def toFiles(cp: Seq[Attributed[HashedVirtualFileRef]])(using conv: FileConverter): Vector[File] =
+    toNioPaths(cp).map(_.toFile())
+end PluginCompat
+</code></pre>
+<p>and here's for sbt 1.x:</p>
+<pre><code class="language-scala">// src/main/scala-2.12/PluginCompat.scala
+
+package sbtfoo
+
+private[sbtfoo] object PluginCompat {
+  type FileRef = java.io.File
+  type Out = java.io.File
+
+  def toNioPath(a: Attributed[File])(implicit conv: FileConverter): NioPath =
+    a.data.toPath()
+  def toFile(a: Attributed[File])(implicit conv: FileConverter): File =
+    a.data
+  def toNioPaths(cp: Seq[Attributed[File]])(implicit conv: FileConverter): Vector[NioPath] =
+    cp.map(_.data.toPath()).toVector
+  def toFiles(cp: Seq[Attributed[File]])(implicit conv: FileConverter): Vector[File] =
+    cp.map(_.data).toVector
+}
+</code></pre>
+<p>Now we can import <code>PluginCompat.*</code> and use <code>toNioPaths(...)</code> etc to absorb the differences between sbt 1.x and 2.x. The above demonstrates how we can absorb the classpath type change, and convert it into a vector of NIO Paths.</p>
 
                     </main>
 

2.x/docs/en/changes/sbt-2.0-change-summary.html

@@ -198,6 +198,7 @@ <h2 id="changes-with-compatibility-implications"><a class="header" href="#change
 <li>sbt 2.x plugins are published with <code>_sbt2_3</code> suffix by <a href="https://github.com/eed3si9n">@eed3si9n</a> in <a href="https://github.com/sbt/sbt/pull/7671">#7671</a></li>
 <li>sbt 2.x adds <code>platform</code> setting so <code>ModuleID</code>'s <code>%%</code> operator can cross build on JVM as well as JS and Native, as opposed to <code>%%%</code> operator that was created in a plugin to workaround this issue, by <a href="https://github.com/eed3si9n">@eed3si9n</a> in <a href="https://github.com/sbt/sbt/pull/6746">#6746</a></li>
 <li>Dropped <code>useCoursier</code> setting so Coursier cannot be opted out, by <a href="https://github.com/eed3si9n">@eed3si9n</a> in <a href="https://github.com/sbt/sbt/pull/7712">#7712</a></li>
+<li><code>Key.Classpath</code> is changed to be an alias of the <code>Seq[Attributed[xsbti.HashedVirtualFileRef]]</code> type, instead of <code>Seq[Attributed[File]]</code>. Similarly, some task keys that used to return <code>File</code> have changed to return <code>HashedVirtualFileRef</code> instead. See <a href="../concepts/caching.html#caching-files">Caching Files</a>.</li>
 </ul>
 <h3 id="dropped-dreprecations"><a class="header" href="#dropped-dreprecations">Dropped dreprecations</a></h3>
 <ul>

2.x/docs/en/concepts/caching.html

@@ -205,6 +205,33 @@ <h2 id="automatic-caching"><a class="header" href="#automatic-caching">Automatic
 [info] demo0.1.0-SNAPSHOT!
 [success] elapsed time: 0 s, cache 100%, 1 disk cache hit
 </code></pre>
+<h3 id="caching-is-serializaiton-hard"><a class="header" href="#caching-is-serializaiton-hard">Caching is serializaiton-hard</a></h3>
+<p>To participate in the automatic caching, the input keys (e.g. <code>name</code> and <code>version</code>) must provide a given for <code>sjsonnew.HashWriter</code> typeclass and return type must provide a given for <code>sjsonnew.JsonFormat</code>. <a href="https://www.scala-sbt.org/contraband/">Contraband</a> can be used to generate sjson-new codecs.</p>
+<h2 id="caching-files"><a class="header" href="#caching-files">Caching files</a></h2>
+<p>Caching files (e.g. <code>java.io.File</code>) requires its own consideration, not because it's technically difficult, but mostly because of the ambiguity and assumptions when files are involved. When we say a "file" it could actually mean:</p>
+<ol>
+<li>Relative path from a well-known location</li>
+<li>Materialized actual file</li>
+<li>A unique proof of a file, or a content hash</li>
+</ol>
+<p>Technically speaking, a <code>File</code> just means the file path, so we can deserialize just the filename such as <code>target/a/b.jar</code>. This will fail the downstream tasks if they assumed that <code>target/a/b.jar</code> would exist in the file system. For clarity, and also for avoiding to capture absolute paths, sbt 2.x provides three separate types for the three cases.</p>
+<ul>
+<li><code>xsbti.VirtualFileRef</code> is used to mean just the relative path, which is equivalent to passing a string</li>
+<li><code>xsbti.VirtualFile</code> represents a materialized file with contents, which could be a virtual file or a file in your disk</li>
+</ul>
+<p>However, for the purpose of hermetic build, neither is great to represent a list of files. Having just the filename alone doesn't guarantee that the file will be the same, and carrying the entire content of the files is too inefficient in a JSON etc.</p>
+<p>This is where the mysterious third option, a unique proof of file comes in handy. In addition to the relative path, <code>HashedVirtualFileRef</code> tracks the SHA-256 content hash and the file size. This can easily be serialized to JSON yet we can reference the exact file.</p>
+<h3 id="the-effect-of-file-creation"><a class="header" href="#the-effect-of-file-creation">The effect of file creation</a></h3>
+<p>There are many tasks that generate file that do not use <code>VirtualFile</code> as the return type. For example, <code>compile</code> returns <code>Analysis</code> instead, and <code>*.class</code> file generation happens as a <em>side effect</em> in sbt 1.x.</p>
+<p>To participate in caching, we need to declare these effects as something we care about.</p>
+<pre><code class="language-scala">someKey := Def.cachedTask {
+  val conv = fileConverter.value
+  val out: java.nio.file.Path = createFile(...)
+  val vf: xsbti.VirtualFile = conv.toVirtualFile(out)
+  Def.declareOutput(vf)
+  vf: xsbti.HashedVirtualFileRef
+}
+</code></pre>
 <h2 id="remote-caching"><a class="header" href="#remote-caching">Remote caching</a></h2>
 <p>You can optionally extend the build to use remote cache in addition to the local disk cache. Remote caching could improve build performance by allowing multiple machines to share build artifacts and outputs.</p>
 <p>Imagine you have a dozen people in your project or a company. Each morning, you will <code>git pull</code> the changes the dozen people made, and you need to build their code. If you have a successful project, the code size will only get bigger over time, and the % of the time you spend building someone else's in your day increases. This becomes the limiting factor of your team size and code size. Remote caching reverses this tide by CI systems hydrate the cache and you can download the artifacts and task outputs.</p>