[WIP]Add provider-wide diagnostic collection

CHANGELOG.md

@@ -4,8 +4,10 @@
 
 ### Added
 
-- include a Coder support bundle for the selected workspace and agent only when using the environment's
-  _Collect logs_ action in Toolbox
+- include a Coder support bundle for the selected workspace and agent when using the environment's _Collect logs_ action
+  in Toolbox
+- add a Coder provider _Collect detailed deployment and Toolbox logs_ action that bundles diagnostics for all accessible
+  workspaces and agents alongside local Toolbox and JetBrains daemon logs on Windows, macOS, and Linux
 
 ### Changed
 

README.md

@@ -385,24 +385,74 @@ in Coder or within the individual workspace view, under the option labeled _Coll
 
 ### Coder Support Bundles
 
-The plugin includes a Coder support bundle **only when you use the environment's _Collect logs_ action** in Toolbox.
-Open the action menu for the environment on Coder's Workspaces page or within its workspace view, then select _Collect
-logs_. The general **Settings > About > Collect logs and diagnostic data** action does not generate a
-Coder support bundle.
-
-The plugin runs `coder support bundle` using the deployment's existing CLI login and targets the selected workspace
-and its agent, when available. It places `coder-support.zip` inside the environment's diagnostic directory in the
-Toolbox log archive. Collection does not start a stopped workspace.
-
-The bundle can contain deployment health and configuration, network diagnostics, workspace build logs, template
-source, and agent diagnostics. Available information depends on your permissions and workspace connectivity.
-Review the bundle before sharing, following
+Toolbox's general **Settings > About > Collect logs and diagnostic data** action does not allow providers to attach
+additional diagnostic bundles. The Coder plugin therefore provides its own action to combine detailed Coder
+diagnostics with local Toolbox logs in one archive.
+
+The plugin offers two collection actions with different scopes:
+
+- **Environment _Collect logs_**: open the action menu for an environment on Coder's Workspaces page or within its
+  workspace view. Coder bundle collection is limited to **that workspace and its selected agent**, when available.
+  The plugin adds the bundle as `coder-support.zip`
+  inside the environment's diagnostic directory in the Toolbox log archive.
+- **Coder provider _Collect detailed deployment and Toolbox logs_**: open the Coder provider menu at the top of the
+  workspace list. The plugin creates `coder-toolbox-logs.zip` with support bundles for **all workspaces accessible to
+  your account on the connected deployment**, regardless of the current workspace filter. The plugin collects each
+  agent separately and includes workspaces without agents. The archive also contains the full local Toolbox and
+  JetBrains daemon log directories, including rotated logs and nested plugin logs. These are the local log sources
+  used by Toolbox's general About action. Remote IDE logs are not collected by this provider action.
+
+The general **Settings > About > Collect logs and diagnostic data** action does not generate Coder support bundles.
+
+The plugin runs `coder support bundle` using the deployment's existing CLI configuration and normal Coder environment
+variable overrides. It passes an explicit workspace and agent, or just the workspace when no agent is available.
+Collection does not start a stopped workspace.
+
+Bundles can contain deployment health and configuration, network diagnostics, workspace build logs, template
+source, and agent diagnostics. Available information depends on your permissions, CLI version, and workspace
+connectivity. Review the bundle before sharing, following
 [Coder's support-bundle guidance](https://coder.com/docs/support/support-bundle).
 
 Support bundles require a Coder CLI that supports `coder support bundle` (Coder 2.10 or newer). Collection runs until
 the CLI finishes or you cancel log collection. If the CLI is unavailable or unsupported, or authentication or
-connectivity fails, the plugin removes any partial bundle and attempts to include
-`coder-support-error.txt` instead. Toolbox's other diagnostics remain available.
+connectivity fails, the plugin removes any partial bundle and attempts to include `coder-support-error.txt` instead.
+Toolbox's other diagnostics remain available.
+
+#### Provider Archive and Collection
+
+The provider archive contains `workspaces/`, `toolbox/`, and `daemon/`. Workspace and agent UUIDs form the directories
+under `workspaces/`; each target's `workspace.txt` identifies it, and `coder-support.zip` contains its bundle.
+Bundles run sequentially without an overall timeout. One failed bundle does not prevent collection of the remaining
+workspaces or local logs. Collection failures and missing or unreadable local logs are recorded in
+`collection-report.txt`.
+
+Local logs are copied after Coder bundles finish so they include messages produced during collection. Copies stop
+at each file's observed size. Symbolic links and the collector's own temporary directory are excluded.
+
+During collection, progress and a Cancel button replace the filters in the expanded Coder header. The filters and
+their previous selections return when collection ends.
+Closing the provider also cancels collection. Cancellation terminates
+the active Coder CLI process and removes the unfinished archive and staging files. A successful archive remains in
+a temporary directory, and Toolbox opens its enclosing folder and selects the archive in your file manager.
+
+#### Local Log Locations
+
+| Platform | Toolbox                                                   | JetBrains daemon                                         |
+|----------|-----------------------------------------------------------|----------------------------------------------------------|
+| Windows  | Local AppData known folder, then `JetBrains/Toolbox/logs` | `LOCALAPPDATA`, then `JetBrains/Daemon/logs`             |
+| macOS    | `~/Library/Logs/JetBrains/Toolbox`                        | `~/Library/Logs/JetBrains/Daemon`                        |
+| Linux    | `${XDG_DATA_HOME:-~/.local/share}/JetBrains/Toolbox/logs` | `${XDG_DATA_HOME:-~/.local/share}/JetBrains/Daemon/logs` |
+
+Toolbox's `toolbox.log.path` JVM property overrides its log location on every platform. On Windows and Linux,
+`toolbox.localAppData.path` overrides the base data location when there is no explicit log path. It does not affect
+the daemon. Windows falls back to `~/AppData/Local` when the native known-folder lookup fails; the daemon uses that
+fallback when `LOCALAPPDATA` is unavailable. JNA provides the same Windows known-folder lookup used by Toolbox,
+including redirected user folders.
+
+For the daemon, `JETBRAINS_DAEMON_SNAPSHOT_BASE_PATH` plus `logs` takes precedence over
+`JETBRAINS_DAEMON_LOG_DIRECTORY`, which takes precedence over platform defaults. `JETBRAINS_DAEMON_DATA_DIRECTORY`
+does not change the log location. The Toolbox defaults above target release builds; development builds using
+`Toolbox-Dev` need an explicit `toolbox.log.path` override.
 
 ### HTTP Request Logging
 

build.gradle.kts

@@ -44,6 +44,7 @@ dependencies {
     implementation(libs.retrofit)
     implementation(libs.retrofit.moshi)
     implementation(libs.bundles.bouncycastle)
+    implementation(libs.jna.platform)
 
     testImplementation(kotlin("test"))
     testImplementation(libs.coroutines.test)

gradle/libs.versions.toml

@@ -18,6 +18,7 @@ mockk = "1.14.11"
 detekt = "1.23.8"
 bouncycastle = "1.86"
 jackson = "2.22.2"
+jna = "5.17.0"
 
 [libraries]
 toolbox-core-api = { module = "com.jetbrains.toolbox:core-api", version.ref = "toolbox-plugin-api" }
@@ -40,6 +41,7 @@ marketplace-client = { module = "org.jetbrains.intellij:plugin-repository-rest-c
 bouncycastle-bcpg = { module = "org.bouncycastle:bcpg-jdk18on", version.ref = "bouncycastle" }
 bouncycastle-bcprov = { module = "org.bouncycastle:bcprov-jdk18on", version.ref = "bouncycastle" }
 jackson-module-kotlin = { module = "com.fasterxml.jackson.module:jackson-module-kotlin", version.ref = "jackson" }
+jna-platform = { module = "net.java.dev.jna:jna-platform", version.ref = "jna" }
 
 [bundles]
 serialization = ["serialization-core", "serialization-json", "serialization-json-okio"]

src/main/kotlin/com/coder/toolbox/CoderRemoteProvider.kt

@@ -2,6 +2,7 @@ package com.coder.toolbox
 
 import com.coder.toolbox.browser.browse
 import com.coder.toolbox.cli.CoderCLIManager
+import com.coder.toolbox.diagnostics.CoderProviderLogCollector
 import com.coder.toolbox.feed.IdeFeedManager
 import com.coder.toolbox.oauth.OAuth2Client
 import com.coder.toolbox.plugin.PluginManager
@@ -377,6 +378,18 @@ class CoderRemoteProvider(
                     context.ui.showErrorInfoPopup(it)
                 }
             },
+            Action(context, "Collect detailed deployment and Toolbox logs") {
+                val activeClient = checkNotNull(client) { "Connect to a Coder deployment before collecting logs." }
+                val activeCli = checkNotNull(cli) { "Connect to a Coder deployment before collecting logs." }
+                coderHeaderPage.collectDiagnostics(
+                    CoderProviderLogCollector(
+                        context.logger,
+                        workspaces = { activeClient.workspaces() },
+                        collectBundle = activeCli::supportBundle,
+                    ),
+                )
+                context.envPageManager.showPluginEnvironmentsPage(true)
+            },
             CoderDelimiter(context.i18n.pnotr("")),
             Action(context, "Settings") {
                 context.ui.showUiPage(settingsPage)
@@ -391,6 +404,7 @@ class CoderRemoteProvider(
      */
     override fun close() {
         val sessionIds = lastEnvironments.currentSessionIds()
+        coderHeaderPage.cancelDiagnosticCollection()
         softClose()
         client = null
         cli = null

src/main/kotlin/com/coder/toolbox/diagnostics/CoderProviderLogCollector.kt

@@ -0,0 +1,194 @@
+package com.coder.toolbox.diagnostics
+
+import com.coder.toolbox.cli.WorkspaceAddress
+import com.coder.toolbox.sdk.v2.models.Workspace
+import kotlinx.coroutines.CancellationException
+import kotlinx.coroutines.Dispatchers
+import kotlinx.coroutines.currentCoroutineContext
+import kotlinx.coroutines.ensureActive
+import kotlinx.coroutines.withContext
+import java.io.IOException
+import java.io.InputStream
+import java.io.OutputStream
+import java.nio.file.FileVisitResult
+import java.nio.file.Files
+import java.nio.file.LinkOption.NOFOLLOW_LINKS
+import java.nio.file.Path
+import java.nio.file.SimpleFileVisitor
+import java.nio.file.attribute.BasicFileAttributes
+import java.util.zip.ZipEntry
+import java.util.zip.ZipOutputStream
+import kotlin.coroutines.CoroutineContext
+
+/** Collects local Toolbox logs and Coder support bundles for all accessible workspaces. */
+internal class CoderProviderLogCollector(
+    private val logger: CoderLogger,
+    private val workspaces: suspend () -> List<Workspace>,
+    private val collectBundle: suspend (WorkspaceAddress, Path) -> Unit,
+    private val logDirectories: () -> Map<String, Path> = { ToolboxLogPaths().directories() },
+    private val createTempDirectory: (String) -> Path = { Files.createTempDirectory(it) },
+) {
+    suspend fun collect(progress: (LogCollectionProgress) -> Unit): Path = withContext(Dispatchers.IO) {
+        val directory = createTempDirectory("coder-toolbox-logs-")
+        val staging = directory.resolve("contents")
+        val archive = directory.resolve("coder-toolbox-logs.zip")
+        var completed = false
+        try {
+            Files.createDirectories(staging)
+            val reports = mutableListOf<String>()
+            collectWorkspaces(staging, reports, progress)
+            progress(LogCollectionProgress("Collecting Toolbox and JetBrains daemon logs…"))
+            logDirectories().forEach { (name, source) ->
+                copyLogs(source, staging.resolve(name), directory, reports)
+            }
+            if (reports.isNotEmpty()) {
+                Files.writeString(staging.resolve("collection-report.txt"), reports.joinToString("\n", postfix = "\n"))
+            }
+            progress(LogCollectionProgress("Creating log archive…"))
+            zip(staging, archive)
+            currentCoroutineContext().ensureActive()
+            completed = true
+            archive
+        } finally {
+            deleteTree(if (completed) staging else directory)
+        }
+    }
+
+    @Suppress("TooGenericExceptionCaught")
+    private suspend fun collectWorkspaces(
+        root: Path,
+        reports: MutableList<String>,
+        progress: (LogCollectionProgress) -> Unit,
+    ) {
+        progress(LogCollectionProgress("Listing Coder workspaces…"))
+        val allWorkspaces = try {
+            workspaces().distinctBy { it.id }
+        } catch (ex: CancellationException) {
+            throw ex
+        } catch (ex: Exception) {
+            logger.warn(ex, "Could not list workspaces for log collection")
+            reports.add("Could not list Coder workspaces (${ex.javaClass.simpleName}). Local logs are included.")
+            return
+        }
+        allWorkspaces.forEachIndexed { index, workspace ->
+            val agents = workspace.latestBuild.resources.flatMap { it.agents.orEmpty() }.distinctBy { it.id }
+            val targets = agents.ifEmpty { listOf(null) }
+            for (agent in targets) {
+                currentCoroutineContext().ensureActive()
+                val address = WorkspaceAddress.from(workspace, agent)
+                val name = address.ownerAndWsName + (address.agentName?.let { ".$it" } ?: "")
+                progress(
+                    LogCollectionProgress(
+                        "Collecting workspace diagnostics…", "${index + 1}/${allWorkspaces.size}: $name"
+                    )
+                )
+                val destination = root.resolve("workspaces").resolve(workspace.id.toString())
+                    .resolve(agent?.id?.toString() ?: "workspace")
+                Files.createDirectories(destination)
+                Files.writeString(destination.resolve("workspace.txt"), "$name\n")
+                CoderSupportBundleCollector(logger) { collectBundle(address, it) }
+                    .collectAdditionalDiagnostics(destination)
+                if (!Files.isRegularFile(destination.resolve("coder-support.zip"))) {
+                    reports.add("Could not collect $name. See its coder-support-error.txt for details.")
+                }
+            }
+        }
+    }
+
+    private suspend fun copyLogs(
+        source: Path,
+        destination: Path,
+        temporaryDirectory: Path,
+        reports: MutableList<String>,
+    ) {
+        val coroutineContext = currentCoroutineContext()
+        val excluded = temporaryDirectory.toAbsolutePath().normalize()
+        if (!Files.exists(source)) {
+            reports.add("Log directory is unavailable: $source")
+            return
+        }
+        Files.walkFileTree(source, object : SimpleFileVisitor<Path>() {
+            override fun preVisitDirectory(dir: Path, attrs: BasicFileAttributes): FileVisitResult {
+                coroutineContext.ensureActive()
+                if (dir.toAbsolutePath().normalize().startsWith(excluded)) return FileVisitResult.SKIP_SUBTREE
+                Files.createDirectories(destination.resolve(source.relativize(dir)))
+                return FileVisitResult.CONTINUE
+            }
+
+            override fun visitFile(file: Path, attrs: BasicFileAttributes): FileVisitResult {
+                coroutineContext.ensureActive()
+                if (!attrs.isRegularFile) return FileVisitResult.CONTINUE
+                val target = destination.resolve(source.relativize(file))
+                try {
+                    // Snapshot the current size so an actively growing log cannot keep collection running.
+                    Files.newInputStream(file, NOFOLLOW_LINKS).use { input ->
+                        Files.newOutputStream(target).use { output ->
+                            copy(input, output, coroutineContext, attrs.size())
+                        }
+                    }
+                } catch (ex: IOException) {
+                    Files.deleteIfExists(target)
+                    reports.add("Could not copy $file (${ex.javaClass.simpleName}).")
+                }
+                return FileVisitResult.CONTINUE
+            }
+
+            override fun visitFileFailed(file: Path, exc: IOException): FileVisitResult {
+                coroutineContext.ensureActive()
+                reports.add("Could not read $file (${exc.javaClass.simpleName}).")
+                return FileVisitResult.CONTINUE
+            }
+
+            override fun postVisitDirectory(dir: Path, exc: IOException?): FileVisitResult {
+                if (exc != null) reports.add("Could not finish reading $dir (${exc.javaClass.simpleName}).")
+                return FileVisitResult.CONTINUE
+            }
+        })
+    }
+
+    private suspend fun zip(source: Path, archive: Path) {
+        val coroutineContext = currentCoroutineContext()
+        ZipOutputStream(Files.newOutputStream(archive)).use { output ->
+            Files.walk(source).use { paths ->
+                paths.filter { Files.isRegularFile(it, NOFOLLOW_LINKS) }.forEach { file ->
+                    coroutineContext.ensureActive()
+                    val name = source.relativize(file).joinToString("/") { it.toString() }
+                    output.putNextEntry(ZipEntry(name))
+                    copyFile(file, output, coroutineContext)
+                    output.closeEntry()
+                }
+            }
+        }
+    }
+
+    private fun copyFile(file: Path, output: OutputStream, context: CoroutineContext) {
+        Files.newInputStream(file).use { input -> copy(input, output, context) }
+    }
+
+    private fun copy(
+        input: InputStream,
+        output: OutputStream,
+        context: CoroutineContext,
+        limit: Long = Long.MAX_VALUE,
+    ) {
+        var remaining = limit
+        val buffer = ByteArray(DEFAULT_BUFFER_SIZE)
+        while (remaining > 0) {
+            context.ensureActive()
+            val count = input.read(buffer, 0, minOf(buffer.size.toLong(), remaining).toInt())
+            if (count < 0) break
+            output.write(buffer, 0, count)
+            remaining -= count
+        }
+    }
+
+    private fun deleteTree(root: Path) {
+        runCatching {
+            Files.walk(root).use { paths ->
+                paths.sorted(Comparator.reverseOrder()).forEach { Files.deleteIfExists(it) }
+            }
+        }.onFailure { logger.warn(it, "Could not remove temporary diagnostic files") }
+    }
+}
+
+internal data class LogCollectionProgress(val message: String, val detail: String = "")