Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 11 additions & 5 deletions Benchmarks/Package.swift
Original file line number Diff line number Diff line change
Expand Up @@ -6,18 +6,20 @@ enum UsePackage {
/// Use a known package hash downloaded from GitHub. Set with env var `USE_PACKAGE`
case useGitHubPackage

/// Use a local package, rooted at `String`. For example ".." to back up one directory from the Benchmark package. This is useful when testing local changes.
/// Set with env var `SWIFTCI_USE_LOCAL_DEPS=<path to root>`, for example `SWIFTCI_USE_LOCAL_DEPS=..`
/// Use a local package, rooted at `String`. `SWIFTCI_USE_LOCAL_DEPS` is also visible to `../Package.swift`, which resolves other package dependencies from local sibling checkouts as well.
case useLocalPackage(String)

/// Use a local swift-foundation checkout, rooted at `String`, but fetch other package dependencies from remote.
case useLocalPackageRemoteDeps(String)

/// Use Foundation.framework (Darwin) or the toolchain (Linux)
case useToolchain

var description: String {
switch self {
case .useGitHubPackage:
return "Using GitHub package"
case .useLocalPackage(let root):
case .useLocalPackage(let root), .useLocalPackageRemoteDeps(let root):
#if os(macOS)
return "Using local package checkout at \(root)/swift-foundation"
#else
Expand All @@ -36,15 +38,19 @@ enum UsePackage {
let usePackage: UsePackage

if let useLocalPackageEnv = Context.environment["SWIFTCI_USE_LOCAL_DEPS"], !useLocalPackageEnv.isEmpty {
// CI mode: local swift-foundation with all other package dependencies also resolved locally
if useLocalPackageEnv == "1" {
usePackage = .useLocalPackage("../..")
} else {
usePackage = .useLocalPackage(useLocalPackageEnv)
}
} else if let usePackageEnv = Context.environment["USE_PACKAGE"], !usePackageEnv.isEmpty {
usePackage = .useGitHubPackage
} else {
} else if Context.environment["USE_TOOLCHAIN"] != nil {
usePackage = .useToolchain
} else {
// Default: local swift-foundation with other package dependencies fetched from remote
usePackage = .useLocalPackageRemoteDeps("../..")
}

print("swift-foundation benchmarks: \(usePackage.description)")
Expand All @@ -61,7 +67,7 @@ var i18nTargetDependencies : [Target.Dependency] = []
var swiftSettings : [SwiftSetting] = [.unsafeFlags(["-Rmodule-loading"]), .enableUpcomingFeature("MemberImportVisibility")]

switch usePackage {
case .useLocalPackage(let root):
case .useLocalPackage(let root), .useLocalPackageRemoteDeps(let root):
#if os(macOS)
packageDependency.append(.package(name: "foundation-local", path: "\(root)/swift-foundation"))
targetDependency.append(.product(name: "FoundationEssentials", package: "foundation-local"))
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This foundation-local is set at the main Package.swift. Its value depends on whether SWIFTCI_USE_LOCAL_DEPS is set. When it is set it's using local check, and remote one if unset.

Expand Down
89 changes: 89 additions & 0 deletions Benchmarks/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# swift-foundation Benchmarks

Benchmarks for `swift-foundation` live in this directory as a separate Swift Package. They use the [`package-benchmark`](https://github.com/ordo-one/package-benchmark) plugin.

All commands below must be run from this `Benchmarks/` subdirectory.

---

## Prerequisites

`package-benchmark` uses [`jemalloc`](https://jemalloc.net) for memory allocation statistics. You can either install it or skip it.

`Benchmarks/Package.swift` declares `package-benchmark` with `traits: []`, which disables the `Jemalloc` trait. Benchmarks build and run without jemalloc installed by default.

To opt in to jemalloc and capture malloc metrics, install it first and then pass `--traits Jemalloc`:

- **macOS:** `brew install jemalloc`
- **Linux:** `apt-get install libjemalloc-dev` (Ubuntu/Debian) or equivalent

See the official [Getting Started](https://swiftpackageindex.com/ordo-one/package-benchmark/1.31.0/documentation/benchmark/gettingstarted#Disabling-jemalloc-Swift-61+) article for details.

---

## Running Benchmarks

**Run all benchmarks:**

```
swift package benchmark
```

**Run a specific target:**

```
swift package benchmark --target JSONBenchmarks
swift package benchmark --filter "URL"
```

**List all available benchmarks:**

```
swift package benchmark list
```

**Full help:**

```
swift package benchmark help
```

---

## Choosing Which Foundation to Benchmark Against

The benchmark package supports four configurations, controlled by environment variables. The active configuration is printed at build time.

### Local changes — benchmark your own swift-foundation checkout (default)

By default, with no environment variable set, benchmarks build against the local `swift-foundation` checkout (the parent of this `Benchmarks/` directory). Other package dependencies (e.g. `swift-foundation-icu`) are fetched from their remote URLs. This is the recommended mode for contributors iterating on a performance improvement.

```
swift package benchmark
```

To point at a different local checkout, set `SWIFTCI_USE_LOCAL_DEPS` to the directory *containing* the `swift-foundation` checkout:

```
SWIFTCI_USE_LOCAL_DEPS=/path/to/parent swift package benchmark
```

On Linux, `SWIFTCI_USE_LOCAL_DEPS` resolves to `swift-corelibs-foundation` instead.

> **Note:** `SWIFTCI_USE_LOCAL_DEPS` is also set by the Swift repo CI when running full toolchain unit tests, so that *all* dependencies (including transitive ones like `swift-foundation-icu`) resolve to locally checked-out copies rather than fetching from git.

### GitHub main branch — compare against the latest published commit

```
USE_PACKAGE=1 swift package benchmark
```

Fetches and benchmarks against the `main` branch of `swift-foundation` (or `swift-corelibs-foundation` on Linux) from GitHub. Useful for comparing a local build against a known top-of-tree commit.

### System Foundation / toolchain

```
USE_TOOLCHAIN=1 swift package benchmark
```

On macOS this uses `Foundation.framework`. On Linux it uses the Foundation from the installed Swift toolchain. Use this only if you specifically want to measure the system-installed version.
14 changes: 5 additions & 9 deletions Foundation_Build_Process.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,16 +122,11 @@ The swift-foundation project is also built internally within Apple as part of th

## Benchmarks

Benchmarks for `swift-foundation` are in a separate Swift Package in the `Benchmarks` subfolder of this repository.
They use the [`package-benchmark`](https://github.com/ordo-one/package-benchmark) plugin.
Benchmarks depends on the [`jemalloc`](https://jemalloc.net) memory allocation library, which is used by `package-benchmark` to capture memory allocation statistics. An installation guide can be found in the [Getting Started article](https://swiftpackageindex.com/ordo-one/package-benchmark/documentation/benchmark/gettingstarted#Installing-Prerequisites-and-Platform-Support) of `package-benchmark
Benchmarks for `swift-foundation` live in the `Benchmarks/` subfolder as a separate Swift Package.

Install `jemalloc` before running benchmarks:
- **macOS:** `brew install jemalloc`
- **Ubuntu:** `sudo apt-get install -y libjemalloc-dev`

Afterwards you can run the benchmarks from CLI by going to the `Benchmarks` subfolder (e.g. `cd Benchmarks`) and invoking:
```
From swift-foundation directory, run
```shell
cd Benchmarks
swift package benchmark
```

Expand All @@ -140,3 +135,4 @@ If `jemalloc` is not available, disable it by setting `BENCHMARK_DISABLE_JEMALLO
BENCHMARK_DISABLE_JEMALLOC=1 swift package benchmark
```

See [`Benchmarks/README.md`](Benchmarks/README.md) for setup and usage instructions.
Loading