Updated Fixes for the Linux version of GRDB

[thinkpractice]

Fixes for the Linux version of GRDB

UPDATE this PR merges some upstream changes into my fork.

This PR contains some fixes for the Linux version of GRDB, mainly these fixes include:

• Fixing linker errors on Linux
• Some fixes for differences between Foundation on Linux and Apple platforms
• Fixing the unittests and making them succeed on Linux
• Ignoring Apple platform specific functionality like Combine or NSFileCoordinator, etc.

Some of these fixes are inspired by #1708 by @marcprux. This PR attemps to make these fixes in
a platform-agnostic way so that GRDB will build with SPM on more platforms than Linux alone,
for instance, on Android and Windows. These patches fix the build of GRDB on Swift 6.1 and later.
There are still some issues building on Swift 6.0, which I will attempt to fix later.

Fixing the GRDB Build

Most of GRDB actually builds on Linux just fine. The only change is that I needed to do to make it build was adding a

@preconcurrency import Dispatch

to handle non-Sendable protocol conformance in GRDB/Core/DispatchQueueActor.swift. Not sure if that's the right approach but it made everything build!

Fixing GRDB linking on Linux

The main issue with GRDB on Linux is actually a linker error. The standard libsqlite3-dev installed with apt or dnf doesn't have WAL enabled and when trying to use GRDB in another swift package or app in Linux, you get a linker error. Functions like sqlite3_snapshot_get, sqlite3_snapshot_open are not found by the linker,
and therefore the package isn't usable.

There would be two ways around it I think, either:

• include a custom sqlite build for GRDB on Linux, or
• fix linking by adding extra compiler directives/conditions for Linux

I chose the latter for now as it is a more easy fix. I updated the following conditional compilation directive:

#if SQLITE_ENABLE_SNAPSHOT || (!GRDBCUSTOMSQLITE && !GRDBCIPHER)

to:

#if (SQLITE_ENABLE_SNAPSHOT || (!GRDBCUSTOMSQLITE && !GRDBCIPHER)) && !os(Linux)

This is a quick fix, but it would be better to change the SQLITE_ENABLE_SNAPSHOT to return false on Linux.

Note this should be better handled as a fix to SQLITE_ENABLE_SNAPSHOT as later on Linux may also provide
a custom sqlite build to enable WAL. In those cases, the code inside these blocks should be compiled.

This compiler directive is present (and updated) in the following files:

• GRDB/Core/DatabasePool.swift
• GRDB/Core/DatabaseSnapshotPool.swift
• GRDB/Core/WALSnapshot.swift
• GRDB/Core/WALSnapshotTransaction.swift
• GRDB/ValueObservation/Observers/ValueConcurrentObserver.swift

Updating the compiler directive as above will fix the build with the standard libsqlite3-dev on Linux. If that's fine we
can keep it at that. I'm not completely sure yet what WAL does but if I understand correctly, using it is mostly a
performance consideration. If it would be a nice to have on Linux too, I can research adding a custom sqlite build for
Linux with WAL enabled. I wonder why it is not included in the standard libsqlite3-dev though? Adding the changes above
will fix linking of the GRDB code.

Fixes for open source Foundation versions

Open source versions of Foundation are slightly different than the Foundation Apple bundles with XCode. Also, some
packages, like for instance CoreGraphics, are not available on non-Apple platforms. Some primitive types like CGFloat
are available on non-Apple platforms and are defined in Foundation. In some cases, there are also differences where a
initializer is defined as convenience init in open source Foundation and as a normal init in Apple Foundation. This
causes some problems with Apple specific code in some cases; it will fail to build on open source Foundation. In GRDB in
many cases this code is surrounded by a #if !os(Linux) compiler directive. The fixes below try to solve the build issues on
Linux and try to get rid of these compiler directives. Also fixing the unit tests at the same time.

CGFloat.swift

CoreGraphics is not available on Linux. But CGFloat is available via Foundation. Changed the compiler directive in CGFloat.swift and its test from:

#if canImport(CoreGraphics)

to:

#if canImport(CoreGraphics)
    import CoreGraphics
#elseif !os(Darwin)
    import Foundation
#endif

The same fix should be added to GRDBTests/CGFloatTests.swift.

Decimal.swift

Removed the #if !os(Linux) compiler directive in the Decimal extension, as it compiles fine under swift 6. As discussed
with @groue the minimum version for GRDB is swift 6.0. I tested these changes with swift 6.1 and swift 6.2 and they work fine.

Note: swift 6.0 doesn't build, we should look into fixing the build for this version.

NSString, NSDate, Date

Some foundation NS* types (NSString, NSData, Date, etc.) were excluding the DatabaseValueConvertible extensions on Linux but build fine on swift 6.1 and swift 6.2.

Removed #if !os(Linux) in:

• NSString.swift
• NSData.swift
• Date.swift

NSNumber

There's a problem with the following code in NSNumber.swift on Linux:

 public static func fromDatabaseValue(_ dbValue: DatabaseValue) -> Self? {
   switch dbValue.storage {
   case .int64(let int64):
         return self.init(value: int64)
   case .double(let double):
         return self.init(value: double)
   case let .string(string):
         // Must match Decimal.fromDatabaseValue(_:)
         guard let decimal = Decimal(string: string, locale: posixLocale) else { return nil }
         return NSDecimalNumber(decimal: decimal) as? Self
   default:
         return nil
   }
}

both the .int64 and .double case above do not compile. The error is the following:

error: constructing an object of class type 'Self' with a metatype value must use a 'required' initializer

The NSNumber(value: Int64) and NSNumber(value: Double) are marked as convenience init on Linux. This may cause the build to fail. The issue with this approach is that it doesn't work for NSDecimalNumber, which is a subclass of NSNumber.
It looks like the main issue to begin with is that NSNumber defines NSNumber(value: Int64) and NSNumber(value: Double as convenience init, whereas their NSDecimalNumber equivalents are non-convenience. I could solve it by changing the code
above like this:

public static func fromDatabaseValue(_ dbValue: DatabaseValue) -> Self? {
        switch dbValue.storage {
        case .int64(let int64) where self is NSDecimalNumber.Type:
            let number = NSDecimalNumber(value: int64)
            return number as? Self
        case .int64(let int64):
            let number = NSNumber(value: int64)
            return number as? Self
        case .double(let double) where self is NSDecimalNumber.Type:
            let number = NSDecimalNumber(value: double)
            return number as? Self
        case .double(let double):
            let number = NSNumber(value: double)
            return number as? Self
        case .string(let string):
            // Must match Decimal.fromDatabaseValue(_:)
            guard let decimal = Decimal(string: string, locale: posixLocale) else { return nil }
            return NSDecimalNumber(decimal: decimal) as? Self
        default:
            return nil
        }
    }
}

This is a lot more verbose than the existing code. Also, this code may fail if there are other subclasses of NSNumber
that need to be supported. For each of these classes we should add a case for .int64 and double. I think an alternative
approach would be better, but I don't have one at the moment.

All tests in:

• Tests/GRDBTests/FoundationNSNumberTests.swift
• Tests/GRDBTests/FoundationNSDecimalNumberTests.swift

succeeded with these changes.

URL.swift

The NSURL is a specific case. Apparently, the NSURL absoluteString property is non-optional on Linux and optional on Apple platforms. The fix is the following:

 /// Returns a TEXT database value containing the absolute URL.
        public var databaseValue: DatabaseValue {
            #if !os(Darwin)
                absoluteString.databaseValue
            #else
                absoluteString?.databaseValue ?? .null
            #endif
        }

UUID.swift

UUID.swift contains a compiler directive excluding the DatabaseValueConvertible from building
on Linux and Windows. On Linux the problem is the following:

 public static func fromDatabaseValue(_ dbValue: DatabaseValue) -> Self? {
        switch dbValue.storage {
        case .blob(let data) where data.count == 16:
            return data.withUnsafeBytes {
                self.init(uuidBytes: $0.bindMemory(to: UInt8.self).baseAddress)
            }
        case .string(let string):
            return self.init(uuidString: string)
        default:
            return nil
        }
    }

The NSUUID(uuidBytes:) constructor takes an optional UnsafePointer<UInt8>, see here, whereas this constructor on Linux takes a non-optional.

This could be fixed with a guard statement. However, similarly to the case for NSNumber in addition, the
self.init(uuidString: string) doesn't compile on Linux. The error is:

Constructing an object of class type 'Self' with a metatype value must use a 'required' initializer

Added the following fix:

public static func fromDatabaseValue(_ dbValue: DatabaseValue) -> Self? {
        switch dbValue.storage {
        case .blob(let data) where data.count == 16:
            return data.withUnsafeBytes {
                #if canImport(Darwin)
                    self.init(uuidBytes: $0.bindMemory(to: UInt8.self).baseAddress)
                #else
                    guard let uuidBytes = $0.bindMemory(to: UInt8.self).baseAddress else {
                        return nil as Self?
                    }
                    return NSUUID(uuidBytes: uuidBytes) as? Self
                #endif
            }
        case .string(let string):
            return NSUUID(uuidString: string) as? Self
        default:
            return nil
        }
    }

This also makes all unittests succeed. It should be checked whethere we would want to support any subclass of NSUUID.
If so, a similar solution as for NSNumber above can be used.

Use of DispatchQueue private APIs

In some of the tests for the DispatchQueue APIs, some private DispatchQueue apis are used to get the label of the
current DispatchQueue. These APIs are unavailable on non-Darwin platforms. It's mainly about some tests in for example DatabasePoolConcurrencyTests.swift. For instance, the following test:

func testDefaultLabel() throws {
        let dbPool = try makeDatabasePool()
        dbPool.writeWithoutTransaction { db in
            XCTAssertEqual(db.configuration.label, nil)
            XCTAssertEqual(db.description, "GRDB.DatabasePool.writer")

            // This test CAN break in future releases: the dispatch queue labels
            // are documented to be a debug-only tool.
            let label = String(utf8String: __dispatch_queue_get_label(nil))
            XCTAssertEqual(label, "GRDB.DatabasePool.writer")
         

I understood from @groue that the goal of those tests is to make sure that the Xcode, LLDB, and crash reports mention the label of the current database in the name of the current DispatchQueue. This is a nicety that aims at helping debugging, and worth a test. The label of the current dispatch queue can only be tested with the private __dispatch_queue_get_label api, which is only available on Darwin systems.

Those tests can be avoided on non-Darwin platforms, as @marcprux did in #1708:

#if canImport(Darwin) // __dispatch_queue_get_label unavailable on non-Darwin platforms
// This test CAN break in future releases: the dispatch queue labels
// are documented to be a debug-only tool.
let label = String(utf8String: __dispatch_queue_get_label(nil))
XCTAssertEqual(label, "GRDB.DatabasePool.writer")
#endif

I implemented this approach for:

• Utils.swift
• DatabasePoolConcurrencyTests.swift
• DatabaseQueueTests.swift
• DatabaseSnapshotTests.swift

NSError

Bridging of DatabaseError to NSError does not seem to work on Linux. The testNSErrorBridging therefore
fails, we should probably ignore the test on non-Darwin platforms as NSError is mostly a left-over from ObjectiveC times
and APIs.

   func testNSErrorBridging() throws {
        #if !canImport(Darwin)
            throw XCTSkip("NSError bridging not available on non-Darwin platforms")
        #else
            let dbQueue = try makeDatabaseQueue()
            try dbQueue.inDatabase { db in
                try db.create(table: "parents") { $0.column("id", .integer).primaryKey() }
                try db.create(table: "children") { $0.belongsTo("parent") }
                do {
                    try db.execute(sql: "INSERT INTO children (parentId) VALUES (1)")
                } catch let error as NSError {
                    XCTAssertEqual(DatabaseError.errorDomain, "GRDB.DatabaseError")
                    XCTAssertEqual(error.domain, DatabaseError.errorDomain)
                    XCTAssertEqual(error.code, 787)
                    XCTAssertNotNil(error.localizedFailureReason)
                }
            }
        #endif
    }

Fixing Unit tests

Tests using NSFileCoordinator

There are some tests using NSFileCoordinator which is not available on non-Darwin platforms. Most notably
testConcurrentOpening in DatabasePoolConcurrencyTests.swift contributes 400+ test failures when run on
Linux. testConcurrentOpening, that runs about 500 tests (a for loop of 50 iterations runs 10 times some things in parallel
with DispatchQueue.concurrentPerform. The test uses the NSFileCoordinator which isn't available on Linux.

These tests are ignored because they test whether NSFileCoordinator works with GRDB. This is Darwin specific functionality:

We're talking about testConcurrentOpening, right? The goal of DispatchQueue.concurrentPerform is just to spawn parallel
jobs. In this case each job opens a DatabasePool, protected by a NSFileCoordinator, and we're asserting that not error is
thrown. This is a test for the correct integration of GRDB and NSFileCoordinator, so that we can confidently recommend
using NSFileCoordinator as a way to protect a database that is shared between multiple processes.

We could say that those tests should be removed, arguing that they are just asserting the behavior of NSFileCoordinator and
that "testing the framework" is a bad practice - but we're also asserting that DatabasePool evolution must be compatible
with user code that uses NSFileCoordinator and that any deadlock must be prevented.)

Arguably DispatchQueue.concurrentPerform could be replaced with any other modern technique that guarantees a minimum level of parallelism, such as withTaskGroup.

We should check whether we need to test similar behaviour on Linux. For now we just ignore the test.
@marcprux solved it like this:

func testConcurrentOpening() throws { 
     #if !canImport(ObjectiveC) 
     throw XCTSkip("NSFileCoordinator unavailable") 
     #else 

We solved it similarly.

Other Unit tests

• XCTIssue is not available on non-Darwin platforms, it's used in:
  • GRDBTests/FailureTestCase.swift ignored this test completely with #if os(Darwin) on non-Darwin platforms
  • GRDBTests/ValueObservationRecorderTests.swift ignored this test also.
• Support.swift actually needs Combine, so all tests that use it need Combine as well:
  • DatabaseMigratorTests.swift has several tests using the Test class defined in Support.swift; used an #if !canImport(Combine) clause to skip several tests in this test suite.
  • ValueObservationTests.swift also uses this Test class
• DatabaseRegionObservationTests.swift tests some Combine functionality in testAnyDatabaseWriter; also wrapped them in a #if canImport(Combine)
• When integrating the upstream changes I had some conflicts in DatabaseMigratorTests.swift I tried to merge them but not sure everything went ok. I was struggling a bit integrating them. This file should be checked if all upstream changes have been merged in.

TODOs

• Fix the build under swift 6.0
• Check if we can provide an alternative for XCTIssue
• See if we can find an alternative for __dispatch_queue_get_label on Linux
• Setup a CI.yml pipeline to build the Linux version of the package automatically and run the unit tests

With this we:

Executed 2768 tests, with 46 tests skipped and 0 failures (0 unexpected) in 234.863 (234.863) seconds

[marcprux]

I did an initial scan, and my initial takeaways:

1. Great work!
2. The reformatting and whitespace changes (probably resulting from a linter run) make this challenging to review. There's probably not much that can be done about it at this point, though.
3. I think that Linux should added to the CI matrix in .github/workflows/CI.yml.

[thinkpractice]

I did an initial scan, and my initial takeaways:

1. Great work!

Thanks!

2. The reformatting and whitespace changes (probably resulting from a linter run) make this challenging to review. There's probably not much that can be done about it at this point, though.

Actually, I tried to check this with SwiftLint but I saw a lot of linting warnings in code I hadn't touched too. So was not sure how I should proceed. How can I improve this?

3. I think that Linux should added to the CI matrix in `.github/workflows/CI.yml`.

That would be nice, I discussed this with @groue earlier, but he wanted to wait a bit with this. It would be good in the long run though. If the build fails I could check it and fix it faster. Well let's see 😉

[groue]

Hello @thinkpractice,

That's huge, and I'm so thankful 🥳

It's a pity that the diff messed up with white space 😬. The git blame will be unusable, and code archeology will become very difficult. I'm very sorry but I have to ask you to redo all your commits and remove all white space changes.

Fortunately I can review the changes by asking GitHub UI to hide the white space changes in the diff view 👍

[EDIT] It looks like you actually applied a code formatter. Again, I'm very sorry, but this is not mergeable. And I can not review in correct conditions with so many unrelated changes. I hope VSCode can be configured so that it does not mess up with existing files.

[groue]

Please modify all commits (rewriting history) so that all code formatting and white space changes are removed, and git blame only shows actual changes. If it can help, modify .vscode/settings.json so that VSCode is well behaved and does not take ownership of code formatting.

[groue]

Let's not have code formatting pollute the review. So I looked at SQLITE_ENABLE_SNAPSHOT. A pull request is on the way.

[thinkpractice]

@groue No worries! Ughh this is due to my inexperience contributing to open source projects like this 😞 (and part of my learning curve). It would help me a ton if you give me some pointers as to what your setup according to code formatting is. I think my main mistake is not disabling swift-format for this project, I will investigate how.

The other thing is configuring SwiftLint. When I ran it yesterday on my PR (and also on the original repo) SwiftLint threw a ton of errors, so I must be doing something wrong. I'll try to fix the formatting, but I'm not sure how much time I will have this weekend. Maybe in the train to Leeds 😉

Looking forward to your SQLITE_ENABLE_SNAPSHOT changes, when the PR arrives I can try them on Linux 😄

[groue]

The pull request for snapshots: #1826

[groue]

@groue No worries! Ughh this is due to my inexperience contributing to open source projects like this 😞 (and part of my learning curve).

😅 I hope I did not sound too bad at you! As I said above, the real problem is the git blame.

It would help me a ton if you give me some pointers as to what your setup according to code formatting is. I think my main mistake is not disabling swift-format for this project, I will investigate how.

There is no formatting spec such as an .editorconfig in the project, because .editorconfig can not be made compatible with the default Xcode formatting. As for Swift formatters, I also think none of them can be made compatible with the default Xcode formatting. Yeah people can be pretty much opinionated.

My best suggestion is to disable all code formatting tools, and to control what you commit.

The other thing is configuring SwiftLint. When I ran it yesterday on my PR (and also on the original repo) SwiftLint threw a ton of errors, so I must be doing something wrong. I'll try to fix the formatting, but I'm not sure how much time I will have this weekend. Maybe in the train to Leeds 😉

I don't think SwiftLint is invoked from SPM. I suppose you can ignore it for now?

Looking forward to your SQLITE_ENABLE_SNAPSHOT changes, when the PR arrives I can try them on Linux 😄

I hope #1826 helps! 🤞

[thinkpractice]

@groue No worries! Ughh this is due to my inexperience contributing to open source projects like this 😞 (and part of my learning curve).

😅 I hope I did not sound too bad at you! As I said above, the real problem is the git blame.

Nah it was really fine. To be honest I didn't even think about the code formatting being an issue. Then when @marcprux told me yesterday that code reviewing would be a challenge I realised that the git diffs would be messed up because of that. My mistake! If I cannot fix it on this PR then I will just redo the changes and submit it again. Hoping the third time's a charm!

It would help me a ton if you give me some pointers as to what your setup according to code formatting is. I think my main mistake is not disabling swift-format for this project, I will investigate how.

There is no formatting spec such as an .editorconfig in the project, because .editorconfig can not be made compatible with the default Xcode formatting. As for Swift formatters, I also think none of them can be made compatible with the default Xcode formatting. Yeah people can be pretty much opinionated.

Oh, I just downloaded a vscode EditorConfig extension in the hope that it would take the .editorconfig in the root of the repo into account. Do I then understand correctly that I should ignore it completely?

My best suggestion is to disable all code formatting tools, and to control what you commit.

The other thing is configuring SwiftLint. When I ran it yesterday on my PR (and also on the original repo) SwiftLint threw a ton of errors, so I must be doing something wrong. I'll try to fix the formatting, but I'm not sure how much time I will have this weekend. Maybe in the train to Leeds 😉

I don't think SwiftLint is invoked from SPM. I suppose you can ignore it for now?

Looking forward to your SQLITE_ENABLE_SNAPSHOT changes, when the PR arrives I can try them on Linux 😄

I found out that you have yml for SwiftLint in the Scripts directory. This works and shows only 2 errors on the branch that I'm using. But for now I will just disable all code formatting and check it before submitting.

[groue]

Oh, @thinkpractice, I certainly do not want to sound like I am "ableist" regarding the tooling. If you have difficulties with formatting, this is quite OK: I will perform the necessary cleanup. Please do not feel embarrassed! This PR is such a boon 👏

[thinkpractice]

I'm mostly trying to get my head around some of the git commands related to PRs. I'm used to using branches but not really use to PR. Trying to figure out how to commit changes to this PR for instance?

UPDATE just found out that I can just change my forked branch and the changes will be reflected here DUH. Well that's really awesome! Will go and check how to get the formatting in a good shape again :)

[thinkpractice]

@groue how do you use git blame to see my changes? And I guess you only want to see the lines I changed right? I'm not sure what would be faster... changing back each file or applying each change I made on a fresh fork. I finally got my head around working with PRs so this time it should be fine :)

[groue]

@groue how do you use git blame to see my changes?

To see your changes, I just use the GitHub interface, that's OK. I do not use git blame at this stage.

git blame is useful when one wonders why the code is like it is, and wants to better understand. The blame helps finding which commits have changed some code, and this greatly help performing "code archaeology" - but only when the commits are meaningful. GRDB is old, more than 10 years old now. Code archaeology is necessary to help my aging memory.

EDIT: So a good practice is to strictly distinguish commits that format from commit that change code meaningfully. This way, archeology is still possible: formatting commits can be ignored, and commits that change code meaningfully have no noise.

This rule is not an absolute, of course. But here all thresholds have been crossed ;-)

[thinkpractice]

@groue as I trial I tried to reformat Core/Support/Foundation/Decimal.swift but if I now look at commit diff here in GitHub I'm seeing the changes in Decimal.swift as related to my earlier changes. I guess this is not what you want to see, but you would like to see the changes as related to your original code... Is there a way to get back to that besides starting of a fresh fork+branch and redoing the changes?

EDIT: I have just discovered the Files Changes tab as opposed to the commits view (sorry for being such a GitHub noob) and seeing that that integrates my changes. Will experiment a bit if that makes cleaning up easier. I'm seeing now what you and @marcprux mean by the code-formatter making it a challenge to review!

[thinkpractice]

@groue I found out there's some weird behaviour with my vscode. One time tabs are converted to spaces, and then another indentation which was 4 spaces gets converted back to a tab 😠 Will need to check how to solve this. Already changed my editor settings, so either I missed some setting or it's a bug in vscode....

Found it apparently there is something like "editor.trimAutoWhitespace" in vscode which removes "trailing auto inserted whitespace"

[thinkpractice]

@groue I have been updating some more files to reflect the original formatting and make it easier for me to see my changes. Before these changes would be synced to this PR, but now I'm not seeing them anymore. Is this something that sound familiar to you? I have already tried the solution mentioned here, i.e. updating the base to the same branch and saving it. So far, this hasn't helped, and if I try it again I'm getting an error saying the base is already being updated. I will check later if I can see the changes, but if they don't show up, do you have any tips?

UPDATE some of my changes are starting to show up... so I guess I it's working again. Let's see if the other changes I made later will show up as well!

[groue]

@thinkpractice I have pushed on the groue:grdb_linux_changes branch all changes of your own branch, fully rebased on the latest development branch, with all code formatting removed. I took care that all commits are attributed to you.

I had to heavily amend your initial changes, so I hope I haven't broken or forgotten anything.

If this branch builds as well as yours on Linux, and if you agree, maybe you could force-push this branch to yours? That will make the review, and future changes, much easier.

[thinkpractice]

@groue did you see I spend some time cleaning up the code as well? Hope you could use that. Thanks for taking care of the rest! Will take care to format my code in the right way on any future changes. I am fine with force pushing the changes to my branch if that makes things easier. I am still traveling so I cannot really test your changes now (only brought my MacBook). I will be home next weekend so then I can test everything. Hope that’s ok!

[groue]

did you see I spend some time cleaning up the code as well? Hope you could use that.

Yes, thanks for your efforts! Now starting over from the development branch was the best way to reach the ideal minimal diff.

Will take care to format my code in the right way on any future changes.

I hope .vscode/settings.json solves this for good, yes.

I am fine with force pushing the changes to my branch if that makes things easier. I am still traveling so I cannot really test your changes now (only brought my MacBook). I will be home next weekend so then I can test everything. Hope that’s ok!

If you look at the new commits, you can see that they perform minimal changes. They do indeed make things much easier. I just don't know if I faithfully reported all your changes. So please take your time, check this branch on your Linux system, and make sure I did not forget anything! 🙏

[thinkpractice]

@groue checked out your branch and it compiled successfully! Also the test project compiled and ran and we now have:

 Executed 2771 tests, with 46 tests skipped and 0 failures (0 unexpected) in 236.873 (236.873) seconds

So it looks like we have the build working on Linux again. I will merge the changes you made into my branch and then hopefully we can merge. Do you think whether there would be any additional things I can do to improve the support on Linux? I'm thinking along the lines of adding some equivalent tests for the Darwin specific tests we commented out, adding a custom sqlite build for WAL, adding a CI for the linux build (can we have separate tags to indicate build failures on different platforms like the package index does?), or maybe investigating the new swift container framework so we can also test the Linux build on macOS?

[groue]

Welcome back, @thinkpractice -:

@groue checked out your branch and it compiled successfully! Also the test project compiled and ran and we now have:

 Executed 2771 tests, with 46 tests skipped and 0 failures (0 unexpected) in 236.873 (236.873) seconds

You can be proud :-)

So it looks like we have the build working on Linux again. I will merge the changes you made into my branch and then hopefully we can merge.

If you do not mind, please force-push my branch to yours instead, so that only "my" commits are in your branch, and only them. This will remove from the git history all the formatting and de-formatting commits.

[EDIT]: I'll help you if this is not quite clear - please just tell.

Do you think whether there would be any additional things I can do to improve the support on Linux? I'm thinking along the lines of adding some equivalent tests for the Darwin specific tests we commented out, adding a custom sqlite build for WAL, adding a CI for the linux build (can we have separate tags to indicate build failures on different platforms like the package index does?), or maybe investigating the new swift container framework so we can also test the Linux build on macOS?

I'm looking forward for all the goos ideas you will suggest :-) For now, let's merge this. In parallel, there was some work in progress regarding SQLCipher, in #1827, in which a serious issue was discovered - it slipped in the development branch while working on SQLITE_ENABLE_SNAPSHOT and package traits. I'll have to deal with this before everything else. Fortunately, I do not think this has any impact on your pull request. It's just that there are several topics to deal with, and I'll address them one after the other.

I'm really curious to see what will be shipped in the next release :-)

[thinkpractice]

Welcome back, @thinkpractice -:

Thanks! Good to be back :) Although I had a blast at SwiftLeeds and in the UK in general 😄

@groue checked out your branch and it compiled successfully! Also the test project compiled and ran and we now have:

 Executed 2771 tests, with 46 tests skipped and 0 failures (0 unexpected) in 236.873 (236.873) seconds

You can be proud :-)

Thanks! Happy it was relatively easy to get it to run, thanks for all the prior work done 😄

So it looks like we have the build working on Linux again. I will merge the changes you made into my branch and then hopefully we can merge.

If you do not mind, please force-push my branch to yours instead, so that only "my" commits are in your branch, and only them. This will remove from the git history all the formatting and de-formatting commits.

[EDIT]: I'll help you if this is not quite clear - please just tell.

Yeah would prefer some help. So I have your repo as upstream so I guess it would be something like this?

git fetch upstream
git reset --hard upstream/grdb_linux_changes

Would this do the job?

I'm looking forward for all the goos ideas you will suggest :-) For now, let's merge this. In parallel, there was some work in progress regarding SQLCipher, in #1827, in which a serious issue was discovered - it slipped in the development branch while working on SQLITE_ENABLE_SNAPSHOT and package traits. I'll have to deal with this before everything else. Fortunately, I do not think this has any impact on your pull request. It's just that there are several topics to deal with, and I'll address them one after the other.

I think I will check first if I can get a CI for Linux to work, parallel to the macOS one. This would allow us to identify whether any of the changes made would break on either platform. This would also not interfere with the work you are currently doing on SQLCipher. If I could help there, just let me know 😄 After that, I'm thinking I will start evaluating GRDB on Linux and see if any issues pop up... If the SQLCipher part works, I think we will be able to use that work to make custom sqlite builds on linux too. That would be than my next focus.

I'm really curious to see what will be shipped in the next release :-)

That makes two of us! Looks like there are a couple of exciting changes coming up!

[thinkpractice]

@groue just in case you missed it. Is this the right way to apply your changes to my branch?

git fetch upstream
git reset --hard upstream/grdb_linux_changes

If so then I can merge and fix the remaining issues this week 😄

[groue]

Hi @thinkpractice, sorry I had a busy weekend. OK so let's go:

My current remotes are origin for groue/GRDB.swift, and R4N because I follow #1827:

9:35 groue@Renart ~/Documents/git/groue/GRDB.swift % git remote -v
R4N	git@github.com:R4N/GRDB.swift.git (fetch)
R4N	git@github.com:R4N/GRDB.swift.git (push)
origin	git@github.com:groue/GRDB.swift.git (fetch)
origin	git@github.com:groue/GRDB.swift.git (push)

Let's add a remote for your own fork, where I can find your branch grdb_linux_changes:

9:35 groue@Renart ~/Documents/git/groue/GRDB.swift % git remote add thinkpractice git@github.com:thinkpractice/GRDB.swift.git

Let's checkout your branch grdb_linux_changes:

9:41 groue@Renart ~/Documents/git/groue/GRDB.swift % git fetch thinkpractice
9:41 groue@Renart ~/Documents/git/groue/GRDB.swift % git checkout -b grdb_linux_changes --track thinkpractice/grdb_linux_changes
branch 'grdb_linux_changes' set up to track 'thinkpractice/grdb_linux_changes'.
Switched to a new branch 'grdb_linux_changes'

Is it what we expect? Yes it is. These are your latest commits:

9:42 groue@Renart ~/Documents/git/groue/GRDB.swift % git log -2
commit 2ddf51026ef0b739b7ea911e6a1417e3266969c2 (HEAD -> grdb_linux_changes, thinkpractice/grdb_linux_changes)
Author: Tim De Jong <gitlab@thinkpractice.nl>
Date:   Fri Oct 10 17:34:12 2025 +0100

    Restored some of the original indentation

commit 1785f63cce380972bcd8c702048571506ce9e838
Author: Tim De Jong <gitlab@thinkpractice.nl>
Date:   Fri Oct 10 17:29:28 2025 +0100

    Restored code layout messed up by swift-format.

Let's reset its state to my branch grdb_linux_changes:

9:43 groue@Renart ~/Documents/git/groue/GRDB.swift % git reset --hard origin/grdb_linux_changes
HEAD is now at 3dd3bbf0c vscode config

Is it what we expect? Yes it is. These are my latest commits:

9:43 groue@Renart ~/Documents/git/groue/GRDB.swift % git log -2
commit 3dd3bbf0c5ab23fb8576f380d9a5564bc83ca846 (HEAD -> grdb_linux_changes, origin/grdb_linux_changes)
Author: Tim De Jong <gitlab@thinkpractice.nl>
Date:   Sun Oct 12 16:43:31 2025 +0200

    vscode config

commit d7cf0f9e59c46054d1dcbbe1b145c19c8ed8de30
Author: Tim De Jong <gitlab@thinkpractice.nl>
Date:   Sun Oct 12 17:17:47 2025 +0200

    NSError bridging is not available, or doesn't work the same on non-Darwin platforms. Skip testNSErrorBridging on these platforms.

We can now force-push the content of my branch to yours:

9:43 groue@Renart ~/Documents/git/groue/GRDB.swift % git push --force
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
To github.com:thinkpractice/GRDB.swift.git
 + 2ddf51026...3dd3bbf0c grdb_linux_changes -> grdb_linux_changes (forced update)

And we're done:

• This pull request now only contains the rewritten commits.
• There's no trace of indentation churn anywhere, no trace at all.
• We're going to merge soon.

On your side, since I did force-push to you branch, you will not be able to pull. You will have to reset your local branch to its new remote state. The latest commit is 3dd3bbf, so you'll write in your terminal:

# Grab latest commits
git fetch
# Ignore all local commits in your local grdb_linux_changes branch,
# and just apply the latest commit from the remote.
git reset --hard 3dd3bbf0c5ab23fb8576f380d9a5564bc83ca846

[thinkpractice]

@groue Thanks so much for the detailed explanation. As you see I'm not that familiar with these specialized git commands, but the above helps me a lot to learn 😄 So I guess the command I showed above would've worked too but was missing the git push --force above. Will reset my branch soon and so nice that we can merge soon! I will have a look this week if I can:

• Add a CI for Linux that builds the code and runs the test
• Use the GRDB from this branch in some toy project I've been developing and see how it works in production
• If the toy project works out fine, I will start testing some more complex examples

[groue]

@groue Thanks so much for the detailed explanation. As you see I'm not that familiar with these specialized git commands, but the above helps me a lot to learn 😄

That's fine! I frequently use an LLM to help me, or an app that puts a GUI in front of git. The hardest part is frequently to know what is even possible to do 😅

I will have a look this week if I can:

• Add a CI for Linux that builds the code and runs the test
• Use the GRDB from this branch in some toy project I've been developing and see how it works in production
• If the toy project works out fine, I will start testing some more complex examples

Yes, please play with it as much as you can - that's not something I can easily do!

For the CI part, I'd prefer, if it is possible, that we do not add Linux tests to the .github/workflows/CI.yml file. Instead, we should create a dedicated Linux workflow with its dedicated file. My goal here is that we can detect Linux build failures (that's what we are currently lacking), but without putting a red dot on GRDB itself — not until Linux is an "officially" supported platform.

[thinkpractice]

@groue @marcprux I've started adding a CI file for Ubuntu. The idea is that it will download Swiftly and use that to install the appropriate Swift version. I now added a matrix with both swift 6.1 as well as swift 6.2. Not sure if that's desired, if I understand correctly the CI.yml for Apple platforms only uses swift 6.1. It's not completely working yet, but this is more meant as a status update.

UPDATE: it looks like it's working now, but I have the feeling that is sometimes fails for some random reason. Is this to be expected?

I also added a batch to README.md for the Linux build. That will look like this

Would this be a way forward for the Linux build? As I'm quite new to GitHub Actions and I shamelessly stole some code from another project and adapted it to GRDB I'm looking forward to your feedback 😄

[marcprux]

I think this looks good, pending a couple suggestions! It will be nice to have this extra validation, and also makes a good first step towards Android support.

[marcprux]

I think this should be false, because it is often useful to see when one variant passes and another fails (e.g., if a change breaks for Swift 6.0 but not 6.1)

[thinkpractice]

I agree, changed cancel-in-progress to false

[marcprux]

You could alternatively just use uses: swift-actions/setup-swift@next as per swift-actions/setup-swift#710. I've been using it in other actions, and it seems quite stable.

[thinkpractice]

I did use setup-swift@v2 before but that didn't support swift 6.2. Also, it threw some random errors with the gpg verification (?) of swift, which made the pipeline fail. UPDATE apparently this gpg issue is also mentioned here, i.e. "Investigate GPG issues on linux". That's why I decided to add the swiftly code myself. I agree it would be better to use the setup-swift action in the future. I guess then @next selects the latest v3 version that uses swiftly as well? I saw on its PR that it's not yet merged into main right?

[thinkpractice]

Oh and I saw there's a possibility to cache the swiftly install but it seems with the current code it doesn't do anything. Not sure whether the path I added (${SWIFTLY_HOME_DIR:-$HOME/.local/share/swiftly}) is evaluated correctly or whether it's an issue with the cache action itself. @marcprux, @groue does any of you have some experience with this?

[groue]

Hello, I did not forget this pull request, but there are a lot of new commits that I did not take the time to review yet. Please hold on.

[thinkpractice]

@groue you've beat me to it 😄 I was going to ask you for the status of this. I will check for any upstream changes and try to merge them! I have been testing a bit with GRDB and Linux and haven't run into any issues yet. However, need to test more.

[groue]

Hello @thinkpractice,

I've merged the initial part of the pull request (3dd3bbf) into the development branch. Thank you! 🙏

This pull request now only contains the CI configuration. It looks like something could be done regarding caching? I'll leave the pull request open for a while, until this question finds a clear answer.

[thinkpractice]

Hi @groue,

thanks for merging! Just for my understanding, this will then merge all my changes into the development branch? I just had a look at this branch the other day to see if I could merge some new changes you made after december into it. It was made a bit more difficult because the development branch didn't contain any of these changes yet and I didn't have enough time (yet) to give it a thorough look.

I've merged the initial part of the pull request (3dd3bbf) into the development branch. Thank you! 🙏

You're more than welcome 😄 I will check-out the development branch tomorrow and try to build it on linux, run the tests. Although if you enable the CI it should pickup if things fail to build.

This pull request now only contains the CI configuration. It looks like something could be done regarding caching? I'll leave the pull request open for a while, until this question finds a clear answer.

Caching in what regard? Do you mean the toolchain installed with swiftly for instance? I've tried to do that in a CI in another project, but ran into some issues. When I know what you mean specifically I can give this a look, it will also benefit the other project, so killing two birds with one stone...

kind regards,

Tim

[groue]

The part that added Linux support is merged in development. It was the state of the pull request before you started working on CI. Click on the "Files changed" tab: it now only contains changes related to CI.

Caching in what regard?

You tell me ;-) This sentence of yours above raises questions:

Oh and I saw there's a possibility to cache the swiftly install

Caching speeds up CI. Like everybody else, I hate waiting. If the Linux CI wants me to look at it, it's better be reasonably fast. And it installs a lot of stuff. Marc also suggested some ideas.

For me this is not a blocker, but it looks like there's some unfinished business. It's all up to you. If you say "I'm done" I'll be OK with that.

[thinkpractice]

Caching speeds up CI. Like everybody else, I hate waiting. If the Linux CI wants me to look at it, it's better be reasonably fast. And it installs a lot of stuff. Marc also suggested some ideas.

For me this is not a blocker, but it looks like there's some unfinished business. It's all up to you. If you say "I'm done" I'll be OK with that.

I will have a look if I can speed it up. I remember updating the CI according to some of the changes @marcprux suggested, but maybe I've missed some. Hope I will be able to look at it this week.

[groue]

Thank you @thinkpractice!

[groue]

⛵ The first part of the pull request has shipped in v7.10.0 :-)

https://hachyderm.io/@groue/116075269263911686