Support SQLCipher using package traits#1708
Support SQLCipher using package traits#1708marcprux wants to merge 14 commits intogroue:developmentfrom
Conversation
|
Update: this was simply because I wasn't building in release mode. See comment below, where the SQLCipher build is actually 10% faster. |
|
@marcprux the Android NDK doesn't seem to bundle sqlite3.h, whereas GRDB and your PR do rely on that being available. How are you providing that right now when building and testing GRDB? |
It is included with the SQLCipher source module: https://github.com/skiptools/swift-sqlcipher/tree/main/Sources/SQLCipher/sqlite |
Isn't it just the cipher? (EDIT: I'm wrong because performance tests don't run on encrypted databases, see comment below.) SQLCipher can also be extremely slow when it opens a connection. Some people have reported up to 0.5 seconds (😱). To the point I had to ship #1350. |
|
I ran the same performance tests, but this time in release configuration: This time, SQLCipher tests are faster. |
Agreed, but the performance hit should only occur when encryption is actually enabled on the database, which shouldn't affect any of the perf tests.
Of course! Foolish of me to not do that. Otherwise, it is comparing the debug build of the sqlcipher/sqlite source build against the release build of the vendored sqlite. It is heartening to see that superior performance can be squeezed out of a source build. I wonder how much more might be managed with strategic performance-related flags (e.g., sbooth/CSQLite#59 (comment)). |
…abilites on non-Dawin platforms missing XCTIssue
… rather than disabiling them altogether
|
Apologies for the long delay in addressing the feedback (I was traveling). I believe I've handled each of the requested changes. Let me know if I can do anything else to help get this PR over the finish line – it has grown quite large, so I'm nervous that rebasing is going to be difficult if it starts to become stale. |
|
We've been using this branch successfully. |
|
I'll follow up on the list of needs at https://mastodon.social/@groue@hachyderm.io/113990450491505387 here, just to keep it in the PR record.
This should work out of the box for anyone that just changes: dependencies: [
.package(url: "https://github.com/groue/GRDB.swift.git", from: "7.0.0"),
]to the (theoretical) locally-maintained fork: dependencies: [
.package(url: "https://github.com/groue/GRDB-SQLCipher.swift.git", from: "7.0.0"),
]
You are welcome to relocate https://github.com/skiptools/swift-sqlcipher.git into the
This isn't part of this PR, but it would only be a couple of lines in the README.
This is part of the PR. The new
The tests do run against SQLCipher, but against an unencrypted database. I didn't see how this is being done with SQLCipher for CocoaPods, but I suppose we could just do the same setup here (perhaps just based on checking the
This shouldn't be an issue. Just activate (i.e., un-comment) the
For the SQLCipher dependency, you just need to make a new amalgamated sqlite.c from sqlcipher, commit it, and then tag a release. This is done with the script: https://github.com/skiptools/swift-sqlcipher/blob/main/scripts/build_sqlcipher.sh An example of this sort of update commit is the bump to sqlite 3.46.1 and sqlcipher 4.6.1: skiptools/swift-sqlcipher@5ea2f8b
Update: the fork is no longer necessary when using package traits (see comment below) |
|
I'm sorry I did not answer yet. I had to cope with the problems created with Xcode 16.3 beta. I'll get back to you soon. |
|
This PR saved me today while integrating a closed source framework which links SQLCipher while we were already extensively using grdb. |
|
I've updated this PR to use the new package traits in Swift 6.1, which seem designed for a scenario exactly like this. Now, rather than having to create and maintain a custom GRDB-SQLCipher fork that enables GRDBCIPHER, a consumer of this package can simply enable the Adding the trait to the dependency will activate the For the purpose of testing and CI, the "GRDBCIPHER" environment variable is still checked, but only to see whether to enable the trait by default in the The performance test environment can also still be used, which demonstrates that the custom SQLCipher build continues to be ~10% faster than the vendored SQLite3, at least on my machine: The only drawback is that since traits are simple booleans, you can no longer specify a custom repository with the "GRDBCIPHER" environment variable, so you need to depend on a specific repository. But an advantage is that the |
|
@groue Hoping you can take another look at this PR with the new Package traits dependency. I think it really improves on the original idea. |
I picked it simply because that is a package that I had already put together to work on Android. Any buildable SQLCipher amalgamation would work, and as I've mentioned before, you could simply pull it into the GRDB package as a C module in order to eliminate the need for an external dependency (which I think would be the ideal solution). I would be happy to expand this PR to provide that option, but it would introduce the potential maintenance burden of keeping up with subsequent SQLCipher releases, which I sense you aren't interested in.
You would be correct in thinking that the two issues (SQLCipher support and Android support) are orthogonal. They just happen to both be solvable through this PR. My ultimate goal is to bring this support to all platforms, including Android. GRDB currently doesn't work at all on Android due to there not being any linkable sqlite library on the platform (unlike Linux), so a source build is currently the only solution for that platform (irrespective of whether the source build is sqlcipher or just a stock sqlite package like https://github.com/sbooth/CSQLite or https://github.com/swiftlang/swift-toolchain-sqlite). As to your other concerns, I feel like they can all be mitigated by simply stating that this support is experimental and optional (just like the current Linux support). Nothing this PR does changes how GRDB functions on Apple platforms by default. It is only when opting-into the SQLCipher trait that the new build behavior would be activated. I'm happy to continue iterating on the PR or provide alternate suggestions/solutions, but I also understand if you would prefer GRDB to stick to narrow path of supporting only the vendored-SQLite on Apple platforms. |
|
All right, @marcprux, we are converging. I agree that we can just flag SQLCipher support as experimental. I'll have a closer look at the PR. |
|
(EDIT: in response to a comment that was deleted by its author) I'm not sure the meaning of "experimental" was correctly understood. Expressions such as "huge deal" and "intolerable" can only be supported by actual contributions. Il n’y a pas d’amour, il n’y a que des preuves d’amour (the proof is in the pudding). EDIT: Given the complete lack of mention of GRDB in the SQLiteData README, I would advise commenters to choose their arguments with more tact and discernement than some of their favorite developers. |
This was an honest mistake on our part and we saw your comment on Hacker News and immediately corrected it. We took for granted the obviousness of GRDB’s presence in the library in the name alone, and so forgot to update the readme with the 1.0 release. We are very sorry for that. |
| XCTFail("Expected DatabaseError") | ||
| } catch let error as DatabaseError { | ||
| XCTAssertEqual(error.resultCode, .SQLITE_ERROR) | ||
| #if canImport(ObjectiveC) // non-Darwin platforms bridge NSError differently and do not set the message the same |
There was a problem hiding this comment.
This PR merges two concerns here: SQLCipher and Android.
NSError is available on non-Darwin platforms, so people can use it. I'd be curious to know if this test can be restored. When a custom SQL function raises an error, it is important that the host application can catch a message that preserves as much as possible from the original error.
There was a problem hiding this comment.
What's the specific issue here? This test does pass on Linux, as the DatabaseError is used here directly. I don't need the compiler directive here. Is this different on Android?
|
|
||
| func testNSErrorBridging() throws { | ||
| #if !canImport(ObjectiveC) | ||
| throw XCTSkip("NSError bridging works differently on non-Darwin platforms") |
There was a problem hiding this comment.
Again, I'm having a similar issue on Linux. The DatabaseError in this test is thrown correctly but casting to an NSError in the catch clause fails. I guess this is either a bug in Swift on non-Darwin platforms or not supported. Why would we need NSError in this case instead of the Swift Error protocol? To preserve the error information in DatabaseError?
| func testEmptyMigratorPublisher() throws { | ||
| func test(writer: some DatabaseWriter) throws { | ||
| let migrator = DatabaseMigrator() | ||
| #if !canImport(Combine) |
There was a problem hiding this comment.
I'd move this check at the beginning of the test method (as all similar checks below).
| // > because DISPATCH_QUEUE_OVERCOMMIT is not a public API. I don't | ||
| // > know of a way to get a reference to the overcommit queue using | ||
| // > only public APIs. | ||
| #if !canImport(Darwin) |
There was a problem hiding this comment.
Should be moved at the beginning of the test function
| /// A XCTestCase subclass that can test its own failures. | ||
| class FailureTestCase: XCTestCase { | ||
| private struct Failure: Hashable { | ||
| #if canImport(Darwin) |
There was a problem hiding this comment.
Instead of turning regular methods into throwing methods, as below, in order to be able to throw XCTSkip, I'd rather remove the whole FailureTestCase from non-Darwin platforms (or Android only, if this limitation is specific to Android). I hope Swift Testing makes it possible to run tests that catch test failures.
|
|
||
| func testNSDecimalNumberPreservesIntegerValues() throws { | ||
| #if !canImport(ObjectiveC) | ||
| throw XCTSkip("NSDecimalNumber unavailable") |
There was a problem hiding this comment.
NSDecimalNumber is part of the GRDB support for decimals.
I find mentions of NSDecimalNumber in https://github.com/swiftlang/swift-corelibs-foundation and https://github.com/swiftlang/swift-foundation.
Is this lack of NSDecimalNumber temporary, or Android specific? Shouldn't we use #if os(android) instead of #if !canImport(ObjectiveC)?
Finally, shouldn't we disable the whole test class, since it is named FoundationNSDecimalNumberTests?
There was a problem hiding this comment.
I'm having a similar issue on Linux as mentioned here The issue is not the availability of NSDecimalNumber but a problem with two constructors which are used in converting int64 and double values from a DatabaseValue to an NSNumber. I have solved it similarly by ignoring these tests on Linux and Windows platforms. Not sure why these constructors are giving problems. Maybe they are not available? In any case it needs more research!
| func testNSNumberDatabaseValueToSwiftType() { | ||
| func testNSNumberDatabaseValueToSwiftType() throws { | ||
| #if !canImport(ObjectiveC) | ||
| throw XCTSkip("NSNumber unavailable") |
There was a problem hiding this comment.
Same question as above about NSDecimalNumber.
|
|
||
| func testNSURLDatabaseRoundTrip() throws { | ||
| #if !canImport(ObjectiveC) | ||
| throw XCTSkip("NSURL unavailable") |
There was a problem hiding this comment.
NSURL is less useful than NSNumber/NSDecimalNumber, so here I'll just recommend that the whole test class is disabled on platforms that do not support NSURL. Or to throw XCTSkip from setUp() so that we can see this test case in the test log.
(I admit I don't quite understand why we can keep on compiling this test, and yet do not compile the NSURL support for GRDB)
|
|
||
| private func assertRoundTrip(_ uuid: UUID) throws { | ||
| #if !canImport(ObjectiveC) | ||
| throw XCTSkip("NSUUID unavailable") |
There was a problem hiding this comment.
Same request as in FoundationNSURLTests above.
| XCTAssertNil(URL.fromDatabaseValue(databaseValue_Int64)) | ||
| XCTAssertNil(URL.fromDatabaseValue(databaseValue_Double)) | ||
| XCTAssertEqual(URL.fromDatabaseValue(databaseValue_Blob)!.absoluteString, "bar") | ||
| XCTAssertEqual(URL.fromDatabaseValue(databaseValue_Blob)?.absoluteString, "bar") |
There was a problem hiding this comment.
It looks like a change that is not strictly necessary.
| } | ||
|
|
||
|
|
||
| #if canImport(Darwin) // @MainActor test cases don't compile non-Darwin platforms: "call to main actor-isolated instance method 'test_mainActor_observation()' in a synchronous nonisolated(unsafe) context" |
There was a problem hiding this comment.
This needs a link to at least one bug report in the Swift repository.
| @@ -1,4 +1,4 @@ | |||
| // swift-tools-version:6.0 | |||
| // swift-tools-version:6.1 | |||
There was a problem hiding this comment.
Not required in this pull request, but some cleanup is needed:
git grep -i 'version.*\b6\.0'git grep 'compiler('README.mdGRDB.swift.podspec
| .library(name: "GRDB-dynamic", type: .dynamic, targets: ["GRDB"]), | ||
| ], | ||
| traits: [ | ||
| .trait(name: "GRDBCIPHER", description: "Use the SQLCipher library rather than the vendored SQLite"), |
There was a problem hiding this comment.
After reading the naming concerns expressed in #1816, and the CamelCase trait names used in the proposal SE-0450, I suggest using the name SkipSQLCipher or SkipToolsSQLCipher, and a description that mentions the repository URL https://github.com/skiptools/swift-sqlcipher
I assume, @sjlombardo, that the names I suggested are acceptable for Zetetic.
There was a problem hiding this comment.
We're about to have multiple SQLCipher versions: the one in this pull request, the one from #1816, and the CocoaPods one (even if it's high time I completely removed it).
To simplify the compiler flags dance, I start working on replacing most uses of #if GRDBCIPHER with #if SQLITE_HAS_CODEC. That last flag ought to be set by all GRDB+SQLCipher flavors. See #1819
cc @R4N
There was a problem hiding this comment.
@groue That sounds good to us. We were initially re-using the GRDBCIPHER flag, but using SQLITE_HAS_CODEC should work better for supporting both flavors. We'll switch over to that.
There was a problem hiding this comment.
@groue thanks, the suggested names look good to us, things will be very clear with the prefixing.
|
@marcprux, I found a pretty important blocker while validating the pull request for the official SQLCipher trait: #1827 (comment). This PR will suffer from the same problem. Basically, the fact that Xcode lacks support for traits is likely to block all progress. Maybe this can be fixed, but this is beyond my skills. |
|
I'm working on a project where we're encountering This vendor has linked an old version of SQLite/SQLCipher into their framework. This breaks GRDB because I see from the The way this is manifesting in our app (if we link this vendors framework) is SQL syntax errors for syntax that is very old and very correct. The only way I can think of this working correctly is if SQLite/SQLCipher is linked into into GRDB statically as a hidden dependency (which means no inline code or exposing of sqlite symbols Though this may not be possible with Swift given this old unresolved bug: swiftlang/swift#43633 |
Hello @groue, I haven’t read all the messages, so maybe someone has already suggested this, but there’s a pretty straightforward way to work around this limitation: you can import a local package into the project to enable the trait. This is the approach we use in our project using one local package as a proxy that depends on the package where we need the traits and enabling them there. This way, we have access to it. So, I don’t think the lack of trait support in Xcode should block this. People who need Linux/Android support can work around the Xcode limitation, and those who don’t need it won’t even be affected which I think applies to most users. I’ve also shared an example project that demonstrates this approach. testTrait.zip |
|
Thank you for your hint, @mackoj :-) The use case that is currently broken is the plain Xcode app that has a plain dependency on GRDB (via the Xcode UI), without any wrapper package. That's what most apps that use GRDB today do. The technique of the wrapper package is not a satisfying solution for those apps, because it would be a painful breaking change. Now, I hope that the use case that is currently broken can be restored while keeping the ability to use SQLCipher through SPM. That's the current discussion in #1827. |
|
This pull request is obsolete, so I'm closing it. Thank you nevertheless @marcprux, your support is very precious! 🙏 The Android situation should be in better shape since I merged the Linux compatibility changes of #1825 into the development branch. #1825 is still open, but it is now re-focused on the CI. The SQLCipher situation is also in better shape: #1845 should be merged shortly. From now on, future progress on the SQLCipher or Android topics should happen independently, in dedicated pull requests. |
This is a less ambitious alternative to #1701 that enables adding a SQLCipher source dependency to the Package.swift based on the
GRDBCIPHERenvironment variable. Tests can be run with the dependency with a command like:Running it without the environment variable will leave things exactly as they are. The majority of changes in this PR are simply re-ordering the import header checks for
GRDBCIPHERandSWIFT_PACKAGE, since we would need the former to take precedence of the latter.An advantage of doing it this way is that it facilitates creating a(this is no longer needed when using package traits; see comment below)GRDB-SQLCipherfork that can follow the releases of the upstream. The only change in such a fork would be to hardwire theGRDBCIPHERsetting in Package.swift to some SQLCipher source repository (either mine, or one that you maintain). When a client package wants to switch between GRDB and GRDB-SQLCipher, they would just need to swap their dependency URL.This PR also adds in support building and testing against Android, mostly by stubbing out the unit tests that can't compile due to missing
NSFileCoordinator,Combine, and the like. Android tests can be run with skip:Most of the tests pass, but there are a few that need to be investigated. I'll look into them next.
Closes #1701