[AutoDiff] Crash when overriding _Differentiation module

[philipturner]

I don't know if we can fix this bug, but it should still be reported. In the current directory, make the following files:

Differentiable.swift

public protocol Differentiable {
  associatedtype TangentVector
}

ArrayDifferentiation.swift

import _Differentiation

extension Array: Differentiable {}

Run the following from the command line. Change FRONTEND to match your toolchain or apple/swift build products.

mkdir .build
export FRONTEND="/Library/Developer/Toolchains/swift-5.7-DEVELOPMENT-SNAPSHOT-2022-06-26-a.xctoolchain/usr/bin/swift-frontend"
$FRONTEND -frontend -emit-module $(pwd)/Differentiable.swift -o $(pwd)/.build/_Differentiation.swiftmodule
$FRONTEND -frontend -emit-module $(pwd)/ArrayDifferentiation.swift -I $(pwd)/.build

Crash

(base) philipturner@m1-max-mbp AutoDiffPackage2 % /Library/Developer/Toolchains/swift-5.7-DEVELOPMENT-SNAPSHOT-2022-06-26-a.xctoolchain/usr/bin/swift-frontend -frontend -emit-module $(pwd)/ArrayDifferentiation.swift -I $(pwd)/.build
Assertion failed: (loc.isValid() && "Expected valid source location"), function checkAndDiagnoseImplicitNoDerivative, file DerivedConformanceDifferentiable.cpp, line 567.
Please submit a bug report (https://swift.org/contributing/#reporting-bugs) and include the project and the crash backtrace.
Stack dump:
0.	Program arguments: /Library/Developer/Toolchains/swift-5.7-DEVELOPMENT-SNAPSHOT-2022-06-26-a.xctoolchain/usr/bin/swift-frontend -frontend -emit-module /Users/philipturner/Desktop/AutoDiffPackage2/ArrayDifferentiation.swift -I /Users/philipturner/Desktop/AutoDiffPackage2/.build
1.	Apple Swift version 5.7-dev (LLVM eecf02df5133efe, Swift ad06e7d7251311a)
2.	Compiling with the current language version
3.	While evaluating request TypeCheckSourceFileRequest(source_file "/Users/philipturner/Desktop/AutoDiffPackage2/ArrayDifferentiation.swift")
4.	While type-checking extension of Array (at /Users/philipturner/Desktop/AutoDiffPackage2/ArrayDifferentiation.swift:3:1)
5.	While type-checking protocol conformance to 'Differentiable' (in module '_Differentiation') for type 'Array<Element>'
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  swift-frontend           0x0000000108dd7144 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 56
1  swift-frontend           0x0000000108dd63a4 llvm::sys::RunSignalHandlers() + 128
2  swift-frontend           0x0000000108dd77a8 SignalHandler(int) + 304
3  libsystem_platform.dylib 0x00000001877974a4 _sigtramp + 56
4  libsystem_pthread.dylib  0x000000018777fee0 pthread_kill + 288
5  libsystem_c.dylib        0x00000001876ba340 abort + 168
6  libsystem_c.dylib        0x00000001876b9754 err + 0
7  swift-frontend           0x000000010910dbec swift::DerivedConformance::deriveDifferentiable(swift::AssociatedTypeDecl*) (.cold.7) + 0
8  swift-frontend           0x000000010548048c swift::DerivedConformance::deriveDifferentiable(swift::AssociatedTypeDecl*) + 4796
9  swift-frontend           0x00000001055913c8 swift::TypeChecker::deriveTypeWitness(swift::DeclContext*, swift::NominalTypeDecl*, swift::AssociatedTypeDecl*) + 352
10 swift-frontend           0x00000001055a82e8 swift::AssociatedTypeInference::computeDerivedTypeWitness(swift::AssociatedTypeDecl*) + 132
11 swift-frontend           0x00000001055ad898 swift::AssociatedTypeInference::solve(swift::ConformanceChecker&) + 252
12 swift-frontend           0x00000001055aeee8 swift::ConformanceChecker::resolveTypeWitnesses() + 168
13 swift-frontend           0x00000001055845fc swift::ConformanceChecker::checkConformance(swift::MissingWitnessDiagnosisKind) + 172
14 swift-frontend           0x0000000105581dc4 swift::MultiConformanceChecker::checkIndividualConformance(swift::NormalProtocolConformance*, bool) + 3440
15 swift-frontend           0x0000000105580e54 swift::MultiConformanceChecker::checkAllConformances() + 120
16 swift-frontend           0x000000010558e324 swift::TypeChecker::checkConformancesInContext(swift::IterableDeclContext*) + 5424
17 swift-frontend           0x000000010554b190 swift::ASTVisitor<(anonymous namespace)::DeclChecker, void, void, void, void, void, void>::visit(swift::Decl*) + 8588
18 swift-frontend           0x0000000105545b34 (anonymous namespace)::DeclChecker::visit(swift::Decl*) + 224
19 swift-frontend           0x0000000105545a40 swift::TypeChecker::typeCheckDecl(swift::Decl*, bool) + 116
20 swift-frontend           0x00000001055f6424 swift::TypeCheckSourceFileRequest::evaluate(swift::Evaluator&, swift::SourceFile*) const + 176
21 swift-frontend           0x00000001055f82bc llvm::Expected<swift::TypeCheckSourceFileRequest::OutputType> swift::Evaluator::getResultUncached<swift::TypeCheckSourceFileRequest>(swift::TypeCheckSourceFileRequest const&) + 400
22 swift-frontend           0x00000001055f805c llvm::Expected<swift::TypeCheckSourceFileRequest::OutputType> swift::Evaluator::getResultCached<swift::TypeCheckSourceFileRequest, (void*)0>(swift::TypeCheckSourceFileRequest const&) + 124
23 swift-frontend           0x00000001055f6268 swift::TypeCheckSourceFileRequest::OutputType swift::evaluateOrDefault<swift::TypeCheckSourceFileRequest>(swift::Evaluator&, swift::TypeCheckSourceFileRequest, swift::TypeCheckSourceFileRequest::OutputType) + 44
24 swift-frontend           0x000000010494a14c bool llvm::function_ref<bool (swift::SourceFile&)>::callback_fn<swift::CompilerInstance::performSema()::$_7>(long, swift::SourceFile&) + 16
25 swift-frontend           0x00000001049465f8 swift::CompilerInstance::forEachFileToTypeCheck(llvm::function_ref<bool (swift::SourceFile&)>) + 224
26 swift-frontend           0x00000001049464ec swift::CompilerInstance::performSema() + 76
27 swift-frontend           0x00000001048f89d0 withSemanticAnalysis(swift::CompilerInstance&, swift::FrontendObserver*, llvm::function_ref<bool (swift::CompilerInstance&)>, bool) + 60
28 swift-frontend           0x00000001048eca78 swift::performFrontend(llvm::ArrayRef<char const*>, char const*, void*, swift::FrontendObserver*) + 3388
29 swift-frontend           0x00000001047f308c swift::mainEntry(int, char const**) + 3764
30 dyld                     0x000000011168108c start + 520
zsh: abort       -frontend -emit-module $(pwd)/ArrayDifferentiation.swift -I $(pwd)/.build

---

There is another crash that's related, and must be worked around in S4TF by making Tensor(stacking:alongAxis:) not differentiable on release toolchains. Replace the contents of "ArrayDifferentiation.swift" with:

import _Differentiation

extension Array: Differentiable {
  public typealias TangentVector = Self
}

struct Array2<Element>: Differentiable {
  // `arrayLiteral2` must be variadic.
  init(arrayLiteral2 elements: Element...) {
    fatalError()
  }
  typealias TangentVector = Self
}

struct Tensor: Differentiable {
  typealias TangentVector = Self
}

@differentiable(reverse)
func Tensor_stack(_ lhs: Tensor) -> Tensor {
  Tensor_init(stacking: Array2(arrayLiteral2: lhs))
}

func Tensor_init(stacking tensors: Array2<Tensor>) -> Tensor {
  fatalError()
}

@derivative(of: Tensor_init(stacking:))
func _vjpStacking(
  stacking tensors: Array2<Tensor>
) -> (value: Tensor, pullback: (Tensor) -> Array2<Tensor>.TangentVector) {
  fatalError()
}

Crash

(base) philipturner@m1-max-mbp AutoDiffPackage2 % $FRONTEND -frontend -emit-module $(pwd)/ArrayDifferentiation.swift -I $(pwd)/.build

Assertion failed: (isa<X>(Val) && "cast<Ty>() argument of incompatible type!"), function cast, file Casting.h, line 269.
Please submit a bug report (https://swift.org/contributing/#reporting-bugs) and include the project and the crash backtrace.
Stack dump:
0.	Program arguments: /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2022-06-08-a.xctoolchain/usr/bin/swift-frontend -frontend -emit-module /Users/philipturner/Desktop/AutoDiffPackage2/ArrayDifferentiation.swift -I /Users/philipturner/Desktop/AutoDiffPackage2/.build
1.	Apple Swift version 5.8-dev (LLVM b2416e1165ab97c, Swift 965a54f037cfa76)
2.	Compiling with the current language version
3.	While evaluating request ExecuteSILPipelineRequest(Run pipelines { Mandatory Diagnostic Passes + Enabling Optimization Passes } on SIL for ArrayDifferentiation)
4.	While running pass #47 SILModuleTransform "Differentiation".
5.	While processing // differentiability witness for Tensor_stack(_:)
sil_differentiability_witness hidden [reverse] [parameters 0] [results 0] @$s20ArrayDifferentiation12Tensor_stackyAA0C0VADF : $@convention(thin) (Tensor) -> Tensor {
}

 on SIL function "@$s20ArrayDifferentiation12Tensor_stackyAA0C0VADF".
 for 'Tensor_stack(_:)' (at /Users/philipturner/Desktop/AutoDiffPackage2/ArrayDifferentiation.swift:20:1)
6.	While generating VJP for SIL function "@$s20ArrayDifferentiation12Tensor_stackyAA0C0VADF".
 for 'Tensor_stack(_:)' (at /Users/philipturner/Desktop/AutoDiffPackage2/ArrayDifferentiation.swift:20:1)
7.	While generating pullback for SIL function "@$s20ArrayDifferentiation12Tensor_stackyAA0C0VADF".
 for 'Tensor_stack(_:)' (at /Users/philipturner/Desktop/AutoDiffPackage2/ArrayDifferentiation.swift:20:1)
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  swift-frontend           0x0000000107319848 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 56
1  swift-frontend           0x0000000107318aa8 llvm::sys::RunSignalHandlers() + 128
2  swift-frontend           0x0000000107319eac SignalHandler(int) + 304
3  libsystem_platform.dylib 0x00000001877974a4 _sigtramp + 56
4  libsystem_pthread.dylib  0x000000018777fee0 pthread_kill + 288
5  libsystem_c.dylib        0x00000001876ba340 abort + 168
6  libsystem_c.dylib        0x00000001876b9754 err + 0
7  swift-frontend           0x00000001074d4e04 swift::autodiff::PullbackCloner::Implementation::accumulateAdjointsDirect(swift::autodiff::AdjointValue, swift::autodiff::AdjointValue, swift::SILLocation) (.cold.1) + 0
8  swift-frontend           0x00000001034e8af8 swift::autodiff::PullbackCloner::Implementation::getArrayAdjointElementBuffer(swift::SILValue, int, swift::SILLocation) + 2152
9  swift-frontend           0x00000001034e81d8 swift::autodiff::PullbackCloner::Implementation::getAdjointProjection(swift::SILBasicBlock*, swift::SILValue) + 2072
10 swift-frontend           0x00000001034e1ab0 swift::autodiff::PullbackCloner::Implementation::getAdjointBuffer(swift::SILBasicBlock*, swift::SILValue) + 228
11 swift-frontend           0x00000001034eede8 swift::autodiff::PullbackCloner::Implementation::visitStoreOperation(swift::SILBasicBlock*, swift::SILLocation, swift::SILValue, swift::SILValue) + 76
12 swift-frontend           0x00000001034e64b8 swift::autodiff::PullbackCloner::Implementation::visit(swift::SILInstruction*) + 200
13 swift-frontend           0x00000001034e2b30 swift::autodiff::PullbackCloner::Implementation::visitSILBasicBlock(swift::SILBasicBlock*) + 680
14 swift-frontend           0x00000001034dfffc swift::autodiff::PullbackCloner::Implementation::run() + 5028
15 swift-frontend           0x00000001034dec2c swift::autodiff::PullbackCloner::run() + 24
16 swift-frontend           0x00000001034f9efc swift::autodiff::VJPCloner::Implementation::run() + 1368
17 swift-frontend           0x00000001034fa52c swift::autodiff::VJPCloner::run() + 24
18 swift-frontend           0x00000001035fbd7c (anonymous namespace)::DifferentiationTransformer::canonicalizeDifferentiabilityWitness(swift::SILDifferentiabilityWitness*, swift::autodiff::DifferentiationInvoker, swift::IsSerialized_t) + 5288
19 swift-frontend           0x00000001035f9f00 (anonymous namespace)::Differentiation::run() + 932
20 swift-frontend           0x00000001036720e4 swift::SILPassManager::runModulePass(unsigned int) + 956
21 swift-frontend           0x0000000103677858 swift::SILPassManager::execute() + 624
22 swift-frontend           0x000000010366f1ac swift::SILPassManager::executePassPipelinePlan(swift::SILPassPipelinePlan const&) + 68
23 swift-frontend           0x000000010366f134 swift::ExecuteSILPipelineRequest::evaluate(swift::Evaluator&, swift::SILPipelineExecutionDescriptor) const + 68
24 swift-frontend           0x0000000103690e38 swift::SimpleRequest<swift::ExecuteSILPipelineRequest, std::__1::tuple<> (swift::SILPipelineExecutionDescriptor), (swift::RequestFlags)1>::evaluateRequest(swift::ExecuteSILPipelineRequest const&, swift::Evaluator&) + 28
25 swift-frontend           0x000000010367a690 llvm::Expected<swift::ExecuteSILPipelineRequest::OutputType> swift::Evaluator::getResultUncached<swift::ExecuteSILPipelineRequest>(swift::ExecuteSILPipelineRequest const&) + 252
26 swift-frontend           0x000000010366f3a8 swift::executePassPipelinePlan(swift::SILModule*, swift::SILPassPipelinePlan const&, bool, swift::irgen::IRGenModule*) + 84
27 swift-frontend           0x000000010367ed88 swift::runSILDiagnosticPasses(swift::SILModule&) + 92
28 swift-frontend           0x0000000102eae610 swift::CompilerInstance::performSILProcessing(swift::SILModule*) + 68
29 swift-frontend           0x0000000102e53540 performCompileStepsPostSILGen(swift::CompilerInstance&, std::__1::unique_ptr<swift::SILModule, std::__1::default_delete<swift::SILModule> >, llvm::PointerUnion<swift::ModuleDecl*, swift::SourceFile*>, swift::PrimarySpecificPaths const&, int&, swift::FrontendObserver*) + 716
30 swift-frontend           0x0000000102e52f04 swift::performCompileStepsPostSema(swift::CompilerInstance&, int&, swift::FrontendObserver*) + 1032
31 swift-frontend           0x0000000102e609b8 withSemanticAnalysis(swift::CompilerInstance&, swift::FrontendObserver*, llvm::function_ref<bool (swift::CompilerInstance&)>, bool) + 160
32 swift-frontend           0x0000000102e54a48 swift::performFrontend(llvm::ArrayRef<char const*>, char const*, void*, swift::FrontendObserver*) + 3388
33 swift-frontend           0x0000000102d3b124 swift::mainEntry(int, char const**) + 3764
34 dyld                     0x000000010fc5908c start + 520
zsh: abort      $FRONTEND -frontend -emit-module $(pwd)/ArrayDifferentiation.swift -I 

Additional Context

This was originally mistaken for SR-14008.

[philipturner]

I found a workaround for the second crash. Array.init(arrayLiteral:) is not differentiable because Element... is not differentiable. The solution: initialize Array in a different way.

@differentiable(reverse)
func Tensor_stack(_ lhs: Tensor) -> Tensor {
  var stack = Array<Tensor>()
  stack.append(lhs)
  return Tensor_init(stacking: stack)
}

This uses Array.init() and Array.append(_:), both of which are differentiable. I will use this patch in S4TF.

[AnthonyLatsis]

@philipturner Why did you close your issues?

[AnthonyLatsis]

@philipturner I’m sorry to bother you, but it wasn’t just this one; there are at least 10 more. Was it an accident? Do you want me to reopen them?

[philipturner]

I felt like they should be closed since I am no longer involved with the Swift compiler. I have no intention to solve them, and there seems to be a low chance most will ever be touched again. Except for one thread with Anton, where I glanced over the fact that it was still actively investigated in January 2023.

I was thinking in the mindset of issues in other repositories, where you generally close issues you no longer see as worth solving. Hence "closed as not planned". I forgot that for Swift, issues are an official bug tracker (like JIRA). Therefore closing them might make people mistake them as already being solved. If someone deems a specific bug worth investigating in the future, I can re-open them on a case by case basis. However, I do not wish to provide assistance to anyone investigating.

[BradLarson]

I'll be glad to review these and see if they are still reproducible in top-of-tree Swift. If so, I can reopen them so that they can be tracked to be fixed as we work through a priority list of differentiable Swift issues.