Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an experimental backend that uses Lean to discharge goals #850

Merged
merged 33 commits into from
Mar 19, 2024
Merged

Add an experimental backend that uses Lean to discharge goals #850

Show file tree
Hide file tree
Changes from 27 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
bf36ddf
Translator sufficient for Bubble.bpl to typecheck
atomb Dec 13, 2023
8309d33
Now Bubble.bpl verifies after translation
atomb Dec 14, 2023
4b2eb9e
Control Lean printing with an option
atomb Dec 19, 2023
89b3595
Translator sufficient to verify all but one textbook example
atomb Dec 19, 2023
f6f8ff6
Reorganize unsupported overrides
atomb Dec 19, 2023
637c415
Various improvements
atomb Dec 20, 2023
e4c34dc
Successfully process trivial Dafny program
atomb Dec 22, 2023
2556b26
Cleanups
atomb Dec 22, 2023
7049b6d
Update exception messages
atomb Feb 13, 2024
b2c5b14
Move prelude to external file
atomb Feb 13, 2024
d2193dd
Add configuration files to check Prelude.lean
atomb Feb 13, 2024
7c096c0
Cleanup comments, strings, exceptions
atomb Feb 13, 2024
7fea48d
Clean up error handling
atomb Feb 14, 2024
05b616c
Reorder cases and improve error messages
atomb Feb 14, 2024
005d80d
Add test suite for LeanAuto backend
atomb Feb 14, 2024
df6bb76
Add README for LeanAuto
atomb Feb 14, 2024
6628fea
Add workflow to test the LeanAuto backend
atomb Feb 14, 2024
300ed90
Fill in proofs and definitions in Lean prelude
atomb Feb 14, 2024
359da35
Clean up lakefile
atomb Feb 14, 2024
4fc941d
Fix Lean installation
atomb Feb 14, 2024
bf46f95
Change setup-lean action name
atomb Feb 14, 2024
154fee4
Ensure Lean dependencies are actually built
atomb Feb 14, 2024
9d58eb5
Pin the version of lean-auto for reproducibility
atomb Feb 14, 2024
2896020
Merge remote-tracking branch 'upstream/master' into basic-lean-auto
atomb Mar 6, 2024
a2a40d6
Rename LeanGenerator -> LeanAutoGenerator
atomb Mar 6, 2024
49e1eec
Rename some methods
atomb Mar 6, 2024
2f8b3a9
Move use of passive program to a separate method
atomb Mar 6, 2024
fd7b4c9
Clarify comment
atomb Mar 7, 2024
dd8359f
Merge remote-tracking branch 'upstream/master' into basic-lean-auto
atomb Mar 7, 2024
dc6b667
Merge branch 'master' into basic-lean-auto
atomb Mar 13, 2024
4cc7547
Generalized processing of passive programs
atomb Mar 19, 2024
29979dc
Allow `goto` to have no targets
atomb Mar 19, 2024
fbafb12
Use `foreach` instead of `ForEach`
atomb Mar 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions .github/workflows/test-lean-auto.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
name: LeanAuto CI

on:
workflow_dispatch:
push:
tags:
- 'v*'
pull_request:
branches:
- master

jobs:
job0:
name: LeanAuto CI
runs-on: ubuntu-22.04
steps:
- name: Setup dotnet
uses: actions/setup-dotnet@v3
with:
dotnet-version: '6.0.x'
- name: Setup Z3
uses: cda-tum/setup-z3@v1
with:
version: 4.12.5
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Checkout Boogie
uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Install tools, build Boogie
run: |
dotnet tool restore
dotnet build -warnaserror Source/Boogie.sln
- name: Setup Lean
uses: Julian/setup-lean@v1
with:
# Lean version will be chosen based on Test/lean-auto/lean-toolchain
default-toolchain-file: Test/lean-auto/lean-toolchain
- name: Test Lean generator on textbook examples
working-directory: Test/lean-auto
run: |
lake build
./testall.sh
15 changes: 15 additions & 0 deletions Source/Boogie.sln
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "CoreTests", "UnitTests\Core
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "ExecutionEngineTests", "UnitTests\ExecutionEngineTests\ExecutionEngineTests.csproj", "{473CF455-4306-46E3-9A44-FB7DBA42CA38}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "LeanAuto", "Provers\LeanAuto\LeanAuto.csproj", "{FF9CFE2D-AB94-4973-B0E3-FB00F707ACDC}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU
Expand Down Expand Up @@ -294,12 +296,25 @@ Global
{473CF455-4306-46E3-9A44-FB7DBA42CA38}.Release|x64.Build.0 = Release|Any CPU
{473CF455-4306-46E3-9A44-FB7DBA42CA38}.Release|x86.ActiveCfg = Release|Any CPU
{473CF455-4306-46E3-9A44-FB7DBA42CA38}.Release|x86.Build.0 = Release|Any CPU
{FF9CFE2D-AB94-4973-B0E3-FB00F707ACDC}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{FF9CFE2D-AB94-4973-B0E3-FB00F707ACDC}.Debug|Any CPU.Build.0 = Debug|Any CPU
{FF9CFE2D-AB94-4973-B0E3-FB00F707ACDC}.Debug|x64.ActiveCfg = Debug|Any CPU
{FF9CFE2D-AB94-4973-B0E3-FB00F707ACDC}.Debug|x64.Build.0 = Debug|Any CPU
{FF9CFE2D-AB94-4973-B0E3-FB00F707ACDC}.Debug|x86.ActiveCfg = Debug|Any CPU
{FF9CFE2D-AB94-4973-B0E3-FB00F707ACDC}.Debug|x86.Build.0 = Debug|Any CPU
{FF9CFE2D-AB94-4973-B0E3-FB00F707ACDC}.Release|Any CPU.ActiveCfg = Release|Any CPU
{FF9CFE2D-AB94-4973-B0E3-FB00F707ACDC}.Release|Any CPU.Build.0 = Release|Any CPU
{FF9CFE2D-AB94-4973-B0E3-FB00F707ACDC}.Release|x64.ActiveCfg = Release|Any CPU
{FF9CFE2D-AB94-4973-B0E3-FB00F707ACDC}.Release|x64.Build.0 = Release|Any CPU
{FF9CFE2D-AB94-4973-B0E3-FB00F707ACDC}.Release|x86.ActiveCfg = Release|Any CPU
{FF9CFE2D-AB94-4973-B0E3-FB00F707ACDC}.Release|x86.Build.0 = Release|Any CPU
EndGlobalSection
GlobalSection(NestedProjects) = preSolution
{FDF25D38-01A2-4EAA-8A3E-6F8F7CD254D2} = {EC2B5ECD-B97D-43D6-86F8-28163710B717}
{65A3F7E9-3E13-4C26-A72A-F60B9943E1B2} = {91134A3E-0D5D-4017-B04E-48BA254B3CF8}
{10F866A2-6A9B-4B3D-AEAB-43314BD0475D} = {91134A3E-0D5D-4017-B04E-48BA254B3CF8}
{D2D77420-CFDB-4DA1-B7E7-844C8E8CC686} = {91134A3E-0D5D-4017-B04E-48BA254B3CF8}
{473CF455-4306-46E3-9A44-FB7DBA42CA38} = {91134A3E-0D5D-4017-B04E-48BA254B3CF8}
{FF9CFE2D-AB94-4973-B0E3-FB00F707ACDC} = {EC2B5ECD-B97D-43D6-86F8-28163710B717}
EndGlobalSection
EndGlobal
2 changes: 1 addition & 1 deletion Source/Core/AST/AbsyType.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2011,7 +2011,7 @@ public class TypeProxy : Type
{
static int proxies = 0;

protected readonly string /*!*/
public readonly string /*!*/
Name;

[ContractInvariantMethod]
Expand Down
9 changes: 9 additions & 0 deletions Source/ExecutionEngine/CommandLineOptions.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ void ObjectInvariant2()
public bool VerifySeparately { get; set; }
public string PrintFile { get; set; }
public string PrintPrunedFile { get; set; }
public string LeanFile { get; set; }

/**
* Whether to emit {:qid}, {:skolemid} and set-info :boogie-vc-id
Expand Down Expand Up @@ -676,6 +677,14 @@ protected override bool ParseOption(string name, CommandLineParseState ps)

return true;

case "printLean":
if (ps.ConfirmArgumentCount(1))
{
LeanFile = args[ps.i];
}

return true;

case "pretty":
int val = 1;
if (ps.GetIntArgument(x => val = x, 2))
Expand Down
28 changes: 21 additions & 7 deletions Source/ExecutionEngine/ExecutionEngine.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,13 @@
using System.Threading;
using System.Threading.Tasks;
using VC;
using BoogiePL = Microsoft.Boogie;
using System.Runtime.Caching;
using System.Diagnostics;
using System.Reactive.Linq;
using System.Reactive.Subjects;
using System.Reactive.Threading.Tasks;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices.ComTypes;
using Microsoft.Boogie.LeanAuto;
using Microsoft.Boogie.VCExprAST;
using VCGeneration;

namespace Microsoft.Boogie
Expand Down Expand Up @@ -572,10 +571,11 @@ public async Task<PipelineOutcome> InferAndVerify(
}

var outcome = await VerifyEachImplementation(output, processedProgram, stats, programId, er, requestId, stablePrioritizedImpls);
if (Options.PrintPassive) {
Options.PrintUnstructured = 1;
PrintBplFile(Options.PrintFile, processedProgram.Program, true, true, Options.PrettyPrint);
}

// We unfortunately have to do any processing of the passive program after verification,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand this comment. These lines seems contradictory, both saying that processedProgram.Program is an already passified program, and that it's not yet passified because that only happens during verification.

By the way, what step do you mean by "verification" in the context of Boogie? The whole pipeline could be considered verification.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I mean is that ProcessPassiveProgram needs to be called after VerifyEachImplementation instead of before, because VerifyEachImplementation turns processedProgram into a passive program, destructively. I would love it if everything happened in PreProcessProgramVerification, but it doesn't right now, and I think it'd take a pretty big refactoring to get it there.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, if you change verification to calling VerifyEachImplementation in the comment, then it would be more clear to me.

Copy link
Collaborator

@keyboardDrummer keyboardDrummer Mar 7, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Btw, extracting out the passive-ication is already done in the method vcGenerator.PrepareImplementation, although it only does it for a single implementation. GetVerificationTasks gives an example of how to call it. Maybe you want to use that, so you don't have to actually generate SMT each time?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would generally make sense to generate Lean code as an alternative to SMT-Lib, so it could make sense for it to be called from the VCGeneration package. I took a stab at refactoring things to do that, but as things currently stand there are two issues:

  • The Lean generator depends on the VCGeneration package, and we can't create a cyclic dependency.
  • The Lean generator currently translates the whole program in one go, which doesn't really fit with the current structure of the VC generator.

I think it's possible to refactor things further to not require that cyclic dependency, and I'm considering whether the Lean generator should work on a per-implementation basis, but I think I'd prefer to leave both for a future PR (probably one in the pretty near future, though).

Copy link
Collaborator

@keyboardDrummer keyboardDrummer Mar 19, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To me it's already a problem that ExecutionEngine is calling out to LeanGenerator, because it bloats the scope of ExecutionEngine.

I would go for the combination of my previous two suggestions, to remove the dependency of ExecutionEngine on LeanGenerator, and make use of vcGenerator.PrepareImplementation

Change

var processedProgram = PreProcessProgramVerification(program);
if (!Options.Verify)
{
  return PipelineOutcome.Done;
}

if (Options.ContractInfer)
{
  return await RunHoudini(program, stats, er);
}

...
var outcome = await VerifyEachImplementation(output, processedProgram, stats, programId, er, requestId, stablePrioritizedImpls);

if (Options.LeanFile is not null) {
  var writer = new StreamWriter(Options.LeanFile);
  LeanAutoGenerator.EmitPassiveProgramAsLean(Options, passiveProgram, writer);
  writer.Close();
}

Into

var processedProgram = PreProcessProgramVerification(program);
if (!Options.Verify)
{
  return PipelineOutcome.Done;
}

if (Options.UseResolvedProgram != null && Options.UseResolvedProgram(processedProgram))
{
  return;
}

...
var outcome = await VerifyEachImplementation(output, processedProgram, stats, programId, er, requestId, stablePrioritizedImpls);

Note that both Houdini and Lean are no longer mentioned.

Then instead of setting LeanFile, set Options.UseResolvedProgram with:

void LetLeanUseResolvedProgram(Program program) {
  foreach(var implementation in program.Implementations) {
    vcGenerator.PrepareImplementation(new ImplementationRun(implementation, options.OutputWriter), ..)
  }

  var writer = new StreamWriter(Options.LeanFile);
  EmitPassiveProgramAsLean(Options, program, writer);
  writer.Close();
}

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this approach, and I've pushed some changes that go in roughly that direction.

// because passification happens during verification. Refactoring the code to passify
// the program as an independent step would be a valuable thing to do eventually.
UsePassiveProgram(processedProgram.Program);

if (1 < Options.VerifySnapshots && programId != null)
{
Expand All @@ -588,6 +588,20 @@ public async Task<PipelineOutcome> InferAndVerify(
return outcome;
}

private void UsePassiveProgram(Program passiveProgram)
{
if (Options.PrintPassive) {
Options.PrintUnstructured = 1;
PrintBplFile(Options.PrintFile, passiveProgram, true, true, Options.PrettyPrint);
}

if (Options.LeanFile is not null) {
var writer = new StreamWriter(Options.LeanFile);
LeanAutoGenerator.EmitPassiveProgramAsLean(Options, passiveProgram, writer);
writer.Close();
}
}

private ProcessedProgram PreProcessProgramVerification(Program program)
{
// Doing lambda expansion before abstract interpretation means that the abstract interpreter
Expand Down
1 change: 1 addition & 0 deletions Source/ExecutionEngine/ExecutionEngine.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
<ProjectReference Include="..\Graph\Graph.csproj" />
<ProjectReference Include="..\Houdini\Houdini.csproj" />
<ProjectReference Include="..\Model\Model.csproj" />
<ProjectReference Include="..\Provers\LeanAuto\LeanAuto.csproj" />
<ProjectReference Include="..\VCGeneration\VCGeneration.csproj" />
<ProjectReference Include="..\Provers\SMTLib\SMTLib.csproj" />
</ItemGroup>
Expand Down
1 change: 1 addition & 0 deletions Source/ExecutionEngine/ExecutionEngineOptions.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ public interface ExecutionEngineOptions : HoudiniOptions, ConcurrencyOptions
string DescriptiveToolName { get; }
bool TraceProofObligations { get; }
string PrintFile { get; }
string LeanFile { get; }
string PrintCFGPrefix { get; }
string CivlDesugaredFile { get; }
bool CoalesceBlocks { get; }
Expand Down
2 changes: 2 additions & 0 deletions Source/Provers/LeanAuto/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
.lake
lake-manifest.json
25 changes: 25 additions & 0 deletions Source/Provers/LeanAuto/LeanAuto.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<OutputType>Library</OutputType>
<AssemblyName>Boogie.Provers.LeanAuto</AssemblyName>
</PropertyGroup>

<ItemGroup>
<ProjectReference Include="..\..\BaseTypes\BaseTypes.csproj" />
<ProjectReference Include="..\..\CodeContractsExtender\CodeContractsExtender.csproj" />
<ProjectReference Include="..\..\Core\Core.csproj" />
<ProjectReference Include="..\..\VCGeneration\VCGeneration.csproj" />
</ItemGroup>

<ItemGroup>
<PackageReference Include="System.Linq.Async" Version="6.0.1" />
</ItemGroup>

<ItemGroup>
<EmbeddedResource Include="Prelude.lean">
<LogicalName>Prelude.lean</LogicalName>
</EmbeddedResource>
</ItemGroup>

</Project>
Loading
Loading