Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run again tests with new default checks #8106

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
93c1c12
Revert "Add --no-standard-checks to regression/cbmc runner scripts"
esteffin Dec 14, 2023
a4de00c
Adjust default flags in regression/cbmc tests.
NlightNFotis Nov 23, 2023
9802d2b
Fixed THOROUGH check gcc_popcount2
esteffin Dec 14, 2023
f92140a
Revert "Add --no-standard-checks to regression/cbmc-shadow-memory run…
esteffin Dec 14, 2023
198c1d9
Adjust flags for cbmc-shadow-memory/ regression tests
NlightNFotis Nov 30, 2023
aa363cb
Revert "Add --no-standard-checks to regression/cbmc-incr-smt2 runner …
esteffin Dec 14, 2023
3409334
Fixed cbmc-incr-smt2 regressions
esteffin Dec 14, 2023
00c9b15
Revert "Add --no-standard-checks to regression/cbmc-with-incr runner …
esteffin Dec 14, 2023
bc89949
Revert "Add --no-standard-checks to regression/cbmc-primitives runner…
esteffin Dec 14, 2023
a074af7
Fixed cbmc-primitives regressions
esteffin Dec 14, 2023
87b192e
Revert "Add --no-standard-checks to regression/cbmc-library runner sc…
esteffin Dec 14, 2023
ca1e877
Fixed cbmc-library regressions
esteffin Dec 14, 2023
0f818ff
Revert "Add --no-standard-checks to regression/book-examples runner s…
esteffin Dec 14, 2023
8750d19
Fixed book-examples regressions
esteffin Dec 14, 2023
ff59861
Revert "Add --no-standard-checks to regression/cbmc-concurrency runne…
esteffin Dec 14, 2023
9acac6c
Fixed cbmc-concurrency regressions
esteffin Dec 14, 2023
24f710d
Revert "Add --no-standard-checks to regression/cbmc-cover runner scri…
esteffin Dec 14, 2023
3ce2e4f
Fixed cbmc-cover regressions
esteffin Dec 14, 2023
241f12c
Revert "Add --no-standard-checks to regression/cbmc-cpp runner scripts"
esteffin Dec 14, 2023
d8f5136
Revert "Add --no-standard-checks to regression/cbmc-incr runner scripts"
esteffin Dec 14, 2023
f2225f6
Revert "Add --no-standard-checks to regression/cbmc-incr-oneloop test…
esteffin Dec 14, 2023
9b0162f
Fixed cbmc-incr-oneloop timeouting tests
esteffin Dec 14, 2023
b7dca9a
Revert "Add --no-standard-checks to regression/cbmc-sequentialization…
esteffin Dec 14, 2023
a7b914f
Revert "Add --no-standard-checks to regression/goto-analyzer test run…
esteffin Dec 15, 2023
ce5f46d
Fixed goto-analyzer regressions
esteffin Dec 15, 2023
7c78ff9
Revert "Add --no-standard-checks to goto-analyzer-simplify runner scr…
esteffin Dec 15, 2023
a037958
Revert "Add --no-standard-checks to regression/goto-cc-cbmc test runn…
esteffin Dec 15, 2023
a6abc82
Revert "Add --no-standard-checks to regression/acceleration test runn…
esteffin Dec 15, 2023
cebb4d2
Fixed acceleration regressions
esteffin Dec 15, 2023
1084f76
Revert "Add --no-standard-checks to regression/goto-harness test runn…
esteffin Dec 15, 2023
956cc92
Added mechanism to pass argument to cbmc in goto-harness regression test
Dec 21, 2023
d3e9237
Fixed goto-harness failing test
Dec 21, 2023
df3f766
Revert "Add --no-standard-checks to ../regression/linking-goto-binari…
esteffin Dec 15, 2023
7f0dabf
Fixed linking-goto-binaries regressions
esteffin Dec 15, 2023
0027d18
Revert "Add --no-standard-checks to regression/validate-trace-xml-sch…
esteffin Dec 15, 2023
4e80066
Revert "Add --no-standard-checks to regresion/contracts-dfcc test run…
esteffin Dec 19, 2023
bb402b2
Removed --no-malloc-may-fail from test contracts-dfcc regression test…
Dec 20, 2023
073c29c
Fixed contracts-dfcc regressions
esteffin Dec 19, 2023
8c0f5de
Revert "Add --no-standard-checks to regression/contracts test runner …
esteffin Dec 19, 2023
0d4f56a
Fixed contracts regressions
esteffin Dec 19, 2023
27e312d
Removed extra arguments from goto-instrument regression and added mec…
esteffin Dec 19, 2023
dee9859
Fixed goto-instrument regressions
esteffin Dec 19, 2023
7a1501a
Revert "Add --no-standard-checks to regression/goto-synthesiser test …
esteffin Dec 19, 2023
7e527f9
Removed --no-malloc-may-fail from goto-synthesizer test runners
esteffin Dec 19, 2023
77705eb
Fixed goto-synthesizer failing regression test
esteffin Dec 19, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion regression/acceleration/accelerate.sh
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ is_windows=$4
shift 4

cfile=""
cbmcargs="--no-standard-checks"
cbmcargs=""

# create the temporary directory relative to the current directory, thus
# avoiding file names that start with a "/", which confuses goto-cl (Windows)
Expand Down
2 changes: 1 addition & 1 deletion regression/acceleration/array_safe1/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
main.c

--no-unwinding-assertions
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
2 changes: 1 addition & 1 deletion regression/acceleration/array_safe2/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
main.c

--no-unwinding-assertions
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
2 changes: 1 addition & 1 deletion regression/acceleration/array_safe3/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
main.c

--no-unwinding-assertions
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
2 changes: 1 addition & 1 deletion regression/acceleration/array_safe4/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
main.c

--no-unwinding-assertions
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
2 changes: 1 addition & 1 deletion regression/acceleration/const_safe1/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
main.c

--no-unwinding-assertions
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
2 changes: 1 addition & 1 deletion regression/acceleration/diamond_safe1/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
main.c

--no-unwinding-assertions
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
2 changes: 1 addition & 1 deletion regression/acceleration/diamond_safe2/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
main.c

--no-unwinding-assertions
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
2 changes: 1 addition & 1 deletion regression/acceleration/functions_safe1/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
main.c

--no-unwinding-assertions
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
2 changes: 1 addition & 1 deletion regression/acceleration/multivar_safe1/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
main.c

--no-unwinding-assertions
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
2 changes: 1 addition & 1 deletion regression/acceleration/nested_safe1/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
main.c

--no-unwinding-assertions
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
2 changes: 1 addition & 1 deletion regression/acceleration/overflow_safe1/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
main.c

--no-unwinding-assertions
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
2 changes: 1 addition & 1 deletion regression/acceleration/phases_safe1/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
main.c

--no-unwinding-assertions
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
2 changes: 1 addition & 1 deletion regression/acceleration/simple_safe1/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
main.c

--no-unwinding-assertions
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
2 changes: 1 addition & 1 deletion regression/acceleration/simple_safe2/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
main.c

--no-unwinding-assertions
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
2 changes: 1 addition & 1 deletion regression/acceleration/simple_safe3/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
main.c

--no-unwinding-assertions
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
2 changes: 1 addition & 1 deletion regression/acceleration/simple_safe4/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
main.c

--no-unwinding-assertions
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
2 changes: 1 addition & 1 deletion regression/acceleration/underapprox_safe1/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
main.c

--no-unwinding-assertions
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
2 changes: 1 addition & 1 deletion regression/acceleration/underapprox_safe2/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
main.c

--no-unwinding-assertions
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
2 changes: 1 addition & 1 deletion regression/acceleration/underapprox_unsafe1/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
main.c

--no-unwinding-assertions
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
2 changes: 1 addition & 1 deletion regression/acceleration/underapprox_unsafe2/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
main.c

--no-unwinding-assertions
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
8 changes: 4 additions & 4 deletions regression/book-examples/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -13,27 +13,27 @@ else()
endif()

add_test_pl_tests(
"$<TARGET_FILE:cbmc> --no-standard-checks --validate-goto-model --validate-ssa-equation" -X smt-backend ${gcc_only} ${exclude_win_broken_tests}
"$<TARGET_FILE:cbmc> --validate-goto-model --validate-ssa-equation" -X smt-backend ${gcc_only} ${exclude_win_broken_tests}
)

add_test_pl_profile(
"book-examples-paths-lifo"
"$<TARGET_FILE:cbmc> --no-standard-checks --paths lifo"
"$<TARGET_FILE:cbmc> --paths lifo"
"-C;-X;thorough-paths;-X;smt-backend;-X;paths-lifo-expected-failure;${gcc_only_string}-s;paths-lifo;${exclude_win_broken_tests_string}"
"CORE"
)

add_test_pl_profile(
"book-examples-cprover-smt2"
"$<TARGET_FILE:cbmc> --no-standard-checks --cprover-smt2"
"$<TARGET_FILE:cbmc> --cprover-smt2"
"-C;-X;broken-smt-backend;-X;thorough-smt-backend;-X;broken-cprover-smt-backend;-X;thorough-cprover-smt-backend;${gcc_only_string}-s;cprover-smt2;${exclude_win_broken_tests_string}"
"CORE"
)

# If `-X` (exclude flag) is passed, test.pl will exclude the tests matching the label following it.
add_test_pl_profile(
"book-examples-new-smt-backend"
"$<TARGET_FILE:cbmc> --no-standard-checks --incremental-smt2-solver 'z3 --smt2 -in'"
"$<TARGET_FILE:cbmc> --incremental-smt2-solver 'z3 --smt2 -in'"
"${gcc_only_string}-X;no-new-smt;-s;new-smt-backend"
"CORE"
)
Expand Down
10 changes: 5 additions & 5 deletions regression/book-examples/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -10,27 +10,27 @@ GCC_ONLY =
endif

test:
@../test.pl -e -p -c "../../../src/cbmc/cbmc --no-standard-checks --validate-goto-model --validate-ssa-equation" -X smt-backend $(GCC_ONLY)
@../test.pl -e -p -c "../../../src/cbmc/cbmc --validate-goto-model --validate-ssa-equation" -X smt-backend $(GCC_ONLY)

test-cprover-smt2:
@../test.pl -e -p -c "../../../src/cbmc/cbmc --no-standard-checks --cprover-smt2" \
@../test.pl -e -p -c "../../../src/cbmc/cbmc --cprover-smt2" \
-X broken-smt-backend -X thorough-smt-backend \
-X broken-cprover-smt-backend -X thorough-cprover-smt-backend \
-s cprover-smt2 $(GCC_ONLY)

test-z3:
@../test.pl -e -p -c "../../../src/cbmc/cbmc --no-standard-checks --z3" \
@../test.pl -e -p -c "../../../src/cbmc/cbmc --z3" \
-X broken-smt-backend -X thorough-smt-backend \
-X broken-z3-smt-backend -X thorough-z3-smt-backend \
-s z3 $(GCC_ONLY)

test-paths-lifo:
@../test.pl -e -p -c "../../../src/cbmc/cbmc --no-standard-checks --paths lifo" \
@../test.pl -e -p -c "../../../src/cbmc/cbmc --paths lifo" \
-X thorough-paths -X smt-backend -X paths-lifo-expected-failure \
-s paths-lifo $(GCC_ONLY)

test-new-smt-backend:
@../test.pl -e -p -c "../../../src/cbmc/cbmc --no-standard-checks --incremental-smt2-solver 'z3 --smt2 -in'" \
@../test.pl -e -p -c "../../../src/cbmc/cbmc --incremental-smt2-solver 'z3 --smt2 -in'" \
-X no-new-smt \
-s new-smt-backend $(GCC_ONLY)

Expand Down
2 changes: 1 addition & 1 deletion regression/book-examples/abs/C1.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
abs.c
--function abs
--no-signed-overflow-check --function abs
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
2 changes: 1 addition & 1 deletion regression/book-examples/lock/unwind1.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
lock.c
--unwind 1
--unwind 1 --no-unwinding-assertions
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
4 changes: 2 additions & 2 deletions regression/cbmc-concurrency/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
if((NOT WIN32) AND (NOT APPLE) AND (NOT (CMAKE_SYSTEM_NAME STREQUAL "FreeBSD")))
add_test_pl_tests(
"$<TARGET_FILE:cbmc> --no-standard-checks --validate-goto-model --validate-ssa-equation"
"$<TARGET_FILE:cbmc> --validate-goto-model --validate-ssa-equation"
)
else()
add_test_pl_profile(
"cbmc-concurrency"
"$<TARGET_FILE:cbmc> --no-standard-checks --validate-goto-model --validate-ssa-equation"
"$<TARGET_FILE:cbmc> --validate-goto-model --validate-ssa-equation"
"-C;-X;pthread"
"CORE"
)
Expand Down
4 changes: 2 additions & 2 deletions regression/cbmc-concurrency/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,10 @@ ifeq ($(filter-out OSX MSVC FreeBSD,$(BUILD_ENV_)),)
endif

test:
@../test.pl -e -p -c "../../../src/cbmc/cbmc --no-standard-checks --validate-goto-model --validate-ssa-equation" $(no_pthread)
@../test.pl -e -p -c "../../../src/cbmc/cbmc --validate-goto-model --validate-ssa-equation" $(no_pthread)

tests.log: ../test.pl
@../test.pl -e -p -c "../../../src/cbmc/cbmc --no-standard-checks --validate-goto-model --validate-ssa-equation" $(no_pthread)
@../test.pl -e -p -c "../../../src/cbmc/cbmc --validate-goto-model --validate-ssa-equation" $(no_pthread)

clean:
find . -name '*.out' -execdir $(RM) '{}' \;
Expand Down
2 changes: 1 addition & 1 deletion regression/cbmc-concurrency/atomic_section_sc3/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
main.c

--no-standard-checks
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
Expand Down
2 changes: 1 addition & 1 deletion regression/cbmc-concurrency/deadlock1/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE pthread
main.c

--no-standard-checks
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
Expand Down
2 changes: 1 addition & 1 deletion regression/cbmc-concurrency/deadlock2/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE pthread
main.c

--no-standard-checks
^EXIT=10$
^SIGNAL=0$
^VERIFICATION FAILED$
Expand Down
2 changes: 1 addition & 1 deletion regression/cbmc-concurrency/dirty_local1/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
main.c

--no-standard-checks
^EXIT=10$
^SIGNAL=0$
^VERIFICATION FAILED$
Expand Down
2 changes: 1 addition & 1 deletion regression/cbmc-concurrency/dirty_local2/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
main.c

--no-standard-checks
^EXIT=10$
^SIGNAL=0$
^VERIFICATION FAILED$
Expand Down
2 changes: 1 addition & 1 deletion regression/cbmc-concurrency/dirty_local3/test-local.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
main.c
-Dlocals_bug
--no-standard-checks -Dlocals_bug
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
Expand Down
2 changes: 1 addition & 1 deletion regression/cbmc-concurrency/dirty_local3/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
main.c

--no-standard-checks
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
Expand Down
2 changes: 1 addition & 1 deletion regression/cbmc-concurrency/invalid_object1/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
main.c

--no-standard-checks
^EXIT=10$
^SIGNAL=0$
^VERIFICATION FAILED$
Expand Down
2 changes: 1 addition & 1 deletion regression/cbmc-concurrency/malloc1/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
main.c

--no-malloc-may-fail
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
Expand Down
2 changes: 1 addition & 1 deletion regression/cbmc-concurrency/malloc2/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE
main.c

--no-malloc-may-fail
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
Expand Down
2 changes: 1 addition & 1 deletion regression/cbmc-concurrency/memory_barrier1/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE pthread
main.c
--unwind 1 --no-unwinding-assertions
--no-standard-checks --unwind 1 --no-unwinding-assertions
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
Expand Down
2 changes: 1 addition & 1 deletion regression/cbmc-concurrency/norace_array1/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE pthread
main.c

--no-standard-checks
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
Expand Down
2 changes: 1 addition & 1 deletion regression/cbmc-concurrency/norace_array2/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE pthread
main.c

--no-standard-checks
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
Expand Down
2 changes: 1 addition & 1 deletion regression/cbmc-concurrency/norace_scalar1/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE pthread
main.c

--no-standard-checks
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
Expand Down
2 changes: 1 addition & 1 deletion regression/cbmc-concurrency/norace_struct1/test.desc
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
CORE pthread
main.c

--no-standard-checks
^EXIT=0$
^SIGNAL=0$
^VERIFICATION SUCCESSFUL$
Expand Down
Loading
Loading