Skip to content

Commit a9ca7d6

Browse files
MatthewDaggittMatthewDaggitt
authored
Collapse warnings in vector properties into single warning (#871)
* Collapse warnings in vector properties into single warning * Fix GHC 8.X compatability
1 parent c971f24 commit a9ca7d6

File tree

14 files changed

+125
-81
lines changed

14 files changed

+125
-81
lines changed

‎ChangeLog.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@
88

99
* Improved the ordering of constraints in generated query files.
1010

11+
* When multiple similar warnings are thrown at different indices of the same property vector (i.e. properties of type `Vector Bool n`), they are now collapsed into a single warning.
12+
1113
## Version 0.15
1214

1315
* Added functions `min` and `max` over rationals.

‎vehicle/src/Vehicle/Compile/Print/Warning.hs

Lines changed: 57 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,18 @@
11
{-# OPTIONS_GHC -Wno-orphans #-}
2+
{-# OPTIONS_GHC -Wno-unrecognised-pragmas #-}
3+
4+
{-# HLINT ignore "Use <$>" #-}
25

36
module Vehicle.Compile.Print.Warning where
47

8+
import Data.Bifunctor (Bifunctor (..))
59
import Data.List (sortOn)
610
import Data.List.NonEmpty (NonEmpty, sort)
7-
import Data.Set qualified as Set
11+
import Data.List.NonEmpty qualified as NonEmpty
12+
import Data.Map qualified as Map
13+
import Data.Tuple (swap)
814
import Vehicle.Data.Assertion (UnderConstrainedVariableStatus, prettyUnderConstrainedVariable)
15+
import Vehicle.Data.Tensor (TensorIndices)
916
import Vehicle.Prelude
1017
import Vehicle.Prelude.Warning
1118
import Vehicle.Verify.Core
@@ -25,19 +32,14 @@ instance Pretty SummarisedCompileWarning where
2532
<+> quotePretty status
2633
<+> "without needing to call the verifier. This usually indicates a fault with either the"
2734
<+> "specification or any external datasets used."
28-
UnderSpecifiedProblemSpaceVariablesSummary propertyAddress unsolvedVars -> do
29-
let varsDoc = concatWith (\u v -> u <> "," <+> v) (fmap quotePretty (Set.toList unsolvedVars))
30-
let pluralisedVarsDoc
31-
| length unsolvedVars == 1 = "variable" <+> varsDoc <+> "is"
32-
| otherwise = "variables" <+> varsDoc <+> "are"
33-
35+
UnderSpecifiedProblemSpaceVariablesSummary _ propertyName unsolvedVars -> do
3436
"In property"
35-
<+> quotePretty propertyAddress
36-
<+> "the quantified"
37-
<+> pluralisedVarsDoc
38-
<+> "not always directly related to the input or output of a network."
39-
<+> "This is frequently indicative of a bug in the specification."
40-
UnsoundStrictOrderConversionsSummary queryFormatID propertyAddress _ ->
37+
<+> quotePretty propertyName
38+
<+> "there are quantified variables that not always directly related to the input or output of a network."
39+
<+> "This is frequently indicative of a bug in the specification. These variables are:"
40+
<> line
41+
<> prettyObjectsAndTensorIndicesList prettyVariables propertyName unsolvedVars
42+
UnsoundStrictOrderConversionsSummary queryFormatID _ propertyAddress _ ->
4143
"In property"
4244
<+> quotePretty propertyAddress
4345
<> ", at least one of the generated queries were found"
@@ -56,18 +58,18 @@ instance Pretty SummarisedCompileWarning where
5658
<> line
5759
<> line
5860
<> "See https://github.com/vehicle-lang/vehicle/issues/74 for further details."
59-
AllConstantNetworkInputVariablesSummary _ propertyAddress queryIDs ->
60-
"In property"
61-
<+> quotePretty propertyAddress
62-
<> ", in"
63-
<+> prettyQueryIDs queryIDs
64-
<+> "all network inputs were fixed to be constants."
65-
<+> "Unfortunately there is a known bug in Marabou that it sometimes erroneously returns"
66-
<+> "'unsat' for these type of queries."
61+
AllConstantNetworkInputVariablesSummary _ _ propertyName queryIDs ->
62+
"In some of the queries generated by property"
63+
<+> quotePretty propertyName
64+
<+> "all network inputs were fixed to be constants."
65+
<+> "Unfortunately there is a known bug in Marabou that it sometimes erroneously returns"
66+
<+> "'unsat' for these type of queries. The queries are:"
67+
<> line
68+
<> prettyObjectsAndTensorIndicesList prettyQueryIDs propertyName queryIDs
6769
<> line
6870
<> line
6971
<> "See https://github.com/NeuralNetworkVerification/Marabou/issues/670 for details."
70-
UnboundedNetworkInputVariablesSummary queryFormat propertyAddress varsByQueryID ->
72+
UnboundedNetworkInputVariablesSummary queryFormat _ propertyAddress varsByQueryID ->
7173
"In property"
7274
<+> quotePretty propertyAddress
7375
<> ", the following network input variables do not always have both"
@@ -85,7 +87,36 @@ instance Pretty SummarisedCompileWarning where
8587
<> line
8688
<> indent 2 (vsep $ dotDotList 5 $ fmap prettyUnderConstrainedVariable vars)
8789

88-
prettyQueryIDs :: NonEmpty QueryID -> Doc a
89-
prettyQueryIDs ids =
90-
(if length ids == 1 then "query" else "queries")
91-
<+> prettyNonEmptyList (fmap pretty (sort ids))
90+
prettyObjects :: (Ord a, Pretty a) => Bool -> Doc b -> Doc b -> NonEmpty a -> Doc b
91+
prettyObjects quote single plural ids =
92+
(if length ids == 1 then single else plural)
93+
<+> prettyNonEmptyList (fmap (if quote then quotePretty else pretty) (sort ids))
94+
95+
prettyQueryIDs :: NonEmpty QueryID -> Doc b
96+
prettyQueryIDs = prettyObjects False "query" "queries"
97+
98+
prettyVariables :: NonEmpty Name -> Doc a
99+
prettyVariables = prettyObjects True "variable" "variables"
100+
101+
prettyTensorIndices :: PropertyName -> NonEmpty TensorIndices -> Maybe (Doc b)
102+
prettyTensorIndices name indices
103+
| indices == [[]] = Nothing
104+
| otherwise = do
105+
let addresses = fmap (quotePretty . PropertyAddress (-1) name) indices
106+
Just $ prettyNonEmptyList addresses
107+
108+
prettyObjectsAndTensorIndices :: (NonEmpty a -> Doc b) -> PropertyName -> (NonEmpty a, NonEmpty TensorIndices) -> Doc b
109+
prettyObjectsAndTensorIndices prettyObject name (ids, indices) = do
110+
let indicesDoc = prettyTensorIndices name indices
111+
prettyObject ids <> maybe "" (" in" <+>) indicesDoc
112+
113+
prettyObjectsAndTensorIndicesList :: (Ord a) => (NonEmpty a -> Doc b) -> PropertyName -> NonEmpty (a, TensorIndices) -> Doc b
114+
prettyObjectsAndTensorIndicesList prettyObject name objectsAndIndices = do
115+
-- Singleton.nonEmpty doesn't exist until base 4.15
116+
let nonEmptySingleton x = [x]
117+
-- First map by indices that share the same objects
118+
let indicesMap = Map.fromListWith (<>) (NonEmpty.toList $ fmap (second nonEmptySingleton) objectsAndIndices)
119+
-- Then find the objects that share the same queries
120+
let objectsMap = Map.fromListWith (<>) $ fmap (second nonEmptySingleton . swap) (Map.toList indicesMap)
121+
let finalGroups = fmap swap $ Map.toList objectsMap
122+
indent 2 (vsep (fmap (\x -> "-" <+> prettyObjectsAndTensorIndices prettyObject name x) finalGroups))

‎vehicle/src/Vehicle/Prelude/Warning.hs

Lines changed: 37 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -6,17 +6,17 @@ module Vehicle.Prelude.Warning
66
where
77

88
import Data.List (sortBy)
9-
import Data.List.NonEmpty (NonEmpty, sort)
9+
import Data.List.NonEmpty (NonEmpty)
1010
import Data.Map (Map)
1111
import Data.Map qualified as Map (insertWith, singleton, toList, unionWith)
1212
import Data.Set (Set)
13-
import Data.Set qualified as Set (singleton)
1413
import Data.Tuple (swap)
1514
import Vehicle.Compile.Context.Bound.Core
1615
import Vehicle.Data.Assertion
1716
import Vehicle.Data.Builtin.Core
1817
import Vehicle.Data.Builtin.Tensor
1918
import Vehicle.Data.Code.Value
19+
import Vehicle.Data.Tensor (TensorIndices)
2020
import Vehicle.Prelude (Name)
2121
import Vehicle.Resource (ExternalResource)
2222
import Vehicle.Verify.Core
@@ -36,10 +36,10 @@ data CompileWarning
3636
data SummarisedCompileWarning
3737
= UnusedResourcesSummary ExternalResource (Set Name)
3838
| TrivialPropertySummary PropertyAddress Bool
39-
| UnderSpecifiedProblemSpaceVariablesSummary PropertyAddress (Set Name)
40-
| UnsoundStrictOrderConversionsSummary QueryFormatID PropertyAddress Int
41-
| AllConstantNetworkInputVariablesSummary QueryFormatID PropertyAddress (NonEmpty QueryID)
42-
| UnboundedNetworkInputVariablesSummary QueryFormatID PropertyAddress [(NonEmpty QueryID, [(Name, UnderConstrainedVariableStatus)])]
39+
| UnderSpecifiedProblemSpaceVariablesSummary PropertyID PropertyName (NonEmpty (Name, TensorIndices))
40+
| UnsoundStrictOrderConversionsSummary QueryFormatID PropertyID PropertyName Int
41+
| AllConstantNetworkInputVariablesSummary QueryFormatID PropertyID PropertyName (NonEmpty (QueryID, TensorIndices))
42+
| UnboundedNetworkInputVariablesSummary QueryFormatID PropertyID PropertyName [(NonEmpty QueryID, [(Name, UnderConstrainedVariableStatus)])]
4343

4444
--------------------------------------------------------------------------------
4545
-- Combinable compile warnings
@@ -48,10 +48,10 @@ type UnderConstrainedSignature = [(Name, UnderConstrainedVariableStatus)]
4848

4949
data CombiningState = CombiningState
5050
{ uniqueWarnings :: [SummarisedCompileWarning],
51-
underSpecifiedProblemSpaceVars :: Map PropertyAddress (Set Name),
52-
unsoundStrictnessConversions :: Map (QueryFormatID, PropertyAddress) Int,
53-
allConstantNetworkInputVars :: Map (QueryFormatID, PropertyAddress) (NonEmpty QueryID),
54-
unboundedNetworkInputs :: Map (QueryFormatID, PropertyAddress) (Map UnderConstrainedSignature (NonEmpty QueryID)),
51+
underSpecifiedProblemSpaceVars :: Map (PropertyID, PropertyName) (NonEmpty (Name, TensorIndices)),
52+
unsoundStrictnessConversions :: Map (QueryFormatID, PropertyID, PropertyName) Int,
53+
allConstantNetworkInputVars :: Map (QueryFormatID, PropertyID, PropertyName) (NonEmpty (QueryID, TensorIndices)),
54+
unboundedNetworkInputs :: Map (QueryFormatID, PropertyID, PropertyName) (Map UnderConstrainedSignature (NonEmpty QueryID)),
5555
inefficientTensorCode :: Map Name [(Builtin, NamedBoundCtx, Value TensorBuiltin)]
5656
}
5757

@@ -70,24 +70,25 @@ addWarningToState CombiningState {..} = \case
7070
{ uniqueWarnings = TrivialPropertySummary r names : uniqueWarnings,
7171
..
7272
}
73-
UnderSpecifiedProblemSpaceVar property var ->
73+
UnderSpecifiedProblemSpaceVar PropertyAddress {..} var ->
7474
CombiningState
75-
{ underSpecifiedProblemSpaceVars = Map.insertWith (<>) property (Set.singleton var) underSpecifiedProblemSpaceVars,
75+
{ underSpecifiedProblemSpaceVars = Map.insertWith (<>) (propertyID, propertyName) [(var, propertyIndices)] underSpecifiedProblemSpaceVars,
7676
..
7777
}
78-
UnsoundStrictOrderConversion queryFormat (propertyAddress, _queryID) ->
78+
UnsoundStrictOrderConversion queryFormat (PropertyAddress {..}, _queryID) ->
7979
CombiningState
80-
{ unsoundStrictnessConversions = Map.insertWith (+) (queryFormat, propertyAddress) 1 unsoundStrictnessConversions,
80+
{ unsoundStrictnessConversions = Map.insertWith (+) (queryFormat, propertyID, propertyName) 1 unsoundStrictnessConversions,
8181
..
8282
}
83-
AllConstantNetworkInputVars queryFormat (propertyAddress, queryID) ->
83+
AllConstantNetworkInputVars queryFormat (PropertyAddress {..}, queryID) ->
8484
CombiningState
85-
{ allConstantNetworkInputVars = Map.insertWith (<>) (queryFormat, propertyAddress) [queryID] allConstantNetworkInputVars,
85+
{ allConstantNetworkInputVars =
86+
Map.insertWith (<>) (queryFormat, propertyID, propertyName) [(queryID, propertyIndices)] allConstantNetworkInputVars,
8687
..
8788
}
88-
UnboundedNetworkInputVariables queryFormat (propertyAddress, queryID) vars ->
89+
UnboundedNetworkInputVariables queryFormat (PropertyAddress {..}, queryID) vars ->
8990
CombiningState
90-
{ unboundedNetworkInputs = Map.insertWith (Map.unionWith (<>)) (queryFormat, propertyAddress) (Map.singleton vars [queryID]) unboundedNetworkInputs,
91+
{ unboundedNetworkInputs = Map.insertWith (Map.unionWith (<>)) (queryFormat, propertyID, propertyName) (Map.singleton vars [queryID]) unboundedNetworkInputs,
9192
..
9293
}
9394

@@ -97,36 +98,37 @@ groupWarnings warnings = stateToWarnings $ foldl addWarningToState emptyState wa
9798
stateToWarnings :: CombiningState -> [SummarisedCompileWarning]
9899
stateToWarnings CombiningState {..} =
99100
sortBy compareWarning $
100-
do
101-
uniqueWarnings
101+
uniqueWarnings
102102
<> fmap combineUnderSpecifiedProblemSpaceVars (Map.toList underSpecifiedProblemSpaceVars)
103103
<> fmap combineUnsoundStrictnessConversions (Map.toList unsoundStrictnessConversions)
104104
<> fmap combineAllConstantNetworkInputVars (Map.toList allConstantNetworkInputVars)
105105
<> fmap combineUnboundedNetworkInputVars (Map.toList unboundedNetworkInputs)
106106

107-
combineUnderSpecifiedProblemSpaceVars :: (PropertyAddress, Set Name) -> SummarisedCompileWarning
108-
combineUnderSpecifiedProblemSpaceVars (property, vars) = UnderSpecifiedProblemSpaceVariablesSummary property vars
107+
combineUnderSpecifiedProblemSpaceVars :: ((PropertyID, PropertyName), NonEmpty (Name, TensorIndices)) -> SummarisedCompileWarning
108+
combineUnderSpecifiedProblemSpaceVars ((propertyID, property), vars) = UnderSpecifiedProblemSpaceVariablesSummary propertyID property vars
109109

110-
combineUnsoundStrictnessConversions :: ((QueryFormatID, PropertyAddress), Int) -> SummarisedCompileWarning
111-
combineUnsoundStrictnessConversions ((queryFormatID, property), number) = UnsoundStrictOrderConversionsSummary queryFormatID property number
110+
combineUnsoundStrictnessConversions :: ((QueryFormatID, PropertyID, PropertyName), Int) -> SummarisedCompileWarning
111+
combineUnsoundStrictnessConversions ((queryFormatID, propertyID, property), number) =
112+
UnsoundStrictOrderConversionsSummary queryFormatID propertyID property number
112113

113-
combineAllConstantNetworkInputVars :: ((QueryFormatID, PropertyAddress), NonEmpty QueryID) -> SummarisedCompileWarning
114-
combineAllConstantNetworkInputVars ((queryFormatID, property), queries) = AllConstantNetworkInputVariablesSummary queryFormatID property (sort queries)
114+
combineAllConstantNetworkInputVars :: ((QueryFormatID, PropertyID, PropertyName), NonEmpty (QueryID, TensorIndices)) -> SummarisedCompileWarning
115+
combineAllConstantNetworkInputVars ((queryFormatID, propertyID, property), queries) =
116+
AllConstantNetworkInputVariablesSummary queryFormatID propertyID property queries
115117

116-
combineUnboundedNetworkInputVars :: ((QueryFormatID, PropertyAddress), Map UnderConstrainedSignature (NonEmpty QueryID)) -> SummarisedCompileWarning
117-
combineUnboundedNetworkInputVars ((queryFormatID, property), constraintsBySignature) = do
118+
combineUnboundedNetworkInputVars :: ((QueryFormatID, PropertyID, PropertyName), Map UnderConstrainedSignature (NonEmpty QueryID)) -> SummarisedCompileWarning
119+
combineUnboundedNetworkInputVars ((queryFormatID, propertyID, property), constraintsBySignature) = do
118120
let result = swap <$> Map.toList constraintsBySignature
119-
UnboundedNetworkInputVariablesSummary queryFormatID property result
121+
UnboundedNetworkInputVariablesSummary queryFormatID propertyID property result
120122

121123
compareWarning :: SummarisedCompileWarning -> SummarisedCompileWarning -> Ordering
122124
compareWarning w1 w2 = compare (warningPropertyId w1) (warningPropertyId w2)
123125
where
124126
warningPropertyId :: SummarisedCompileWarning -> Maybe PropertyID
125127
warningPropertyId w =
126-
propertyID <$> case w of
128+
case w of
127129
UnusedResourcesSummary {} -> Nothing
128-
TrivialPropertySummary address _ -> Just address
129-
UnderSpecifiedProblemSpaceVariablesSummary address _ -> Just address
130-
UnsoundStrictOrderConversionsSummary _ address _ -> Just address
131-
AllConstantNetworkInputVariablesSummary _ address _ -> Just address
132-
UnboundedNetworkInputVariablesSummary _ address _ -> Just address
130+
TrivialPropertySummary address _ -> Just $ propertyID address
131+
UnderSpecifiedProblemSpaceVariablesSummary propertyID _ _ -> Just propertyID
132+
UnsoundStrictOrderConversionsSummary _ propertyID _ _ -> Just propertyID
133+
AllConstantNetworkInputVariablesSummary _ propertyID _ _ -> Just propertyID
134+
UnboundedNetworkInputVariablesSummary _ propertyID _ _ -> Just propertyID

‎vehicle/tests/golden/compile/issue712/MarabouQueries.err.golden

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11

2-
Warning: In property 'isMalicious', in query 1 all network inputs were fixed to be constants. Unfortunately there is a known bug in Marabou that it sometimes erroneously returns 'unsat' for these type of queries.
2+
Warning: In some of the queries generated by property 'isMalicious' all network inputs were fixed to be constants. Unfortunately there is a known bug in Marabou that it sometimes erroneously returns 'unsat' for these type of queries. The queries are:
3+
- query 1
34

45
See https://github.com/NeuralNetworkVerification/Marabou/issues/670 for details.
56


‎vehicle/tests/golden/compile/issue765/MarabouQueries.err.golden

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11

2-
Warning: In property 'p', in query 1 all network inputs were fixed to be constants. Unfortunately there is a known bug in Marabou that it sometimes erroneously returns 'unsat' for these type of queries.
2+
Warning: In some of the queries generated by property 'p' all network inputs were fixed to be constants. Unfortunately there is a known bug in Marabou that it sometimes erroneously returns 'unsat' for these type of queries. The queries are:
3+
- query 1
34

45
See https://github.com/NeuralNetworkVerification/Marabou/issues/670 for details.
56


‎vehicle/tests/golden/compile/simple-arithmetic/Marabou.err.golden

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11

2-
Warning: In property 'property', in query 1 all network inputs were fixed to be constants. Unfortunately there is a known bug in Marabou that it sometimes erroneously returns 'unsat' for these type of queries.
2+
Warning: In some of the queries generated by property 'property' all network inputs were fixed to be constants. Unfortunately there is a known bug in Marabou that it sometimes erroneously returns 'unsat' for these type of queries. The queries are:
3+
- query 1
34

45
See https://github.com/NeuralNetworkVerification/Marabou/issues/670 for details.
56


‎vehicle/tests/golden/compile/simple-constantNetworkInput/Marabou.err.golden

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11

2-
Warning: In property 'p', in query 1 all network inputs were fixed to be constants. Unfortunately there is a known bug in Marabou that it sometimes erroneously returns 'unsat' for these type of queries.
2+
Warning: In some of the queries generated by property 'p' all network inputs were fixed to be constants. Unfortunately there is a known bug in Marabou that it sometimes erroneously returns 'unsat' for these type of queries. The queries are:
3+
- query 1
34

45
See https://github.com/NeuralNetworkVerification/Marabou/issues/670 for details.
56


‎vehicle/tests/golden/compile/simple-foreach/Marabou.err.golden

Lines changed: 2 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,6 @@
11

2-
Warning: In property 'index!0', in query 1 all network inputs were fixed to be constants. Unfortunately there is a known bug in Marabou that it sometimes erroneously returns 'unsat' for these type of queries.
3-
4-
See https://github.com/NeuralNetworkVerification/Marabou/issues/670 for details.
5-
6-
7-
Warning: In property 'index!1', in query 1 all network inputs were fixed to be constants. Unfortunately there is a known bug in Marabou that it sometimes erroneously returns 'unsat' for these type of queries.
2+
Warning: In some of the queries generated by property 'index' all network inputs were fixed to be constants. Unfortunately there is a known bug in Marabou that it sometimes erroneously returns 'unsat' for these type of queries. The queries are:
3+
- query 1 in 'index!0' and 'index!1'
84

95
See https://github.com/NeuralNetworkVerification/Marabou/issues/670 for details.
106

0 commit comments

Comments
 (0)