Yosys SV Support Misrepresented Due to $display

[sifferman]

Hello, and thank you for assembling this very useful record of SV support in tools! It has been super interesting to follow.

Problem

There are several tests that Yosys is incorrectly failing due to using $display. Yosys does not support the $display statement for non-constant values:

ERROR: Failed to evaluate system task `$display' with non-constant argument.

Many more tests would pass if their values were verified by other means than $display. To list a few:

• tests/chapter-5/5.6.4--compiler-directives-preprocessor-macro_1.sv
• tests/chapter-7/arrays/packed/equality.sv
• tests/chapter-7/arrays/packed/onebit.sv
• tests/chapter-7/memories/read-write.sv
• tests/chapter-7/structures/packed/basic.sv
• tests/chapter-7/structures/packed/signed.sv
• tests/chapter-7/structures/packed/unsigned.sv
• tests/chapter-7/unions/packed/basic.sv
• tests/chapter-10/10.4.1--blocking-assignment.sv

Many more tests correctly fail because the feature is unsupported, but will eventually incorrectly fail because they use $display. To list a few:

• tests/chapter-7/arrays/multidimensional/copy.sv
• tests/chapter-7/arrays/multidimensional/subarrays.sv

Possible Solutions

To fix this, a custom assert macro/function should be added that would operate differently depending on the tool choice. Here are a couple possibilities for what the custom assert macro/function should do:

output ERROR Port

Possibly all the tests mentioned could have an output ERROR port added to the module definition, and the assert function/macro could just run ERROR |= !(<assertion>); for Yosys runs. Then, eval could be run to check the value of ERROR.

External Script

Another possibility is that the custom assert macro/function does nothing when run by Yosys. Instead, an external script will parse all the asserts and run eval on each one.

(Note that this will not work for the tests onebit.sv and read-write.sv because there are modifications to values after they have an assertion)

[tgorochowik]

Hi, thank you for the report.

Let me start by saying that originally all the tests that use display with asserts inside were meant for simulator tests. Then we added "elaboration-only" class which made it possible to run those in other tools (like yosys). So we could also just disable those tests for yosys, the things they cover (besides the asserts meant for simulation) should already be covered by other test cases.

That being said, I'm not opposed to adjusting those tests to make then usable for tools that only run elaboration. I'm not sure if having an error output port makes sense for yosys? would we be able to read the calculated value there?

One additional idea - maybe we could wrap those display/assert calls with some ifdefs (e.g. ifdef SIMULATION) and define this flag for all the simulators we test.

Feel free to propose a PR with a fix or a workaround for the problem.

[sifferman]

I'm not sure if having an error output port makes sense for yosys? would we be able to read the calculated value there?

Yes! Yosys' eval command calculates the values of all output ports of a specified module. If we add an ERROR port, then Yosys can calculate the value

I just made an example repo that demonstrates this: sifferman/eval-example. I adjusted the tests mentioned above, then run synthesis and eval -set-undef top.

we could also just disable those tests for yosys, the things they cover (besides the asserts meant for simulation) should already be covered by other test cases.

I guess I am also concerned about false positives when the synthesis is incorrect but Yosys doesn't throw an error when parsing. Using eval allows for extra confidence.

[sifferman]

I would be happy to make these changes if you think it sounds good

[sifferman]

I spent some time sorting through the tests to determine which should be enabled/disabled for Yosys.

These should be modified for Yosys:

• chapter-11/11.10--string_bit_array-sim.sv
• chapter-12/12.6.1--casex_pattern.sv
• chapter-12/12.6.1--casez_pattern.sv
• chapter-12/12.6.1--case_pattern.sv
• chapter-12/12.6.2--if_pattern.sv
• chapter-12/12.7.1--for.sv
• chapter-12/12.7.2--repeat.sv
• chapter-12/12.7.3--foreach.sv
• chapter-12/12.8--break.sv
• chapter-12/12.8--continue.sv
• chapter-12/12.8--return.sv
• chapter-12/12.8--return_val.sv
• chapter-13/13.4--function.sv
• chapter-13/13.4.1--function-return-assignment.sv
• chapter-13/13.4.1--function-return.sv
• chapter-13/13.4.1--function-void-return.sv
• chapter-13/13.4.2--function-automatic.sv
• chapter-13/13.4.2--function-recursive.sv
• chapter-13/13.4.2--function-static.sv
• chapter-13/13.4.3--const-function.sv
• chapter-20/20.15--random.sv
• chapter-20/20.5--itor.sv
• chapter-20/20.5--real-bits-conv.sv
• chapter-20/20.5--rtoi.sv
• chapter-20/20.5--shortreal-bits-conv.sv
• chapter-20/20.6--bits.sv
• chapter-20/20.6--bits_type.sv
• chapter-20/20.7--array-queries-multi-dim.sv
• chapter-20/20.7--array-queries.sv
• chapter-20/20.8--atan2.sv
• chapter-20/20.8--ceil.sv
• chapter-20/20.8--clog2.sv
• chapter-20/20.8--exp.sv
• chapter-20/20.8--floor.sv
• chapter-20/20.8--hypot.sv
• chapter-20/20.8--ln.sv
• chapter-20/20.8--log10.sv
• chapter-20/20.8--pow.sv
• chapter-20/20.8--sqrt.sv
• chapter-20/20.9--countbits.sv
• chapter-20/20.9--onehot.sv
• chapter-20/20.9--onehot0.sv
• chapter-21/21.2--display-boh.sv
• chapter-21/21.2--display.sv
• chapter-21/21.3--feof.sv
• chapter-21/21.3--ferror.sv
• chapter-21/21.3--fpos.sv
• chapter-21/21.3--sscanf.sv
• chapter-21/21.3--ungetc.sv
• chapter-22/22.4--check_included_definitions.sv
• chapter-22/22.5.1--define-expansion_1.sv
• chapter-22/22.5.1--define-expansion_10.sv
• chapter-22/22.5.1--define-expansion_11.sv
• chapter-22/22.5.1--define-expansion_12.sv
• chapter-22/22.5.1--define-expansion_13.sv
• chapter-22/22.5.1--define-expansion_14.sv
• chapter-22/22.5.1--define-expansion_15.sv
• chapter-22/22.5.1--define-expansion_16.sv
• chapter-22/22.5.1--define-expansion_17.sv
• chapter-22/22.5.1--define-expansion_18.sv
• chapter-22/22.5.1--define-expansion_2.sv
• chapter-22/22.5.1--define-expansion_21.sv
• chapter-22/22.5.1--define-expansion_24.sv
• chapter-22/22.5.1--define-expansion_25.sv
• chapter-22/22.5.1--define-expansion_26.sv
• chapter-22/22.5.1--define-expansion_3.sv
• chapter-22/22.5.1--define-expansion_4.sv
• chapter-22/22.5.1--define-expansion_5.sv
• chapter-22/22.5.1--define-expansion_6.sv
• chapter-22/22.5.1--define-expansion_7.sv
• chapter-22/22.5.1--define-expansion_8.sv
• chapter-22/22.5.1--define-expansion_9.sv
• chapter-22/22.5.1--define_and_resetall.sv
• chapter-22/22.6--ifdef-chained-nested.sv
• chapter-22/22.6--ifdef-nested.sv
• chapter-26/26.3--package-ref.sv
• chapter-5/5.13-builtin-methods-strings.sv
• chapter-5/5.6.3--system-functions.sv
• chapter-5/5.6.4--compiler-directives-debug.sv
• chapter-5/5.6.4--compiler-directives-preprocessor-macro_1.sv
• chapter-5/5.9-string-basics.sv
• chapter-5/5.9-string-broken-line.sv
• chapter-5/5.9.1-string-special-chars.sv
• chapter-6/6.12--real_edge.sv
• chapter-6/6.13--void.sv
• chapter-6/6.24.2--cast_fn.sv
• chapter-7/arrays/multidimensional/copy.sv
• chapter-7/arrays/multidimensional/subarrays.sv
• chapter-7/arrays/packed/equality.sv
• chapter-7/arrays/packed/onebit.sv
• chapter-7/arrays/packed/operations.sv
• chapter-7/arrays/packed/querying-functions/dimensions.sv
• chapter-7/arrays/packed/querying-functions/high.sv
• chapter-7/arrays/packed/querying-functions/increment.sv
• chapter-7/arrays/packed/querying-functions/left.sv
• chapter-7/arrays/packed/querying-functions/low.sv
• chapter-7/arrays/packed/querying-functions/right.sv
• chapter-7/arrays/packed/querying-functions/size.sv
• chapter-7/arrays/packed/querying-functions/unpacked-dimensions.sv
• chapter-7/arrays/packed/slice-equality.sv
• chapter-7/arrays/packed/slice.sv
• chapter-7/arrays/packed/treat-as-integer.sv
• chapter-7/arrays/packed/variable-slice-zero.sv
• chapter-7/arrays/packed/variable-slice.sv
• chapter-7/arrays/unpacked/assignments.sv
• chapter-7/arrays/unpacked/equality.sv
• chapter-7/arrays/unpacked/onebit.sv
• chapter-7/arrays/unpacked/operations.sv
• chapter-7/arrays/unpacked/ordering-methods/reverse.sv
• chapter-7/arrays/unpacked/ordering-methods/rsort.sv
• chapter-7/arrays/unpacked/ordering-methods/shuffle.sv
• chapter-7/arrays/unpacked/ordering-methods/sort.sv
• chapter-7/arrays/unpacked/reduction-methods/and.sv
• chapter-7/arrays/unpacked/reduction-methods/or.sv
• chapter-7/arrays/unpacked/reduction-methods/product.sv
• chapter-7/arrays/unpacked/reduction-methods/sum.sv
• chapter-7/arrays/unpacked/reduction-methods/xor.sv
• chapter-7/arrays/unpacked/slice-equality.sv
• chapter-7/arrays/unpacked/slice.sv
• chapter-7/arrays/unpacked/variable-slice.sv
• chapter-7/memories/read-write.sv
• chapter-7/structures/packed/basic.sv
• chapter-7/structures/packed/signed.sv
• chapter-7/structures/packed/unsigned.sv
• chapter-7/structures/unpacked/basic.sv
• chapter-7/structures/unpacked/default-value.sv
• chapter-7/unions/packed/basic.sv
• chapter-7/unions/tagged/basic.sv
• chapter-7/unions/tagged/packed.sv
• chapter-7/unions/unpacked/basic.sv
• chapter-8/8.10--static_methods.sv
• chapter-8/8.16--cast_func.sv
• chapter-8/8.23--scope_resolution.sv
• chapter-8/8.5--parameters.sv
• chapter-8/8.5--properties.sv

These should be disabled for Yosys:

• chapter-12/12.7.4--while.sv
• chapter-12/12.7.5--dowhile.sv
• chapter-13/13.3--task.sv
• chapter-13/13.3.1--task-automatic.sv
• chapter-13/13.3.1--task-static.sv
• chapter-13/13.4.4--fork-invalid.sv
• chapter-13/13.4.4--fork-valid.sv
• chapter-15/15.4--mailbox-blocking.sv
• chapter-15/15.4--mailbox-non-blocking.sv
• chapter-16/16.11--sequence-subroutine-uvm.sv
• chapter-18/18.16--random-weighted-case-randcase_0.sv
• chapter-18/18.16--random-weighted-case-randcase_2.sv
• chapter-18/18.17--random-sequence-generation-randsequence_0.sv
• chapter-18/18.17.1--random-production-weights_0.sv
• chapter-18/18.17.2--if-else-production-statements_2.sv
• chapter-18/18.17.3--case-production-statements_0.sv
• chapter-18/18.17.4--repeat-production-statements_0.sv
• chapter-18/18.17.5--interleaving-productions-rand-join_0.sv
• chapter-18/18.17.5--interleaving-productions-rand-join_2.sv
• chapter-18/18.17.6--aborting-productions-break-and-return_0.sv
• chapter-18/18.17.6--aborting-productions-break-and-return_2.sv
• chapter-18/18.17.7--value-passing-between-productions_0.sv
• chapter-20/20.15--dist_chi_square.sv
• chapter-20/20.15--dist_erlang.sv
• chapter-20/20.15--dist_exponential.sv
• chapter-20/20.15--dist_normal.sv
• chapter-20/20.15--dist_poisson.sv
• chapter-20/20.15--dist_t.sv
• chapter-20/20.15--dist_uniform.sv
• chapter-20/20.3--realtime.sv
• chapter-20/20.3--stime.sv
• chapter-20/20.3--time.sv
• chapter-20/20.4--printtimescale-hier.sv
• chapter-20/20.4--timeformat.sv
• chapter-20/20.6--isunbounded.sv
• chapter-20/20.6--typename.sv
• chapter-20/20.6--typename_type.sv
• chapter-20/20.9--isunknown.sv
• chapter-21/21.6--test.sv
• chapter-21/21.6--value.sv
• chapter-24/24.3--program.sv
• chapter-6/6.20.6--const.sv
• chapter-7/arrays/associative/alloc.sv
• chapter-7/arrays/associative/arguments.sv
• chapter-7/arrays/associative/assignment.sv
• chapter-7/arrays/associative/literals.sv
• chapter-7/arrays/associative/locator-methods/find-first-index.sv
• chapter-7/arrays/associative/locator-methods/find-first.sv
• chapter-7/arrays/associative/locator-methods/find-index.sv
• chapter-7/arrays/associative/locator-methods/find-last-index.sv
• chapter-7/arrays/associative/locator-methods/find-last.sv
• chapter-7/arrays/associative/locator-methods/find.sv
• chapter-7/arrays/associative/locator-methods/max.sv
• chapter-7/arrays/associative/locator-methods/min.sv
• chapter-7/arrays/associative/locator-methods/unique-index.sv
• chapter-7/arrays/associative/locator-methods/unique.sv
• chapter-7/arrays/associative/methods/delete.sv
• chapter-7/arrays/associative/methods/exists.sv
• chapter-7/arrays/associative/methods/first.sv
• chapter-7/arrays/associative/methods/last.sv
• chapter-7/arrays/associative/methods/next.sv
• chapter-7/arrays/associative/methods/num.sv
• chapter-7/arrays/associative/methods/prev.sv
• chapter-7/arrays/associative/methods/size.sv
• chapter-7/arrays/associative/methods/traversal.sv
• chapter-7/arrays/associative/nonexistent.sv
• chapter-7/arrays/dynamic/op-delete.sv
• chapter-7/arrays/dynamic/op-new.sv
• chapter-7/arrays/dynamic/op-size.sv
• chapter-7/arrays/unpacked/index.sv
• chapter-7/arrays/unpacked/subroutines.sv
• chapter-7/queues/bounded.sv
• chapter-7/queues/delete.sv
• chapter-7/queues/delete_assign.sv
• chapter-7/queues/insert.sv
• chapter-7/queues/insert_assign.sv
• chapter-7/queues/max-size.sv
• chapter-7/queues/persistence.sv
• chapter-7/queues/pop_back.sv
• chapter-7/queues/pop_back_assing.sv
• chapter-7/queues/pop_front.sv
• chapter-7/queues/pop_front_assign.sv
• chapter-7/queues/push_back.sv
• chapter-7/queues/push_back_assign.sv
• chapter-7/queues/push_front.sv
• chapter-7/queues/push_front_assign.sv
• chapter-7/queues/size.sv
• chapter-7/queues/slice.sv
• chapter-8/8.11--this.sv
• chapter-8/8.12--assignment.sv
• chapter-8/8.12--shallow_copy.sv
• chapter-8/8.13--inheritance.sv
• chapter-8/8.14--override_member.sv
• chapter-8/8.15--super-default-new.sv
• chapter-8/8.15--super.sv
• chapter-8/8.17--constructor_const_arg.sv
• chapter-8/8.20--virtual_method.sv
• chapter-8/8.21--abstract_class.sv
• chapter-8/8.21--abstract_class_inst.sv
• chapter-8/8.22--dynamic_method_lookup.sv
• chapter-8/8.24--out_of_block_methods.sv
• chapter-8/8.25.1--parametrized_class_invalid_scope_resolution.sv
• chapter-8/8.25.1--parametrized_class_scope_resolution.sv
• chapter-8/8.26.2--implements.sv
• chapter-8/8.26.2--implements_extends.sv
• chapter-8/8.26.2--implements_multiple.sv
• chapter-8/8.26.3--type_access_implements.sv
• chapter-8/8.26.3--type_access_implements_invalid.sv
• chapter-8/8.26.4--illegal_forward_def_implements.sv
• chapter-8/8.26.4--illegal_implements_parameter.sv
• chapter-8/8.26.5--cast_between_interface_classes.sv
• chapter-8/8.26.5--implemented_class_handle.sv
• chapter-8/8.26.6.1--name_conflict_resolved.sv
• chapter-8/8.26.6.1--name_conflict_unresolved.sv
• chapter-8/8.26.7--partial_implementation.sv
• chapter-8/8.5--properties_enum.sv
• chapter-8/8.6--methods.sv
• chapter-8/8.7--constructor.sv
• chapter-8/8.7--constructor_param.sv
• chapter-8/8.7--constructor_super.sv
• chapter-8/8.8--typed_constructor.sv
• chapter-8/8.8--typed_constructor_param.sv
• chapter-8/8.9--static_properties.sv
• chapter-9/9.4.1--delay_control-sim.sv
• chapter-9/9.4.1--delay_control-two-blocks-sim.sv
• chapter-9/9.4.2--event_control_sim.sv
• chapter-9/9.4.2--event_control_sim_minimal.sv
• chapter-9/9.4.2.4--event_sequence.sv
• chapter-9/9.7--process_cls_await.sv
• chapter-9/9.7--process_cls_kill.sv
• chapter-9/9.7--process_cls_self.sv
• chapter-9/9.7--process_cls_suspend_resume.sv

Note: it's a bit difficult what tests should be enabled/disabled for synthesis tools, because ideally they should be able to run all constant functions/constructs that don't depend on input ports. I will go back through this list and check how other synthesis tools handle ambiguous cases. Or is there a place to see how the Synopsys/Cadence/Siemens tools respond to all tests in this repo? That could be a useful metric.

[tgorochowik]

Thank you for the work. You suggestions sound good, just make sure the tests are not getting much more complicated than they are now (at least in terms of readability and number of different SV constructs).

Or is there a place to see how the Synopsys/Cadence/Siemens tools respond to all tests in this repo? That could be a useful metric.

The licenses of the proprietary tool disallow using them to run in any benchmarks or comparisons so we cannot have this here.

If you do wish to compare (other) tools that allow it, take a look at how the runners in the tools directory are implemented. Adding your own tool should be fairly straightforward - so you should be able to run some comparisons locally

[daglem]

I just added a simulation pass to the Yosys runner in #5498, in anticipation of merge of YosysHQ/yosys#3963 in Yosys.
I have tested the Yosys PR, and it solves the current issues with $display in sv-tests, and gives a nice boost in passed tests for Yosys.
I believe this negates the need to change any tests.

[daglem]

The recent boost in passed tests for Yosys is due to YosysHQ/yosys#4129.

By default, output from $display et al. in initial blocks is produced both from elaboration and simulation. The tests still work, however duplication of the output is confusing, and could possibly cause problems later on. For simulation tests, #5525 suppresses the output from elaboration, leaving only output from simulation.