0.5.0a1

This is a preview of some of the new features in the upcoming 0.5.0 release.

New Features

• Greedy case statements: explicit ambiguity resolution can be turned on as an option in case statements now (52e5418)
• Finish codes: explicit finish statements can now return custom status codes, instead of just always giving _DONE (1ffe4d4)
• Yield: parsers can now interrupt execution with a return code and be restarted; for example to build a lexer (1ffe4d4, 41b2c23)

Bugs Fixed

• Loops that have (possibly conditional) breaks before any matching statements no longer think they're infinite (a687b3b)

Infrastructure Changes

• Tests now use finish codes to better check correctness (9f1c20e)
• Various examples/docs have been updated to use finish codes instead of result enumerations (976fc60, a777a5e)
• New lexer example demonstrating yield functionality (a9a96a8)
• Various documentation improvements:
  • Better syntax highlighting (2527e74, 4d52e5c, ee68d8f, dd6223f, d779af1)
  • New tutorial describing the lexer example (b693a8a)
  • Documentation for the new greedy case and yield/finish codes (b9e96a8, 4e47875)
• Updated to lark 1.0 (f3dd61e)
• Appimages now build against Ubuntu 20.04 as their base (9dc256c)
  • Note snaps may be removed in a future release due to issues in reliably building them in CI

0.4.1

(0.4.0 was missing a commit and so is not listed here as a release)

This is a major feature release, vastly improving NMFU's capability for parsing binary protocols, in addition to fixing a number of bugs. There's also now a proper documentation site for NMFU, available at nmfu.rtfd.io.

New Features

• Expanded type system: (ea5e297)
  • Integers can now have a custom size and signedness: out int{size 4, unsigned} wow;
  • Strings can now be unterminated: out unterminated str[56] look_no_nulls;
• Conditional blocks: if/elif/else statements now exist (be41210, f3f71de).
• Expanded expressions:
  • Boolean expressions: || and &&, as well as comparison operators (baa2ba1, 3369969)
  • Better math: unary not and negation (5f24078), string length and indexing (353f391) and bitwise logic/bitshifts (6936042).
  • Better integer literals; now supporting hexadecimal and binary: 0b0110, -0x54 (1e61988)
• Binary matches (8b2f09b, e202041):
  • Strings and regexes can now be made "binary", where instead of them having characters they instead only contain hex.
  • For example, "f0 0ffa"b or b/ff+ (3456|56 77{2-3})/.
  • This also adds support for \xHH escapes within normal strings.
• Character constants now support escape sequences (45196d6, 1cdd324)
• Improved debugging tools for working on NMFU itself:
  • Option for dumping the parse tree for debugging (1b7242a).
  • DFA-dumper can now be asked to show all literals as hex codepoints (3450fa5)
  • Gave conditional actions proper debug names (151eaae)
  • Added dumper for viewing the DTAG tree (c45b4d2)
• New optimizer that tries to shortcircuit fallthrough transitions if possible (3043adc, db126e2)

Bugs Fixed

• Better ambiguity detection and resolution for chaining matches, especially those with inverted matches (1026f83).
• Properly handle actions after certain blocks when there is nothing following them (c5f1806).
• Correct short-circuit behaviour for action side-effects (such as outofspace handling) (bbda784).
• More accurate line/column information for errors concerning regexes (92cd8f2).
• Ensure chaining DFAs doesn't drop error handling-transitions in certain circumstances (64e89f2)
• Fix DFA.dfs() not properly considering MAY_GOTO_TARGET-type actions, causing the optimizer to mistakenly drop states (ca72b3b)
• Fixed loop body start actions being ignored (ccb54a3)
• Macros now bind arguments in a sane way, fixing various problems with nesting them. (8f87354, 6191b42)
• Fallthrough loop detection now considers multi-state cycles. (222d971)
• Action-only conditional nodes now properly propagate subsequent actions (ad3a917)
• Foreach nodes only apply their actions to non-fallthrough transitions to ensure they only run once per character (709d314)
• Rewrote the majority of the loop implementation to make it more compatible with action side-effects (5b520bb)
• DTAG system now does more checks for garbage-collection and uses weakrefs to be slightly more memory-efficient (e5d58f3)

Infrastructure Changes

• Proper documentation, powered by mkdocs (ce5429c) as well as a basic tutorial (a1ea939, 3af8af2, 3897641)
• New example showcasing binary features which parses TripUpdate messages from a GTFS Realtime feed (f4a2edb, e7c8b1d)
• Demonstration of more advanced string manipulation in the RDF example (92ff838, 5fa891e)
• Full integration tests now will run the generated code through an actual compiler to check for correctness. While they're skipped if one isn't found, please make sure any potential patches are tested with one installed. (acbff31)
• Certain integration tests now check for both compilation success and correct matching (a2fb61a)
• Code coverage is now reported in CI (2b9152f)
• More tests, both for new and old features. (lots)

0.4.0a2

This is another early preview of some more new features in the upcoming 0.4.0 release, as well as some fixes for regressions in 0.4.0a1.

New Features

• More math operators: unary not and negation (5f24078), string length and indexing (353f391) and bitwise logic/bitshifts (6936042).
• Character constants now support escape sequences (45196d6)
• Expanded type system: (ea5e297)
  • Integers can now have a custom size and signedness: out int{size 4, unsigned} wow;
  • Strings can now be unterminated: out unterminated str[56] look_no_nulls;
• Better integer literals; now supporting hexadecimal and binary: 0b0110, -0x54 (1e61988)
• Improved NMFU-debugging:
  • DFA-dumper can now be asked to show all literals as hex codepoints (3450fa5)
  • Gave conditional actions proper debug names (151eaae)
  • Added dumper for viewing the DTAG tree (c45b4d2)
• New optimizer that tries to shortcircuit fallthrough transitions if possible (3043adc, db126e2)

Bugs Fixed

• Ensure chaining DFAs doesn't drop error handling-transitions in certain circumstances (64e89f2)
• Fix DFA.dfs() not properly considering MAY_GOTO_TARGET-type actions, causing the optimizer to mistakenly drop states (ca72b3b)
• Fixed loop body start actions being ignored (ccb54a3)
• Macros now bind arguments in a sane way, fixing various problems with nesting them. (8f87354, 6191b42)
• Fallthrough loop detection now considers multi-state cycles. (222d971)
• Action-only conditional nodes now properly propagate subsequent actions (ad3a917)
• Foreach nodes only apply their actions to non-fallthrough transitions to ensure they only run once per character (709d314)
• Rewrote the majority of the loop implementation to make it more compatible with action side-effects (5b520bb)
• DTAG system now does more checks for garbage-collection and uses weakrefs to be slightly more memory-efficient (e5d58f3)

Infrastructure Changes

• New test showcasing binary features which parses TripUpdate messages from a GTFS Realtime feed (f4a2edb, e7c8b1d)
• Demonstration of more advanced string manipulation in the RDF example (92ff838, 5fa891e)
• Full integration tests now will run the generated code through an actual compiler to check for correctness. While they're skipped properly if one isn't found, please make sure any potential patches are tested with one installed. (acbff31)

0.4.0a1

This is an early preview of some of the new features in the upcoming 0.4.0 release.

New Features

• Conditional blocks: if/elif/else statements now exist (be41210, f3f71de).
• Boolean expressions: || and &&, as well as comparison operators (baa2ba1, 3369969)
• Binary matches (8b2f09b, e202041):
  • Strings and regexes can now be made "binary", where instead of them having characters they instead only contain hex.
  • For example, "f0 0ffa"b or b/ff+ (3456|56 77{2-3})/.
  • This also adds support for \xHH escapes within normal strings.
• Option for dumping the parse tree for debugging (1b7242a).

Bugs Fixed

• Better ambiguity detection and resolution for chaining matches, especially those with inverted matches (1026f83).
• Properly handle actions after certain blocks when there is nothing following them (c5f1806).
• Correct short-circuit behaviour for action side-effects (such as outofspace handling) (bbda784).
• More accurate line/column information for errors concerning regexes (92cd8f2).

Infrastructure Changes

• Certain integration tests now check for both compilation success and correct matching (a2fb61a)
• Code coverage is now reported in CI (2b9152f)
• More tests, both for new and old features. (lots)

0.3.1

This is a minor release that tweaks a few things in code generation.

New Features

• jpto_ labels are only emitted when used in outputted code, preventing -Wunused-label warnings. (1b93e4b)
• Allow customizing the output format of debugging graphs. (5005bae)

0.3.0

This is a feature release which adds a bunch of miscellaneous features, in addition to changing the public-facing API.

New Features

• New API: (cbd45c4)
  • Instead of taking in single characters, the _feed function now takes in blocks of characters at once. The function can also
    optionally report how many characters it read in case of early finishes, to aid in chaining parsers.
  • Instead of having a parameter for is_end, there is now an optional separate _end function for handling EOF detection, enabled
    by the -feof-support flag.
  • These changes are designed to improve generated parser efficiency.
• Fallthrough semantics: (cce57c4, af6728f)
  • Instead of else matches in case nodes and catch blocks consuming characters, they now forward the error/else to the first match
    in their block.
  • This makes error handling more robust, as well as removing a lot of the uses for some_var += [$last], simplifying parsers.
• Revamped regexes: (50f219c, 5603d44)
  • Operator precedence is now more consistent with typical regexes, so (option1|option2) now works correctly.
  • Repetition syntax (\d{4-6}) is now supported.
• Foreach statement: (9cda4ac)
  • Parsers can now execute statements on each character matched by some region.
  • This greatly simplifies things like reading numbers:

    foreach {
      \d+\;
    } do {
      num = [num * 10 + ($last - '0')];
    }
    

• Macros can now take arguments. (07e4a04)
• Character constants can now be used in math expressions. (fc93c91)

Bugs Fixed

• Fixed problem where having a wait match after a try except could break due to incomplete error-handling redirection. (87de753)
• Fixed internal error with allow_replace in DFState.transition which could hide ambiguous transitions. (0e1ad3b)
• Fixed problems with incomplete else handling on case nodes with regex-matches with inverted character sets/wildcards. (417fb41, 3fa4d93)

Infrastructure Changes

• NMFU now also builds AppImages (57ff2e7, 193910a)
• The test suite has been thoroughly expanded, and now covers a lot more. (lots)
• There is now an automated version update script. (2556d66)

0.3.0a2

This is another early preview of some more changes in the upcoming 0.3.0 release.

New Features

• Added a mode for making the start pointer indirect, allowing _feed to report how much of its input it consumed. (e589c6f)
• Changed regex grammar to more closely match other regex syntax (by fixing the precedence of the | operator.) (50f219c)
• Added support for regex repetition matches (5603d44)
• Added a new block syntax, foreach, which can execute statements on each character consumed by a block. (9cda4ac)
• Added character constants in math expressions. (fc93c91)

Bugs Fixed

• Various problems with the new fallthrough transitions from 0.3.0a1. (4203c2e, 68e95a3)
• Fixed problem where having a wait match after a try except could break due to incomplete error-handling redirection. (87de753)
• Fixed internal error with allow_replace in DFState.transition which could hide ambiguous transitions. (0e1ad3b)
• Fixed problems with incomplete else handling on case nodes with regex-matches with inverted character sets/wildcards. (417fb41, 3fa4d93)
• Fix end matches, which were broken in 0.3.0a1. (4e3b6db, 4f4dba9)

Infrastructure Changes

• The test suite has been thoroughly expanded, and now covers a lot more.
• There is now an automated version update script.

0.3.0a1

This is an early preview of some of the breaking changes in the upcoming 0.3.0 release.

New Features

• New API: instead of taking in single characters, the _feed function now takes in blocks of characters at once, to improve efficiency
• end support is now optional. This is part of the API changes, because now end handling is through a separate _end function.
• Fallthrough semantics: the else path of a case node, as well as jumps into catch blocks no longer consume a character, and instead
  properly forward it to the first match in the target block.

Infrastructure Changes

• NMFU now also builds AppImages

0.2.1

This is a minor bugfix release with a few new features

New Features

• New option -fuse-cplusplus-guard which makes the generated headers C++ compatible

Bugs Fixed

• Previously, a try-except block with an empty handler would cause an error; now it correctly does nothing and exit on an error.

0.2.0

This is a feature release which adds hooks and a few miscellaneous features.

New Features

• Hooks:
  • Parsers can now call arbitrary callbacks at arbitrary points like macros
  • These hooks get a pointer to the current state and input character
  • In the future, these may be used for things like buffering large outputs
• Append-character expressions
  • Append expressions can now add individual character codes from a math expression
  • e.g: out_var += [$last - 3];

Bugs Fixed

• Case labels with inverted character classes (or non-erroring Else transitions) would accept too few inputs
• Macros with more than one statement would be truncated
• Custom Else transitions were not properly dealt with by the ambiguous detection logic and so a variety of things involving wildcards would fail in strange ways.

Infrastructure Changes

• NMFU now has a test suite, install the necessary dependencies with the tests extra and run with pytest
• NMFU now has CI, see the badge in the README
• NMFU can now build as a snap