code lines removed with partial line comment

[loziniak]

I'm trying to add Red support for extractor.
During CI tests extractor removed these lines altogether, not only comments and trailing whitespace:

x: 1 ; line comment 1
x: 2   ;-- line comment 2
x: 3		;@@ line comment 3

see discussion

My language config looks like this:

!e
;\p
Red [\p-->>]\pj
Red +[\p-->>]\pj
comment [\p-->>]\pj
comment +[\p-->>]\pj
comment {\p-->>}\pj
comment +{\p-->>}\pj

[ErikSchierboom]

@joshiraez Thoughts?

[joshiraez]

Looking at it.

[joshiraez]

Ok so the problem is in this rule. I'm having problems setting the code up (I'm etting problems with the json library just like in old times in this laptop and ashdkjahejknsakjdnskjad I'll get to slack to get support eventually as I plan to do some progress on this during xmas, but i'm digressing) but from the trace I'm quite sure this is what's happening

So Red [\j-->]\pj is parsed as follows.

Look for Red[ and skip it. (It's just, it's looking for something even if there's no whitespace)
Look for a \n, and skip the whole line up to that point. (There's no "partial" flag, so it will look for the next line break before anything else)
Look, in the following lines, for ] (Just)
As it's a "partial" clause, remove up until after the second token is found.

This means that in the test case:

          Red [] ; a comment
          ; single line comment
          x: 1 ; line comment 1
          x: 2   ;-- line comment 2
          x: 3		;@@ line comment 3
          comment ['this
          	'is 'multiline
          	'comment]
          comment   {and this
          	as well}
          function add100 [x [integer!]] [
          	"; this should not count as comment "
          	{ comment [and neither
          	 this] }
          ]

What is being skipped by that rule is this:

          *Red [**** -->>**] ; a comment
          ; single line comment
          x: 1 ; line comment 1
          x: 2   ;-- line comment 2
          x: 3		;@@ line comment 3
          comment ['this
          	'is 'multiline
          	'comment-**->>****]*
          comment   {and this
          	as well}
          function add100 [x [integer!]] [
          	"; this should not count as comment "
          	{ comment [and neither
          	 this] }
          ]

In fact with the current rules this will be the result I believe

          function add100 [x [integer!]] [
          	"
          	{ }
          ]

It should work if you set the just flag for everyone (this is something I have had in mind for a while and plan on "improve" in the new version of the extended extractor). I'm thinking on how to avoid the inside comments right now.

!e
;\p
Red [\pj-->>]\pj
Red +[\pj-->>]\pj
comment [\pj-->>]\pj
comment +[\pj-->>]\pj
comment {\pj-->>}\pj
comment +{\pj-->>}\pj

I can't try it out right now (because the project doesn't compile in my pc because it can't find the json library and ahhgahghagh) but please try that out and tell me if it works.

Also the second part, it's really not supported right now in the extractor as we have no way to tell it "hey, don't parse stuff in string literals". I have been thinking about that also, to add some kind of negative to skip places where it shouldn't parse. I'm seeing now that it'll have to be supported

Also sorry it took this so long but I moved recently ON TOP OF EVERYTHING so I barely have any time these days. I'll work on the snippet extractor as well with some new ideas that should help with this.

[loziniak]

Hi! Thanks for looking at this. I did not expect string bypass to work, I just wanted to emphasize the issue. I'll check your language setup in a minute.

[loziniak]

This seems to be OK now. Closing, thanks.