Start and End properties are inaccurate

[kla7]

Bug Description

When running spacy, the start and end values for each token are inaccurate.

For example:

        {
          "@type": "http://vocab.lappsgrid.org/Token",
          "properties": {
            "start": 26444,
            "end": 31989,
            "pos": "DT",
            "lemma": "a",
            "text": "a",
            "id": "to_5546"
          }
        }

Reproduction steps

Run spacy on any txt or mmif file.

I ran it on:

1. Sample text from wikipedia with the following output MMIF
2. Whisper output MMIF with the following output MMIF

Expected behavior

The end value should be the start value + the length of the token.

Log output

No response

Screenshots

No response

Additional context

No response

[marcverhagen]

At some point some code was added to deal with pre-tokenized input to the spaCy app. This builds a token index and the length of that index is used to calculate the end offset. With non-tokenized input that index is build token by token, and since the length of the entire index is used the first token is length 1, the second is length 2 and so on.

app-spacy-wrapper/app.py

Lines 75 to 78 in ce95ecc

	if n not in tok_idx:
	a.add_property("start", tok.idx)
	a.add_property("end", tok.idx + len(tok_idx))
	tok_idx[n] = a.id

The code added to deal with tokenized input probably was not tested to confirm that it did the right thing with non-tokenized input.

[marcverhagen]

The "end" property is fixed, the "start" property was never wrong. Will test a little bit more before releasing a new version (specifically for pre-tokenized input).

[marcverhagen]

Well, the pretokenized parameter seems to be broken independent of what is going on in here so I will make that a separate issue.