We also discussed how much name normalization should be performed by the "heuristic understanding" app.
In my opinion, we should (eventually) go for the best performance possible. The difference between heuristic understanding and processing by LLM is not that the heuristic understanding is somehow bad. It is just that it is intuitive and purely rule-based -- the kind of algorithm for which there could have been an efficient implementation 25 years ago.
However, for now, something simple will suffice. I think it should do something along the following lines to the "name as written" string:
- Remove from the beginning any word ending with a period, such as "Dr.", "Mrs.", "Rep.", etc.
- Remove from the end any characters starting with the first "(" or ",", such as "(D)" or ", MD".
- Convert the case of each word to "title case" (i.e., only first letter capitalized).
- Split the remaining string on the final internal space character, yielding two strings
s1 and s2.
- Return
s2 + ', ' + s1.
To me, something along those lines is a plausible (but still very naive) interpretation of the instructions given to the human annotator for the "second datum". As such, I think it would provide an appropriate baseline for comparison.
Originally posted by @owencking in #1
Originally posted by @owencking in #1