Deadlock under TEDIT.QUIT in specific corner case

[fghalasz]

There is deadlock that occurs under TEDIT.QUIT when both of the following conditions are true:

1. TEDIT.QUIT is called from a function called by TEDIT.DEFAULT.MENUFN which has been invoked by a mouse click in the TEdit window title bar. This matters because TEDIT.DEFAULT.MENUFN sets the textobj EDITOPACTIVE flag before TEDIT.QUIT is called.

2. The TTY.PROCESS does NOT belong to the TEdit window (primary pane?) when TEDIT.DEFAULT.MENUFN is invoked. This matters because the BLOCKing described just below is skipped if the TTY.PROCESS does belong to the TEdit window.

With both of these conditions true, TEDIT.QUIT (running in the menu process) sets the EDITFINISHED flag to T and then BLOCKs waiting for the TEdit process to exit.

Meanwhile, the \TEDIT.COMMAND.LOOP in the TEdit process also BLOCKs waiting for the EDITOPACTIVE flag to be reset before it can deal with the EDITFINISHED flag and exit.

Hence, a deadlock. The TEDIT.DEFAULT.MENUFN won't reset the EDITOPACTIVE flag until TEDIT.QUIT returns. But TEDIT.QUIT is blocked waiting for the EDITFINISHED flag to be processed. But the EDITFINISHED flag won't be processed until the EDITOPACTIVE flag is reset.

I have worked around this in Notecards by ensuring that condition 2 is never true.

Probably safest true fix is to have TEDIT.QUIT always reset the EDITOPACTIVE flag when it sets the EDITFINISHED flag - even though this is not necessary 99% of the time.

[rmkaplan]

Maybe TEDIT.QUIT should not set the EDITFINISHED flag if EDITOPACTIVE is true? I.e., you can't quit if something is going on.

Would that eliminate the deadlock?