Avoid the full ancestor chain walk when removing elements from the Project Explorer

Avoid the full ancestor chain walk when removing elements from the Project Explorer #5154