We tried set is_enabled to false for rule without restart elastalert. however, the rule sill take effect.

https://github.com/Yelp/elastalert/blob/master/elastalert/elastalert.py#L982

It should remove the disabled rule from the active rules, not just continue.