It's a shame you can't make the wither arena an instance like some MMOs do to keep uninvited people from joining.