Giants ownership was reportedly concerned that Zaidi couldn't get the job done