As a manager, can I delete feedback given for an employee's performance goal?

No. You need the Administer Requested Feedback aggregate privilege to delete feedback requests of performance goals.