Is the source assignment impacted when you delete the End Temporary Assignment record from the destination assignment?

No, the deletion doesn’t impact the source assignment from where the temporary assignment was initiated. You can delete the source assignment update related to End Temporary Assignment.