I cannot find any nice words for the previous owners! This is going to take some time, as he will need to learn that you are his new mom. While working on that, instead of a halter, drape a lead rope around his neck, and entice him to follow you (can also be done without the rope). Once he begins to follow you reliably, then start petting his face, resting a rope on it, and eventually (hopefully not too long) he will accept it and then proceed to the halter. Also, while he is young, you can also drape towels on his back, expose him to brushes, start working on picking up his feet, etc. Make it all fun for him, and don’t overload him with too many new things at once. We’ve always started bonding foals with a person as close to the time of birth as possible, so they just take it all for granted. So much easier.
It is never the horse's fault