I would get a second opinions from your vet, not just your farrier. He may have thin walls or just weaker feet and shoes may help especially if he is only sore on harder footing. It may simply be that he need shoes because harder ground bothers his feet. Other than that abscesses are always a possibility and stone bruises too. I would just consult your vet and maybe try shoeing him and go from the results of that.
The triple threat of riding = EVENTING! 😉