Maybe he is just not a trail horse. My horse is really not a trail horse. Retired roping. Arena horse. He follows my husband when he walks on a trail and I ride but horse won’t follow other horses. It’s like he has his own job to do and it’s not about following others. And it’s not about being on the trail with horses either. You need a horse that really enjoys being out and one that can work with other horses around. The round pen and around the barn there is usually no one else that bothers you. Horse is by himself. Maybe your horse thinks he has to dominate with others and can’t concentrate on the job at hand?? Just a thought.