A few suggestions:
It is really important to make sure you aren’t causing your horse pain when you ride him. Horses are easily trainable in both good and bad and if he is expecting pain when you get on then he won’t want to move. Double check his teeth, back, legs and feet for possible injuries.
You also may need to reevaluate your tack. I suggest changing your nose band to correct the opening of the mouth. A crank, flash or grackle (figure 8) nose band can prevent a horse from opening its mouth if properly adjusted. It is also important to make sure your bit is effective and does not cause your horse unnecessary pain. Make sure it is the right width (a pinky finger on either side) and that the joint (if it is not a loose ring) does not wiggle, because that can pinch the horse’s mouth and cheeks. You should always use the softest but possible to protect and preserve your horse’s mouth.
Your issues on the ground show that your horse does not respect you at all. That is big problem for the reasons described above. If he does not respect humans, he can easily become an extremely dangerous horse. He needs to learn that you are in charge and that you cannot be ignored.
Based on your description, my guess is your biggest problem is that you may need to be more clear in your instructions, both on the ground and in the saddle. However, you need to fix your problems on the ground before attempting to fix the problems undersaddle.
When you ask him to do something, ask nicely and easily and if you do not get an immediate response, use more force until you do. However, the most important thing is that the moment you get any sort of progress, LET UP. For example, if you are asking your horse to move over, start by pushing gently. If he does not move, push harder, smack him or stick your thumb in his side. Quickly increase the severity of your request until he moves. He will most likely move to evade the pressure in his side which may hurt. The second he takes even a baby step, remove the pressure and reward him. Then ask again, first gently and then quickly increasing pressure. He will learn that when you ask gently, you want him to do something and that when he does it he is rewarded. Do not reward with food though. Treats undermine your relationship with your horse. Reward with pats and “good boy”.
As far as gaining respect, teach him that if he ignores you he will be punished and that if he listens he will be rewarded. you can ask him to submit to you in others ways. Examples of submissive behavior are backing up and lowering the head. When you lead him, he needs to walk next to you. Do not let him drag behind you or walk in front of you. Walking behind you is dangerous, especially if he does not respect you or your space. If he spooks he will have no issues running you over. Allowing him to walk in front of you removes your control however and lets him think he is in charge. You also should be controlling his pace. If he is normally a slow walker in hand, it may be a good idea to carry a whip to get him moving. If he tries to move ahead of you, make him back up and move away. If he refuses to back up, do not give up or get scared. Keep asking until he does it and reward him for responding. He ignores you both on he ground and under saddle because you have taught him that you will allow it. By allowing him to get away with it, he wins and in that way you will never gain his respect.
You should also work with him on a lunge line with side reins. Do this at the walk and trot with a lunge whip. He needs to be out on a large circle. That is the most important thing about lunging for training. I recommend finding someone to help you with lunging the first few times, and to help with the fit of the side reins. It is really important that the side reins do not pull the horse’s head in. If you cannot find someone to help you, some general guide lines: the side reins are the last thing to go on and the first to come off. When leading a horse with side reins, do not make tight turns. Start your horse off without them and once he trots calmly on the lunge line, put them on. If he starts off bucking and cantering do not panic. In any situation, panicking is the worst thing you can do. YOU MUST ALWAYS REMAIN CALM AROUND HORSES. They can sense your emotions and be set off by them. If your horse starts cantering, calmly say whoa, and allow him to come back to trot, placing the whip under your arm if needed. When working, keep him out on the large circle (as large as possible) by pointing the whip at his shoulder and keep him going by pointing the whip at his leg. Try to stay stillin the center of the circle, but if your horse will not stay out of refuses to move, walking around in a small circle is fine.
From the moment you lead your horse to the mounting block, you need to have his respect. When you mount, it is extremely important that your horse stands patiently for you to sit down in the saddle. Do not let him walk away as you mount or before you are settled and have your reins. This is dangerous and the first sign your horse does not respect you. As you mount, ensure you have contact on the reins and that you always have control. If he walks away then you have given control to him before you have even begun.
While working, again it is important to make sure to ask gently, then if you do not get a response, reprimand quickly and as harshly as needed but release the pressure as soon as you get any sort of response. YOUR RELEASE OF PRESSURE IS THE MOST EFFECTIVE AID YOU HAVE. You also need to make sure that you are very clear in how you ask for something. You should always ask the exact same way every time. Accuracy is key.
As far as whips and Spurs, I recommend that you get very big Spurs. However, you need to makes sure that you know how to use them. When you ride with Spurs, you need to make sure you turn your toes in enough that you do not accidentally jab your horse with Spurs. Using Spurs is a punishment for not listening to your leg and it is extremely detrimental to training if you punish your horse at the wrong time or incorrectly. As far a whips, remember that a whip is also a punishment. YOU USE YOUR WHIP TO REPRIMAND YOUR HORSE FOR IGNORING YOUR LEG. ALWAYS USE YOUR WHIP BEHIND YOUR LEG. You should never use it on the shoulder or butt. Always hit behind your leg an as close to it as possible. You mentioned that your horse reacts negatively to this and walks sideways. This can happen for a few reasons. You may have hit harder than you need to and your horse is responding. It may be that he has been trained (by you or someone else) that when he is hit in that area with a whip he is supposed to move sideways. Thirdly and most likely, he k owe that if he does this you will not hit him and he wins. If it one of the first two possibilities, then you need to teach him that using a whip means go forward. Place yourself along a wall and carry the whip in the inside hand, so he cannot move away from the whip without hitting the wall. Ask him to move forward by squeezing, slowly increasing the pressure. If he does not move forward, hit him and if he does not move forward, spur him and use your whip again. Once he move slowly forward, even a little bit, soften your legs on his side (Spurs turned out) and rest the whip on your thigh. Then ask again. Make sure that when you squeeze, you do not draw your leg back but instead squeeze in with your calf muscles. You may need to build up muscles you do not generally use. It is very important that you stay very patient and do not get frustrated if you do not see immediate results. Horse’s learn things much more quickly then they forget them and you need to untrain bad habits your horse has learned.
To fix your issue with turning:
I had the exact same problem with a horse I was training about a year ago, where I would ask him to turn and he would bend his head in and not turn. This problem happens because of the way you ask your horse to turn. When you ride, your reins have different functions. Your outside rein is you controlling rein for both speed and bend. YOU SHOULD ALWAYS HAVE CONTACT ON YOUR OUTSIDE REIN. Do not allow your horse to evade your outside rein by hanging on your inside rein. The inside rein is you directional rein. It tells your horse which direction to bend. You do not need a heavy contact with your inside rein. In fact, with a properly trained horse, you technically could get away without really using it at all. This is because bend and rounding comes from the leg. Your horse does not respect your leg, which is why he doesn’t respond to it. When you ask your horse to turn, you should be asking with your legs, back and seat. Riding is 98% body and 2% hands. You should be able to ask your horse to make both upwards and downwards transitions as well as turns with only your body and without your hands completely. When you ask your horse your turn, the outside leg should go back to prevent the horse’s hindquarters from swinging out and the inside leg should come forward slightly to keep your horse upright in the turn, similar to when you ask for canter. The difference between a canter transition and a turn is that in the turn you should use your inside leg exclusively to keep the horse upright, balanced and moving forward while in an upwards transition to canter you use the outside or both legs. When you are working at the trot, you should be using only your inside leg to keep your horse moving, not both. This is the other reason you should carry your whip in the inside hand, as it is an aid to your most active leg.
When you begin your work, you need to be in charge from the start. One of the most common mistakes I have seen with people who train horses is that they forget the walk. When you are warming up, resting, and, to an extent, finishing up, you need to control the walk. Your walk needs to be forward and active. It is the easiest fair to ruin and the hardest to fix, but it is the basis of all of your horse’s other gaits. Use alternative leg aids in the walk, so inside, outside, inside, outside. Use your legs and, if necessary, whip or Spurs to achieve a forward march, asking your horse to come together only after he is moving forward with impulsion. Once he is walking, keep your legs still on his side. If he slows down, speed him up again. Your legs should never really come off the horse, even if you are not using them. Make sure you do not get into a rhythm of using your legs without response however. This is another trap that is easy to fall in to. It is your horse’s job to keep moving forward, not yours. If he does not keep a good tempo and rhythm, correct him and then give him the chance to maintain the pace himself.
The other possibility is that your horse views moving forward in a negative way. This may be because you have punished him for not moving forward but when he did, you still punished him or did not let up. A common example is when a rider hits a horse with a crop and when the horse jolts forward, the rider loses balance and grabs the reins accidentally. It could also be that you missed a minute response and continued to punish the horse for not moving forward. In both of these situations the horse will not want to move forward because he expects punishment no matter what and he is confused and often scared. The key concept to training is not to scare your horse, but to teach him that he needs to listen and that there are consequences if he does not because you are in charge.
To treat the herd-bound issue:
Once he begins to respect your leg, you can work on he here problem. Start by taking him outside alone to hack out. Move away from the barn at a trot or canter and head home at a walk. Make sure that he is moving at a forward and energetic pace away from the barn and that he is not allowed to pull you home. If he tries, use half-halts to slow him down. If he does not listen to your reins, you can stop him and ask him to back up. If you are walking home and he trots, turn him in a circle away from home and ask him to walk. Once he is walking calmly, head home again and repeat as necessary. Once he is no longer trying to rush home, do this around other horses. Do upwards transitions heading away from a group and downwards transitions towards them. Make sure you warn other members of the group of your plans to avoid spooking other horses and never take “no” for an answer from your horse. You are in charge. With this exercise, stick to walk and trot. Another tip is when you are in a large group lesson, ensure that you are not allowing your horse to follow another by doing something slightly different (change your speed, circle away without allowing him to decrease speed or pass without allowing him to decrease speed). In a group jumping lesson, do not allow your horse to stand in the center with the other horses. Keep him walking at a forward pace out on the rail. It will also help jumping by preventing buildup of lactic acid in his muscles.
So a summary of what you should do:
1. Have your farrier check your horse’s feet and make sure his teeth do not need to get floated (they should get floated about once a year). If you see your horse dribbling food as he eats, this is a sign his teeth hurt. Check his back and legs by running your hands down them and watching to see if he flinches. If he does, try again after a moment. If he still flinches, it may be worth a call to your vet. It may also be a good idea to have your vet (if you notice back soreness) and/or farrier watch you ride as they may be able to give you a better idea of what your horse needs.
2. Double check that your tack fits properly and is appropriate for your horse’s needs as I described above. Make sure it is clean and free of dirt and sweat that could irritate your horse (please note that glycerin soap does not clean. It seals moisture into the leather to keep it healthy. Use leather cleaner to clean tack).
3. Do ground work to gain your horse’s respect on the ground. Teach him to stand calmly when you ask and move only when you ask him to.
4. Work him on a lunge line to aid training and allow him to move forward without interference from a rider. He will be able to learn that moving forward is something he has to do himself and will not be punished.
5. Begin under saddle work at the walk and trot, focusing on controlling speed in walk and trot. Include lateral movements such as turn on the forehand, turn on the haunches and leg yield to teach him to respond to your legs. Make sure to focus on how you are using your aids and that you are consistent.
6. At this stage, you can also use poles to encourage your horse to shorten and lengthen his stride. Just make sure you allow your horse access to his head over poles and that you are out of his way. Always ask for impulsion and forward momentum before asking for a proper outline. And remember that outline and energy come from legs. Also, energy is not speed. Once you have a good tempo, you should still ask for more energy by using your legs and controlling speed with the outside rein. Inside leg also moves your horse off of your inside rein and into your outside rein.
7. Begin incorporating canter transitions back in. Make sure hat you have contact with your outside rein and do not allow your horse to run into canter. You also need to make sure you allow your horse to be forward. Do not pull or hang on his mouth. That will be crucial at this stage.
8. Return to jumping after you have his respect in your flat work. If you do not have good flat work, you cannot have good jumping and attempting to do so is very dangerous and most likely will be detrimental to your horse’s training. Continue to focus on being soft on your horse’s mouth and controlling speed with your body as much as possible.
9. Work on moving away from the barn and not being attached to it during work. Then work on moving away from other horses during work. It is fine for your horse to be attached to his friends but when he is working he needs to be focused on you and on his job, nothing else. Make sure to differentiate this for him.
I hope this helps and if you have questions or other issues, let me know!