It sounds like your horse is bored and doesn’t trust you. A spooky horse will be spooky no matter what, but you can affect what he does when he spooks.
When you work, make sure you keep his focus 100% of the time. Incorporate lots of circles and transitions into your work so he never knows what is coming next. As a result, he will always be listening for your next cue.
You mentioned that he takes a lot of leg to get going. I had a mare that was very similar and just as spooky. What worked for me was a big pair of Spurs and keeping her busy, especially when I was worried about a spook. You may need to work on getting your horse’s respect for your leg a little more. You can teach him to respond to a light request by asking gently and then reprimanding quickly if he does not respond. However, you need to make sure you are always completely clear and consistent in your aids. Your horse is young enough that he will learn something extremely quickly which means that if you let him do something wrong once he will think that is ok. He may have learned that by spooking he can get out of work. The truck is to never be complacent. Give him the benefit of the doubt but always be ready. Then when he spooks, do your best to ignore it and continue working. Try not to acknowledge spooking so he can’t get out of work by being scared. And you will become a constant reassuring presence so when he is nervous, you can easily and quickly reassure him.