Important distinction: Just because a garment is the right size for you does not mean that it actually FITS you.
It will help to take your jacket to a tailor and have it adjusted to your particular shape. Tell them how you need to move in it. A good tailor should be able to raise the armpits and rotate the sleeves to accommodate the forward arm movement that you need, as well as make the jacket more flattering than you ever expected.
Jackets are not made for the movement we need when riding. They are made to look tidy and wrinkle-free when stand at the mirror with our arms hanging straight down at our sides. This is why the sleeves bind and the sides lift up when we use our arms. If a jacket is cut to give the proper arm mobility for riding, we tend to think it looks sloppy when we stand and look in the mirror, because there is a little extra fabric across the back shoulders and small wrinkles in the armpits. But when you are in the saddle, where it matters, the jacket will look tidy and proper, rather than tugging across the upper arms and bunching at the shoulder and chest.