If you are a 45 colt BP shooter then annealing 45 colt brass absolutely makes a huge difference, maybe even a ginormous difference in blow back which is especially useful in your rifle.
I've been annealing for 5 or 6 years now and I would not go back. Actually I did this past weekend when I forgot my rifle ammo and had to borrow non-annealed rounds. The weather was horrid, very hot and what felt like 100% humidity..something my 1860 Henry and I don't care for anyway but on the first stage by the 9th round I had to fight to lever the gun. Didn't think much of it because I was so miserable from the weather but after struggling through the second stage I took at look at my horribly fouled rifle and realized that I needed to clean and Ballistol it after every stage.
When I got home I had to take the rifle apart and clean it....the first time I did that in 4 years because my annealed brass keeps the receiver clean...I shoot at least 30 matches a year plus in the backyard and go through about 30lbs of BP a season so I think I've got enough practical experience to make this claim.
I did try various crimps, bigger bullets etc but only annealing made a difference. I also use Winchester brass which is supposed to have the thinnest neck wall.