Keep the bushings.
Extra caster does wonders. Helps with tracking, highway stability, handling, etc.
Downside is that steering effort could go up if you have a lot. Power steering this is not relevent, but trying to park the truck in a parking lot with big tires and a little steering wheel will be noticable.
As for blowing out a U-joint, don’t worry about it. Just think how many people out there running 3-1/2” lifts with 7 degree bushings and they live just fine. Yes, the angle is a little steeper, but you can live with it just fine.
Caster is Camber in turns. I will try to explain. This will be a pavement discription. When you make a hard turn, the tire wants to roll under the wheel. Outside edge biting hard, inside part of the tire wanting to lift. What caster does is lay the tire over a little bit during that hard turn. It plants the inside harder and lifts the outside. When you combine this with the forces of the tire, they partially cancel each other out. This gives the front tire a better bite on the ground by loading the tread more evenly.
It is also the factor that helps the steering self center itself while driving down the road, minimizing the need to straighten the wheel at the end of a turn. That is less of the having to steer out of a turn