You just have to change your "take everything off" mentality if you are going to join the ranks of the true trailer sailor who raises and lowers every time they go out.
My mentality is leave absolutely everything possible in place. When I lower my mast the fewest lines possible are slacked to allow the mast to be rolled to the pulpit. Once secure there nothing else is removed. The mast raising lines stay in place bungee'd up to the mast. The pole stays right where it finishes. Sheets, spreaders, halyards, nothing is undone or removed, all are simply secured in place for the road. You can save all the spreader lifeline hassles with a simple set of keypin shackles that allow the lifelines to be slacked so the mast can set right over them.
The goal is to optimize the process so the next time you show up at the ramp you have as little setup as possible. When I pull into the parking lot I release about 12 of my special BWY bungee's, roll the mast back to the mast step, and crank away on the raising tackle. I have to attach nothing to do this, it's all in place, adjusted to the right length and ready to go because I left it all there the last time I lowered the mast.
Keep the furler bungee'd to the mast with at least one bungee at a height you can reach and you will not have any issues with the mast twisting or being pulled to the side when rolling it back and inserting the bolt or when raising it.
Likewise limit the amount of things that have to be stored or removed from the boat as part of the setup and teardown process. There are only 4 things I remove when setting up. The tiedown strap, the furler wood support, and the two quick pins that lock the rudders in the up position. These are removed and put in the truck before I even climb on board to start the setup process.
Likewise I never remove my baby stays or my mast raising pole. Once the mast is up the pole simply folds up against the front of the mast and is secured there with a bungee while out on the water (Note the X raising system is more conducive to this than the M raising system). That way when it comes time to lower it everything is in place. It also is handy to have this in place if while out you need to lower the mast for maintenance or a low bridge.
I usually have two people doing the setup, myself and my wife or one of my sons. I have a setup and a teardown checklist that lists all the steps to be done in order and assigns various ones in tandem to each of the people doing the setup. That way there is no waiting to do your task because the other person hasn't done there's.
Build a custom one of these for your boat and use it each time.
Here is mine,
http://www.ddunn.org/BoatProcedures.pdf .
Likewise there is a page for each step in the tear down process. Two copies of each are on board in plastic sleeves for easy reference. We used to use them a lot, but now we know the routine and each just work in tandem to get the boat quickly in and out of the water.