Quote:
... Slipping the rotor back between the brake pads, ...
That is the problem. Remove the caliper, then bolt it back up after the wheel is in place. Same goes for the front wheel too.
Have the wheel up under the fender with the chain on the chain ring. Then slide the axle through the left drop out enough to hold the caliper plate in place. Place the spacers where they belong. Then slowly lower the bike over the wheel. You'll have to shift the tire to the right as you lower the bike. Having the wheel perpendicular will allow the right spacer to clear the adjuster bracket. Once that occurs, lower the bike until the drop out holes are at the height of the wheel center. Then while pulling the wheel backwards (you didn't loosen the adjusters did you?) (that's why you have to pull the wheel rearward) push the axle through the left side of the wheel. Then look to ensure that the right side lines up. Then using the heel of you hand, push the axle home. DONE. Easy as pie.
The time it takes to remove the caliper, is less than the time it takes to readjust the chain. Leave the chain adjusters alone when removing the wheel. BTW you can leave the caliper in place, just make sure you had a Dr. graft another arm on your torso.  (or have a helper near by)
Blowing gravel off rural roads
| 
