Hey Siggy...
You've received some good advice on this website thus far. I have the gunfighter on my 04 Speedmaster. Initially, I took my lady for a ride without the sissy bar. Her complaint was that she was afraid that she might fall/slip off. ...and I'm 5'9" 180lbs, she's 5'4" 110lbs., so we both fit just fine. I've given her friends (who are built much like her) a ride and they were worried about falling/slipping off, too.
So... I bought the tall sissy bar as well as the luggage rack. Personally, I don't use the luggage rack, but my lady does.... and I'll give ya a guess what she puts on it! Her purse--bungeed and cargo netted to the bike.
Also, something really important... when riding alone and carrying a few things, the tall sissy bar comes in VERY handy because you can cargo net whatever you have, to the tall sissy bar. Fill up a duffle bag full of whatever and strap it on. Personally, I use the Rev Pac that's specially designed to ride against the sissy bar and on the seat. You should check Rev Pack out.
Oh yeah... kind of off the subject... but it you decide to use easy brackets for saddlebags, make sure that you tell the guys at EZ Brackets that you have the tall sissy bar. Although, if you have EZ Brackets and want to remove the the sissy bar, contact the guy at EZ Brackets and request the standard bolt and fixture(only $20). You're going to need them and believe me, it makes a huge difference.
But you're right, bottom line is that the tall sissy bar takes away from the streamline look of the bike. That's why I keep a hex bolt and a socket handy, so I can take them all off when I arrive to my destination.
It's all trial and error my friend... find what fits your lifestyle.
Happy riding...
Jason