Hmmm... I've never seen anything but hex head bolts on any of the luggage rack/sissy bars.. and I've seen a few. Always an 8mm head on the bolt and a 10mm nut...Maybe inverted torx? Either way vice grips will mangle anything you use them on.
As far as the mounting setup in general, there are two "L" shaped brackets that attach to the rear of the three rear fender bolts on each side and the sissy bar and or luggagge rack attach to these via four 6 mm bolts with nyloc nuts... I'd love to see a pic of your setup.