Personally I'd use some sort of assembly lube at the contact point(I like Lubri Plate assembly lube) using straight motor oil will cause premature cam wear. Not a serious issue given the 16,000 mile valve adjustment interval and the fact that most bikes wont have more than 4 or 5 adjustments over their useful life but an issue none the less.
As far as not worrying about it if it's not making excessive noise........ bad idea. Exhaust valves work into the seat and actually loose clearance over time while (no noise) intake valves will loosen up or gain clearance over time (these make the noise). Eventually the exhaust valves will recess into the seats to the point that they will not fully close, then you end up with burned exhaust valves and a hefty repair bill.
JMHO
