It definitely makes a difference who you ride with. I enjoy riding with my buddy Jake a lot. He's a very experienced rider (over 15 years) and showed me the ropes a lot last year, which was my first year. We know each other's riding styles and it's always an enjoyable ride. I enjoy riding with him as much as I enjoy riding alone.
At my work there are a lot of riders. When they organize a ride, there are usually 10-15 people. Last year I went on a few rides with them and learned they all have very different riding styles, and some of them are downright frustrating to ride with.
On one trip they asked me to lead (since I knew the route really well and some of the others didn't). I said I wasn't totally comfortable in doing so since it was my first year of riding and I hadn't lead a group before. They all encouraged me and said it wasn't anything special. So, I lead the group for a stretch, only to find out it was very irritating because people didn't listen. Every time we pulled up to a stoplight everybody started discussing if we should turn because somebody knew a cool road up one way or another. One time about half the group went one way while I lead the other half the other way (down the planned route). Then when we recollected they said "I thought we were going to go up this way - didn't you hear us?" I finally got irritated enough that I gave up. It was poorly organized and I'm not so sure I'll ride with them again, even though they are my friends at work.
