My 05 Speedmaster had four baffles.

Your bike could well have three. Apparently three isn't unusual, depending upon the year of manufacture.

If you decide to remove the third or final baffle, all baffles gone will indeed be very loud.