I paddle four kayaks, two without rudders (ok, one is a surf kayak...), and I can tell you that paddling is DEFINITELY more efficient with a rudder. As Doug mentions, you track better, and you can compensate for an uneven paddle stroke due to injury, misplacement of the hands on the paddle, bad technique, etc.
I would not consider a fishing kayak without a rudder unless I was just looking to piddle around in a small bay or pond.
-Brent