One suggestion Jimmy, when you get your chance, hold the gaff between you and the fish with the hook pointed away from you. I see guys all the time try to gaff the fish from the far side with the hook pointed back at the yak and its pretty comical....until the fish breaks off.
So if I read your post correctly are you saying that you kind swing up and away from yourself?