it sounds as if the sail(s) are in too tight if you are not pointing.
What's the speed like relative to other similar boats?
Have you checked leeway by visual inspection / trailing a long thin line and a small float ?
Is the underwater hull fair, and clean ?
Propellor in the optimum position ? Rudder not too heavily engaged ?
Any other drag factors - eg rope round keel ?
Have you got telltales on all sails at the important places ? Are you using them ?
I'd try to get the boat footing freely and then play with one item at a time, logging the difference it makes.
The old adage "If you don't measure, you can't manage" applies to racing as much as to business. Keep records.
I think, therefore I am. I am, therefore I sail.