I use 28 petites for almost all cross stitching and most beading. I will use a 26 or 24 for bigger threads used on canvas work - #8 braids, bigger perle threads and such. I don't really fit needle to fabric, I fit it to the thread instead, going as small as possible in all cases, just because a regular-sized needle is HUGE to me now
I have just moved up to 36 ct linen for a new project and use size 28 needles on that. The only thing I find is that it is hard to get a grip on them when finishing off on the backside and I have bent one or two this way. But it is easier to stitch with them otherwise on higher count material.
Have you tried the little rubber circles that you can fold around the needle and that really helps pull the needle through on the back. Can't remember what they are called. Mine came in a package of 3 they are about 1.50" and are a daisy shape.