To add on to what Koleos said, Affection is generally infinitely better than increasing a stat as long as they have some moves. I got an ace of hearts, but already got Light of Chaos, why would I boost their lust by 1 if getting another affection would allow me to add 1 to 6 lust every tournament. Even besides that, Covet alone is better than a flat +1 increase in one stat, because you get to choose where it goes.
You could make Affection less powerful by making it so they can only do a move once per tournament unless you learn a move multiple times, but that would make all the tournaments a lot harder as you just don't have enough ways to learn moves.
Another solution would be to make it so you can't just decide to get affection after not getting a face card, maybe you have to decide not to draw a card to gain affection. I think that is the best option, makes you want to draw for face cards but even if you don't it's possible to get a stat increase. It also fits the idea that you choose to rest instead of even bothering to try to train.
If you don't like that, you could instead roll to get a stat if it's a 6 or below, and if it's 7 or above, you can roll or get affection. Lastly, if you wanted to make it so you aren't likely to get much affection at all, you could have it so you can only get them with face cards, choosing to get affection instead of a move.