As of right now with the current skill cycles, getting high AP with only a semi-decent amount of HP is risky. Like you said, AP has more of an impact on the card skills, which in turn means higher recoil damage from Full House. With either a pure HP or HP/HV build, you'll at least be dealing a decent amount of damage but maintaining a lower recoil damage.
Go into Chaos Tower with high AP with semi-decent HP, use Full House only to get recoiled, than get hit. You'll either be nearly dead or dead, whichever comes first depending on the stats.
AP has always been easy to stack so it has no purpose for a Charm to even stack crazy amounts of it. If you want to Elemental Melee grind, AP is still easy to stack. With a HP/HV build, you also won't have to worry about anything that decreases your AP significantly (if monsters can...can't remember).