Ah, seal breaker gems.. QQ They actually used to be easier to hunt, since you would get random gems after every run based on which character you used. (I.e. If you used Lire, you'd get a few green ones). Unfortunately, I believe they've changed this recently and now they're random drops.
Farming trial forest might not be the best place to find gems. The drops are so random that I personally can't even tell you where to start. If you're pressed for time, then just do runs as fast as possible, but if you're not, then just do normal runs and you'll accumulate them over time.