From what I am told, if you used points across the 4 trees at the point you are at, it would be better for you to use a Reset Capsule and focus on one tree. Elemental Mastery(s) are important to help improve that damage output and Magic Upgrade from the Misc can help as well.
If you have any Dagger skills...they could result in why you aren't getting the desired damage out of your fire skills. I would say trade points into Fire tree or Magic Upgrade to help out. If you want to use daggers skills...or just dagger, maybe the Fire Enchant buff will be useful to you.
If you are really on the fence about changing element try these posts:
(I linked a specific post but look throughout the thread for more.
(a few testimonies from elements people use)
I would say stick it out with Fire. Try not to be detoured by people using other elements and combos. If it is really the element you like, try to learn it the best you can. I can't say I seen many Fire mages to compare damage with. However, I can say that the damage difference between Wind and Earth Magic is quite a bit and I heard fire is even bigger. As far as combos, Wind is the best.
Hope this helps.