well its def gonna matter. just look at how many fortifcation stones it takes to upgrade and youll kno how easy or hard it will be to upgrade.
1 fortification stone: 1st degree(lvls 1-10) Should be safe from destruction at like up to +5
2 fortification stones: 2nd degree(lvls 10-20), Safe up to +4 without destruction
4 fortification stones: 3rd degree(lvls 21-30), Safe up to +3 without destruction
and so on. armor as i kno should go up to around 8th degree max. so yea, lvl/degree does matter