I found out a way to go up ladders twice as fast as usual.
BEFORE: Jump before getting on the ladder.
DURING: Jump off the ladder, and then go back on it. Might be difficult at start.
Hold on left or right, push jump, then INSTANTANEOUSLY push up. Do it repeatedly and fast for maximum benefit.
AKA: Hold left or right, then spam jump, up, jump, up, jump, up, jump, up, etc.
After: When you're about a body length below the surface, jump off and land on it.
To go down is a little more simple, but it only works for tall ladders. IE: Ladder going underground at Elias.
Jump down after getting more than about a body length below the surface.
To go up ladders that are short and to keep going fast...
While running right, jump up, push up to get on the ladder, and then instantaneously push right and then jump, and then hold on right. You should jump further than usual, as if you are running.
Use it as much as you can. It can give you a pretty big boost as to how fast you can reach your destination. I can climb the underworld sewer ladder (at Elias) in about 4 seconds (or less, depends on my latency) compared to 12.
Jumping down ladders is good only if the ladder is really long.