It's your network distance, internet connection, or both.
By network distance, I mean the Host-to-Party connection. Since the Host is the "host" of the entire room, you use their connection as your lag. If you're too far, well, you're basically screwed and lag a lot.
Internet connection is an obvious.
Not sure what there is to help other than closing as many programs as you can...