Why don't you hang around your friends AND Joe? You can be with someone and have them be your everything and the most important, but still have other people there that you care about. By sectioning off and being alone all the time, if anything did happen between the two of you, you'd have nobody to fall back on, or anywhere to go to find comfort.
Play with your friends, and invite Joe, or do something with Joe and then suggest inviting others along as well; let him get to know the people that like you. You can still have ~alone~ time with him, but don't completely leave your friends behind for him.