If you want to have a group YouTube party using your Chromecast, you usually need to get everyone on the same Wi-Fi network, but a recent update to the Chromecast Android app lets anyone cast to the Chromecast as long as they're in the same room.
Basically, once you enable the setting on the Android Chromecast app, devices can pair with the Chromecast using ultrasonic sound. From there, anyone can cast to your TV as long as they're nearby. The update is rolling out to devices today.
Chromecast, now with guest mode [Google Chrome Blog]