This is another limitation of GitHub's OAuth permission scopes. Believe it not, there is no GitHub API permission that asks for read-only or list access to your private repositories.
The only time we ever "write" to a private repo is to add a webhook integration. We will never, ever modify your code. Ever. Just like you, we're developers and entirely respect the privacy of your code.
Hopefully GitHub will address this in the future by offering more granular scopes.
We've got a longer article on the subject here.
GitHub have also recently introduced the ability to limit the scope of applications to organisations. Take a look here.
Alternatively, you can create private channels that are not based on GitHub resources, this way you can create private rooms that don't map to a private repo. You can read here for more information on this.