This is how community membership works. I need to document all this but just getting it cleaned up is taking my time.
Anyway, starting or joining a conversation in someone's community joins you to them. It's one way so both need to join conversations.
Everyone gets two communities to start. One public. One private. Anyone can join anyone's public community. You have to be invited to private communities.