In the 1001 Nights, "converse" is used as a euphemism for sex. I think it has more to do with the need to talk around the word "sex" than it does the words from many sources that are put to that use. And of course fraternities aren't related.
Though maybe "fraternite" in the motto of the French Revolution could be another reason. You're an American in Europe and you hear "liberte, egalite, fraternite" and you know what the first two mean, leaving you to guess about the third. And they are French...