It's right for once. The relative pronoun "whoever/whomever" fills two grammatical slots—two object slots or an object and a subject. In your sentence, it's the subject of the clause "is in charge of scheduling this" and the object of the preposition "to." In these cases, the traditional rule is that the subject role trumps the object role.