The problem is the phrase is often used to assert that a person should stop having a disparate value on all the options. Like a detective walking in to a room and a person asking, "Who are you?"
"If it's all the same to you, I'll be asking the questions here."
Or if you're on a walk and suddenly a person walking with you doesn't feel well,
"If it's all the same to you, I'll just sit down here for a minute."
Is it used in such a way that a person *isn't* asserting a position and asking you to be OK with it?