I think it's that way for a lot of people. We generally have an easier time understanding that "oy" and "ow" are two vowel sounds, because they're written with two letters. But we think of the sound in "I" as simply a "long" i, not a combination.
I really think they should teach more about linguistics in school. I was trying to explain the concept of a diphthong to a coworker the other day, using the word time as an example, and he thought that when I was talking about two vowels, I was referring to the silent e. I had to explain that I was talking about sounds, not letters. It seemed natural to me after having so many linguistics classes, but sometimes I forget that not everybody knows all about phonology.
So, Poter, what did you call diphthongs before you learned that term?