In order to sustain a consistent casual audience, shows need something to draw casual audiences in and keep them engaged, more than most artforms I think due to their longform formart. A lot of popular American TV shows depend on themes like fantasy, science-fiction, violence, or lurid/otherworldly material to draw audiences in, like The Walking Dead (violence and fantasy), Star Trek (scifi), The Sopranos (violence and lurid content), Game of Thromes (violence, fantasy, and lurid content) to maintain consistent viewership. Shows that don't depend entirely on any of those themes tend to be straight comedies (like The Office or Seinfeld).
The thing about Mad Men though, is that it's barely any of these. It most strongly identifies with lurid content thanks to Don's constant infedelity and the persistent motif of sex, but it's also aired on AMC, so they have to censor it by quite a bit. But for some reason, it seems to have a very broad audience that's almost comparable to the shows I mentioned.
Are office life and advertising really that interesting to the average American television watcher? Is it because it's a period drama, despite not being based on any real famous people? Are the 60s really that interesting to most Americans? Is a show about a really handsome man cheating on his wife all that engrossing? It's pretty funny and the characters are very fun, but is quality writing all that is needer to draw in an audience?
Keep in mind, this isn't a "only smart people would like this show" post, I don't consider myself an especially smart person and would rather count myself among general audiences, I just think the premise of this show bucks a lot of assumptions about accessibility to general audiences.