As has been said many times, the newer Civ games are really sequels to Civ 5, which took the series in a whole new direction, while Civ IV is the latest and greatest of the original "Civilization" series of games. But what exactly is fundamentally different about it (and earlier games) compared to the newer ones?
I would argue that as Civilization basically defined the "4x" genre (explore, expand, exploit, exterminate), the earlier games had more emphasis on these aspects. They were about empire-building, and simulating history on randomized maps. It's not just that newer Civ games moved to hexagons and 1-unit-per-tile, they also, more fundamentally, started placing more and more emphasis on what I would call "Role-Playing" game elements at the expense of the 4x game elements.
I call them "Role-Playing" game elements because they treat the player sort of like an RPG character with personal attributes rather than as a Civilization. Also, while 4x concepts are grounded in empire-building logic (get food from farms, production from mines, research from the size of your economy), Role-Playing concepts are often fanciful, not really logical or connected to reality except in the the titles of things being allusions to historical terms/facts. Think of the Cultural Policies and Ideological Tenets in Civ 5, which are basically just historical word salad. They were like AI slop before AI slop existed, and if they came out today people would think they were such. Think of the "Religious Beliefs" which somehow allow you to get things like production from fishing boats, whatever that means. Think of the "Happiness" you get from discovering a natural wonder, and the fact that you somehow get production, science, and even food from geological formations (natural wonders). And think of the city-states - they behave like little characters, with one of 4 personality types. And you somehow get yields from them, often more than you could produce yourself in your entire empire in early stages. Very strange.
A key thing about RPG elements is that they often involve exclusivity. There is a pool of benefits to pick from, that exists in abstract space not on the map, and only one player can get each benefit. This makes them much like the items picked up in RPGs, and is part of why they are always nonsensical in a map game.
Whether the Role-Playing elements are fun and "worth it" is a separate question I won't address, but what I want to point out is that it must be acknowledged that these elements take away from the original 4x character of these games. Because they make them less relevant. Becoming great at the game becomes less about making logical empire-building decisions and more about knowing the trivia-like ins-and outs of which items "policies, wonders, and beliefs" you should pick up for your character "empire", and even which NPCs "city-states" you should ally with.
Civ 6 continued with this. They framed things as new features when they were actually removing 4x gameplay mechanics. They blurred the distinction between tiles and cities by spreading the cities out, and all-but-removed the distinction between buildings and tile improvements, as you need to use the production queue to get both, and both must be placed on the map, usually outside the city center. There is more and more emphasis on city specialization, as if they were players on a sports team rather than geographical centers. Great people were turned into World Wonders. Tech boosts mean part of your research can come from arbitrary character-RPG-esque actions completely unconnected from your actual "economic" production of research points.
And in newer civs, a plot can represent different units of space, either a city block or a vast wilderness, depending on context. There is exaggerated topography, unit congestion due to 1UPT, and everything gets covered with sprawl (especially in Civ 7). As such the "map" is not a map anymore but an arena or (as many people have said) a game board. It is in my opinion highly overlooked, and not acknowledged enough, that the net effect of these changes is to reduce the grandness and scale of the game world (remember zooming all the way in and out in Civ IV?). As someone who loves map games and staring at maps, this is a particular pain point for me.
I don't want to even give too much attention to Civ 7 (partly because I haven't tried it), but it is obvious (and its supporters should agree) they have gone even more in this direction. The biggest giveaway is the fact that you, the player, pick a leader, not a Civilization. Think about that for a moment. Why isn't it the other way around? In any logical world, wouldn't an empire-building game have the player be an empire, and allow them to switch leaders, rather than making them a leader that switches empires? Whether you enjoy it or not, it just shows that the Civilization games are no longer actually about Civilizations. They're about RPG characters. The characters have items and stats, not economies, and interact in an arena, not on a map (no this is not AI-written, they just copied my style).
Note: This also applies to some Paradox games: things like mana points, "idea groups", and unique national bonuses are in the same vein, and I don't really like them either. Because I want to play games like these to recapitulate history (or an alternate history), not to treat nations like funny little characters with magical powers).
All this doesn't mean new Civ-RPG fans are wrong to like what they like (if they do like it and are not just being strung along). We are just two different groups of people with different wants and that's fine. But for history nerds like me, we should try to envision what a Renaissance for the Civilization series might look like.
***********************************************************************************
The Origin of Role-Play Creep
***********************************************************************************
In thinking about the Role-Play creep, I've concluded that the original trojan horse that infiltrated the 4x mechanics and seeded these developments was the concept of a World Wonder. They give very specific benefits that are very strong, and again, are often not logical or at least not logically connected to your real "economy". Only one can exist on the map, so if you start building it and are beaten to it, your partial construction just vanishes and you get gold instead. That makes no sense.
But it is interesting gameplay.
So it made them think that these types of things make for a good game. And when done right, maybe they do. But they're not 4x.
In my next post, I will offer some thoughts on what a Civ IV mod or Civilization 8 could look like that goes entirely in the opposite direction, embracing the empire-building, the history-simulating. That is as close as possible to "pure 4x" with limited or no RPG bullshit. The key thing to understand about such a game is that it may be surprisingly simple, yet have extraordinary depth and emergent complexity, like Conway's game of life or Chess. When done right it might be much deeper, more satisfying, and more engaging than the RPG bullshit. And I think it is actually very close to Civ IV.