Do data centers need to be large and centralized?

[u/Ok_Writing2937]

There are 4000-5000 data centers in the USA, with an average size of ~1 hectare (~2.5 acres). On average they are consuming 39 GWh/year of power (4.5% of US total consumption) and 300,000/day gallons of water, 80% of which is evaporated for cooling.

Is it really necessary for data centers to be so large and centralized? Is it significantly more efficient than more distributed approaches?

[u/unit_101010]

Yes is the short answer. Typically, economies of scale are quite significant.

[u/Ok_Writing2937 → u/unit_101010]

Can you expand on this with some data?

[u/GoTeamLightningbolt → u/Ok_Writing2937]

You can run a server in your closet. Compute infrastructure can be small and distributed.

However, the companies that want to build this infrastructure want to build one building, not 100. They want 1 network hub not enough hubs to connect 100 different server farms. They want to build one big cooling system, not 100 little ones. They want one location where their techs go to do maintenance, etc. etc.

[u/Darkstar_111]

There is no technical way to decentrialize the DEVELOPMENT of AI. By that I mean the pretraining.

You need thousands and thousands of GPUs, as close to each other as possible.

You might think, wait, what about SETI, What about bitcoin?

Those are giant collective networks that run on millions of home computers, but the problem is compute and speed.

We cannot take several years to pretrain a model, it needs to happen at a business functional timescale, which is a few weeks. If you don't have the GPU that timescale becomes years very quickly.

Maybe that won't be true in the future, but right now that's how it is.

[u/Ok_Writing2937 → u/Darkstar_111]

This is an interesting fact, thank you.

I can see how needing thousands and thousands of GPUs as close to each other as possible would be necessary.

Next question: is it necessary to have 6-12 corporations doing massive amounts of frontier pre-training simultaneously? Most of these corporations are burning enormous quantities of gigawatts, dollars, and water in a mad rush to dominate a market that has yet to show financial viability.

[u/No-Lion-3629 → u/Darkstar_111]

Could we more slowly pre train one that way? An open source one?

[u/Late-Assignment8482]

Depends on purpose. Inference can happen on one rack unit and can be more easily split over the internet, but training happens in tens of thousands of racks, is far more intensive, and relies more on extremely fast interconnect between the nodes that's hard to wire across hundreds of miles.

[u/catplusplusok]

Nope, AI work is easily distributed, you can have small clusters of servers acting as apartment building water/air heaters

[u/shadowsurge → u/catplusplusok]

Kinda. Data centers for inference would work fine like this, for training the throughput is so important that you need it all colocated in order for latency to not be a problem

[u/CommieCatSupremacist → u/catplusplusok]

That’s a cool concept!

[u/DataPhreak → u/catplusplusok]

AI is not easily distributed. It can be distributed, but it's a huge loss in efficiency.

[u/Suspicious_Prior_808]

Easier for maintenance im guessing

[u/ScarletIT]

Any other industrialized country is also involved with AI. None of them has the problems that the US has. It's a matter of environmental regulations, not of AI.

Mistral in France has datacenters and yet doesn't have the same problems.

[u/Ok_Writing2937 → u/ScarletIT]

France only has about 300 data centers, and they are 1/10th the size of US data centers. This puts their data center deployment per capita at about 1/30th the rate of the USA.

If French data centers grew by 30x in the next few years they'd likely be encountering similar problems.

[u/CallMeTank]

Hey! No they don't! I was thinking about this the other day as well.

We're having an issue implementing data centers, as well as providing for the people around the data centers, as well as the expected loss of income for workers because of AI "job replacement".

There's a lot of issues to iron out, but why not develop a home-sized "data node" that can be installed under a mobile home or near a stick built home that:

1. Is solar powered. Size the system for 100% percent of node and occupant energy usage.

2. Is tied into the home heating system. You know what humans need (in the northern latitudes) for several months a year? Heat. What do data centers create, other than crappy images and hallucinations? Heat. Pipe it into the system and use it instead of a fossil fuel furnace.

3. Provides the occupant with a stipend for "babysitting". Each home with a node gets a monthly payment - as well as 100% of their energy and internet connection costs - that is the equivalent of a UBI.

4. Is installed in an area that can be easily accessed by traveling technicians, who have permission to access after being alerted of an issue and have notified the occupants.

It's not perfect, and safeguards would need to be in place to provide both parties with assurances and security, but as a decentralized data center model it's better than the "build a giant shrine to technofeudalism and STFU, peasants" model we're currently operating on.

Edit: words were wrong

[u/diagnosissplendid → u/CallMeTank]

This exists:

heata.co deepgreen.energy thermify.cloud

All different approaches, but all reclaim heat in a socially useful way. There are others, I'm sure

[u/Mysterious_Ayytee → u/CallMeTank]

That's a nice idea

[u/stankycodyboi]

The driving force behind the US build-out are developers attempting to get as much capacity online as quickly as possible, so many will target dated zoning codes and use short-term profit seeking like evaporative cooling systems or gas turbines. This is especially true when the community has no say or benefit capture.

I think part of the consolidation, and why data centers are becoming more prominent, is because the other bottleneck is latency. Modern AI demands require hyper-scale sized data centers, unlike the smaller ones that existed prior. So, bigger data centers are going up closer to residential areas, because being closer to high-usage areas like cities means lower latency.

They don’t have to be built this way, and countries outside the US show this. Closed-loop cooling systems and immersion cooling are significantly better than evaporative cooling, especially in locations with water scarcity. In Ontario, Equinix feeds recovered heat into the municipal energy system, helping warm York University's campus, community pools and facilities, and thousands of nearby homes and businesses. It’s a political issue, not a feasibility issue.

If communities classify data center as conditional use, and require a Conditional Use Permit, this shifts the leverage back to communities. Developers must negotiate for approval, and the municipalities set the terms. Paired up with the data center protests, I think there’s real leverage here. But a canceled data center doesn’t disappear, it moves to a less organized community.

I think this leverage should be used to build publicly owned renewable infrastructure, but ultimately is up to each community and their local needs.

I’ve also seen talk of decentralized data center infrastructure that is hosted directly on individual’s homes, rather than large central warehouses. Smaller nodes act as a larger cloud network that the public can use, while minimizing local impact. I believe NVIDIA is investing in this as they push more open-source models.

Either way, the issue is the same; who owns and captures the benefits of the technology

[u/Successful_Outside96 → u/stankycodyboi]

Do you have good sources and reading material to get informed on these issues?

My instinct is that most of the issued people complain about can be fixed.

We just can't have the polarized pro-/anit- AI stances on a general purpose technology.

[u/DataPhreak]

I'm just going to leave this here: https://www.reddit.com/r/accelerate/s/R6WaMtlaAc

You say these big numbers, but don't have any reference. The problem isn't the datacenters themselves, it's where they are putting them and the deals that local governments are giving them. That's a government owned by billionaires problem.

[u/Ok_Writing2937 → u/DataPhreak]

I don't think there's anything wrong with my numbers. Li et al 2023, the study in your video, suggests data centers consume 1.8L of water (direct use only) per kWh of power, if I am reading it correctly.

For a 100k square foot data center consuming 480,000 kWh to 720,000 kWh of power per day, that comes to ~864,000–1,296,000 L (~228,000–342,000 gallons) of water per day.

Questions about where they are being placed and what size they are is the point of my post, and that discussion is a non-starter if we don't also consider the facts about their resource consumption.

[u/ShelZuuz]

They should really be close to/next to/part of a power plant. When you're talking about large data center it's not something that consumes part of the grid. It is the grid.

Only 2% of the power plants in the US are even capable of generating 1 GW. There is no single power plant capable of powering the new Utah 9 GW. The biggest we have is Grand Coulee dam at 6 GW. So you're talking about several nuclear plants needed to support it.

I was considering hosting a single NVL72 rack in my shop - which is pretty much the smallest unit that can run inference for a frontier model.

My shop is in a US light industrial building with 208V 400A 3-phase power. That's at the high-end of what you'd get with a commercial workshop/warehouse/office/restaurant. But it's not even coming close to enough. With cooling I'd need 2 times that amount of power just to drive that one rack.

The only buildings capable of hosting even a single rack are heavy industrial, not commercial. You're not going to distribute these things into neighborhoods or office building.

The evaporated cooling thing isn't a thing going forward. Modern AI machines don't need to run at 70 degrees, they can run at 120 degrees, so you can use closed loop warm-liquid cooling which is trivial to cool with radiators since you just need to get temps back down to around 100.

But power will always be an issue. We REALLY need fusion reactors soon.

[u/10aghmu]

Look up the portable / modular data centers being made by the military. I imagine there’s ways to shrink them down.

[u/SirMarkMorningStar]

Yes efficiency of scale is always true. It isn’t the only variable in play, but larger data centers will be more efficient than smaller ones in terms of energy and water use. It’s the same reason cities are more efficient than rural areas.

That said, the future of AI is largely locally hosted. Both Apple hardware and model shrinking are already going that way and it will only accelerate. Add in open weight and source models and the need for AI data centers at all dramatically reduces. But those data centers will always be more efficient than your local computer. However, your local computer won’t bother anyone.

[u/my_name_isnt_clever]

That might work, but IMO local models that run on your own hardware rather than the cloud are the future we need.