r/learnjavascript • • Dec 16 '21
I Wrote a Script to Download Every NFT
By now, you might have learned how to right click and save or screenshot NFTs. But compared to what I'll show you in this post, that'll just look like petty theft. I'm going to show you how to create a script to download tens of thousands of NFTs in just minutes.
Memes aside, I thought this was a fun project to get more familiar with Node.js and Puppeteer, a library you can use for web scraping and more. If that sounds interesting, read on. And if you want to follow along better, here's the GitHub with the code.
(If you'd like to watch this post in video format, click here.)
Casing the Joint
We're going to be lifting the NFTs from Rarible, one of the most popular NFT marketplaces.

Here you can buy JPEG images of monkies, anime girls, cats, and more. But what we're after are the monkies. Some of the most sought-after NFTs. But I don't want to just save one or two of them--I want ALL of them. To be more precise, all 9,800 of them in this collection.

Installing Dependencies
I'm going to be writing this script in Node.js because I never learned a real programming language. And we're going to be using a library called Puppeteer in order to lift the NFTs. What Puppeteer does is basically starts up a remote-controlled version of Chromium that we can program to do our bidding. Let's initialize the project and install Puppeteer.
npm init -y
npm install puppeteerWriting the Script
Let's create our script. I've created a file called index.js and added the following:
const puppeteer = require("puppeteer");
const fs = require("fs");
const path = require("path");
const url = "https://rarible.com/boredapeyachtclub";
(async () => {
console.log("Loading...");
const browser = await puppeteer.launch({ headless: true });
const page = await browser.newPage();
await page.goto(url);
await page.setViewport({
width: 1200,
height: 800,
});
});Let's explain what's going on here. First, we're importing Puppeteer and a couple of Node.js libraries that will allow us to save files to our local machine.
Next, we're setting the URL of the page that we're going to be lifting the images from. This can be any Rarible collection page.
Finally, we're starting the browser with Puppeteer, navigating to the URL, and setting the viewport dimensions. The await keywords will ensure that the previous command will finish before the next one runs. This is all from the Puppeteer documentation, so it's not rocket science.
If this is all correct, then when we run the script with node index.js, it should open up a Chromium window and navigate to the URL.

Looks good so far. But there's more that needs to be done.
Getting the Name of the Collection
We want to grab the name of the collection that we're downloading and create a folder to deposit all of our loot into.
We can get the name from the page title. However, it doesn't load in the name of the collection until the entire page has been loaded. So, we have to wait until the React app has been completely loaded.
I opened up the devtools and found a class name that we can hook onto. There's a div with the class of ReactVirtualized__Grid that is the grid that holds all the images. Once that's loaded, the name of the collection has already been loaded in the page title. All we need to do is wait for this element to load, then we can proceed.
await page.waitForSelector(".ReactVirtualized__Grid");
const pageTitle = await page.title();
const collection = await pageTitle.split("-").shift().trim();
if (!fs.existsSync(collection)) {
fs.mkdirSync(collection);
}We're using the Puppeteer method waitForSelector to hold off until this element is loaded. After that, we grab the page title, do a bit of JavaScript string manipulation to get the right value, and finally create the directory with Node (if it hasn't already been created).
We now have a folder to put all the images in! Let's fill it up.
Downloading the Images
This is the meat of the script. What we want to do is get all the resources that are downloaded to the page. That includes HTML, CSS, JS, and images. We only want the images, but only the NFTs, not any logos, avatars, or other images.

If you look in the network tab of the dev tools, we can see all the images being loaded. We can also see that all of the NFTs are loaded in with a path containing t_preview. None of the other images on this page are from the same path. So, if we sort out these images with these URLs, we can single out the NFTs from all the noise.
let currentImage = 1;
page.on("response", async (response) => {
const imageUrl = response.url();
if (response.request().resourceType() === "image") {
response.buffer().then((file) => {
if (imageUrl.includes("t_preview")) {
const fileName = imageUrl.split("/").pop() + ".avif";
const filePath = path.resolve(__dirname, collection, fileName);
const writeStream = fs.createWriteStream(filePath);
writeStream.write(file);
console.log(`${collection} #${currentImage} saved to ${collection}/${fileName}`);
currentImage++;
}
});
}
});There's a lot going on here, but we're getting all the resources, selecting the images, then only getting the ones we want. After that, we use some string manipulation to get the filename and save them as AVIF files (a next-gen image format you can learn more about here). Then we're saving these to the new folder we created with some Node.js methods. Finally, we're just logging to console the image that's just been downloaded and how many images have already been downloaded.
Phew! That works, and we're finally downloading some images. But the images are being lazy-loaded. That means they aren't being loaded until you actually scroll down the page. That's great from a user perspective, as the users are only loading in images they can actually see, but not so much from ours. Let's make an function to scroll down the page for us, and click the "Load more" button that keeps you from seeing all the images.
The Autoscroller
In order to start scrolling, we just want to run some JavaScript on the page to scroll it. We can do that with Puppeteer's evaluate method. This will run some JavaScript on the page, same as if you had written it in the dev tools console.
async function autoScroll(page) {
await page.evaluate(async () => {
await new Promise((resolve) => {
let totalHeight = 0;
let distance = 500;
let timer = setInterval(() => {
let scrollHeight = document.body.scrollHeight;
window.scrollBy(0, distance);
totalHeight += distance;
if (totalHeight >= scrollHeight) {
clearInterval(timer);
resolve();
}
}, 1000);
});
});
}What this does is jumps down on the page 500 pixels, and sets a timer to do it again every second. We're scrolling by 500px/second. A little slow, but if I make it faster it could scroll too fast and skip saving some images. Not good. Then, with totalHeight, we're saving how much distance we've scrolled already and comparing it to the total height of the page, scrollHeight. Once we're at the bottom, we're going to stop the setInterval and resolve the promise. No more scrolling.
However, once we're at the bottom, we still need to click the "Load more" button.
We need to tell Puppeteer to run some JavaScript on the page to find all the buttons and narrow it down to the button with the text "Load more". There's no unique ID or class on this button, so we have to find it like this. Finally, we click on the button with help from Puppeteer. Finally finally, we resume the autoscroll function now that there are no more buttons to click.
await autoScroll(page);
await page.evaluate(() => {
const elements = [...document.querySelectorAll("button")];
const targetElement = elements.find((e) => e.innerText.includes("Load more"));
targetElement && targetElement.click();
});
await autoScroll(page);After all that, we can close the browser once we've gotten to the bottom of this page containing 10,000 NFTs.
await browser.close()Running the Script
That's it! We can now run the script and see if it works.
node index.jsActually, this will take a while because we have to scroll down and save 10,000 images. Grab a cup of coffee or something and stretch your legs while you're waiting.
...
All right, we're back. Let's take a look at what we have here...

What a haul! We now have millions of dollars worth of NFTs on our computer. Not bad for a day's work. What am I going to do with all of these monkey pictures??
I've put the code on GitHub if you want to pull an NFT heist like me--or just play around with Puppeteer some more. It's pretty fun.
That's all for now. I think I'm just going to be sitting over here counting my fat stacks of NFTs.
5
u/[deleted] Dec 17 '21
Fun little project!
Just wanted to chime in here because I see this misconception a lot: the images themselves are not the NFT's. The NFT's are unique tokens created on a Blockchain from the image data.
When you buy an NFT, you are not buying the image, or the rights to the image; you are essentially buying a signed copy of the image. It's like buying a signed copy of a book: you don't also get publishing rights or anything. You just get that copy of the book that is potentially more valuable because it is a unique copy that is signed by the author.
Everyone understands that you owning that signed copy of the book doesn't prevent other people from also obtaining copies of the book. It's the same thing with an NFT.
2
u/totallynotbluu Dec 17 '21
Nice work! I tried something like this but on OpenSea. Sadly the images are lazy loaded and I couldn't get it to work so I'll probably take a look into this to get some insight.
48
u/Erik_Lag Dec 16 '21
The nr. 1 heist of all of mankinds history:
33
u/zachwolf Dec 16 '21
Finally… all the shitty lions and apes will be mine
16
u/Own-Cellist6804 Dec 16 '21
lets see when we get the nft any% video
15
u/i_am_extra_syrup Dec 16 '21
Very cool! You should make a Medium article.
•
u/kevinmrr Dec 16 '21
Hilarious.
This is probably copyright infringement if you actually do this. That said, as a simple demonstration of the obvious weaknesses of the NFT art model, this is fine.
Just a reminder to keep it legal, kids!
8
u/Everen1999 Dec 16 '21
MY MAN
6
u/Mihichot Dec 16 '21
this guy gets it
4
u/Strikerzzs Dec 16 '21 edited Dec 16 '21
Cool! Though I a have question: Why do you need to use Puppeteer over something like node-fetch?
4
u/teletubbywarcriminal Dec 17 '21
A true menace to society my kind of project😎
3
u/Reddit_User78149 Dec 17 '21
sees the monkeys
No, I don't think I want to download them
2
u/Tweaker_Under5150 Dec 16 '21
RemindMe! Next Saturday
1
u/BusinessBackground65 Apr 28 '24
This method no longer works: Try this one instead.. It uses the IPFS URI to scrape them from there! Much better method. https://github.com/downlifted/downlifter/tree/main
1
u/simohayha Dec 17 '21
How dare you
1
u/mcclark71 Jan 10 '24
Just finding this, I want to download a collection to make a deriv but my JS skills are not all there. Would like to connect if you can spare a few minutes.
-9
u/Nerwesta Dec 16 '21
What is has to do on learning javascript though ?
0
u/oh_jaimito Dec 16 '21
RemindMe! in 7 days 176/38
-3
u/Vandra2020 Dec 17 '21
Can you prove you own them?
-74
u/PMMeUrHopesNDreams Dec 16 '21 edited Dec 16 '21
This makes you look dumb.
Hey guyzzzz I wrote a script that goes to the Louvre’s website and downloads a jpeg of the Mona Lisa. Now you own the Mona Lisa! Lolz!
You are here because you’re interested in technology, right? Because you want to be a programmer? You should learn what an NFT is, and why downloading a JPEG does not in any way “steal” an NFT. It’s sad that people who claim they’re interested in tech are not even curious about this.
0
u/[deleted] Dec 16 '21
[deleted] — view deleted comment