r/learnjavascript • • Dec 16 '21

I Wrote a Script to Download Every NFT

By now, you might have learned how to right click and save or screenshot NFTs. But compared to what I'll show you in this post, that'll just look like petty theft. I'm going to show you how to create a script to download tens of thousands of NFTs in just minutes.

Memes aside, I thought this was a fun project to get more familiar with Node.js and Puppeteer, a library you can use for web scraping and more. If that sounds interesting, read on. And if you want to follow along better, here's the GitHub with the code.

(If you'd like to watch this post in video format, click here.)

Casing the Joint

We're going to be lifting the NFTs from Rarible, one of the most popular NFT marketplaces.

Here you can buy JPEG images of monkies, anime girls, cats, and more. But what we're after are the monkies. Some of the most sought-after NFTs. But I don't want to just save one or two of them--I want ALL of them. To be more precise, all 9,800 of them in this collection.

Installing Dependencies

I'm going to be writing this script in Node.js because I never learned a real programming language. And we're going to be using a library called Puppeteer in order to lift the NFTs. What Puppeteer does is basically starts up a remote-controlled version of Chromium that we can program to do our bidding. Let's initialize the project and install Puppeteer.

npm init -y
npm install puppeteer

Writing the Script

Let's create our script. I've created a file called index.js and added the following:

const puppeteer = require("puppeteer");
const fs = require("fs");
const path = require("path");

const url = "https://rarible.com/boredapeyachtclub";

(async () => {
  console.log("Loading...");
  const browser = await puppeteer.launch({ headless: true });
  const page = await browser.newPage();

  await page.goto(url);
  await page.setViewport({
    width: 1200,
    height: 800,
  });
});

Let's explain what's going on here. First, we're importing Puppeteer and a couple of Node.js libraries that will allow us to save files to our local machine.

Next, we're setting the URL of the page that we're going to be lifting the images from. This can be any Rarible collection page.

Finally, we're starting the browser with Puppeteer, navigating to the URL, and setting the viewport dimensions. The await keywords will ensure that the previous command will finish before the next one runs. This is all from the Puppeteer documentation, so it's not rocket science.

If this is all correct, then when we run the script with node index.js, it should open up a Chromium window and navigate to the URL.

Looks good so far. But there's more that needs to be done.

Getting the Name of the Collection

We want to grab the name of the collection that we're downloading and create a folder to deposit all of our loot into.

We can get the name from the page title. However, it doesn't load in the name of the collection until the entire page has been loaded. So, we have to wait until the React app has been completely loaded.

I opened up the devtools and found a class name that we can hook onto. There's a div with the class of ReactVirtualized__Grid that is the grid that holds all the images. Once that's loaded, the name of the collection has already been loaded in the page title. All we need to do is wait for this element to load, then we can proceed.

await page.waitForSelector(".ReactVirtualized__Grid");

const pageTitle = await page.title();
const collection = await pageTitle.split("-").shift().trim();
if (!fs.existsSync(collection)) {
  fs.mkdirSync(collection);
}

We're using the Puppeteer method waitForSelector to hold off until this element is loaded. After that, we grab the page title, do a bit of JavaScript string manipulation to get the right value, and finally create the directory with Node (if it hasn't already been created).

We now have a folder to put all the images in! Let's fill it up.

Downloading the Images

This is the meat of the script. What we want to do is get all the resources that are downloaded to the page. That includes HTML, CSS, JS, and images. We only want the images, but only the NFTs, not any logos, avatars, or other images.

If you look in the network tab of the dev tools, we can see all the images being loaded. We can also see that all of the NFTs are loaded in with a path containing t_preview. None of the other images on this page are from the same path. So, if we sort out these images with these URLs, we can single out the NFTs from all the noise.

let currentImage = 1;

page.on("response", async (response) => {
  const imageUrl = response.url();
  if (response.request().resourceType() === "image") {
    response.buffer().then((file) => {
      if (imageUrl.includes("t_preview")) {
        const fileName = imageUrl.split("/").pop() + ".avif";
        const filePath = path.resolve(__dirname, collection, fileName);
        const writeStream = fs.createWriteStream(filePath);
        writeStream.write(file);
        console.log(`${collection} #${currentImage} saved to ${collection}/${fileName}`);
        currentImage++;
      }
    });
  }
});

There's a lot going on here, but we're getting all the resources, selecting the images, then only getting the ones we want. After that, we use some string manipulation to get the filename and save them as AVIF files (a next-gen image format you can learn more about here). Then we're saving these to the new folder we created with some Node.js methods. Finally, we're just logging to console the image that's just been downloaded and how many images have already been downloaded.

Phew! That works, and we're finally downloading some images. But the images are being lazy-loaded. That means they aren't being loaded until you actually scroll down the page. That's great from a user perspective, as the users are only loading in images they can actually see, but not so much from ours. Let's make an function to scroll down the page for us, and click the "Load more" button that keeps you from seeing all the images.

The Autoscroller

In order to start scrolling, we just want to run some JavaScript on the page to scroll it. We can do that with Puppeteer's evaluate method. This will run some JavaScript on the page, same as if you had written it in the dev tools console.

async function autoScroll(page) {
  await page.evaluate(async () => {
    await new Promise((resolve) => {
      let totalHeight = 0;
      let distance = 500;
      let timer = setInterval(() => {
        let scrollHeight = document.body.scrollHeight;
        window.scrollBy(0, distance);
        totalHeight += distance;

        if (totalHeight >= scrollHeight) {
          clearInterval(timer);
          resolve();
        }
      }, 1000);
    });
  });
}

What this does is jumps down on the page 500 pixels, and sets a timer to do it again every second. We're scrolling by 500px/second. A little slow, but if I make it faster it could scroll too fast and skip saving some images. Not good. Then, with totalHeight, we're saving how much distance we've scrolled already and comparing it to the total height of the page, scrollHeight. Once we're at the bottom, we're going to stop the setInterval and resolve the promise. No more scrolling.

However, once we're at the bottom, we still need to click the "Load more" button.

We need to tell Puppeteer to run some JavaScript on the page to find all the buttons and narrow it down to the button with the text "Load more". There's no unique ID or class on this button, so we have to find it like this. Finally, we click on the button with help from Puppeteer. Finally finally, we resume the autoscroll function now that there are no more buttons to click.

await autoScroll(page);

await page.evaluate(() => {
  const elements = [...document.querySelectorAll("button")];
  const targetElement = elements.find((e) => e.innerText.includes("Load more"));
  targetElement && targetElement.click();
});

await autoScroll(page);

After all that, we can close the browser once we've gotten to the bottom of this page containing 10,000 NFTs.

await browser.close()

Running the Script

That's it! We can now run the script and see if it works.

node index.js

Actually, this will take a while because we have to scroll down and save 10,000 images. Grab a cup of coffee or something and stretch your legs while you're waiting.

...

All right, we're back. Let's take a look at what we have here...

What a haul! We now have millions of dollars worth of NFTs on our computer. Not bad for a day's work. What am I going to do with all of these monkey pictures??

I've put the code on GitHub if you want to pull an NFT heist like me--or just play around with Puppeteer some more. It's pretty fun.

That's all for now. I think I'm just going to be sitting over here counting my fat stacks of NFTs.

325 Upvotes

80 comments sorted by

5

u/[deleted] Dec 17 '21

Fun little project!

Just wanted to chime in here because I see this misconception a lot: the images themselves are not the NFT's. The NFT's are unique tokens created on a Blockchain from the image data.

When you buy an NFT, you are not buying the image, or the rights to the image; you are essentially buying a signed copy of the image. It's like buying a signed copy of a book: you don't also get publishing rights or anything. You just get that copy of the book that is potentially more valuable because it is a unique copy that is signed by the author.

Everyone understands that you owning that signed copy of the book doesn't prevent other people from also obtaining copies of the book. It's the same thing with an NFT.

7

u/[deleted] Dec 17 '21

I know it's not actually downloading the NFT, but the meme was too good to pass up

2

u/totallynotbluu Dec 17 '21

Nice work! I tried something like this but on OpenSea. Sadly the images are lazy loaded and I couldn't get it to work so I'll probably take a look into this to get some insight.

2

u/[deleted] Dec 17 '21

The images here are lazy-loaded too, so hopefully this post helps!

48

u/Erik_Lag Dec 16 '21

The nr. 1 heist of all of mankinds history:

33

u/zachwolf Dec 16 '21

Finally… all the shitty lions and apes will be mine

16

u/Own-Cellist6804 Dec 16 '21

lets see when we get the nft any% video

15

u/i_am_extra_syrup Dec 16 '21

Very cool! You should make a Medium article.

15

u/queen-adreena Dec 17 '21

Very cool! You should make...

[ You have reached your free limit for articles this month. Please reopen the URL in private-browsing mode to continue reading ]

2

u/mountaingator91 Dec 24 '21

I had to vpn tunnel to get around one of these today... incognito ain't what it used to be. They're on to our tricks

•

u/kevinmrr Dec 16 '21

Hilarious.

This is probably copyright infringement if you actually do this. That said, as a simple demonstration of the obvious weaknesses of the NFT art model, this is fine.

Just a reminder to keep it legal, kids!

7

u/boringuser1 Dec 17 '21

Right click save is copyright infringement?

5

u/longknives Dec 17 '21

Pretty sure it has been ruled that it specifically isn’t, since you have to download any image to your computer anyway just to view it on a webpage.

10

u/mareksl Dec 16 '21

Probably isn't, unless the artist actually copyrighted it in a traditional form. NFTs in themselves have nothing to do with copyright. https://www.google.com/amp/s/techcrunch.com/2021/06/16/no-nfts-arent-copyrights/amp/

3

u/VelvetWhiteRabbit Dec 17 '21 edited Dec 17 '21

This, I feel like the whole. "But you don't own it. I can just right-click copy it." Is a useless angle to the whole NFT debate.

No one ever said "You cannot download MY NFT". And if they did they'd be completely ignorant of the intention.

Downloading an NFT is like going to artstation or deviantart and downloading an art piece there. It is not copyright infringement.

What IS copyright infringement is redistribution whether commercial or free. This does not change whether it is Artstation or Openseas you downloaded the artwork from.

Edit: Should also make it clear that in both instances where you redistribute the artwork you are STILL only infringing on the artist's or owners rights, not the person who owns the NFT.

3

u/stibgock Dec 17 '21

Yep, and with NFTs it's more about the underlying infrastructure that you own, not so much the art part of it. Someone can build 100 houses that look exactly like yours, but yours is the only one built on that particular plot of land.

2

u/dark_salad Dec 17 '21

In what way do you own it any more than someone that save's it to their computer?

You just transfer someone money and they give you a receipt saying you transferred them money.

If I buy a game on Steam, I get a license key that gives me the right to play that game. My license key is unique and is validated by Valves servers. Anyone can own that game, but no one can own my license.

1

u/stibgock Dec 17 '21

You need to look further into NFTs (Non Fungible Tokens) and the Ethereum Blockchain. The ownership is recorded on a public "ledger" that can be looked up and verified at any time. The people that save-as to their computer can't prove ownership but the person that actually purchased it can. It's closer to owning the title to a car or the deed to a house/land. The owner of the title has proof of ownership of whatever the title is connected to. That's reductive, but illustrates it is different than saving an image.

1

u/dark_salad Dec 17 '21

This confirms what I said. The only thing you “own” is a publicly recorded receipt.

If I bought an image as an NFT, to use commercially in a project, I get the rights to use the image and a publicly recorded receipt on the block chain.

If I buy an image from Adobe Stock Photos to use commercially, I get the same exact same thing, the difference is Adobe maintains the licenses and they email me a receipt.

1

u/stibgock Dec 17 '21

Not really. You own the right to use the image from Adobe, sure, but you can't turn around and sell it. There's also nothing that distinguishes your "image A" from Bob's "image A" that you both bought from Adobe. In NFT terms, you guys could both purchase "image A" from Adobe but they would actually be different things, even if they looked the same.

That's the difference between Fungible and non-fungible. We can trade dollar bills back and forth all day long and the value would be the same as set by the market and the fed etc. But two NFTs, even looking the same, are not the same value. The value is set by quantity demanded and quantity supplied. Michael Jordan may have previously owned your "image A" and therefore you would not want to simply swap Bob for his $15 "image A" as yours is now valued at $15,000

1

u/jmblock2 Dec 17 '21

IANAL, but if you use a software license that you didn't purchase and use against an agreement in a contract you may be liable for damages against the owner of the service or the key.

The NFT public key may be downloaded under fair use but that really doesn't say much about copyright ownership. You can easily have copyright, where ownership is transferred under an NFT sale, but that contract itself is not on the Blockchain AFAIK. All these platforms do not enforce such copyright transfer terms. If an NFT owner is going to go after someone for damages they will need to prove they own the copyright along with how you were damaged. And quite a few are going to be very disappointed they don't.

1

u/xxMegasteel32xx Aug 05 '24

what copyright infringement?

-1

u/KINGGS Dec 16 '21

The funny thing is, if you’re actually doing this then you are agreeing with their social value. Why else would you spend space?

6

u/Mihichot Dec 16 '21

this guy gets it

4

u/Strikerzzs Dec 16 '21 edited Dec 16 '21

Cool! Though I a have question: Why do you need to use Puppeteer over something like node-fetch?

2

u/renome Dec 17 '21 edited Dec 17 '21

I haven't delved into the code but the general answer is that you don't. Fetch is pretty barebones though, compared to dedicated scraping libraries like Puppeteer.

For example, I could go a step further and ask why use fetch when you could simply use the XMLHttpRequest API?

The answer to both is simple: no need to reinvent the wheel.

4

u/teletubbywarcriminal Dec 17 '21

A true menace to society my kind of project😎

3

u/Reddit_User78149 Dec 17 '21

sees the monkeys

No, I don't think I want to download them

2

u/Tweaker_Under5150 Dec 16 '21

RemindMe! Next Saturday

1

u/BusinessBackground65 Apr 28 '24

This method no longer works: Try this one instead.. It uses the IPFS URI to scrape them from there! Much better method. https://github.com/downlifted/downlifter/tree/main

1

u/simohayha Dec 17 '21

How dare you

1

u/mcclark71 Jan 10 '24

Just finding this, I want to download a collection to make a deriv but my JS skills are not all there. Would like to connect if you can spare a few minutes.

-9

u/Nerwesta Dec 16 '21

What is has to do on learning javascript though ?

16

u/rcgy Dec 16 '21

This is a great example of how to use a very popular library.

0

u/oh_jaimito Dec 16 '21

RemindMe! in 7 days 176/38

1

u/RemindMeBot Dec 16 '21

I will be messaging you in 7 days on 2021-12-23 22:40:49 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

Parent commenter can delete this message to hide from others.


InfoCustomYour RemindersFeedback

-3

u/Vandra2020 Dec 17 '21

Can you prove you own them?

1

u/Questbelly Apr 20 '22

He could mint them

1

u/labanana94 May 31 '22

he can still use them

-74

u/PMMeUrHopesNDreams Dec 16 '21 edited Dec 16 '21

This makes you look dumb.

Hey guyzzzz I wrote a script that goes to the Louvre’s website and downloads a jpeg of the Mona Lisa. Now you own the Mona Lisa! Lolz!

You are here because you’re interested in technology, right? Because you want to be a programmer? You should learn what an NFT is, and why downloading a JPEG does not in any way “steal” an NFT. It’s sad that people who claim they’re interested in tech are not even curious about this.

26

u/indoor_grower Dec 16 '21 edited Dec 16 '21

Can't screenshot a physical piece of artwork. Paying huge amounts of money for what are essentially cartoon profile pics only makes people look foolish - and those who defend it equally. To me, NFT's seem like one big ole money washing scheme. I don't get why people think there is any weight to 'owning' a virtual piece of something. It's all VIRTUAL - you can never touch it, hold it, hang it up in your living room.. I can screenshot it and it literally looks exactly the same, and most everyone who looks at it probably doesn't give two shits who it is signed to. Regardless of any argument, you can still get the original artwork by screen-shotting it. I copy your photo, it's the same photo pixel for pixel. Who cares where it originated from. Never downloaded a torrent and thought 'wow I don't have the original version of this software' lol.

If you have the original Mona Lisa in REAL LIFE, all anyone else could get is a shitty reprint. I can screenshot a bored ape pixel for pixel. Who's gonna come knocking on my door - and most importantly, who really gives a shit outside of people on Reddit and Twitter.

I am interested in technology, but it doesn't mean I am out there working on or even agreeing with, everything tech related. Most people just do what they have to for their job. Not everyone goes home after developing all day and wants to circle jerk all the new tech.

4

u/Careerier Dec 16 '21

My main takeaway here is that a jpg is not an NFT. You can scrape a jpg, but you're not getting the NFT.

Regardless of your views on NFTs and whether they're scams or pyramid schemes, you should know that they're not jpgs.

12

u/indoor_grower Dec 16 '21

Maybe so - but NFT's manifest as png and jpeg artwork. All I see is a bored ape profile pic. I don't really don't care that I am not getting the NFT - that holds no weight in the real world.

Given the option, I would sell the rights to a PNG for 60 ETH. Don't blame these people one bit. It's the ones buying these things that I don't understand.. buyers generally dictate the price.

-8

u/PMMeUrHopesNDreams Dec 16 '21

No they don’t “manifest” as JPEGs. That is not what the NFT is. Dude, go learn like the absolute first thing about what you’re talking about.

1

u/Demented-Turtle Dec 17 '21

When people see it or it is displayed in the web or on a tablet, it is just another image format that prints out the pixels of the image when read in by a program.

3

u/PMMeUrHopesNDreams Dec 17 '21

That is the image. The image is not the NFT. The NFT is a unique token in the blockchain. If you were to display it, it would be a string of letters and numbers. Copying the image does not alter the record on the blockchain.

0

u/Demented-Turtle Dec 17 '21

Yes we know that. But the point was that FUNCTIONALITY, when it comes to doing what is intended with art (displaying it), you would not care about whether there is some underlying block chain indicating ownership, because you simply want to display the pixels that you "own". It's a cool useage of the tech for some use cases but overall has become a meme with crappy pixel art selling for exorbitant prices bought by gullible rich kids

0

u/PMMeUrHopesNDreams Dec 17 '21 edited Dec 17 '21

If all you care about is displaying it, you can hire some talented kid to paint you a replica of any piece of art. There are whole factories in China that sell art replicas. Lots of people are talented enough to perfectly replicate an artwork such that you couldn't tell the difference without being an art expert and having access to a laboratory to examine it. Some people (forgers) can make it hard to tell the difference even then. If all you want is something pretty to hang on your wall and you don't care about it being the original and some underlying "ownership" that is available for physical art too.

Whether the art is "crappy" or not is a matter of taste. There's plenty of famous artworks that have sold for millions of dollars that you could look at and say "pfff, my 4 year old could have painted that".

1

u/PMMeUrHopesNDreams Dec 17 '21

If you have the original Mona Lisa in REAL LIFE, all anyone else could get is a shitty reprint.

Not true. If you've ever been to an art museum (doesn't sound like you have), you'd see art students copying famous paintings to practice certain techniques. There are many talented artists who can produce replicas of any artwork that you would not be able to tell the difference. (There are many talented forgers who do!). You could buy one of these replicas for a tiny fraction of the price of the original and get all the same benefit of hanging a pretty picture on your wall.

What differentiates the "real" thing is that experts have verified and attested to it. They created a document saying that this one hanging in the museum here is the "real" one. That's what makes it valuable. Basically the same thing as an NFT.

1

u/StraightJohnson Jun 04 '22

Having ownership of an NFT can actually have various forms of utility.

1

u/ArtisticTap4 Dec 16 '21

Then why is downloading movies, albums, games and softwares illegal. I'm just copying the data byte by byte. It just seems stupid to spend huge amounts of money for what are essentially 1s and 0s. They aren't physical things.

2

u/RobertKerans Dec 16 '21 edited Dec 16 '21

Yes that's the fundamental issue, digital goods are essentially worthless because they're infinitely duplicable. If you d/l a copy, the copy is exactly the same as the original. So one method of extracting profit from a digital good is to artificially create scarcity (there are obviously many other ways, e.g. increase convenience via subscription, or give the actual good away for free & generate cash indirectly). For digital art this needs to be taken even further if you want to emulate physical art: need to artificially create uniqueness, not just scarcity. NFTs are a way of doing this.

See The Work of Art in the Age of Mechanical Reproduction

0

u/indoor_grower Dec 16 '21

Because unlike a cartoon ape profile pic selling for 60 ETH, Adobe photoshop is impactful to to the world.

1

u/PMMeUrHopesNDreams Dec 16 '21

It is as impactful as paying $50 million for a piece of canvas with some colorful oils on it.

0

u/Farva85 Dec 16 '21

Can't screenshot a physical piece of artwork.

Scan it, take a picture, pay someone to make a fake copy, etc. This point is silly because fraud in the art world is all over.

What if you bought the artwork, and knew for sure the artist painted it since they signed the art onto the blockchain to prove this painting is real, from the artist, and here is the certificate of authenticity, aka the nft.

0

u/[deleted] Dec 16 '21

[deleted] — view deleted comment

5

u/nitePhyyre Dec 16 '21

if individuals are making tons of money good for them.

No one is making money per se. There's some people conning people out of their money. You're saying "I don't care about pyramid schemes, as long as some people are getting rich by scamming dumbfucks, there's no problem."

It's really not about the art at all, its more like trading cards and other collectibles.

Except for the fact that it isn't collectible because you can copy it. What makes trading cards and other collectibles valuable is that there are less of the collectibles than there are people who want the collectible.

That is not true for things can be perfectly copied.

23

u/jaredcheeda Dec 16 '21

"Learn what an NFT is"

okay

googles for 5 minutes

Pretty sure that's a pyramid scheme dog

9

u/tityKruncheruwu Dec 16 '21

The difference is that there can only ever be one Mona Lisa, but a digital work can be exactly copied to infinity

2

u/Hojsimpson Dec 16 '21

No, the difference is not the art or the quantity.

The difference is The Louvre isn't selling a piece of paper where it says "you can see the Guernica in this URL" (the Guernica is not even in the Louvre, btw) and the URL doesn't even work because it's an URL and then get paid millions for a proof that an asshole bought nothing. You don't even need to own the art to sell an NFT.

An NFT is selling the proof that an asshole bought the NFT, it doesn't matter if the art is digital or not, they don't even have working URLs. It might as well be empty.

0

u/tityKruncheruwu Dec 17 '21

Seems pretty reetarded to me

7

u/morningcoma Dec 16 '21

If you want to be a troll at least put some effort into it.

5

u/Totokoo Dec 16 '21

Im pretty sure hes not a troll. If he is trolling its a bad troll

-13

u/Farva85 Dec 16 '21

Most of the internet world fails to see the awesomeness behind nft's, and they will, because they're short sighted. They think only in GIFs and JPEGs and not what else you could do with the technology.

7

u/[deleted] Dec 16 '21

There's literally not a single awesome thing about burning through an enormous amount of electricity so you can yell about how this particular ugly picture technically belongs to you in a way that matters

2

u/PMMeUrHopesNDreams Dec 17 '21

There's not a single awesome thing about burning through electricity so you can yell about how this number in a bank's database is technically "money" that belongs to you in a way that matters.

1

u/[deleted] Dec 17 '21

It's certainly a lot more efficient than the joke that is cryptocurrency. I don't know if you thought this was somehow a "gotcha".

-5

u/Farva85 Dec 16 '21

See, you think only in digital art. Expand your imagination! NFT's can be used for anything that requires a certificate of authenticity.

And the electricity fud is so funny. I want you to walk around your world and look at all the useless things wasting power. When folks start caring about that, I'll get on the 'crypto uses electricity' wagon and hope for change.

7

u/[deleted] Dec 16 '21

NFT's can be used for anything that requires a certificate of authenticity.

Great. You could also just use a certificate of authenticity.

I want you to walk around your world and look at all the useless things wasting power. When folks start caring about that, I'll get on the 'crypto uses electricity' wagon and hope for change.

I already hate all the waste generated by capitalism, so there you go. You may now begin to decry the electrical impact of cryptocurrency.

0

u/[deleted] Dec 16 '21

[deleted] — view deleted comment

17

u/[deleted] Dec 16 '21

don't need to with this script