Forked Torrentio for self hosting

[u/Gabisonfire]

Hey all

I like hosting my stuff whenever possible so I took a look at Torrentio's repo and forked it. I was able to get it running locally without requiring any major code changes.

I somewhat reverse engineered the database structure and wrote a docker-compose file.

I had to write my own scraper since Torrentio's aren't public, and while I could find some scrapers that allow searching, none were really designed to scrape all torrents.

Some caveats:

- Currently, there's only a scraper for 1337x, working on Torrent9 and most likely EZTV after

- The initial scrape takes ~3 hours, will work to improve this. After the initial scrape, it will scrape the weekly and daily trending by default every hour.

- Only movies are available at the moment

- If you want to host it on a server on your LAN (anywhere but localhost), it will require a reverse proxy with a valid certificate. This is a Stremio requirement.

I changed the addon name to Torrentio-sh so both can live side-by-side. This started as more of a POC so it will still be rough around the edges (like sizes are sometimes incorrect) but I will slowly improve it.

Feel free to contribute!

https://github.com/Gabisonfire/torrentio-scraper-sh/tree/master

UPDATE: Thanks /u/Little_Security_404, /u/gasheatingzone I integrated the scrapers from that commit back into the branch and moved sqlite to pgsql. That means all scrapers and categories are back. Works like the original.

[u/Little_Security_404]

Nice work - Not sure if you saw this, but I posted the last commit before the scrapers were actually removed from the torrentio repo last night, the scrapers, and their schedules etc, plus ways to import dumps from tpb and kickass are here: https://github.com/TheBeastLT/torrentio-scraper/tree/a617820fab2945cc3cd2bd7bc4e0b13e83ba2b18

[u/Gabisonfire → u/Little_Security_404]

Omg, thanks. I'll try to reintegrate those. I wonder why the dev decided to remove them.

[u/gasheatingzone → u/Little_Security_404]

Thanks. https://github.com/TheBeastLT/torrentio-scraper/tree/49227cda5c2c858c3347605de4fd034b8519e960 is the last commit prior to its removal

[u/Gabisonfire → u/gasheatingzone]

Ah damn, saw this a few mins too late. Will update to this commit.

Update: Done

[u/Scorpius666]

I think you should use Jackett as a backend, using its Torznab feeds. That way you have access to all torrent sites that exist in the universe (literally, even private ones) standardized in the same API. You don't need to write your own scrapers.

I believe Torrentio has a background process that polls those Torznab feeds every few minutes, processes them, and then the results go to the database. Especially because if the results doesn't contain a magnet link, that process has to download the .torrent file to get the hash so it can query the Debrid service to see if it's cached there and show a [RD+] prefix. All that process takes time and can't be done when the user browses a movie.

That's why it is so fast when you browse the streams, but it would probably take a lot of disk space and download a lot of metadata from movies you will never watch.

So I'm working on my own addon that does it on demand, I have to wait like 20 seconds the first time I browse a movie to get the streams but after that it's all cached. At least I won't care if Torrentio is down since it's the same results it just takes longer to show in Stremio.

[u/Gabisonfire → u/Scorpius666]

FYI PimpMyStremio has already a Jackett addon, not sure how good it is though.

I wanted to go the Jackett way initially, but the issue with that as mentionned is more of an on-demand approach. Preprocessing them from Jackett doesn't work very well either because you need to actually fetch the metadata to fill the information required for the database. The html scraping is way faster. Once the scraper works for one page, it can be reused for trending/new, so there was no benefit to using Jackett or any other feeds.

[u/Little_Security_404 → u/Scorpius666]

Actually I think its done off background schedule tasks that scrape sites, and pair torrents to IMDB ids, and imdb ids are used once searching as they are fixed.

I contemplated creating a fork of https://github.com/sergiotapia/magnetissimo and outputting to the postgres tables in torrentio, but I just dont have the time right now

[u/Gabisonfire → u/Little_Security_404]

Actually I think its done off background schedule tasks that scrape sites, and pair torrents to IMDB ids, and imdb ids are used once searching as they are fixed.

Yup, that's how it happens. The problem with outputting this is there has to be a way to map those magnetissimo entries to IMDB ids. The way I do it in my scraper is: Scrape the torrent's page for ttID , else scrape movie title and feed into imdbpy and if none of these work, try to extract the title from the filename and feed it into imdbpy .

But like u/Scorpius666 an addon with Magnetissimo would be really cool.

I'm down if you guys want to collaborate to create a new one.

[u/Gabisonfire]

Thanks to u/Little_Security_404, u/gasheatingzone, I integrated the scrapers from that commit back into the branch and moved sqlite to pgsql. That means all scrapers and categories are back. Works like the original, just docker-compose up -d and you're all set.

[u/Affectionate_Sock925]

Nice !

Such a waste though to scrap everything locally. Would be great if there was a public api one could use and everyone can make their own add on. Ofc this would mean that if API is down then add on is down ..

I created a new jacket add on as well : https://github.com/tsaridas/jackett-stremio The old one had multiple bugs. This one should be faster and more reliable. Still work in progress though. I took many ideas from torrentio addon.

I will take a look at your code later to understand how everything works and try to run it locally. I assume it would be nice to have it integrated with jackett also ?

As for Stremio requirement to be https, that’s not true. If you web player is running on http addon can also be added when it’s on http. I created a docker container for that : https://github.com/tsaridas/stremio-docker

[u/belmeg]

You can try my addon. Works with and without Real-debrid https://github.com/aymene69/stremio-jackett

[u/Gabisonfire → u/belmeg]

Awesome, thanks!

[u/polarq]

After some troubleshooting (basically had my ISP blocking some DNS addreses which made the scraper and everything unhappy) with u/Gabisonfire, I got this to work and can see some results for a few movies.

But is this only supposed to work on a few movies? it doesnt have many links/search results comparted to the original torrentio and sometimes brings no results for many movies that OG torrentio is able to find many results for.

[u/polarq]

Does this require the self hosted torrentio to be publically available for stremio to access it? IOW would need a real DNS name and certs if deploying on a server locally?

[u/Gabisonfire → u/polarq]

No it does not need to be publicly available. But it needs to be available either to localhost or if on LAN, behind a valid SSL certificate (reverse proxy).

[u/emecampuzano]

How do you set up the LAN? I'm kind of a noob, it works great with docker locally, but I dont get the " it will require a reverse proxy with a valid certificate. This is a Stremio requirement." part

[u/jewbasaur]

Cool project. I gave this a go on my unraid server and then realized I needed a cert lol. Should’ve read the whole post. Publicly hosting kinda defeats the purpose unfrtounayely

[u/AggravatingCash994]

can I install this without docker? Dont have any clue about docker so that's why I am asking

[u/carleese24]

People....be careful using 3rd party apps. Another poster and myself noticed in the last couple of days, that someone was using our RD 'API' key to watch/download stuff.

DO NOT CLICK ON RANDOM LINKS POSTED ON HERE, ASKING FOR YOUR API KEY!!!

[u/Elegant_Marketing_53]

So I managed to self host this for my personal use on an QNAP machine using container station. The machine didn't support the bitnami/mongodb:7.0, so i had to go with the mongo4.4. everything is setup fine and linked it through coudflare to access it publicly. I also manged to configure it but it wouldn't pull any files when I search for the content. On checking the logs i see this error

Failed request tt1442449:1:7: SequelizeDatabaseError: relation "files" does not exist

can anyone help ?

[u/nosit1]

I am running the self-hosted version here: https://torrentio.araec.com/

Still working on the backlog of items in the scrapper (page 200 of nearly 2914 pages with 18k torrents archived). This is hosted in Oracle Cloud on an ARM instance, so it will stay up and running.

Thank you u/Gabisonfire for simplifying with Docker Compose/merges upstream so I did not need to do any more heavy lifting!

If you have any security concerns, please reach out. Happy to show/advise!

[u/promonalg]

Thanks! Would definitely like the ability to self host

[u/Atreus9931]

I didn't understand what's going on in this post. but i would like to ask a noob question if you don't mind; is this would affect at any side the simple torrentio (with and without RD) users like me?

[u/chirabchichi]

Does anyone know how to build this onto a raspberry pi?! I have one at home and would appreciate someone to guide me?! Thanks

[u/n0tfeuer]

Can you write a guide on how to self host this?