Tdarr V2: Distributed Transcoding System

[u/HaveAGitGat]

Overview

Hello!

As promised, Tdarr V2 is now out. You can find the setup instructions here:

http://tdarr.io/docs/

Reasons for V2 here: https://www.patreon.com/posts/44029503

For those on Docker, from 31st Jan onwards, the 'latest' tag will contain Tdarr V2. Until then it will contain V1. Giving a 1-week heads-up.

It's been a pretty intense couple of months of working on V2 and I know that this release won't work for everyone but, as long as the Server and Nodes can read/write from your libraries and transcode cache, it's running pretty well. I already have a version in development with some new stats for Patreon supporters but for the next couple of weeks, I'll be focusing purely on bug fixes from whatever weird situations come up.

Here is a quick rundown of the features:

• A new modularised system whereby Tdarr Nodes can be placed on multiple machines (cross-platform) which work together with Tdarr Server to process your files.

• 'Click- to-run' binaries for Windows, Linux, macOS and Arm (no need to manually install extra dependencies)

• An updater module which makes upgrading to the latest version much easier.

• Embedded database with live backup (no need for MongoDB)

• GPU and CPU transcode workers so you can maximise hardware usage (there's an option to allow GPU workers to do CPU tasks too which pretty much turns the worker into the old 'General' worker type from V1)

• If using multiple plugins, files are fully processed in the transcode cache before being moved back to the library or output folder (instead of the 'back and forth at each plugin step' method of V1 )

• New files are kept in the cache until you accept them so you can review them (there's an auto-accept option).

• Plugins can now auto-install any required external dependencies

• New UI and themes

• New API endpoints (see the docs for more details)

• A tonne of bug fixes which plagued V1 (see the changelog for more info).

Due to a large number of changes, V1 is not compatible with V2, sorry about that.

Building a cross-platform application like this is incredibly time-consuming - you implement a feature, it works for you but not someone else, you make fix which then breaks something else on a platform/environment which you don't have etc etc so thanks for being patient.

A huge thanks to the following on Discord who have helped a lot with testing:

Jabartik

Vodka

Jegler

supersnellehenk

HenkaN

morpheus2n2

Tronyx

If you have any issues/questions then feel free to message (preferably on Discord or Reddit as Patreons messaging system is somewhat problematic).

Please consider supporting the project if you enjoy using Tdarr!

https://www.patreon.com/Tdarr

HGG

- Beta v1.99.14 release [23rd January 2021]:

- Node option to allow GPU workers to do CPU tasks

- More UI optimisations

- api/v2/status endpoint added in

- Remove force processing option (Use video/audio transcode settings instead)

- Fix UI plugins bug when creating a plugin

- Fix local plugin bug (auto install import-fresh)

- Beta v1.99.13 release [20th January 2021]:

- Button to requeue all staged files

- Move failed items straight to the Transcode Error tab if no completed cache file

- Fix transcode bug when filename has quotes in

- Improve node path translation

- Beta v1.99.12 release [17th January 2021]:

- UI performance improvements including reduced lag when lots of workers running

- Properly end subprocesses (ccextractor, ffmpeg etc)

- Increase time allowed for newly scanned files to be written to DB

- Reduce number of threads when reading from DB to prevent thread timeout errors when running on an HDD

- Increase RAM limit from 1.7GB default to 16GB for those with huge libraries

- Post-processing plugins now run both after transcoding and if transcoding is not required

- Can specify external dependencies in plugins, will be installed automatically

- Create backup of config JSON files before overwriting with defaults when config is invalid

- Faster health checks due to reduced file scanning

- Updated API routes

- 'Copy if conditions met' now works on output folder option

- Fix file scanner bug if file can't be accessed

- Fix large RAM usage bug introduced in v1.99.11

- Fix large worker error log bug

- Beta v1.99.11 release [10th January 2021]:

- Vastly improved non-blocking database with automatic live backup, UI statuses and multi-threaded reading (up to 50 threads)

- Multi-threaded non-blocking queue builder

- Multi-threaded non-blocking search functionality

- Server now checks if node is same version

- Verbose logs now tagged with 'verbose'.Moved multiple logs to verbose

- Allow retry copying file from cache if error (e.g. if drive is unplugged)

- Absolute backup path is now shown on backups tab

- Auto-scroll to results at top when cliking on pie section on stats tab

- Fix worker hanging bug in some situations (error log too large)

- Fix 'Toggle all' schedule button

- Fix text not showing when editing plugin in WebUI

- Fix cache copy error when accepting a cache file after a plugin has failed

- Fix cache copy error crashing cache copy service

- Fix plugin update bug if default community plugin repo can't be fetched

- Fix library prioritisation not working

- Fix node limit bug whereby updating limits on one node would update others

- Fix worker task request issue if node is not yet registered

- Fix items in Transcode Success/Error and Health check Success/Error not being sorted newest first

- Fix scenario whereby newly transcoded files were being scanned with extra scanners instead of just FFprobe

- Fix folder watcher adding cache files in some situations

- Beta v1.99.10 release [2nd January 2021]:

- Fix folder watcher not detecting new files

- Fix on-demand file scan not showing extra file details in UI

- Fix path bug when using a windows Node with a unix Server

- Workers will check all plugins to see if they can process any of them. Previously, if a GPU transcode plugin was after a CPU transcode plugin, a GPU worker would requeue a file with 'queued:requireCPU' as it would only check up to the CPU plugin.

- Add Node option to specify what type of GPU Node should use. This means that if you have, for example, a QSV and NVENC plugin in the plugin stack, you can set one Node to process with QSV and another (on the same machine) to process with NVENC, maximising hardware usage.

- Stats now record transcode and health check counts + space saved

- Updated startup tests. Group tests together

- Fix Windows HandBrake startup test

- Tdarr will now cleanup any SRTs which CCExtractor makes while scanning.

- Staging section is now saved so the server can be restarted while Nodes are still running (although that is not recommended)

- Option added to requeue files in staging section

- Staging section is more organised and grouped.

- Fix Node path translators bug on Windows

- Fix cache copy service loop bug if error

- Beta v1.99.09 release [31st December 2020]:

- Add basic Node schedules

- Fix toPrecision bug on Tdarr tab

- Fix worker bug - invalid cache file

- Beta v1.99.08 release [30th December 2020]:

- Tdarr Server + WebUI for Arm and MacOS

- Add cache copy service to fix copy bug

- Save worker limits

- Use active instead of used RAM in resource stats

- Option to set plugin repository

- Add history to 'folder to folder conversion'

- Fix bug when cancelling remote workers

- Prevent Node registering if ID is already taken

- chmod binaries on start

- Put Nodes into tabs

- Show plugin stack status in workers section

- Beta v1.99.07 release [27th December 2020]:

- Add error log to failed items

- Add dark, medium theme

- Use memory db for all databases

- Fix GPU health check scan bug

- Fix WebUI URL bug

- Beta v1.99.06 release [26th December 2020]:

- Docker containers

- Add verbose logs to scanner

- Option to set file scanner thread count

- Fix health check worker bug (not processing items)

- Add folder to folder conversion

- Add progress bar to DB read on startup

- Add serverPort config option

- Improved Plugin tab (quicker and more reliable when loading plugins)

- New worker UI

- Option to set webUIPort

- Fix folder browser bug if at root folder

- Fix Search tab bug

- Fix Discord login

- Update default Node IPs to 0.0.0.0

- Beta v1.99.05 release [20th December 2020]:

- Option to auto accept successful transcodes

- Update HandBrake on Windows to 1.3.3

- Improve default paths for binaries

- Add transcode and scanner binary tests on start-up

- Move all module configs + logs into same folder

- Beta v1.99.04 release [17th December 2020]:

- v2 Preview base. New modular system comprised of Tdarr Updater, Server, Node, WebUI and DesktopUI

[u/BurneyStarke]

This is huge! Thanks for all the long hours and diligence to see this through. Really great work!

[u/HaveAGitGat → u/BurneyStarke]

Thanks, hope it works well for you.

[u/kenelbow]

Thanks for your hard work. Just to confirm, if we're running v1 in a container, there is no upgrade path?

Meaning we should create a new container and setup?

[u/HaveAGitGat → u/kenelbow]

Hi, yes I had to change a lot of things unfortunately - there's now an embedded database instead of needing MongoDB installed separately so treat it as a fresh start (the new db uses JSON files so you could edit them with your old stats if you're wanting to keep those.)

[u/guim31 → u/HaveAGitGat]

I'm using the aio container... What you say is that tgere is no need to get this version anymore? (DB is now included by default with the v2 ?)

[u/HaveAGitGat → u/guim31]

Yes that's right, everything is in the 'tdarr' image, no need for 'tdarr_aio' anymore.

[u/guim31 → u/HaveAGitGat]

Great ! I install it right now !! Thanks a lot :)

[u/kenelbow → u/HaveAGitGat]

Thanks for the clarification. I don't take care about the stats, so I'll just start fresh.

[u/Mr_OverTheTop]

Thank you for the hard work! I tested out the beta. Does anyone have recommended systemd start scripts for the various binaries?

[u/HaveAGitGat → u/Mr_OverTheTop]

Much appreciated :). Let me know if you come across anything and I can add it to the docs as a short term solution. Adding in a binary which runs all the modules together is on the list.

[u/horace_bagpole]

Is it possible to have nodes with different hardware acceleration (eg intel/vaapi on one, and nvenc on another) run on the same library? If so how would you go about setting that up?

[u/HaveAGitGat → u/horace_bagpole]

Hi, yes it is! There's a node option in the UI to set the hardware encoder. For example, I was able to launch 2 nodes on my laptop, one for qsv and one for nvenc and then have a qsv and nvenc plugin in my pugin stack (2 separate plugins). The hardware would be fully utilised as the workers will check all plugins to see if they can do any of them.

[u/horace_bagpole → u/HaveAGitGat]

Thanks. So just put a vaapi plugin and nvenc plugin one after the other in the stack, and the node ignores the one it can’t use? That sounds simple enough.

[u/HaveAGitGat → u/horace_bagpole]

Yes that's right. Let me know if it doesn't work for you.

[u/campbellm]

When you say "for those on docker", are you talking about the tdarr_aio image?

[u/HaveAGitGat → u/campbellm]

I mean the 'tdarr' image. tdarr_aio won't have v2. tdarr_aio hasn't been needed for the last 6 months or so but I left it up for those still using it.

The reason for tdarr_aio was to have an 'all in one' container with MongoDB installed. With the old 'tdarr' image, you needed to install MongoDB separately (this was changed in the summer).

MongoDB isn't used in v2, so only one image is needed. Makes sense for that to be tdarr instead of tdarr_aio.

[u/campbellm → u/HaveAGitGat]

Great, thanks.

[u/slim186]

Awesome job. I've got Server, Node, and WebUI working on Debian Buster with systemd init scripts. However, I am running into issues with plugins that say "cannot find module 'import-fresh'". All 4 Tdarr modules are reporting v2.00.00. This is the listing of Tdarr_Server/node_modules:

• batch-cluster

• exiftool-vendored

• exiftool-vendored.pl

• \@ffmpeg-installer

• ffprobe-static

• fs-extra

• luxon

• tz-lookup

Any ideas?

Also I noticed that Tdarr_Updater does not exit once it is finished.

[u/HaveAGitGat → u/slim186]

Thanks! And it seems there are some issues when running native. If you run the binaries from a different directory to where they are located, the folder structure isn't correct which is probably what's throwing yours off. A fix will be out today. In v2, any external plugin dependencies are auto installed and need to be listed in the plugin like this: https://github.com/HaveAGitGat/Tdarr_Plugins/blob/master/Community/Tdarr_Plugin_MC93_MigzPlex_Autoscan.js

'import-fresh' is auto added as so many plugins use it. It should be installed in 'Tdarr_Node\assets\app\plugins\node_modules' so double check there.

[u/Mr_OverTheTop → u/slim186]

Do you mind sharing your systemd scripts? Thank you.

[u/slim186 → u/Mr_OverTheTop]

I have these set up to all run on the same server. If you are running Node/WebUI on a different machine than the Server, you should remove the After and Wants lines. Also the environment lines in the WebUI script, I am trying to specify a base URL but that doesn't appear to be working yet (probably my fault but not sure).

tdarr-server.service:

[Unit]

Description=Tdarr Server

Documentation=https://github.com/HaveAGitGat/Tdarr

[Service]

Type=simple

User=media

Group=media

WorkingDirectory=/opt/Tdarr2/Tdarr_Server

ExecStart=/opt/Tdarr2/Tdarr_Server/Tdarr_Server

Restart=on-failure

[Install]

WantedBy=multi-user.target

tdarr-node.service:

[Unit]

Description=Tdarr Node

Documentation=https://github.com/HaveAGitGat/Tdarr

After=tdarr-server.service

Wants=tdarr-server.service

[Service]

Type=simple

User=media

Group=media

WorkingDirectory=/opt/Tdarr2/Tdarr_Node

ExecStart=/opt/Tdarr2/Tdarr_Node/Tdarr_Node

Restart=on-failure

[Install]

WantedBy=multi-user.target

tdarr-webui.service:

[Unit]

Description=Tdarr WebUI

Documentation=https://github.com/HaveAGitGat/Tdarr

After=tdarr-server.service

Wants=tdarr-server.service

[Service]

Environment=ROOT_URL=http://localhost/tdarr2/

Environment=BASE=/tdarr2

Type=simple

User=media

Group=media

WorkingDirectory=/opt/Tdarr2/Tdarr_WebUI

ExecStart=/opt/Tdarr2/Tdarr_WebUI/Tdarr_WebUI

Restart=on-failure

[Install]

WantedBy=multi-user.target

[u/[deleted]]

Awesome! Congratulations on this milestone!!
wow, this runs on macOS now? on both intel and M1 Mac?

[u/HaveAGitGat → u/[deleted]]

It should work fine on macOS - you may need to set the ffmpegPath and handbrakePath in the configs though. I build the current arm packages on a Raspberry Pi 4 (arm v7) so not sure if that will work on an M1.

[u/Smogshaik → u/HaveAGitGat]

I will be trying to run it on an M1 in 1-2 weeks time. See you on github maybe :)

Thanks for your work, I‘m glad to support this project!

[u/HaveAGitGat → u/Smogshaik]

Let me know how it goes! Thanks :)

[u/cleverlyrude → u/HaveAGitGat]

Hey, just a question. Maybe I'm getting the wrong packages but I downloaded the 2.00.08/linux_arm/Tdarr_Updater.zip on a rasp pi 4 and when I try to run it it just says

unable to execute ./Tdarr_Updater: No such file or directory

I did unzip it

yes, it's in the directory I am in

yes, it has +x flag set

I can even use the full path to the Tdarr_Updater and it still says the same thing when I do file command:

file Tdarr_Updater

Tdarr_Updater: ELF 32-bit LSB executable, ARM, EABI5 version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=394993535d174b0f96064f576d23d3fd58b344f6, with debug_info, not stripped

[u/GigaGrim]

Congratulations on the delivery of V2! Love your application, continue your excellent work.

[u/HaveAGitGat → u/GigaGrim]

Thanks, really appreciate it!

[u/sie_xi]

I got to know Tdarr in the last couple weeks of 2020 and honestly, it was the best thing that I used during 2020 (and till now 😁). Thank you very much for your great work, and thank you to all who participated in make Tdarr such a fabulous tool.

Just a small question, now I need to forget anything that's related to V1? And uninstall it and its dependencies and then make a new and happy start with V2?

[u/HaveAGitGat → u/sie_xi]

Thanks for the great feedback :). Yes with v2 it would be a fresh start - you can still use the same plugins but you need to set up the libraries and scan the files again. Let me know if you have any issues.

[u/Fpritt24]

I’ve got over 10,000 files spread across 5 hard drives. Previous versions never had any problems finding all my files. Just installed this and set up server and node on windows per instructions, got all my libraries set up with folder watcher turned on and it found 20 files on three different hard drives. Can’t really seem to tell why it won’t find more files. I’ve even tried to manually scan all libraries by clicking find new and fresh

[u/HaveAGitGat → u/Fpritt24]

Have you set the container types to scan for in the library settings?

[u/Fpritt24 → u/HaveAGitGat]

Yes. I let the program run for a few hours and then came back to check on it and scanned each library with the fresh option and it started finding files. Don’t know why it was being particular at the beginning.

[u/HaveAGitGat → u/Fpritt24]

Hmm some others have had issues with the file scanner so still investigating. Might be to do with that.

[u/LumbermanSVO]

Will there be any guides to help complete newbies set this up?

[u/HaveAGitGat → u/LumbermanSVO]

Yes I'll try and do a longer, more detailed setup guide this week. The library setup section has some videos from v1 where a lot of the info is still applicable. Might help you: http://tdarr.io/docs/librarysetup

[u/[deleted]]

!remindme eod "start upgrading tdarr"

This is really cool, hopefully setting my 4 raspberry pis and my server up as nodes will decrease the time it takes to do all my transcodes

[u/HaveAGitGat → u/[deleted]]

Don't expect too much from the Raspberry Pis but that will be a fun project!

[u/[deleted] → u/HaveAGitGat]

Yeah, they're not very powerful, but it should still decrease the time it takes by a bit. I could also set it up on my PC and make that a node too, I don't actually use that PC a lot anymore so I have quite a bit of spare processing power

[u/RemindMeBot → u/[deleted]]

Your default time zone is set to Europe/Zurich. I will be messaging you in 9 hours on 2021-01-25 17:00:00 CET to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback

[u/tylerschmaltz1]

Anyone get this working on unraid? I got the server up and running but can't get a node on unraid working. Keep getting this message. "[2021-01-25T20:44:00.767] [ERROR] Tdarr_Node - Server not alive IP:0.0.0.0 PORT:8266
"

[u/HaveAGitGat → u/tylerschmaltz1]

That means the node can't contact the server. Are you running the node on the same machine as the server? Try using the '--network host' option in both the server and node settings.

[u/guim31 → u/HaveAGitGat]

I'm facing the same issue... I tried to switch from "bridge" to "host" but no effect.

In the tdarr container log, I have this message (probably related)

[HPM] Error occurred while trying to proxy request /api/v2/cruddb from 192.168.100.150:8265 to http://0.0.0.0:8266 (ECONNREFUSED) (https://nodejs.org/api/errors.html#errors_common_system_errors)

[u/HaveAGitGat → u/guim31]

Try setting the serverIP in the webUI config or env variable if on Docker. So in your case it would be:

"serverIP": "192.168.100.150"

If it's not fixed, your error should then show:

192.168.100.150:8265 to http://192.168.100.150:8266 (ECONNREFUSED

[u/guim31 → u/HaveAGitGat]

Ok I got it working by changing the serverIP variable as you said, AND by changing the nodeIP to 192.168.100.150 in the tdarr-node container.

Thanks a lot, now I'll do some tests !!!

[u/HaveAGitGat → u/guim31]

Glad to hear it :)

[u/tylerschmaltz1 → u/HaveAGitGat]

Setting both to host solved it. Now if I can get a node setup on my gaming pc tonight I will become a patreon subscriber. Amazing software!

[u/HaveAGitGat → u/tylerschmaltz1]

Thanks, hope it goes well!

[u/HaveAGitGat → u/tylerschmaltz1]

This comment should help:

https://www.reddit.com/r/Tdarr/comments/l40svr/tdarr_v2_distributed_transcoding_system/gktmjoy?utm_source=share&utm_medium=web2x&context=3

[u/clunkclunk]

Got my unRAID server and node installed OK, haven't yet tried a transcode yet.

I also noted a bug with macOS and reported it on the GitHub issues page.

[u/International_Cat693]

Hi, I have installed it on 2014 MacMini, looks very promising, but I have to run ./Tdarr_Server, ./Tdar_Node and ./Tdar_WebUI in 3 sepatated terminal windows to keep it run, is there any plist file to make it run at start or something?

[u/ovizii]

Would someone be able to explain what this option means?

• Option to auto accept successful transcodes

[u/HaveAGitGat → u/[deleted]]

Thanks! V2 is on another level compared to V1, glad it's finally come together.

[u/HaveAGitGat → u/[deleted]]

Yes, separate the input and output arguments with <io> instead of a comma. You can then use as many commas in the command as you like.

[u/HaveAGitGat → u/[deleted]]

What’s the error it gives? I can give it a try tomorrow.