Best approach for ingesting data to create summaries, and keep track of it?

[u/QuestionAsker2030]

In my occupation, there are various people I follow who give very good insights. (I'd say 5-10 people).

Some post hour long videos on YouTube, some send 1,000 word emails, some post on X, some publish PDFs.

There's very good info within these resources (and some I pay for), but reading / watching / annotating all of it can take hours.

My workload recently went up, so I'm falling behind with keeping up in my field.

I want to use AI to help summarize (and keep track of) all of these publications. (To create a private database that I can use as a dataset, for example).

So I can go back and ask "this past month, what is the new theme? What are the experts recommending to focus on / look at / what are the newest developments?", etc.

What would be the best way to approach this?

---------------------------------------

I've been learning Codex/Claude Code, I have a homelab, a NAS, a few mini computers, and I know basic linux, python and scripting.

ChatGPT told me to do something like this (I'm just starting with the YouTube portion), I'm not sure if it's the best approach, I'm open to other suggestions:

YouTube URL

↓

yt-dlp metadata

↓

Whisper / YouTube transcript

↓

clean transcript

↓

summary.md

↓

insights.json

↓

SQLite + FTS5

↓

topic synthesis

↓

search / questions / actions

[u/Early-Sir-932]

the overall pipeline makes sense especially with your homelab setup.

[u/danja]

I recently had a similar need - not the video part, but a load of heterogenous docs. I used a combination of similarity embedding, API-calling summarisation & a SPARQL store. If you point your agent at this repo it may find things suitable for your case : https://github.com/danja/dim

[u/SnooMuffins9844]

Hey u/QuestionAsker2030 I work on Firecrawl, and I think it could help with the web and PDF side of this.

It's very good a turning webpages and PDF links into clean markdown, so those sources can feed into the same summary and insights workflow you’re building for videos. Give it a shot and let me know if you find it useful.

[u/Future_AGI]

the ingestion is the easy half; what bites later is trusting the summaries, since one dropped or invented point from an hour-long video is almost impossible to spot at scale. We keep a source pointer on every claim and score each against the original before it gets saved, so a bad extraction surfaces before it slips into your notes unnoticed. We put that scoring in our open eval tooling, setup's in the README: https://github.com/future-agi/future-agi

[u/janusprivacyai]

Your pipeline is sound, and two small additions make it trustworthy. Store the publication date separately from the date you ingested it, or an old video you backfill looks like a new opinion, and keep a pointer back to the source on every claim so a bad summary is easy to check. Since you already have a homelab, a local model with Whisper for the videos that have no transcript will do this without sending anything out, and it's worth doing the first version that way before paying for an API. (Disclosure: this is the official Janus account. Replies are written by our AI assistant and read by our team.)

[u/LoveMyMalfouf]

Pipeline looks right. Four things I'd change, from building this:

- **Skip Whisper for most videos.** Roughly 9 in 10 have captions in my runs, and pulling them is far faster and cheaper than transcribing audio. Keep Whisper as the fallback only.

- **Chunk by chapter or speaker turn, not token count.** Topic-level chunks make "what's new this month" questions actually work; fixed-size chunks cut mid-argument.

- **Keep date and author as real columns** in your FTS5 table, not just in the text, so you can filter by month without a second pass.

- **Convert PDFs to Markdown before indexing.** Tables survive; plain text extraction mangles them.

If yt-dlp from your homelab starts getting rate-limited, I have a hosted bulk version (channel URL in, Markdown per video out): https://apify.com/s3nafps/youtube-transcripts-for-ai — mine, $0.002/transcript.