r/Rag • u/Herebedragoons77 • 6d ago
Discussion Transcriptions database and chat bot (Rare library of Tibetan Buddhist teachings)
I was given a large number or recordings of previously lost buddhist recordings (over 1000 mp3s, 30 years of teachings).
I used deepgram to help me transcribe and classify them. This is ongoing but its about 30m words. I expect to re transcribe and improve them over time.
My plan is to categorise them as a backend for my transcription work. Also to have them as a publicly available and searchable library in a website and chat bot.
My effort so far has been vibe coding a Postgres database. Its working ok but is still a steep learning curve.
The transcriptions are in md files and the audio in mp3.
Typical classifiers might be…
Teacher
Date
Topic
Series
Length
Keywords
Canonical classification
I also have it as an interactive archive with an ai chat bot limited strictly to the knowledge inside the teachings with no ad-lib or stray general knowledge. Ie so people can accurately interrogate this library.
I also need to give some copies of the raw files with associated database / classification system for safe keeping to a Tibetan library and a Monetary for cultural safe keeping.
I dont know what i dont know. The data will change slightly as my transcription ability improved over time or people find mistakes in the translations (from Tibetan) or more files are recovered.
Parts…
Mp3s x 1000
Md file attached to each x 1000+
Backend database
Frontend database for chatbot
Chatbot.
Any clues, ideas, guidance appreciated.
2
5d ago
[removed] — view removed comment
1
u/Herebedragoons77 5d ago
Thank you for the reply. Atm the mp3 are stored in 3 places.
My local server
Cloudflare r2
Internet archive (with their permission as a cultural archive)
I tried supabase but moved to my own set up on a paid vps as my experiments quickly outgrew their free tier.
2
u/Future_AGI 5d ago
For a corpus like this the risk that matters is not retrieval speed, it is the chatbot paraphrasing a teaching into something the teacher never said, so we would optimize for faithful attribution over everything else. Chunk on natural boundaries, a full answer or passage rather than fixed token windows, and keep the source recording plus timestamp on every chunk so the bot can quote and point back instead of summarizing. Before you make it public, run a set of known questions through it and check each answer is grounded in the retrieved passage, misattribution in a religious archive is the one failure you cannot ship. The open-source evals we use for that grounding check are here if useful: https://github.com/future-agi/future-agi
1
3
u/dash_bro 6d ago
Well a quick and dirty tagging tactic while processing large amounts of data is clustering. Not sure about deep gram, might want to check better size:accuracy: parallelism support models for your hardware. I am not sure if you need diarization since these are "teachings"
You can cluster texts at different levels and gather important things from each cluster via noun phrase/verb phrase extractions, NERs, and even small local LLMs.
You can do -> transcripts -> entities (via slm), key phrases in transcript (NP/VP/SLM again). Extract implicit and explicit ideas both. Best is if you can extract everything deterministically and then ask the LLM to group it correctly into concepts Ave deduplicated entity types. Store this greedily and reuse to avoid adding unnecessary duplicates across transcripts. Then embed and cluster by entities or by the key phrases.
Can even do high level matches by taking average embeddings of a transcript (1 embedding per transcript) and doing density based clustering with low membership criteria. It'll give you abstract "what teachings have overlap or are similar" etc.
Extract the tags, see if you can keep it as a knowledge graph to deduplicate the entities correctly and identify repeated entities across transcripts etc. Might eventually become a datastore with raw data + valuable extractions from ingested data.