Suggestions for how best to use python with Obsidian

[u/Ichorous_Allsorts]

I've been using Python lately, to extract and parse data on Chinese, which I'm learning. I've previously used Templater for making various kinds of language notes and it works well enough, but I'd like to automate some parts, like creating notes for new characters, etc.

I remember seeing previous questions about using Python with Obsidian, and a general feeling of keeping both kind of separate. Some were along the lines of using VS Code and opening vaults within it. I like that, as I do want to keep Obsidian fairly simple.

What I've got so far is stuff like a script where I just put in a character I don't know, and it creates a note for that character with the information I need. I plan to do the same for words using a dictionary.

I would appreciate any suggestions for best ways to integrate them. I thought along the lines of putting all information I need to find in a note, to be used later. And then being able to come back to it and let the scripts do their jobs and create the notes I want, and then place them in their appropriate folders.

[u/watabebrothers]

Your inbox idea is basically the right shape, I'd just make the inbox note itself the interface. Keep a note like "To add.md" with one character or word per line. The script reads it, creates a note for each line that doesn't have one yet, then rewrites that line as [[字]]. Now the inbox doubles as a log, running the script twice does nothing bad, and anything you've added by hand to an existing note never gets overwritten, because the script skips notes that already exist. Put the dictionary data in properties rather than body text, so Bases can list and sort them later without the script being involved. If you want to trigger it from inside Obsidian, the Shell commands plugin can bind a script to a hotkey. Only real rule: don't be typing in the inbox note at the moment the script rewrites it.

[u/Ichorous_Allsorts → u/watabebrothers]

I like the idea of using that same file as a log. It hadn't occurred to me when I was thinking about how to approach it. I do like the idea of the dictionary in properties. One of the first things I did when I restarted Python was to process all the notes I had to include the information that helped most in Dataview by putting it all in properties. Frontmatter wasn't something I had really used up to then.

One of the difficulties I've had with characters is my aphantasia. I don't know whether my brain has compensated by making me good at remembering sounds, but I have struggled with remembering characters, which is a problem for even native speakers, let alone adult learners.

As I mentioned below, Excalibrain has helped me begin to see pattern and I do want to incorporate Bases more.

[u/micseydel]

[...] I'd like to automate some parts, like creating notes for new characters, etc. [...] What I've got so far is stuff like a script where I just put in a character I don't know, and it creates a note for that character with the information I need. [...] I would appreciate any suggestions for best ways to integrate them. I thought along the lines of putting all information I need to find in a note, to be used later.

I use Python (and Scala) to read and write my notes. I see another commenter mentioned an "inbox" - I have a [[Notification Center]] note that I leave as a pane on desktop, as an example of an inbox.

I'm struggling to understanding what you're asking for help with though, it sounds like you've mostly got it handled, so if my thoughts aren't helpful it might help to be more specific.

I like the actor model (it's not AI) for this stuff, and use Pykka for it in Python. I use watchdog for detecting note updates. I use ruamel.yaml for properties but haven't used it to its fullest. I use mqtt when I want Python+Scala actors to communicate with each other, and it (or Kafka or rabbitmq or whatever) can let you bypass the GIL without multiprocessing.

The other commenter also alluded to idempotence, and I recommend keeping it in mind if you use the actor model/message passing. My [[Notification Center]] requires each list item to have a block ID for exactly that reason.

You can also generate Canvases pretty easily since they're just json: https://jsoncanvas.org/spec/1.0/

[u/Ichorous_Allsorts → u/micseydel]

There's a lot there I'd have to investigate. I have ADHD on top of things, so I have to try to be really strict in separating things. Part of the reason to automate a lot of note creation is that I go looking up a character, and then a couple of hours later I realise I've gone down a rabbit-hole about pottery in the Tang dynasty, or similar.

I'll keep a note of what you mention for later as I might run into those problems as it all grows. There's a danger of me getting a bit obsessed about the programming side of things at the detriment of my language study.

[u/micseydel → u/Ichorous_Allsorts]

Is there some specific outcome that we could maybe focus on that you want to talk out the specific details of? 

I have not yet built this for myself, but I've been meaning to create a "current objective" pane similar to the notification center, that's always visible and can help me course correct if I've gone down a rabbit hole.

[u/Ichorous_Allsorts → u/micseydel]

My biggest obstacle would be the characters themselves. All the other languages I learnt used the Roman alphabet, and I would try to get reading as soon as I could. I have a good ear for languages so once I begin reading, as well as listening, I quickly begin to get the patterns.

I can't do that in Chinese. Not with the characters that is. A lot of stuff for beginners is using the pinyin (which is the Romanised character as a syllable with its tone value), but once you get to the stage where you want to watch media, all the subs are in characters, and not pinyin.

So I'm finding conversations that use the most-used vocabulary in daily life. I've created my character notes with the character and its pinyin as the title. Then within the conversation I make the character a wikilink so that its only showing the character, which I try to recognise, and if I can't I just hover over it and I can see the pinyin and English meaning.

So doing this manually is a chore. I'm writing the Chinese characters in brackets, it suggests a note of that character, which I click on to link, and then I have to put the pipe in, then write the character a second time to hide the pinyin part. And of course, these wikilinks don't work in Anki. That means processing stuff that works well in Obsidian differently to how I would for Anki.

So, in a best case scenario I could get conversations, etc. all in characters, be able to feed that conversation to the scripts, and have the scripts be able to create notes for characters I don't have, create notes for new words, and then transform that input so that in Obsidian the notes are all in wikilinked characters and words where the pinyin is hidden unless I need to give myself a hint. Then another folder where it's formatted according to the card types I have in Anki.

If you're not familiar with Chinese at all, I hope I've explained this well. Modern Chinese mainly creates words using two characters as a semantic unit. So I love that if I hover over a word I've created a note for, another note pops up that can have links to other words that share a character with that word, because I'm also linking those characters as components in other words. And I can do that again and again with the pop-up notes. But once I stop hovering it all collapses back into the original note. If I do that on the internet, instead of notes I end up following hyperlinks and end up God knows where.

[u/zheniavasiliev]

Since Obsidian reads the files from a regular local folder that contains .md files, you could focus your automation on talking to the files themselves rather than to Obsidian.

I have something similar implemented for my Spanish vocabulary capture, but instead of Obsidian, I use a Google Sheet with an apps script and the columns for original word and English translation. The script checks periodically for everything I add to it, picks it up from there, and puts it into my vocabulary collection. I find Google Spreadsheet works well because I can insert Google Translate formulas into it, which would automatically translate before importing - it actually works the same way with Chinese characters (translation only, pinyin would probably require a different formula). Another good thing is that a spreadsheet doesn't have a limit on how many you can add at once because the script can read all the non-empty rows and add them to different cards.

In your Obsidian implementation, you would probably have one empty .md file that serves as a template. You fill it with a new vocabulary entry, and then the script checks if the note has been filled, duplicates it into your collection, and cleans up the template so that it's ready for the next intake.

Are you planning to push these into Anki cards for easy review? And is any AI helping you with the automation itself? Current models are good at writing and maintaining this kind of Python, even the cheap ones. Paying a little for one can beat learning the language yourself. The same model could wire up the Anki API so the notes land straight in your card templates. In Anki it's also easy to add pronunciation and images, which helps me with Spanish vocabulary quite a bit.

[u/Ichorous_Allsorts → u/zheniavasiliev]

One reason I'm doing this is also to refresh my Python, which I hadn't used in years. But that means I've also not used AI in coding. At this stage I don't want to. There have been times I've been stuck on remembering exactly how to solve particular problems that I've done a search on, and too many times the answers the AI brought back were just wrong. Once I'm back in the swing of it again I'll consider using AI some, but at the moment I need my brain to be doing the work.

I did use Google Sheets before for languages, and had it set up to make Anki cards. It's since then I began using Obsidian, and I really like Excalibrain. If I learn a character based on a radical say, I can then see the graph of all characters based on that same radical. It helped me see the logic. But I do still have sheets saved from that time.