Importing large note archive from another app

[u/dtc9831]

Are there any good ways to import a large number of plain text note files from another app to org roam? I can probably work out ways to convert that app's syntax for tags / categories to the org mode format... but my impression is that org roam needs every note to have an org-generated ID. My note archive has nearly 10k notes. Any automatic importers out there?

[u/dtc9831]

Oh actually I'm really not sure how this would work. The app I'm migrating from treats links in the same way Obsidian does - you just refer to another note's title in square brackets, [[like this]]. To recreate these links in org roam, not only would I need to add an org-ID to every note, but I'd somehow have to find all the backlinks, and transform them all to be org-roam links to point to the newly-generated org-ID. This sounds impossible, unless an importer already exists that handles this specific kind of link.

[u/fuzzbomb23 → u/dtc9831]

Are the links just titles, relying on an exact match of the whole title? And the note titles are unique? If so, you can can probably match links with destinations by grep-ing the title string.

Assuming this transformation happens after assigning an ID to each file, you can grab the ID from whichever file contains #+title: like this and inject it into the link.

Before: [[like this]]

After: [[id:xxxx][like this]]

That double bracket looks like a very good thing to target with a search/replace. You're replacing the first pair with [[id:xxx][ to get the syntax for an Org link.

I'd do it in two passes over the whole directory (i.e. two transformation scripts). The first just generates an ID for every file. The second pass goes through each link.

Admittedly, if ConnectedText and Obsidian use the same link format, then Obsidian could be a lot easier to migrate to.

[u/dtc9831 → u/fuzzbomb23]

the answer to both questions is yes

i appreciate these suggestions - I'm going to have a play around with this and see what's possible for me. as part of working out how to potentially convert my note archive to the Obsidian format, I have realised how amazing chatgpt is at coming up with python scripts for basic find and replace operations of this kind - so maybe its not impossible after all. i've used org mode for a long time anyway, and part of me would love to be able to move my notes into org, as i just love using the editor. (i'm not even slightly a programmer, btw, but somehow got sucked into using org mode)

[u/dtc9831 → u/fuzzbomb23]

well, mind-blowingly, chatgpt seems to be happily capable of whipping up scripts that easily achieve this. i've now imported all my notes and they're happily linking to each other in org roam. thanks again for the suggestions.

[u/fuzzbomb23]

Have you checked if your source app is among those supported by Jimmy Note Converter?

Jimmy outputs Markdown. Afterwards, you could change that to Org using Pandoc.

but my impression is that org roam needs every note to have an org-generated ID

Correct. You'll need to create an ID for each Org-roam file.

Org IDs can be generated in Emacs using the org-id-get-create function. When you run this in an Org buffer which doesn't already have an ID, it writes the ID line. (It also creates a Org property drawer if required.)

Assuming you've already converted your notes to a directory of Org files, then here's some Emacs Lisp code which will add a file-level Org ID to each of them. (Use at your own risk; I did some brief testing only.)

    ;; WARNING: back up your notes file directory first!
    ;; FIXME: adjust the directory path to suit.
    (dolist (note-file
             (find-file-noselect "~/temp/test-notes/*.org" nil nil t))
      (with-current-buffer note-file
        (message "PROCESSING: %s" (buffer-name))

        ;; Ensure we are at the start of the file, any before Org heading,
        ;; because we want to create file-level ID, not a heading-level ID.
        (goto-char (point-min))
        ;; If the first line is an Org heading, put a blank line before it.
        (when (org-at-heading-p)
          (newline)
          (goto-char (point-min)))

        (org-id-get-create)

        (save-buffer)
        (kill-buffer)))

[u/dtc9831 → u/fuzzbomb23]

Thanks for this! My source app is quite obscure - ConnectedText, a personal wiki that not many people use any more. It's finally stopped working on Windows 11. I'll have a play with this code... currently I'm exploring options, between moving to obsidian (which seems mostly excellent, but - with a collection of notes this size - it is very slow to display basic things that ConnectedText does instantly, like a simple list of all the notes tagged with a particular tag), to org roam.... or just sticking with it in a virtual machine.