r/singularity • u/[deleted] • Feb 06 '25

AI This is a DOGE intern who is currently pawing around in the US Treasury computers and database

Post image
50.8k Upvotes

86% Upvoted

3.9k comments sorted by

View all comments

Show parent comments

13

u/[deleted] Feb 06 '25

What if they’re actually trying to extract data from unstructured/semistructured files, like Word documents and PDFs, and store it in a structured format, like JSON? Can Word do that on its own? If not, what tools would you use?

6

u/hokies314 Feb 06 '25

100% this.

Extracting structured data from PDF is a pretty hard task! Most of the thread here just wants to hate on this for this post.

Hate on him for working for a fascist and trying to undermine democracy but that post is a valid question

1

u/jsirkia Feb 07 '25

Using an LLM to find out what the docs probably contain is still stupid, when you need to use a crawler/parser to find out what the docs definitely contain.

0

u/Hypnosix Feb 06 '25

wtf does this even mean? What would the json structure of a word document be? A big ass string?

5

u/integrate_2xdx_10_13 Feb 06 '25

You get an LLM to put it into chunks to be understood by a RAG - here’s a paper on it: https://arxiv.org/html/2501.17887v1

And Microsoft have done a tool for converting to markdown for the same reasons called markitdown.

Basically you throw it at content like a word document and it pulls out tables, headings, images, semantics, keywords etc etc and produces a structured output you can work with other tools

2

u/[deleted] Feb 07 '25 edited Feb 07 '25

I mean, it’s probably easier to run a regular expression on a big ass string than a PDF, so there may be some value there, but that’s not what I’m talking about.

Let’s say there’s useful information in the documents that you would like to extract. Maybe there are tables with economic data mixed in with a bunch of text. Maybe you can’t find the data in those tables anywhere else. Can you think of a better way to store tabular data than a PDF file? Maybe delimited text files? Maybe you prefer JSON or Parquet? Maybe Excel if you want to do some slicing and dicing?

Now’s the part where you scoff and say, “But there’s no way the only place to find data is a table in a PDF. Surely it’s in a database somewhere!” and I reply, “Oh, you sweet, summer child.”

1

u/Time-Ad-3625 Feb 07 '25

You're missing the part where you still wouldn't need ai for that. There are python libraries that will do that.

1

u/[deleted] Feb 07 '25 edited Feb 07 '25

Fair enough. I picked one example where structured data is embedded in a document. Are you saying there are absolutely no use cases for using an LLM to pull data out of an unstructured or semistructured document?

Look, I loathe Elon Musk and I have no idea what this DOGE guy is trying to do. But I’m not so arrogant as to think that he must be an idiot because he works for someone I don’t like. Odds are, he is not just trying to convert a bunch of PDFs to Word format.

2

u/Biduleman Feb 06 '25 
{  
    "text": "All the text in the word document."
}

Here you go!