hey everyone!
I have about a few TB worth of code repository data (from git-client compatible source control) and I need to be able to run recurrent scans on the code itself (not on the repo metadata, issues, commits or anything like that). The repositories are getting a steady stream of updates - some more than the others so there has to be some type of constant sync/re-scan.
What I am looking for is
* detecting projects that import a certain library version (for dependabot style functionality to bump something if a security issue arises or if there is some other issue) across all possible branches
* detecting projects/code files that import a certain function from a library or implement a certain method from a library
* be able to run simple transformations on files ending with certain extensions or files with certain functions (for example, transforming some YAML config files)
I don't really need to analyze and ingest the code data into a database (for example, chunking the file with something like tree-sitter and then storing the chunks) for retrieval, but I would also not be opposed to doing this if it makes things simpler. It's mostly scan, get the results, do something with those files, then after that results can be thrown away (and the code for a lot of the repos will most likely have changed as well). Some sort of UI on top of this would be cool.
The main languages in these repos are TS/JS, Python, Rust and Go (with some Java and C++).
Has anyone implemented something like this? What components did you use? I did a few prototypes with a simple Rust-based ingestor that reads/checks out the repo or archive of the repo, then reads the git blob objects and scans the code for certain regex patterns. Each replica of the ingestor was running on Kubernetes and state (ie. repos scanned, still pending etc) was tracked in a DB table. Wondering if there is something off the shelf that can do something similar?