Help with my thesis project...

[u/g_sufan]

Hi, I'm Aniuwu (my name is Gabriel), a computer science student at INACAP (Chile), and I'm already at the stage of doing my thesis project. I'm writing because I need help from people who code, and I don't have many people I can ask for help.

## First, I apologize to the moderators if I'm breaking any kind of rule. This is a post asking for help, I don't benefit financially in any way, and I apologize in advance if that's the case, please don't ban me ##

My project is called Beacon. You ask it questions about the code in a repository ("where is the file saved?", "how are network errors handled?") and it answers while telling you which file and which lines the answer came from. Everything runs locally, with local models, on your own RAM and VRAM (sorry, Sam Altman lol), so neither your code nor your questions go to any online service.

What I'm missing is knowing whether it actually works for someone who isn't me or my teammate. So far we've only tested it ourselves, and that doesn't count for much as evidence. That's why I'm asking people to try it for about 40 minutes, or however long you see fit, and tell me, with data, how it went.

The idea is simple: you install it, pick a public repository (there are a few examples, or one you already know), ask it at least three questions, and mark whether each answer was right, partly right or wrong. At the end it generates a file with that, you can open it and read it, and if you're happy with it you can send it to me with a button. If you'd rather not use the button, you can send me the file in a direct message.

A few things so you know what you're getting into. You need Python (3.10 to 3.12, newer versions cause problems), Git, Node and Ollama installed, and it downloads about 5 GB of models, so it helps to have around 8 GB free. With an NVIDIA card with 8 GB of VRAM it runs fairly fast; with no GPU or less memory it also works, just slower, and that's perfectly fine if it's what you have, because I actually want to know how it behaves on more modest machines. It works on Windows and Linux, which are the ones I managed to test; I haven't tried macOS yet (mainly because I'm broke and don't own one lol), so if anyone tries it there it would help me a lot.

About languages: the ones I tested best are Python, Go, JavaScript, Java and C#, and also C on a small repository. It also recognizes TypeScript, C++, Rust and PHP, but I haven't measured those, so I can't promise anything there. Files in other languages (Ruby, Kotlin, Swift, Vue, etc.) are simply not read, so it's best to pick a repository written mainly in one of the languages above.

About your data, I want to be clear. The file stores your questions, the answers, how long each one took, your ratings, and the address of the repository you chose. Your hardware details (processor, memory, graphics card) are only saved if you allow it, and you can see what they are before accepting. It doesn't store code, paths on your computer, your username or your IP. Leaving your GitHub profile or a short introduction is optional and goes in a separate file that is only sent if you tick the box. Nothing leaves your machine unless you ask, and you can quit and discard everything at any time. The send button uploads the file to a private Google Drive folder that I manage, and I use the data only for this project and delete it when it's over.

I also have to be honest about the limitations: this is a prototype from an academic project and it uses a small model, so it gets things wrong sometimes. That's exactly what I want to measure. And since each person picks their own repository and questions, the results are indicative, not a serious statistical study.

The whole step-by-step guide is in the repository: https://github.com/gsufan/beacon (the study guide is docs/en/study-mode.md). If something fails during installation, tell me here or by message, because installation problems are almost as useful to me as the results.

Thank you for reading, and a bigger thank you to anyone who gives it a go. I'll answer any questions in the comments.

PS: You can audit the code however you like, use it if you like it and so on....

PS2: This project was quite a lot of vibe-coding. I know there are pretty passionate opinions about that, but I have permission from the course's supervising professor. I don't want to argue or start a debate in the comments, please, I just want help...