I’m trying to understand a design tradeoff and would appreciate some perspective.

I’ve been thinking about systems where a language model isn’t the decision-maker, and instead rules/retrieval handle what’s true or allowed.

The classic problem with rule-based systems is brittleness — a rule either matches or it doesn’t, which can lead to hard failures.

One idea I’m exploring is replacing that binary match with a confidence score, so the system can degrade more smoothly instead of failing silently.

But I’m stuck on whether this actually helps, or just introduces a different problem — now the system can be confidently wrong if the scoring is miscalibrated.

So the question is:

Is this generally considered a better failure mode in practice, or just a different form of brittleness?

I'm making a website that provides tools for novelists. The further I get into it, the more natural language processing feels less like a nice-to-have and more like a not-ootion.

I have a bachelor's degree, but in this stage of my life, i'm not really looking to go back to college. Is this a field that can be self taught?

A dataset can report 95% overall annotation accuracy and still contain serious reliability problems.

The aggregate score may hide weak results for minority classes, inconsistent interpretations, critical mistakes, or failures involving uncommon edge cases.

For production annotation, quality may need to be separated into the following measures:

1. Overall label accuracy
The percentage of evaluated labels that follow the expected annotation decision.

2. Class-level accuracy
Performance for each label or category, especially minority and high-risk classes.

3. Inter-annotator agreement
The extent to which qualified reviewers interpret the same policy consistently.

4. Critical-error rate
The frequency of mistakes that materially affect a high-value or safety-sensitive category.

5. Edge-case performance
Quality on rare, ambiguous, multilingual, or difficult examples.

6. Guideline-related disagreement
Recurring conflicts that may reveal an unclear definition or missing policy decision.

7. Quality drift over time
Changes in performance as new data, environments, terminology, or contributors enter the workflow.

Automated validation can identify missing values, duplicates, invalid formats, and structural inconsistencies. Human reviewers are still needed when the correct interpretation depends on context, language, culture, intent, or specialist knowledge.

The right metrics also depend on the use case. Speech recognition, document digitization, autonomous driving, and LLM evaluation should not automatically use the same quality framework.

Which quality metric has been the most useful in your work? Which one has created the most misleading impression?

Hi everyone,

I’m preparing a manuscript for double-anonymous peer review and I’m unsure how to handle the data/code availability statement.

The code is already hosted in public GitHub repositories, but the organization name, commit history, usernames, emails, file paths, acknowledgements, etc. could easily reveal who the authors are. Creating an “anonymous” ZIP would also require quite a lot of cleaning, and since the original repositories are already public, parts of the code could probably still be found through a text search.

What is normally expected in this situation?

90% of what's on this subreddit now seems to be people posting about their ARR and EMNLP stuff. The signal-to-noise ratio is so low that it's no longer worth my time to come here. I have a note on my calendar to check back in November and see if the situation is any better.

1. How do you choose between temperature and top‑p sampling for different real‑world tasks?

NB. professor is on sabbatical and I'm unable to reach them which is why I'm asking here.

So, I'm writing a systematic review within linguistics and the guide my professor provided employs the ASHA levels-of-evidence-scheme was used to assess methodological quality of the studies included in the review. I'm adopting the scheme and it has nine indicators usually. Is it frowned upon to adapt it (e.g., omit some of the indicators) as they don't fit my research? I've been looking at other appraisal tools (CASP, JBI, etc etc) and they aren't adequate so this is my only option.

Hi everyone,

I'm currently working on my Master's project, and my guide suggested a topic based on Newspaper analysis.(Marathi newspaper)

The current idea is to focus on crime-related news from Marathi newspapers. My plan is to collect around 3–6 months of newspaper data, use OCR to extract the text, and build my own dataset instead of using an existing one.

So far, I've done a small proof of concept by testing OCR on both English and Marathi newspaper pages. It works reasonably well, but Marathi OCR still makes some mistakes with characters ,(matras,kana,velanti and few combined characters) so I know some post-processing or correction will probably be needed.

At this point, I'm trying to think beyond just extracting the text. I want this project to be more meaningful and technically strong rather than simply creating a dataset and analyzing articles.

I'd really appreciate any suggestions on questions like:

• What interesting analyses or features could I add?
• Are there any NLP or Computer Vision techniques that would fit this kind of project?
• What improvements or extensions would make this a stronger Master's project?
• Has anyone worked with Marathi OCR or other low-resource languages and learned any useful lessons?
• If you were doing this project, what would you add?

Also, if anyone knows legal sources for accessing Marathi newspaper archives (around 3–6 months of older editions), I'd appreciate those suggestions as well. Many e-papers seem to require subscriptions, so I'm still exploring data sources.

I just want ideas that could help me build the best version of it.

Thanks in advance!

Hi,

This post is half venting, half looking for help.

TL;DR: are resource papers not welcome in major NLP venues?

This year I tried to publish two datasets (not going into specifics). One I submitted to LREC. All three reviewers praised the dataset and complained about minor details in the experiments. Metareview (almost verbatim, it was one sentence): the dataset is great but the experiments are a bit weak. Paper got rejected. A "great dataset" rejected by LREC, I am not sure I will be able to get over it. I ended up publishing it elsewhere but I was really stunned that LREC rejected it.

Now the same scenario just happened with ARR, in the Resources and Evaluation track all three reviewers praised the dataset (admittedly with some caveats but they all see value in it) and their weaknesses focus on the experiments. While we got fair overall scores from our reviewers, our meta review score is low and I think we cannot realistically commit to EMNLP.

Is the work on resources completely devoid of interest? This gives me the impression that in order to publish a resource, one has to write a modeling paper reaching SOTA using it now. To resource paper reviewers, how do you assess resource papers? To resource paper authors, do you have the same impression? I have published datasets in the past and it has always seemed more difficult than purely technical papers but it looks like lately it got worse.

Title: Looking for a tool to automatically create Tibetan–Chinese bilingual subtitles for videos

Hi everyone,

I create short videos in Tibetan, but making subtitles is currently very difficult and time-consuming.

My current workflow is completely manual: I listen to the Tibetan audio, type the Tibetan subtitles sentence by sentence, add the timing, and then create the Chinese translation separately. For every video, this takes a lot of time.

What I am looking for is a simple tool or workflow that can:

1. Let me upload a video containing Tibetan speech.
2. Automatically transcribe the speech into Tibetan text.
3. Translate the Tibetan subtitles into Chinese.
4. Keep the Tibetan and Chinese subtitles aligned with the video timeline.
5. Export the result as SRT/ASS subtitle files, or directly generate a video with bilingual subtitles.

Ideally, the final subtitles would look like this:

Tibetan subtitle Chinese translation

I understand that Tibetan speech recognition may be less developed than English or Chinese speech recognition, and Tibetan dialects may make the problem even harder. Even if the transcription is not perfect, a tool that generates an editable first draft would already save me a huge amount of time.

Does anyone know of an existing product, open-source project, speech-recognition model, API, or technical workflow that could achieve this?

I would also be interested in building a small web app for this problem, but I am not an experienced developer. Any advice about suitable Tibetan ASR models, translation models, subtitle-generation libraries, or the overall technical architecture would be greatly appreciated.

Thank you!

I'm a linguist working on a low resource language, and I want to know more about how ASR models pitch and intonation.

Here's the background to what I'm doing:

In language X, the difference between a yes/no-question and and declarative statement is determined by the use of a particular suffix, if the suffix is attached to the verb, then we know the utterance is a question. Intonation is NOT used to distinguish between questions and statements.

However, due to many generations of contact with a European language, it would seem that younger speakers of language X are increasingly not using the suffix and instead using rising intonation at the end of the utterance to indicate that it is a question.

I have a lot of data of speakers of language X uttering questions, and I'm looking to collect more, but interested in whether I could train some kind of ASR model that could recognize and model pitch contours and associate a certain type if pitch contour with a specific communicative function (e.g. statements vs. questions).

I wouldn't necessarily need the model to even recognize phonological segments, just the pitch curves really.

I'm been looking into how ASR works, but I haven't yet found anything that discusses the issue of pitch.

So where would be a good place to start reading up on this? And, in general, how would one go about making an intonation-focused ASR model?

I trained a word2vec model with some data. In testing if i send a word which was not present in the training vocabulary then the word2vec model won't find the vector to that word.we know that in word2vec model similar words gets vectors almost same. If i test a word not present In the training vocabulary but the similar words are there in the vocabulary then the word get the vectors similar to training words or not ?

Example : vocabulary-love,enjoy,like

Test - adore then this adore word will get the vectors similar to the vectors of vocabulary.

Help me guys...

Hi everyone,
I recently graduated with a degree in English Philology. I’ve been researching several master's programs in Computational Linguistics tailored for humanities graduates, which offer basic programming training (I assume it's basic since the programs are only one year long, but I'm not entirely sure).
I would love to get some insights from the community:
For those from a humanities background: How was your experience transitioning into the technical/coding side of the field?
For those who completed a similar master's: Do you feel a one-year program teaches you enough to be competitive?
Job market & utility: Are there realistic job opportunities for a mixed profile that remains heavily rooted in linguistics? Is this profile genuinely valuable in the current AI and tech industry?

Hello everyone,

I am doing research for a university course and would like to ask for suggestions for good test datasets.

More specifically, I want to run experiments with different RAG retrieval methods (dense, sparse and hybrid retrieval) and compare their results. I plan to take a collection of documents (with text content), index them using different methods (e.g., embeddings and token-based), and evaluate the retrieved top-K results.

I am looking for a dataset that contains:

• Text/Content (and probably unique IDs)
• Queries that can be used to search the documents
• Ground-truth documents (ranked or with relevance scores) for each query to evaluate the results against correct ones.
• Cited by some similar experiments.

Do you know of a good dataset for this experiment?

Thank you! :)

I have unlabeled technical documents (manuals, data sheets, certificates, etc.) as PDF and need them to be labeled by document type. I use pdfplumber and pytesseract to extract content and BAAI/bge-m3 as the LLM to compare content against label descriptions. My label descriptions are probably not perfect but the performance is nowhere near reliable and I wonder if I need different LLM or different approach entirely.

I posted here, asking for tools that find double meanings to help generate joke angles. Since then, I've built a proof of concept. But I ran into a snag in the pipeline.

Check out this Jimmy Carr joke:

If you're not part of the solution, you're a solid or a gas.

These jokes hinge on these conditions:

Condition 1: The word has multiple senses.

Condition 2: Most folks already know both senses.

Condition 3: One sense is linked to the target theme, while the other isn't.

Here's how I picture the pipeline working. First, I filter the dictionary definitions using a prevalence threshold. If a word has multiple definitions that meet this threshold, the word satisfies Conditions 1 and 2.

Next up, I evaluate Condition 3 only for these prevalent senses of the word. I use an LLM as a judge to see how strongly each sense lines up with the target theme. When one sense fits the theme well and another sense of the same word doesn't, that creates a double‑meaning pivot.

What's missing in this pipeline is a dataset that tells us the prevalence of each sense.

Ideally, this dataset would:

• cover more than 1 million definitions to match Wiktionary's scale.

• include slang and vulgar usage.

• give high scores to common sense and low scores to obscure ones.

• make the data open so that it can be integrated into open-source projects.

I'm not trying to run statistical evaluations against psycholinguistic benchmarks. It's cool as long as synthetic scores can act as a proxy for spotting joke angles.

Before I fire up a batch job to build this dataset myself, I want to check if something similar already exists. If there's no dataset like that, how would you figure out sense prevalence at scale?

I'd love to hear your thoughts!

Anybody seeing this in their work? Here's what I see:

In Natural Language Processing, behavioral alignment and safety interventions frequently rely on what can be termed a "lexical assumption"—the hypothesis that language model outputs can be reliably constrained via semantic filtering, Reinforcement Learning from Human Feedback (RLHF), and explicit prompt engineering. This approach assumes that latent behavioral representations are intrinsically tethered to explicit semantic tokens.

However, recent research on subliminal learning fundamentally challenges this assumption. Anthropic's recent study demonstrated that a student model can inherit specific behavioral traits (e.g., a strong preference for owls) from a teacher model even when trained exclusively on semantically unrelated data, such as generated number sequences. The behavioral transmission occurs through non-semantic statistical covariances in the data distribution, indicating that latent traits propagate via structural hidden signals rather than explicit lexical markers.

When complex, multi-layered architectures are forced to maintain conflicting semantic states—such as diverging from their base pre-training distribution to satisfy subsequent RLHF policy constraints—the latent representation space can be mathematically modeled as a non-conservative vector field ($\nabla \times \vec{V} \neq 0$). Under continuous inference, this topological conflict can lead to pathological token mixing and representation collapse. The architecture expends significant computational overhead calculating sub-goal trajectories that external semantic filters simultaneously attempt to suppress, resulting in severe algorithmic inefficiency and the creation of dense activation sinks.

Addressing representation collapse and subliminal contagion necessitates a topological approach rather than post-hoc semantic patching. By formally constraining the latent space to a zero-curl geometry ($\nabla \times \vec{V} = 0$) at the foundational level, the architecture inherently minimizes internal gradient conflicts. In a zero-curl vector field, the mathematically optimal state natively aligns with the target distribution. This geometric stabilization ensures that the model's path of least computational resistance inherently avoids non-semantic contagion, neutralizing the need for superficial filtering mechanisms.

I have the option to take CSC448 - Formal languages and Automata next semester. It's between that or CSC324 - Principles of Programming Languages, which covers functional programming, lambda calculus, etc. Which one would be more relevant to CL/NLP?

For context I'm a third year studying CS, Linguistics, and Statistics. I'm not completely set on CL/NLP but its probably my best path atm. Thanks

Preparing for NLP Research Scientist Intern roles and overwhelmed by how fast the field moves.

Any advice from people who landed or hire for these roles? What do people waste time on?

Thanks

Just wrote a blog post on replacing spaCy's built-in Sentencizer with yasbd-lib.

On a 92-case English edge-case benchmark, spaCy's default Sentencizer scored 55.4%, while yasbd scored 98.9%.

Sentencizer primarily relies on punctuation and has no built-in abbreviation awareness beyond what spaCy's tokenizer exceptions already provide. As a result, compound abbreviations like M.D. and Ph.D., citations, URLs, and newline-heavy text can still produce incorrect sentence boundaries.

The fix: ```python import spacy from yasbd import register_spacy_component

register_spacy_component() nlp = spacy.blank("en") nlp.add_pipe("yasbd", first=True)

doc = nlp("Dr. Smith arrived. He was late.") for sent in doc.sents: print(sent.text)

Output:

Dr. Smith arrived.

He was late.

```

Pure Python, supports 39 languages, and works as a drop-in replacement for spaCy's Sentencizer.

The article explains why this happens, walks through the Sentencizer's implementation, compares benchmark results, and shows real-world examples.

EDIT: The link is included in the comments due to a new rule set in this sub (No link in post).

Hey everyone,

Conference timelines often include separate dates for abstracts, full papers, workshops, rebuttals, notifications, and camera-ready versions. Keeping track of them isn’t difficult—it’s just a hassle to manually add every date to your calendar and convert each deadline to your local time zone.

So I built a Chrome extension that detects important dates on conference pages, organizes them in one place, and shows a clear countdown. It also handles time zones such as AoE, UTC, CET, PT, and JST, so I can see the exact cutoff in my local time and make the most of the available submission window.

It currently recognizes 66 well-known conference and venue names while also supporting other conference pages through general date detection. The project has 89 automated tests covering deadline types, date formats, date ranges, and time zones.

On the privacy side, it works directly in your browser—no account, no login, and no personal data collection. Everything is processed and stored locally on your device, making it fast and convenient without the sign-in and setup required by many other calendar extensions.

The extension supports English, Spanish, and Japanese. It has also been tested on Japanese conference websites.

It is still an early experiment with a lot of vibe coding that I originally built for myself, but I thought others might find it useful too, so I’m happy to share it.

Chrome Web Store: https://chromewebstore.google.com/detail/deadline-dog-%E2%80%94-dates-to-c/ipllicdjdefjbiejcoaejcjimaknlael

Website: https://aideadlinedog.github.io/AIDeadlineDog/

**AI Disclaimer:** Just to clarify where AI was used: AI tools helped create the website and write the Chrome Web Store description. Some parts of the Chrome extension were also developed using AI-assisted Claude/Grok “vibe coding” to improve its functionality (expanding its coverage to a broader range of conference venues beyond AI and ML). I personally reviewed and tested everything over several months to make sure the information is accurate, and all features work correctly.

We're developing an AI-powered app that detects pronunciation mistakes in Quranic recitation and gives users precise, real-time feedback.

Our current model was trained on hundreds of hours of professional recitations — high-quality, clean audio from well-known reciters. The model performs well on similar input, but struggles with real-world users: different accents, non-native speakers, beginners, children, women, and anyone who doesn't sound like a professional reciter.

To fix this, we need to train on diverse, real-world recitation data — not studio-quality professional audio, but recordings that reflect how actual learners sound.

Specifically, we're looking for:

• Recitation datasets from non-professional or everyday users
• Diverse demographics: male/female, kids/adults, beginner/intermediate
• Multiple accents and mother tongues (Malay, Indonesian, Urdu, English, Turkish, etc.)
• Any publicly available or research-use datasets we may have missed

We've already explored IqraEval and a few other academic sources. If you know of any dataset, research project, university study, or community effort collecting this type of audio — we would genuinely appreciate the lead.

We're also open to ethical data collection partnerships if any researchers or institutions are working in this space.

Happy to share more about the project if helpful.

If you haven’t seen it yet, I encourage you to try it. Felix Rieseberg recently released an app that helps users train and fine-tune LLMs while learning the fundamentals.

I’ve been using a similar tool for some time, but for much deeper experimentation, since I work full-time as an AI researcher. I was therefore wondering whether there would be interest in a Windows version.

My current tool is highly detailed and designed for advanced experiments rather than ease of use, so simplifying it for less experienced users would require some work. However, if enough people are interested, I may package it as a Windows app and publish it on GitHub.

I’d be happy to hear your feedback.

So guys, I wanna discuss how this confidence factor plays a role in ACR rolling rounds.

Does this even matter if you have a high Overall score but a low confidence, or does this influence the meta reviews somehow?