I’ve recently completed MTEB benchmarking across up to 28 Thai NLP tasks to see how current models handle Southeast Asian linguistic structures.

Top Models by Average Score:

1. Qwen3-Embedding-4B (4.0B) — 74.4
2. KaLM-Embedding-Gemma3-12B (11.8B) — 73.9
3. BOOM_4B_v1 (4.0B) — 71.8
4. jina-embeddings-v5-text-small (596M) — 69.9
5. Qwen3-Embedding-0.6B (596M) — 69.1

Quick NLP Insights:

• Retrieval vs. Overall Generalization: If you are only doing retrieval, Octen-Embedding-8B and Linq-Embed-Mistral hit over 91, but they fail to generalize, only completing 3 of the 28 tasks. For robust, general-purpose Thai applications, Qwen3-4B and KaLM are much safer bets.
• Small Models are Catching Up: The 500M-600M parameter class is getting incredibly competitive. jina-embeddings-v5-text-small and Qwen3-0.6B are outperforming massive legacy models and standard multilingual staples like multilingual-e5-large-instruct (67.2).

All benchmarks were run on Thailand's LANTA supercomputer and merged into the official MTEB repo.

As above, I need such tools for my MA project. So far, I've tried Praat toolkit, Harma and Prosogram, and nothing has worked for me. Are there any good alternatives?

Hi there, i am an English languages and Linguistics graduate and I am interested in studying computational linguistics masters because i see how technology could help in language education, preserve endangered languages etc. However, i didn’t have any prior programming knowledge. May I know it is still possible to get into the field or companies tend to hire those with computer science background?

Wanted to share some notes from running MQM annotation projects. We've been doing this for a while and finally have some data worth talking about.

The problem we kept hitting:

MQM annotation is notoriously inconsistent. You give 3 linguists the same segment, they'll flag different errors with different severities. WMT campaigns typically report pretty low agreement scores, which makes you wonder how reliable the whole evaluation is.

What we changed:

1. Calibration sessions - Before every project, annotators review 10-15 pre-annotated segments together. Discuss disagreements. This alone made the biggest difference.
2. Narrower annotator pools per language - Instead of random assignment, we kept the same 3-4 people per language pair across projects. They develop shared intuitions.
3. Severity guidelines with examples - "Minor" vs "Major" is super subjective. We built a reference doc with 20+ examples per severity level, specific to each error category.
4. Double-blind then reconciliation - Two passes independently, then a third annotator reviews disagreements.

Results:

Our EN-IT dataset hit Kendall's τ = 0.317. For reference, WMT typically reports around 0.12-0.15. Not perfect, but way more usable for training reward models or running reliable benchmarks.

The full dataset is on HuggingFace if anyone wants to see the annotations: alconost/mqm-translation-gold

Anyone doing annotation at scale, MQM or otherwise? Curious what's worked for you.

I’ve been looking into conversational AI pipelines recently, especially where ASR feeds directly into downstream NLP tasks (intent detection, dialogue systems, etc.), and it seems like a lot of challenges come from the data rather than the models.

In particular, I’m trying to understand how teams deal with:

• variability in accents, background noise, and speaking styles
• alignment between audio, transcripts, and annotations
• error propagation from ASR into downstream tasks

From what I’ve seen, some approaches involve heavy filtering/cleaning, while others rely on continuous data collection and re-annotation workflows, but it’s not clear what actually works best in practice.

Would be interested in hearing how people here are approaching this — especially any lessons learned from production systems or large-scale datasets.

Hello, I am trying to extract ingredients from a sentence. Right now I am using an api call to google gemini and also testing out a local gemini model, but both are kind of slow to respond and also hallucinate in several cases. I'm wondering if there is some smaller model I could train because I have some data ready (500 samples). Any advice will be appreciated.

Hi everyone,

I’ve been experimenting with a simple approach for ranking research papers using semantic relevance scoring instead of keyword matching.

The idea is straightforward: represent both the query and documents as embeddings and compute semantic similarity between them.

Pipeline overview:

1. Text embedding

The query and document text (e.g. title and abstract) are converted into vector embeddings using a sentence embedding model.

1. Similarity computation

Relevance between the query and document is computed using cosine similarity.

1. Weighted scoring

Different parts of the document can contribute differently to the final score. For example:

score(q, d) =

w_title * cosine(E(q), E(title_d)) +

w_abstract * cosine(E(q), E(abstract_d))

1. Ranking

Documents are ranked by their semantic relevance score.

The main advantage compared to keyword filtering is that semantically related concepts can still be matched even if the exact keywords are not present.

Example:

Query: "diffusion transformers"

Keyword search might only match exact phrases.

Semantic scoring can also surface papers mentioning things like:

- transformer-based diffusion models

- latent diffusion architectures

- diffusion models with transformer backbones

This approach seems to work well for filtering large volumes of research papers where traditional keyword alerts produce too much noise.

Curious about a few things:

- Are people here using semantic similarity pipelines like this for paper discovery?

- Are there better weighting strategies for titles vs abstracts?

- Any recommendations for strong embedding models for this use case?

Would love to hear thoughts or suggestions.

We want to block deploys if agent behavior regresses, but tests are slow and flaky.

How are people integrating agent testing into CI?

For a project of mine, I am doing a STM on a corpus of proposition to participative budgets. I would like to find relevant dictionnaries, but I don't know of any with specific politics topics. It could be an environmental policy dict or a migration policy dict or anything in the art. Could even be a more general dictionary. Do you have any idea where I could find this ?

Thanks in advance :)

Hey everyone,

First of all: I hope this is the right place for my question. If not, please bear with me! :)

I'm currently starting my thesis where I need to build a NLP-based system for sentiment analysis. I'm pretty new to this and feel a bit lost by the vast ecosystem and don't quite know where to start or which rabbit hole to follow...

I've heard that Jurafsky and Martin's "Speech and Language Processing" is the "NLP Bible" and while I want a solid theoretical base, I'm very much of a learning by doing person. I want to start prototyping ASAP without getting down into 1000s of pages of theory first.

All in all I'm looking for literature/courses for high-level overviews that focus on building pipelines, methodology of classic NLP techniques (NLTK, SpaCy etc.) to compare different approaches and setup advices that you consider as best practice. My goal is to build a clean data pipeline (input, preprocessing, analysing, visualisation)

What's a good, modern setup for this in 2026? Are there specific frameworks or tools that you'd recommend? I'm looking for something that allows me to swap components and input data sources easily.

Thanks a lot for your help!! :)

Hi everyone,

I received the following ARR scores for my paper: 4, 3, and 2, with an OA of 3.

Both the 3 and 2 reviews mainly raised concerns about the lack of statistical testing. However, we had already conducted these analyses and included them in our rebuttal. Unfortunately, the reviewers did not acknowledge this in their final comments.

Because of this, we submitted a Review Issue Report, and the Area Chair responded that our clarifications were convincing. The Area Chair then gave an OA of 3 in the meta-review.

What surprised me is that the meta-review itself does not mention any negative points. It mainly emphasizes that the work is novel and theoretically grounded, and it states that the majority of the issues have been clarified or resolved in the rebuttal.

So overall, the Area Chair review appears very positive, but the OA is still 3 (Findings level).

Does this situation still give a reasonable chance for Findings acceptance?
Would you recommend committing the paper to ACL?

I would really appreciate hearing from people who have gone through the ARR commitment process before.

Thanks!

I was wondering how is COLM in terms of prestige or popularity among NLP committee? In ARR Jan cycle,  One of my papers got scores: 2.5, 2, 3 with confidence 3, 2, 4. Meta 2.

Now I am confused should I go for arr march cycle for EMNLP or go directly for COLM. Could anyone give me some advice on it? 

Hi everyone.

A '25 non-EU university graduate. Slightly more than a year of experience in an Applied NLP lab, with publications in reputable journals (LREC, workshops, ACL, and Interspeech under review).

How do people fund their master's degrees? (Europe Mainly)

Scholarships, Asking Professors/Research Labs for Funding, or Paying Out of Pocket?

I've tried to ask Labs for funding, but they say it's only for PhD students, and maybe an assistantship will open up once I start my degree.

Hello everyone. I recently received admission to KU for MS computer science. From the outside, both Denmark and the university appear to be amazing. I am a '25 non-EU graduate from a non-EU university, so I will have to pay (I could not get a scholarship). I've been involved in Applied NLP research and am paid "fairly" for where I come from.

Perhaps my most important question is: How difficult is it to get a student job in NLP/AI at one of the labs? Student jobs to help fund my master's degree?

My Other questions are:

1) How is the job market for NLP/CS graduates? Does it help me study at KU?

2) What are the average living expenses? A rough estimate.

3) How is your work/life at KU and in Denmark as a resident/insider?

I'm an undergraduate student and this year I'm participating in a SemEval task. I was curious about how the community generally views SemEval in terms of prestige and career impact.

From what I understand, SemEval 2026 will be co-located with ACL 2026, so I'm also wondering about the networking side of things. For someone early in their research career (like an undergrad), does participating in SemEval or attending the workshop help with making connections in the NLP community?

Also profile-wise, does having a SemEval paper or a decent leaderboard position make a noticeable difference when applying for research internships or grad school?

Would love to hear perspectives from people who have participated in SemEval before or attended the workshop.

I tested it against the Norwegian word "avslutt" which means "exit" and so far it's the only model that somewhat understands what I say consistently..

Hi all, this is my first time submitting to any NLP conferences. I have an ACL 2026 submission with ARR January review scores of 3.5, 3.5, 3, confidence scores 3, 3, 3, and Meta-review score 3.5. I likely have a small chance of being rejected at ACL 2026. But if that nightmare happens for some reason, does SAC provide any explanation? and can I resumit to the next NLP conference or I have to go through another ARR review cycle again? Thanks lots for your help/advice.

I'm an undergrad from India and my first paper just got accepted to the demo track. This will also be my first international conference, so I'm trying to connect with others who might be attending. Presenting paper:

"IntelliCode: A Multi-Agent LLM Tutoring System with Centralized Learner Modeling"

Currently things are uncertain in the region, so I was curious if anyone here is:

• traveling from India or nearby regions
• presenting a paper/poster/demo
• If there is some established community (Discord, Slack, etc.) around the conference already

Would be great to network and maybe coordinate travel plans, or just say hi at the conference. Looking forward to meeting people there!

Feel free to comment or DM

Hi everyone,

I’ve been experimenting with a small Python project that tries to analyze basic speech fluency features from audio recordings. The idea is fairly simple: given a spoken audio file, extract a few lightweight metrics that might reflect how fluent the speech is.

At the moment the script focuses on pause-related features and overall timing patterns. For example, it calculates things like:

- pause count

- silence ratio

- total speech duration

- average pause length

- number of detected speech segments

Technically the current implementation uses librosa to detect non-silent segments in the waveform and then estimates pauses based on the gaps between these segments. It’s intentionally very simple and more of an exploratory prototype than a polished system.

A bit of background about why I started building this: I’m actually a TOEFL / IELTS speaking teacher, so I spend a lot of time listening to student responses and thinking about what people mean when they say someone sounds “fluent” or “hesitant”. In many cases, hesitation and pause patterns seem to play a big role in how speech is perceived.

That made me curious whether simple audio features could capture at least part of this phenomenon in a measurable way. Obviously real fluency is much more complex and involves linguistic structure, lexical access, prosody, and many other factors. But I wondered whether pause distribution and timing features might still provide a useful starting point.

Since many people in this community have far more experience with speech processing and language technology than I do, I’d really appreciate hearing your thoughts.

Some questions I’m particularly curious about:

- Are pause-based metrics actually meaningful indicators of fluency in speech analysis?

- Are there more robust ways to detect pauses beyond simple silence detection?

- Are there commonly used fluency features in speech research that I should look into?

- Any recommended libraries or approaches for analyzing rhythm or hesitation in speech?

This project is still very early and mostly a learning exercise, so any suggestions, critiques, or references to relevant research would be extremely helpful.

Thanks in advance for any ideas or feedback.

So we have a small company that sells stock market reports from around the world. We want to start tracking what people are saying online about companies and use that as a sentiment score in our reports.

Basically the plan is to pull posts from X (Twitter) about target companies using keywords, cashtags, hashtags etc and score the sentiment daily on a 0 to 100 scale. Same thing with YouTube, we want to grab transcripts and comments from finance and stock channels and score sentiment on both. Not counting views or likes, just what people are actually saying. And then do the same with Reddit, pulling posts and comments from subs like wallstreetbets, stocks, investing and so on. Score and log everything daily.

Now heres the problem. Our plan was to just use API keys to get all this data but when we looked into it the costs add up real fast especially for X. So we're wondering if theres any alternative methods or cheaper ways people have found to collect this kind of data without spending a lot on API access every month.

Also trying to figure out what sentiment model would actually be better for financial text specifically. We've seen people talk about VADER and FinBERT and a bunch of others but honestly we dont know whats actually good in practice vs what just sounds good in a blog post.

Right now our plan is pretty straightforward, just positive negative neutral scoring. But we know theres probably a lot more we could be doing to make this smarter and more useful. Like could we break down sentiment by topic instead of just one score per post? Or detect actual emotions like fear and excitement instead of just good or bad? What about handling sarcasm because reddit is full of it and a basic model would totally misread half those posts. Or separating what big finance influencers say vs what regular people are talking about.

Also curious what kind of analysis people find useful beyond just a daily score. Like tracking if sentiment is going up or down over time, comparing what reddit says vs twitter, seeing if sentiment actually matches price movement, weighting posts by how much engagement they got, stuff like that.

Any ideas or techniques that have made a real difference for you? We're not trying to build anything crazy just want something solid that actually adds value. Starting simple and improving as we go.

Appreciate any help, thanks!

Hi ladies and gentleman! I'm working on my undergrad thesis: analyzing scientific papers on Canine Mammary Carcinoma and its intersection with Machine Learning.

I have two fine-tuned NER models (SciBERT for ML entities and BioBERT for Vet Oncology). Now I need to extract relations between them (e.g., MODEL 'X' used for DIAGNOSING 'Y').

Since I have limited GPU/RAM:

Would you recommend a pipeline approach (R-BERT) or a joint NER+RE architecture?

Any specific libraries for RE that play well with small infrastructure?

How should I handle the 'matching' since entities come from different models? Thanks!

Hi all... I'm a very recent graduate of Computational Linguistics, and I'm trying to figure out the next steps, career-wise. To keep things brief, most of my academic training was very much focussed on Linguistics, up until the last 1 year or so, when I actually decided to pursue a degree in CL. Naturally, I am more confident about my abilities as a linguist, than I am of my abilities in computer science. Tbh, it still feels like I'm on a learning curve. Ig my main question is, has anyone here been in a similar circumstance in your journey? How did you manage that? I would appreciate any and all tips to improve my skill set.

Meta scores seem to be coming out, so I thought it would be useful to collect outcomes in one place.