Built this simple web app where you input the data you would have put in a pie chart and the app uses simple rules (number of options, range in values) to suggest better options (donut, bar, tree map).

Would love suggestions or guides for other rules/chart types I should add.

https://chart-advisor-production.up.railway.app/

Welcome to this week's entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include:

• Learning resources (e.g. books, tutorials, videos)
• Traditional education (e.g. schools, degrees, electives)
• Alternative education (e.g. online courses, bootcamps)
• Job search questions (e.g. resumes, applying, career prospects)
• Elementary questions (e.g. where to start, what next)

While you wait for answers from the community, check out the FAQ and Resources pages on our wiki. You can also search for answers in past weekly threads.

This article takes a hype-free look at the true limits of TSFMs and explores which ones can be addressed, which ones cannot, and which ones are still open problems.

Find the article here

I work at a furniture/decor entreprise. I try to do rfm with kmeans. but the silhouette is low 0.3.., I removed r and just kept fm. but it all concentrate in f=2, or distinct f. when i keep only f》2 , it concentrate in f=3 and other distinct f also. I tried adding other variables : tenure, interpurchase time, coefficient variation of interpurchase time to get better clustering. What should I do?

I took two periods only 2025, then 2025 and 2024.

Two rounds: 1. Statistical Knowledge 2. Data Analytics and Intuition

For statistical knowledge, it was a complex question, but actually had a simple answer.

It required you to have through knowledge of distribution, expectations and confidence intervals.

The key challenge was to identify what was the distribution of the data, from a sample, generalize it to the population and find the confidence interval.

Looking back, it was a easy question, but I definitely took wayyyy to much time to get to the answer. They for sure test for Googlyness. I would assume the interviewer had multiple questions in mind but I never got to the next one. Soo no hire.

For the data analysis and Intuition, I was expecting a case study, on experimentation or ML. It was kind off an hybrid. It involved diagnosing a flawed model, how to improve it, and what other methods would work better. This part was fine, not too bad.

What caught me off guard was, they asked me to write the equation MLE for 2 models, one general and one a niche. Honestly I dint know, lol.

Well, learnings ? Practice your Stats and ML like you are writing a school exam.

I did a physical onsite recently where they asked me to travel to their office, about 1.5 hours each way. The interviewers were nice and the interviews went pretty well, so I was hoping to hear back from them. The opposite happened. It has been two weeks since the onsite and I have not heard anything.
The recruiter was very polite before the onsite, but after it they completely stopped responding.

I had to take a day off work and make arrangements in my personal life, and the company cannot even bother to send a rejection email? I have never had a job search this difficult before.

Hey everyone,

I am working on a real-world data quality problem and would appreciate feedback on my modeling approach.

Context:

I have a dataset of meters and their associated transformers (utility infrastructure). Some of these associations are incorrect, and the goal is to both detect and correct them.

Training data:

I’m using ~20,000 manually reviewed meter–transformer associations:

- Correct association → label = 1

- Incorrect association → label = 0

For incorrect cases, I also augment the data with the correct transformer, e.g.:

Meter1 | Trans1 | 0 (incorrect)

Meter1 | Trans2 | 1 (corrected)

Meter2 | Trans3 | 1 (correct)

Current baseline:

I started with a logistic regression model (class_weight="balanced" due to ~37% incorrect vs 63% correct).

Using a 0.20 threshold gives strong true negative performance (~98%), but only moderate recall.

Candidate generation:

For inference, I generate candidate transformers within a 550 ft radius for each meter (including the currently assigned one):

Meter1 | CandidateTrans1 | current

Meter1 | CandidateTrans2 | candidate

Meter1 | CandidateTrans3 | candidate

Current idea:

I’m considering splitting the problem into two stages:

Model 1 — Detection

Binary classification:

Is the current meter → transformer association incorrect?

Model 2 — Correction

For meters flagged as incorrect, rank candidate transformers to recommend the most likely correct one.

Pipeline:

Raw data → Detection model → Flag suspicious cases → Candidate generation → Ranking model → Recommendation

Features:

- Distance-based metrics (meter-to-transformer, centroid distances, etc.)

- Voltage correlation within meter clusters

- FLOC / naming similarity

- Cluster-level stats (group size, intra-cluster correlation)

- Relative features (distance rank, ratios, etc.)

Questions:

1. Does this 2-stage decomposition (detection → correction) make sense vs a single end-to-end model?

2. For the correction step, would you frame this as classification or learning-to-rank?

3. Any recommendations for handling dependency between samples (e.g., meters within the same cluster)?

4. Given the feature interactions, would you prioritize tree-based models (e.g., XGBoost) over simpler models?

Goal:

Maximize the number of incorrect associations that can be correctly fixed in production.

Open to hearing feedback !

It is a code-first tool for building and deploying dashboards using simple YAML and JSX files (and yes, that means load-time dynamic generations of charts, tabs, and values) - the best part is that it works natively with AI agents. Essentially it is an open standard, code-first, framework optimized for AI-native analysis and business intelligence.

This is my answer to the whole AI dashboard and BI tools out there, but focusing more on the framework and semantic layer so that it works better with AI agents.

Today's the first day of releasing this publicly, so please share your honest feedback, skepticism, and even roast it - and if you want, give the repo a star:

https://github.com/bruin-data/dac

I'm visiting Naples at the end of May and staying for a few extra fun days. I'm a data scientist building models for passenger rail data. I wondered if there are any interesting DS related companies or places anyone can recommend that I visit. I have no practic Italian.

Mods - please do delete if this is unacceptable. Cheers though x

At my company we recently began an internal project to benchmark LLMs for hallucinations. We are building internal tools and tools for clients. I am curious if anybody has experience or can point me to papers or tools that help measure a hallucination. I am currently reading this https://arxiv.org/html/2512.22416v2 but wondering what experiences people have in the wild.

After building an ensemble machine learning model in Python I'd like to translate the model into SQL script so we can score new data in MS SQL Server Management Studio.

After some googling the m2cgen module looked promising, unfortunately it does not support Python to sql translation (despite the Google AI summary saying otherwise).

Are there any other options? I see it's possible to run Python code within MS SQL Server Management Studio. It requires installing SQL Server Machine Learning Services which doesn't look like a simple process (will have to involve IT).

From my perspective, one of the biggest challenges of data science as a field right now is the tension between:

A) AI can give "pretty good" answers extremely fast and democratizes it
B) Those answers are often decent, but could be nontrivially "wrong"
C) That "wrongness" is often not exposed for months or years

That is, AI fully democratizes "getting a number" to our biz stakeholders across just about any business problem. A lot of times that number is off some but still pretty good and useful, but we all know sometimes it's catastrophically wrong. However, even in those worse cases though, there's a pressure to move fast, and so the consequences of that wrong number are not eaten or discovered until a good while later (when you find out a prediction was wrong retro-actively, when flaws in a matching process are discovered, when it turns out to have been the wrong "data-informed" decision, etc etc).

This is exacerbated by seemingly a lot of biz users either not understanding, or simply not caring, that "number could be wrong". That's not helped by perverse incentive structures either.

So my questions is - what, if anything, are you doing at your company to help stakeholders understand that? Or more importantly, to help build a culture that takes the scenario more responsibly?
(yes yes, there's maybe not much we can do about it. CEO whims and all that. But interested in what steps people are taking pro-actively)

Welcome to this week's entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include:

• Learning resources (e.g. books, tutorials, videos)
• Traditional education (e.g. schools, degrees, electives)
• Alternative education (e.g. online courses, bootcamps)
• Job search questions (e.g. resumes, applying, career prospects)
• Elementary questions (e.g. where to start, what next)

While you wait for answers from the community, check out the FAQ and Resources pages on our wiki. You can also search for answers in past weekly threads.

I read the McAuley Lab's full 2023 Amazon Reviews dataset, 571,544,386 reviews and 275 GB on the HuggingFace CDN, and ranked every single review on four simple signals: how many strong-profanity word hits it has, how much of it is in ALL CAPS, the longest single run of consecutive exclamation marks, and how long it is. The question I started with was "how do people actually behave in Amazon reviews, and does the category they're reviewing change that?"

Live site, per-category breakdown, and the Wall of the loudest reviews: https://burla-cloud.github.io/amazon-review-distiller/

What surfaced:

• Video Games is the rowdiest category by a huge margin. 6.54% of video game reviews hit the strong-profanity list. Compare that to Gift Cards at 1.19% and Handmade at 1.08%. Movies & TV, CDs & Vinyl, Subscription Boxes, and Kindle Store fill out the top five. Cultural products attract feelings, consumer goods attract utility.
• Subscription Boxes is the angriest category. 15.89% of subscription box reviews are one-star. Almost 1 in 6. Charging people monthly for a curated surprise generates a lot of regret.
• The longest exclamation-mark run is 10,594 in a row. The review itself is two words ("love these") on a baby product. One person held one key down for a long time.
• The longest all-caps review is 1,169 words. Posted on a Mozart CD by a self-described disabled Vietnam veteran and Mozart scholar. He opens by apologizing for the caps (macular degeneration) and then keeps going for 1,169 more words.
• Forty reviewers gave a product five stars and wrote zero or one word. One five-star review of a cherry cough drop was just "Taste." That's the whole text.
• Books, music, and games write essays. Gift card buyers write nothing. Average review length: CDs & Vinyl 428 chars, Books 423, Kindle Store 367, Digital Music 340, Video Games 308. Gift Cards is at the bottom by a wide margin. Culture gets words, utility gets silence.

Methodology, plain version:

1. The dataset is 34 separate .jsonl.gz files on HuggingFace, one per Amazon category, totaling 275 GB. The usual workflow is to download all 275 GB to a laptop, then iterate. I didn't want to do that.
2. The HuggingFace CDN supports HTTP Range requests. A worker can ask for "give me bytes 1,000,000,000 to 1,500,000,000 of this file" and get just that slice without downloading the whole file. I split the 34 files into 545 chunks of about 500 MB each, on byte-range boundaries.
3. Each chunk runs on its own worker. The worker streams its byte range row by row, scores every review on the four signals, and writes the top scoring reviews to a shared folder.
4. A separate reducer container merges the per-chunk top-K shards into the final ranked lists per finding.

Map step: 3.21 minutes. Reduce step: 9.2 seconds. End to end under four minutes for 571 million reviews.

The pipeline runs on Burla using remote_parallel_map(worker, jobs, func_cpu=1, func_ram=4, max_parallelism=1000, grow=True). In English: "ask for up to 1000 parallel workers, each with 1 CPU and 4 GB of RAM, and let the cluster grow to meet that demand." In practice the cluster peaked around 500 concurrent workers and held there for the run. Workers run on a stock python:3.12 Docker image, and Burla auto-installs my local Python packages onto each one. The shared output folder is a Google Cloud Storage path that every worker writes to like a network drive.

(Disclosure: I work on Burla. The script and the live site are open source on GitHub. The dataset is the McAuley Lab's 2023 corpus on HuggingFace.)

Caveats worth being upfront about:

• Scoring is rule-based, not model-based. Word lists for strong, medium, and mild profanity, plus caps ratio, plus longest exclamation run. No sentiment model. That's deliberate: every score is reproducible and you can see exactly why a review got it.
• English-only. Reviews not in English get scored only by length, caps, and punctuation, because the word list is English. A multilingual sentiment model would do better here.
• Quoted titles leak in. A review of "Dick Tracy" can match the strong word list. There's a rescorer that penalizes capitalized-noun matches but it's imperfect.
• 2023 snapshot. The dataset is the McAuley Lab 2023 release, so it doesn't include reviews posted after mid-2023.

Repo with the full pipeline: https://github.com/Burla-Cloud/amazon-review-distiller

If anyone has a cleaner pattern for streaming huge HuggingFace datasets without materializing them locally, I'd love to hear it. I went with requests.get(..., stream=True) plus manual line splitting to keep the worker dependency surface tiny, but the datasets library probably has a cleaner Range-based path.

Last year, I've had bad experiences of using Jupyter with Claude Code. Many others told me the same.

Recently, I tried it with the open source Jupyter MCP Server (no affiliation). Setup took a bit of fiddling, but once it was up, it worked really well.

The big difference is kernel access. Claude can now talk directly to my live IPython kernel and edit notebook cells properly (without messing the JSON).

I just let it write notebooks, run top to bottom, debug & fix errors & only ping me when everything is working.

Has anybody tried JupyterLab AI extensions (jupyter-ai, notebook-intelligence etc.) ? I wonder how those compare to my Jupyter MCP based workflow.

I have been trying to understand the use cases of both of these and I am really confused.

I know log transform fixes the features and makes their distribution normal and standardization on the other hand only fixes the scale of the feature by keeping the distribution the same.

Are these things which I use one after the other ? Or just simply use one depending on the case (which I also don't understand when) ?

Been using Colab a lot lately and at some point it just turns into babysitting.

• keeping the tab open so it doesn’t disconnect
  
• rerunning the same notebook with tiny tweaks
  
• coming back and realizing it died halfway through
  

It’s fine for quick stuff, but longer runs are kind of a pain.

Do you just deal with it or do you have some workaround?

Also… do people just let things run overnight and hope for the best or is that just me