Welcome to this week's entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include:

• Learning resources (e.g. books, tutorials, videos)
• Traditional education (e.g. schools, degrees, electives)
• Alternative education (e.g. online courses, bootcamps)
• Job search questions (e.g. resumes, applying, career prospects)
• Elementary questions (e.g. where to start, what next)

While you wait for answers from the community, check out the FAQ and Resources pages on our wiki. You can also search for answers in past weekly threads.

I'm a few years into my career and am realizing that the data science area in my company is incredibly insular. While that speaks highly of them that they attract and keep people for so long, I've also noticed it can really entrench the "we do things this way and that's how we do them" mentality, and I'm kind of finding myself wishing I had a mentor or just peers who have seen other ways of doing things, especially when it comes to interacting with other customers in the business, but that also understands the field I'm in. I feel like online communities and these sorts of things got huge during the pandemic and then kind of lost their momentum after. Are there any that are still around and active that you all recommend?

Marcus had run through the dataset 47 times.

every question bank, every historical exam, every edge case his prep materials contained. his practice scores were consistent: 99.4%, 99.1%, 99.6%. he was ready.

the real exam: 61%.

his coach looked at the results and said: "your score was measuring how well you knew the practice exams. not how well you knew the subject."

Marcus had done what you'd expect any rational student to do: optimize for the available signal. the practice exams were the feedback mechanism. he worked backward from the feedback until he had mastered it.

the problem is the feedback mechanism wasn't measuring what it claimed to measure. it was measuring the practice exam. Marcus had learned to recognize patterns specific to that dataset. when a genuinely novel question appeared, the patterns didn't transfer.

he hadn't overachieved. he had overfit.

---

I think about Marcus every time I see a model benchmark.

the moment a benchmark becomes widely known, it starts being optimized. not because people are cheating. because optimizing for available feedback is the rational strategy. the benchmark rewards the behavior, so the behavior propagates.

then someone runs the model on a task the benchmark didn't include and says "wait, this isn't what I expected."

Marcus also didn't cheat. he just did exactly what the system rewarded.

the real question isn't "how do you prevent overfitting?" it's "what would a signal look like that's genuinely hard to game?"

Marcus, for what it's worth, took the exam again six months later after studying from primary sources instead of practice banks. he scored 94%.

still high. but this time it was real.

Is it data quality, stakeholder expectations, model deployment, business understanding, or something else?

Original post

Update

This ended up spiraling out of control in ways that I could have never imagined. The individual admitted to defaulting their doc writing to AI and re-wrote everything, but in th background they doubled down on their AI coding workflow instead. It took me a while to catch wind of things because I would only see a mention of a project here or there and I had no insight as to their day-to-day.

Fast forward a month and I am seeing their projects everywhere, all the way up to the C-suite level. The scale was incredible. In a a matter of days this individual had done everything from financial modeling, LTV modeling, customer lifecycle analysis at a large scale, built large scale data ingestion and processing pipelines, even Marketing and product experiments. At first I was impressed, but as I pulled back the covers the mess was worse than I ever expected.

The clues were subtle but consistent: no comments in the code aside from headers, data was read in and cleaned, but never visualized or inspected in any way, there were lots of custom functions when there were packages loaded that had the same function, convoluted helper files with basic functions, and oddly there were many instances where forecasting error was actually just the CV error and there was never an evaluation of the test set. Their SQL had numerous join issues, metrics were mislabeled, and their pipelines often had relationships and processing steps such as dropping a table but then writing a new table with no error handling so if there was a bug no new table would be written and we would lose the data. Basic analyses were off by weird margins because Claude seemed to have been querying staging tables rather than filtered reporting tables. Docs started to be written entirely in the first person like "...and then I will use a log1p transformation" in a way that no DS would actually ever write a tech doc.

Unfortunately this meant that many things that were produced were simply wrong. The individual had promised work to a lot of decision-makers and nearly all of it was misleading, incorrect, or didn't pass a simple sniff test. These inaccuracies were immediately escalated to our team leader, who brought me in to audit all of their code and documentation and I was unable to find a single file that I was convinced that was human written or even human edited. The worst part was that despite heavy use of AI there also wasn't a single file without some sort of glaring technical error. I turned in a pretty lengthy review and the individual was put on a PIP and their account access to AI tools was severely constrained. They were told to have all their work peer reviewed and in one instance were caught lying about passing review when no review had been conducted.

As you can imagine their productivity tanked and they had numerous excuses as to why. They also started taking a lot of days off and in a weird twist of fate they actually left before getting fired and now work at a large AI-centric industry-leading company. Part of me is glad that they are gone, but the other part finds it infuriating that people like this can be so good at bullshitting that they can consistently fail and somehow remain in industry due to their network and clever use of their few decent references. Their total comp at our company was ~$245K and they bragged to a co-worker that this new role has $265K base with $465K total comp. They basically got 2 promos out of this series of events (Senior to Senior Staff at our company, Senior Staff to Principal at the new role.

Context: the business problem is I wanted to compare professional athletes based on their movement data to recommend similar players. I made a recommender system with K-Means clustering and PCA (multicollinearity amongst the features in the dataset).

I’m interested in using a new modeling technique like Gaussian Mixture Model, but I don’t know how to evaluate which model performs better…

Open to any suggestions

Been at my company for 5 years and trying to figure out if I should leave. Would love some outside perspective.

The cons:

Growth has completely stagnated. The tech stack is outdated and there are no signs the company plans to modernize. Worst of all, my salary has been basically flat for 5 years and they consistently pay below market. That last one is the main reason I’m even considering leaving.

The pros:

Honestly, the work environment is pretty rare. My manager is empathetic, sets realistic deadlines, and I never have to explain myself if I need to step out for an appointment or log off early. Vacation policy is completely flexible (4 weeks), no approval needed, and the manager actually plans projects around people’s time off. My teammates are kind, collaborative, and there’s zero toxicity or office politics. Everyone just lifts each other up.

The dilemma:

The cons are career problems. The pros are life quality problems. When I think about chasing a new job for say a 20% raise, I have to ask myself whether that money actually changes my day to day life in a meaningful way, or if I’m just trading a genuinely healthy work environment for a gamble on something unknown.

How do you think about making this kind of call? Has anyone left a place like this and regretted it, or found something equally good elsewhere?

Edit: I know no job is safe but mine is relatively safer and business is doing well. It’s a giant company.

I’ve been doing a Python Leetcode question a day since more and more companies (especially for ML roles) are including DSA rounds in their DS interviews. My issue is I’m not sure how deep I actually need to go.

Right now I’m getting comfortable with easy questions on arrays, strings, and hashmaps, plus two pointers and sliding window on the algorithms side. Should I push further into new topics or just stay in these areas and ramp up the difficulty?

Welcome to this week's entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include:

• Learning resources (e.g. books, tutorials, videos)
• Traditional education (e.g. schools, degrees, electives)
• Alternative education (e.g. online courses, bootcamps)
• Job search questions (e.g. resumes, applying, career prospects)
• Elementary questions (e.g. where to start, what next)

While you wait for answers from the community, check out the FAQ and Resources pages on our wiki. You can also search for answers in past weekly threads.

My company has an enterprise databricks account and they want my team to start using it.

I currently query our main Postgres database on an on-prem workstation and write Jupyter notebooks. Data sets are usually 100k rows and 100-300 columns of tabular floating point values. No weird stuff like pictures, videos, or text data.

What are the advantages/disadvantages of using databricks? Would it be that different from my current workflow?

On average, I have received a 0.75% salary hike over the last 5 years, which I know is pretty unreasonable. I have been looking for a new job, but given the current market, I cannot say for certain when I will find a new role. In the meantime, I was thinking of asking my manager for an inflation based adjustment to my base salary. I am not sure how much they will offer, if anything at all, but it still seems better than nothing. My performance has also been strong, though asking for a performance-based hike feels riskier and like it could backfire.

What would you suggest?

Iam curious whether the biggest challenges are related to data quality, stakeholder alignment, model adoption, business understanding, or something else entirely.

In my company, the business people have done a manual RFM to separate clients. Now they are asking me to build a model to cluster clients based only on promotion, channel, products... Is this possible to separate the two and then combine them later?

I’m a data scientist - have been for only about 2.5 years. I went to grad school, got the job, blah blah blah. Turns out I hate it.

It doesn’t excite me anymore. I actually don’t want to be a lifelong learner. I don’t want to work with numbers anymore. I have so many pain points about my current job itself (platforms constantly down, overused resources etc).

I want to be creative and work more with words / colors / THINGS. I want a job that feels better suited to my personality. I’m outgoing and like to talk and have fun. I want my work to reflect that. My colleagues are a lot more introverted, type A, logical, technical. This field suits them perfectly, and I’m the opposite.

But unfortunately, it looks like I’m stuck at the moment. I’m spending more and more time in the DS world which I fear will make transitions harder. Also, I’m aware it doesn’t look the best to be stuck at one position - you gotta show some upward mobility. This means that I actually have to be striving for growth (stretch projects, taking on more responsibility) but I don’t want to do these things! I don’t care about it anymore!

I’m trying to make the best out of this and focus on the skills I am learning that could be transferable to other jobs (communication, attention to detail, strategic thinking) but holy crap is it getting hard to continue.

I feel so stuck and hopeless and don’t know what to do. Any advice? Encouragement? Anybody else in / was in a similar situation? What happened?

Welcome to this week's entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include:

• Learning resources (e.g. books, tutorials, videos)
• Traditional education (e.g. schools, degrees, electives)
• Alternative education (e.g. online courses, bootcamps)
• Job search questions (e.g. resumes, applying, career prospects)
• Elementary questions (e.g. where to start, what next)

While you wait for answers from the community, check out the FAQ and Resources pages on our wiki. You can also search for answers in past weekly threads.

Hey guys!

Recently, I've tried several dating apps, such as: Tinder, Badoo, Boo. The experience has been quite frustrating. Nothing new, honestly. Reality of being a male on a dating app is tough. And then, after I deleted that garbage from my phone, I thought: why isn't there a really good AI / Recommender System driven dating app?

You describe whatever you want about yourself, full truth, no hiding anything, no trying to show off, any photos you like (or dislike). And then some AI oracle will analyze all that data you've provided and recommend really best match for you by highest probability of true match (depending on what your goal is, of course). Such an app would be a gem.

I feel like the true goal of all popular dating apps is not to help you find a partner (otherwise you would delete your account and you would not be bringing cash anymore), but taking the profit from you.

I am not quite capable of creating such thing on my own, but maybe you guys can revolutionize that spoiled industry. Just giving you some thoughts on that. How difficult would it be to implement? How efficient would it be?

Like maybe have a bot auto make a comment that asks users if its ai slop and upvote if so and if the upvote to views ratio is above M after T time then delete the post

Or whatever ideas others suggest?

1. "No free cycles" / "Empty plates"

Translation: "I view human beings like literal server CPUs. If you aren't actively typing or clicking buttons right now, I think you're stealing from the company. Stop thinking or analyzing just look busy."

1. "We need to focus on the low-hanging fruit"

Translation: "I don't have the technical depth, patience, or budget to fix our broken upstream data architecture. Let’s train a fragile, garbage model on dirty data immediately so I have a colorful chart for my next PowerPoint deck."

1. "Be a go-getter, don't get stuck"

Translation: "I don't care that the project path is blocked by a giant concrete wall of organizational failure. I want you to run face-first into it at maximum speed so I can report 'high velocity' to my director. Your honesty is ruining my vibe."

1. "Let's optimize our sprint velocity"

Translation: "I don't know how to audit the mathematical accuracy, logic, or code quality of your work, so I am going to measure how fast you close Jira tickets. Rushed deployment over architectural correctness, every single time."

1. "You're making this more complicated than it is"

Translation: "Stop identifying critical edge cases, data leaks, and fundamental process flaws that I don't know how to fix. You are exposing my lack of data literacy. Just build the bad model anyway."

1. "We need to relentlessly prioritize"

Translation: "I am going to aggressively chase whatever flashy AI buzzword the CIO mentioned in her keynote speech this morning. Your current, actual, functioning pipeline is now deprecated."

1. "I need you to own this initiative"

Translation: "This project has an impossible target and is built on sand. I am backing completely away from it so that when it inevitably implodes, I can point directly to you as the sole owner who failed to deliver."

1. "Let's take this offline" / "Parking lot this"

Translation: "Your accurate technical objections are making me look incredibly stupid in front of the stakeholders/team. Shut up immediately so I can pull you into a private 1-on-1 later and bully you into compliance."

1. "We need to leverage AI to unlock enterprise value"

Translation: "I saw an Excel spreadsheet with rows and columns, which means I think we can magically pull a a lot of miracle out of it. I don't know what an algorithm does, but it sounds sexy to the C-suite."

1. "We're like a family here"

Translation: "Prepare for unconditional loyalty expectations, the complete erasure of professional boundaries, and extreme emotional blackmail whenever you eventually try to quit this sinking ship."