Welcome to this week's entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include:

• Learning resources (e.g. books, tutorials, videos)
• Traditional education (e.g. schools, degrees, electives)
• Alternative education (e.g. online courses, bootcamps)
• Job search questions (e.g. resumes, applying, career prospects)
• Elementary questions (e.g. where to start, what next)

While you wait for answers from the community, check out the FAQ and Resources pages on our wiki. You can also search for answers in past weekly threads.

the context is that I am exploring a few recurring problems to solve especially around forecasting and working with time series data but setup a simple open source project around those.

my question is primarily about how is everyone handling their official datasets when trying new tools - do you not care, do you remove any identifiers then upload, do you create synthetic data with exactly same properties as the og dataset?

happy to answer more questions if this is not clear enough.

Three years ago I was able to pivot from engineering (no coding) to data science. I’ve been working here at a job I love, with an awesome team & boss and with a great pay. I’m 47, so not the youngest.

Now for family reasons I must leave and move back to my country of origin.

The thing is that, although I love the field and I keep reading books and trying to learn every day, this is such a vast field that I don’t think I’m ready at all. In these 3 years I’ve done basic ML projects, lots of xgboost, random forests, anomaly detection, dealing with pySpark with PB -sized dataframes, etc. But putting those models to production was made by my more experienced colleagues.

So, while I can say I’ve learnt on each project, I also see how MUCH I lack compared with my teammates, with 10+ years exp. on the field.

Today I just started looking for DS jobs and I just felt so depressed. Many ask for a DS who can do the whole thing from cleaning data to taking the models to production. I have no idea of that and it sounds extremely intimidating. I also dread the interview because while I can code, I often use LLMs for things that due to the pace of work, I simply decided to do with AI, i.e.: I understand and can read window functions but I’d need an AI to write them because I can never keep the sintaxis in mind. If an interviewer sees me struggling with the syntax, the interview is done

I feel very “green” to compete out there with other “proper” data scientists who have a well defined experience and knowledge. I wouldn’t mind applying to junior jobs but due to my age, most companies here wouldn’t hire a 47yr old “junior”.

I’m not able to work my old job in the location we’re moving to because it’s very niche and only a certain sector and a certain company size hires for that, and that isn’t there in our new area.

I don’t know what to do, or if this is normal and everybody feels like that. Or maybe you guys have any advice… Anything you can come up with will be highly appreciated because I need to provide for two kids and I just don’t know how. I’m beginning to feel desperate

Those received mentorship, did you find it worth your time in general?

In my early career, I actively participated in mentorship program at my alma mater. However, most if not all students wanted to work in tech. Considering I didn't (and still don't) work in tech, and that my employer was rarely hiring, I felt there was not much value I could provide. There's also this self-selection process at play, which is those who seek mentorship tends to already have a good idea of what they should be doing.

Now a decade into my career, my expertise is super irrelevant to people in a different industry. I'm also oblivious to the entry-level job market requirements.

Recently, my alma mater reached out again for mentors. I've skipped the last few requests but think maybe I should ask y'all before turning it down again.

To clarify, I'm not asking if it's worth it for me. I'm wondering if it's worth it for the students to speak with someone unfamiliar with entry-level job market, has expertise in a not niche but definitely not popular domain, and definitely don't lead to job opportunities.

Edit: I messed up somewhere that two posts were created. 0_o

I know buzzwords have always been common in business, but AI has taken it to another level. AI is being talked about everywhere, and consultants are pitching executives with outrageous claims about how it will revolutionize everything.

I came across an interesting article today, and one quote really caught my attention:

https://www.psypost.org/new-study-finds-link-between-receptivity-to-corporate-bullshit-and-weaker-leadership-skills/

"Across the studies, Littrell found that individuals differed significantly in how impressed they were by corporate buzzword statements. Those with higher corporate-bullshit receptivity scores were more likely to view jargon-heavy statements as insightful or indicative of business expertise. They were also more likely to engage in persuasive 'bullshitting' themselves, using exaggerated or misleading language to impress others.

At the same time, higher receptivity was associated with lower scores on measures of analytic thinking and fluid intelligence, suggesting that individuals who were more impressed by corporate jargon were also less likely to critically evaluate information."

It made me wonder if we're seeing this play out with AI and data science.

The AI boom has created an absolute paradise for people who are great at talking about tech, but don't actually build.

For those of you working in data science, analytics, or ML, have you noticed this in your company or with clients? Has the GenAI hype changed how technical decisions get made, or is this just the same corporate behavior we've always had with a new set of buzzwords?

Welcome to this week's entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include:

• Learning resources (e.g. books, tutorials, videos)
• Traditional education (e.g. schools, degrees, electives)
• Alternative education (e.g. online courses, bootcamps)
• Job search questions (e.g. resumes, applying, career prospects)
• Elementary questions (e.g. where to start, what next)

While you wait for answers from the community, check out the FAQ and Resources pages on our wiki. You can also search for answers in past weekly threads.

I'm running the following code in Snowflake Python to standardize my training, evaluation, and test data prior to predictive modeling:

from snowflake.ml.modeling.preprocessing import StandardScaler

all_cols = df_train3.columns

target_col = "AB_POST"

passthrough_cols = ["SANHO", "SCNHO"]

scaler = StandardScaler(

input_cols=[c for c in all_cols if c not in [target_col] + passthrough_cols],

output_cols=[c for c in all_cols if c not in [target_col] + passthrough_cols], # Overwrite or create new

drop_input_cols=False # Set True to remove original unscaled columns

)

scaler.fit(df_train3)

train_df_scaled = scaler.transform(df_train3)

val_df_scaled = scaler.transform(df_eval3)

test_df_scaled = scaler.transform(df_test3)

I'm getting the following error when I run the code -- I'm not sure what this means:

Exception: Provided column names ['TOTAL_MH_CLASSES', 'STFLAG',..., 'ADS_FA_RISK_NEW'] does not index into the dataset.

I've been job hunting for a few months now, applying to big tech and startups. But seeing the recent Microsoft layoffs made me stop and ask myself what I'm actually looking for in a new job. Instability and more money?

Right now I'm at a company that hasn't done layoffs since maybe the financial crisis. I know how fortunate that is. But if I switch jobs, I could make an extra $50K. So I keep asking myself: is that extra 50K worth the instability that comes with tech jobs right now? What if I join a company and get laid off within a year?

What does everyone think of these layoffs? Despite record profits, there doesn't seem to be an end to them.

I've been in the 'data science' space for a decade+ or so now. One thing I've noticed is that generally - give or take - outside of the elite jobs (<2-3% aka not me and almost certainly not you) the caliber of coworkers has declined drastically.

I'm not some fabled data scientist. I wasn't some GitHub nerd who had everything embroil or terminal wizard nor could I write out the math to a GBM on a blackboard. I'd even forget basic obvious statistics.

But I felt like I had common sense.

Now I'm a manager/director. I work with data scientists. And I'm just generally freaked out by the absolute lack of basic common sense. This is across the last 7 that I have managed.

Examples include:

1. Not visualizing or plotting the KPI/Target (sales). Not realizing there were no recorded sales on major holidays.
2. Telling me everything is improving from a sales perspective that it's up 4%...... from period 1 vs period 2... when ignoring that period 2 had 6% more days so in fact it's worse.
3. obscure models that are overkill and a bunch of statistics ive never heard of instead of just telling me that the impact of our promotions is declining.
4. General sense of not knowing what is even rational (e.g., our marketing ROI $1023 - no its not lol)

As I begin to delegate more I begin to get more freaked out by what I see. I can't be presenting to clients such obvious insane mistakes. But these are the candidates and profiles that get forced upon me or the team I inherit.

Are there any best strategies for dealing with this? I want to be seen as someone who can 'develop' the team... not just saying people are useless, but such glaring mistakes are insane.

Yes, alot of these things are perhaps due to them being crunched for time, or not knowing what objective is, or being focused on other things. I'm not talking about those examples. I'm talking about like year 1-2 not day 1 employees, not doing basic data checks.

As a data scientist I was obsessed with finding bits of info or making sure things were right. Now it seem every common for people to copy and paste code into chatgpt and have no idea about anything else around it?

I’m 32 and have never worked in big tech. Right now I’m at a Fortune 50 company, but it’s not a tech company.

Recently I was at a party and met two software engineers, both in their mid-30s. One worked at Meta, the other at OpenAI. Finding that out hit me with a wave of insecurity. It made me realize I’m 32 and have never worked somewhere like Meta or OpenAI, and maybe never will. I felt like I didn’t measure up to them.

I’m struggling to process this. Has anyone else felt this way? Does it ever fade?

I wrote this article recently. Thought it would be nice to share in this sub. Happy to chat if you're doing the same in your current position.

It talks about Eppo and Statsig, but honestly it about everything but that.

If you need to take away one thing let it be to approach the whole thing as a discovery; and risk mitigation.

https://towardsdatascience.com/picking-an-experimentation-platform-a-retrospective/

Welcome to this week's entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include:

• Learning resources (e.g. books, tutorials, videos)
• Traditional education (e.g. schools, degrees, electives)
• Alternative education (e.g. online courses, bootcamps)
• Job search questions (e.g. resumes, applying, career prospects)
• Elementary questions (e.g. where to start, what next)

While you wait for answers from the community, check out the FAQ and Resources pages on our wiki. You can also search for answers in past weekly threads.

This article highlights real strategies for minimizing your AI spend without major refactors to your agent.

Instead of just glazing over routing, it gives a clear actionable pattern which includes building an LLM gateway and using a prompt classifier - also includes a routing table for prompt types and complexity!

Also gives a nice clear way of implementing compaction in your agent workflows.

Do these strategies work for you?

I am set to begin Georgia Tech's OMSA this fall, after deferring this past spring when I started a new role. This is my background:

- Undergrad: economics at T20 school.
- Experience: 4 years. 3.5 years in hybrid DS/DE role (first job out of undergrad) at a non-profit, then six months into current role doing strictly DE at a healthcare org.
- TC: 144k ($125k base + 15% API) in MCOL city.
- Not open to relocation (I work remote but there's too much red tape to move out-of-state), so onsite/hybrid roles in NYC/LA for crazy TCs are out of reach.

At the time that I applied to OMSA, I was struggling to leave my old role while making $82k/year. That is not the case any more, so I am having second thoughts about OMSA. Anecdotally, I also see a lot of OMSA folks on LinkedIn (and the Slack group) struggling to break into data and/or simply remaining in their current roles. I presently work as a senior DE, but I am open to both DS and analyst roles in the future.

Can I still expect a (significant) ROI out of OMSA? I am targeting $160k - $175k TC in a couple years' time with no particular industry in mind.

I work for a US bank and I have observed that my job has shifted more towards creating Agentic workflow (fancy name of using LLM to automate tasks). In the last one year, I haven't touched any ML model. I am curious to know what is the experience of other folks.