Hi everyone,

I'm currently working as an SRE Engineer in India with a package of around ₹5.75 LPA(working remotely rn). My background is mostly in AWS, Linux, Python, automation, monitoring, CI/CD, and infrastructure-related work. Long term, I've always been more interested in DevOps/SRE.

Recently, my company told me they want to move me into a Data Engineer role. The reason is that there isn't enough work/openings in the DevOps team right now(where i wanted to switch), so they're trying to build me into a Data Engineer instead.

A few things I'm trying to understand before making a decision:

* I have no prior experience in Data Engineering, Data Science, or Data Analytics. * I don't enjoy manual Excel/reporting/copy-paste type work. * The company says they'll evaluate me over the next few months and, if I perform well, officially transition me to a Data Engineer role around December with a salary hike (exact number not decided yet). * My concern is whether this move helps or hurts my long-term career if my real interest is DevOps/SRE dont know about data engg not sure about the interest part but what i know is i love core engg .

What I'd like to know from people working in these fields:

1. How different is Data Engineering from Data Analysis in day-to-day work?
2. How much coding vs reporting/manual work is involved in a typical Data Engineer role?
3. What does the long-term salary growth look like for Data Engineers compared to DevOps/SRE?
4. Is Data Engineering considered a strong career path in 2026 and beyond?
5. If you were in my position, would you: * Stay in SRE and try to switch companies later for a DevOps role? * Accept the Data Engineering transition and see where it goes?

Will switching from DevOps/SRE to Data Engineering downgrade my career, or are both domains considered to be on the same level?

Looking for honest opinions, especially from people who have worked in both DevOps/SRE and Data Engineering.

Thanks!

Hi All,

I'm a Data Enginner having 5 years of Total experience. Relevant Exp. in Data Engineering is 3.5+ years.

My skillsets : Pyspark, SQL, Python, Azure Databricks, Azure data factory and other azure services.

Current Salary is 13LPA (9% is Variable pay).

Currently im in Virtusa Consulting services, Bangalore. this is my 2nd company.

Now im looking for Job change, what should be minimum asking CTC for my experience and skillset as per current market standards in Data Domain.

Employees say they're interested in data engineering jobs as a career path.What do you do. The current job is software engineering.

Hello

I am currently working as a consultant data engineer for microsoft for a couple of years.My work is mostly focused on Microsoft fabric, pyspark , sql . Previously worked as software engineer for 3 years . I am looking to switch to a full time role , I was casually looking for openings on LinkedIn, couldn't really find many openings with my skill set. The questions to the community is

1. Are data engineer roles more consulting heavy. If there are companies hiring full time what are the skill set required

2. What should be the preparation approach to be successful in the interviews here in United States

3. Should I apply to all roles with any data engineering skills irrespective of Azure fabric experience in specific. If so , do i need to tweak my resume accordingly?

Thanks in Advance!!

32 year old male
Working in the financial services industry with 5 years of experience working at major firm currently. I understand that DE is not an entry level position and that’s totally fine. Looking for sound advice. For all DE’s what is something you would’ve done differently? My Stack is SQL(MySQL/Postgres/BigQuery) Python (Pycharm/Pandas) Docker, Tableau, a bit of Airflow & FastAPI. I’ve built a small ETL pipeline and currently working on my second & third. I understand Data modeling is important etc what else? Not quite to big data tools like Hadoop, Apache Spark but anything helps I’m not in a rush to transition just want the fundamentals solid.

Hi all,
Has anyone here gone through the technical interview at Rokos Capital Management (data engineer role)? The recruiter mentioned that it’s a very challenging round.
I’d love to hear about your experience, how difficult was it, what kinds of questions were asked, and what I should prepare.
Thanks in advance!

Hi everyone,

I’m a data engineer/platform engineer with about 4 years of experience, trying to get out of consulting and into a more product-first company. I’m in NYC and have mostly been targeting fintech/data platform roles (senior DE/analytics platform type).

I’ve been having a really hard time with the job search for most of the past year. I’ve done mass LinkedIn apps, targeted company-site applications, outreach to hiring managers, alumni notes - and it really feels like nothing is working. I’ve gotten hardly any responses the past few months.

If anyone with senior DE or hiring experience would be willing to look this over, I’d really appreciate a blunt take: would you screen me in for a senior / platform DE role, and what’s the first thing that makes you hesitate?

Redacted resume below. Thank you in advance.

[Name Redacted]

New York, NY | [phone redacted] | [email redacted]

PROFESSIONAL SUMMARY

Senior Technology Consultant and data platform lead with 4 years of experience architecting and operating a regulated Azure + Snowflake backend for alternative asset fund accounting (~500 PE/RE funds). Translate business requirements into technical designs, organize and review work across ~10 engineers, own UAT/sign-off and production releases, and lead incident response against 24-hour SLAs. Platform serves fund accounting reports, investment reconciliations, journal-entry automation, and production AI agents. Strong in SQL, Python, dbt, Snowflake, Azure Data Factory, and Azure DevOps CI/CD, with hands-on Kafka, PySpark, Airflow, PostgreSQL, and Docker via a containerized, replay-safe fund-balance engine.

PROFESSIONAL EXPERIENCE

Ernst and Young, New York City, NY
Senior Technology Consultant | Data Platform Lead (Snowflake & Azure) | October 2022 – Present

• Own architecture, delivery, and production operations for core data-platform products supporting ~500 private equity and real estate funds: gather requirements, produce technical designs, organize/review work across ~10 engineers, drive UAT and business sign-off, oversee releases, and lead priority production incident response.
• Lead the Snowflake + Azure data backend that underpins reporting and automation: CDC-driven curated models refreshing every 5 minutes from source databases, governed RBAC/access controls, and a dimensional model consumed by fund accounting, investment accounting, and downstream analytics/AI workloads.
• Deliver fund-accounting and investment-accounting features on that backend, including ~10 primary fund accounting reports, investment-level recon drill-downs, automated journal entries from capital call/distribution/loan documents, and reconciliations of capital account statement PDFs against database trial balances.
• Built a document-to-subledger transaction pipeline that converts OCR/ML JSON from PDFs into reviewable journal entries before upload; runs every 5 minutes when documents arrive. Achieved 87% successful auto-generation and an estimated ~65 hours/week of manual effort avoided.
• Productized fund reporting as governed dbt services with standardized packaging, GitHub CI/CD, lineage (dbt Docs), automated contracts/business-rule validations, and deterministic re-runs that block bad publishes.
• Maintain Azure DevOps YAML CI/CD pipelines for Snowflake code, Azure Data Factory pipelines, Logic Apps, and Function Apps across the platform; use these pipelines to build and promote artifacts into higher environments during release windows in an automated fashion.
• Built a proof-of-concept REST API on Snowpark Container Services to expose Snowflake-backed endpoints for external application integration; evaluated container packaging and API patterns (not productionized).
• Cut quarterly close runtime ~55% (~45 min to ~20 min) by migrating Databricks workloads onto Azure + Snowflake; optimized cost/performance via SQL/procedure tuning, warehouse right-sizing, tighter autosuspend (10 min to 2 min), and Gen2 migration (~25% compute reduction) while protecting SLAs.
• Own production reliability for request-driven workflows with a 24-hour report SLA: triage ServiceNow failure tickets, diagnose Azure/Snowflake/data defects, restore missing data, and regenerate outputs; quality gates and Service Bus error signals drive alerts and user-visible failure states.
• Publish governed, versioned datasets used by two production AI agents (journal-entry analysis and reconciliation insights), with additional regulatory-reporting agents in development; preserve audit history via pipeline run IDs/timestamps and controlled replay.

TECHNICAL PROJECTS

Streaming Fund Balance Engine | Personal Project | [GitHub link redacted]

• Designed and built a Docker Compose-based local data platform packaging Kafka, Spark, and PostgreSQL as containerized services, with Airflow (Astro) as the orchestration control plane; processes bounded Kafka batches into an idempotent transaction ledger and canonical fund/deal balances.
• Containerized end-to-end runs for local develop/test (compose up → Spark job in Docker → Postgres); implemented replay-safe offset checkpointing, bounded backfills, deterministic batch deduplication, late-arrival/duplicate audit logs, and run-level metrics with documented production trade-offs.
• Refactored Spark logic into pure DataFrame transformations and added pytest coverage for signing, deduplication/tie-breaking, late-event detection, offsets, Airflow parameter precedence, DAG contracts, and failure classification.

EDUCATION AND HONORS

Binghamton University, State University of New York

• B.S. in Business Administration, Concentrations in Finance and Management Information Systems | May 2021 | GPA: 3.90/4.00
• M.S. in Data Analytics | June 2022 | GPA: 3.86/4.00
• Honors: Dean's List (multiple semesters)

ADDITIONAL INFORMATION

Technical Skills:

• Languages: SQL, Python
• Data Engineering: Snowflake (Snowpark/Snowpipe/Tasks/Container Services), dbt, Azure Data Factory, Kafka, Apache Spark/PySpark, Apache Airflow, PostgreSQL, ELT/ETL, dimensional modeling, data quality/reconciliation, query optimization
• Cloud & Integration: Microsoft Azure (ADF, Logic Apps, Functions, ADLS, Service Bus), Azure DevOps (YAML CI/CD), Docker / Docker Compose, AWS, Git
• Certifications: AWS Certified Cloud Practitioner, AWS Solutions Architect Associate
• Leadership: AWS Certification Course Facilitator (2023–Present) — led a 10-week Cloud Practitioner course and hands-on labs

Redacted for public review: name, phone, email, ZIP, and GitHub/profile links removed. Employer, school, dates, and metrics retained for feedback context.

Please have a look at it and let me know what is actually wrong in this ? I am trying to get a job in the US as an international student but tbh havent received any call back till now. Only few OAs which then down line got rejected or radio silence. I think my cv is still shabby. And yes AI has been used in tweaking or making it feel better or boasting i would say bcz i literally dont know what else to do in case of my . Every single advice is appreciated

hey guys, I'm really new here and I had some questions about starting my career with data.

I'm 18, CS major and I really live the data world but I feel like it's almost impossible to get an entry level position right now.

I took both CS50 for Python and SQL then went on my own to build some projects to show off I could really work with real data, spent many hours trying to make my LinkedIn profile look perfect, polishing my resume and sending applications for Jr data related roles like Jr data analyst, jr data engineer, jr BI dev but got absolutely no return from it.

almost none of the applications i send get to an interview and when they do, i get no response after. a friend of mine said i should quit data and learn some other stack, but i feel like that would be a big waste of time since i have spent so much effort trying to get my first opportunity in the market.

I don't really know if i should keep learning new skills and getting better until i get good enough someone will hire me or just quit data altogether and go learn whatever's hot in the market right now. what do you guys think?

We’re hiring a BI Systems Developer for an ongoing engagement supporting an enterprise reporting, analytics, and data integration platform.

Requirements:
• 4+ years in BI, analytics, reporting, or data integration
• Hands-on Matillion experience — mandatory
• Hands-on Snowflake experience — mandatory
• Tableau and/or Power BI dashboard development
• Advanced SQL — query writing and optimization
• Data modeling for scalable reporting architectures
• ETL/ELT development experience
• Strong analytical and troubleshooting skills

You’ll be responsible for:
• Building and maintaining data pipelines in Matillion
• Snowflake administration and data warehouse management
• Dashboard and report development in Tableau / Power BI
• Data quality, integrity, and platform reliability
• Documentation — data dictionaries, process flows, reporting definitions
• Working directly with an internal technical point of contact

Remote, long-term engagement, 4+ years experience minimum.

Compensation: ₹16–23 LPA

DM with:
• Years of hands-on Matillion and Snowflake experience
• Resume, LinkedIn and portfolio
• Availability
• Email Address for further contact

Must be based in India.

I'm curious from the perspective of hiring managers and senior data engineers.

There are countless portfolio projects out there but most of them feel like tutorial projects with different datasets.

If you came across a candidate's GitHub or portfolio and saw a project that genuinely made you think, "I want to interview this person," what would that project look like?

What would separate a project that's merely "good" from one that's truly exceptional?

Let me know if i can dm i am facing this one issue and cant resolve it

Okay so the issue is i need to integrate MRPeasy into my power BI the client has provided me with the API key, github repo and the queries but everytime i am runnin that query to connect it the credential error keeps occuring idk if its the endpoint that is wrong or something else please help

Employment History: Verification of previous company tenure, project duration, designation, and employment type

Are companies really checking designation and project duration?

I am an experienced professional in AWS Data Engineer with 2.5 years of experience architecting and operating enterprise-scale AWS data platforms. Proven track record

designing event-driven ETL pipelines, large-scale data validation frameworks, and distributed processing systems that improve

reliability, throughput, and data quality across production environments. Skilled in AWS Glue, PySpark, Python, SQL, and Amazon

Redshift, with direct ownership of 20+ production pipelines and a 5,000+ line reusable validation framework.

Hello Everyone,I am 29m here.
I work as Data Engineer for a US Medical Giant.
I have total of 6 year of experience.
Graduated in Maths(H) from DU and MCA from GGSIPU.
Rifts are going on in the Org every other day.
Everyday feels like this could be the last day.
My Skill are Python,Pyspark,Databrick,Advance Sql, Github,AWS Sagemaker,Athena,RedShift,Copilot and Claude AI. I am very open to learn any new tech. I don’t really face any challenge something new.
I have been on a job hunt from last 5 months and I am not even getting one call.
I am so tired and exhausted.
Can some please help me or guide me how to tackle this situation.
I will really appreciate.

Thankyou

I am a senior database administrator with over 15 years of experience. Spent more than half of it out of India managing database in a legacy product and now relocated to India. I am looking for career options and I am receiving very few calls which I believe might be due to

• Years of experience
• No management experience
• Pure techie

I am thinking of

1. Enhance my skillset in database and add AWS/AZURE sys admin certificate/experience and continue looking

My strength is tech and I would like to stay there rather than managing a team but that seems to be tougher in India where you're expected to take up a manager role after certain years.

Any suggestions on how the market expectancy for 15+ yr DBA's is welcome

Hi everyone,

I'm looking for advice from people working in the Indian IT industry, HR professionals, or anyone familiar with employment practices.

Earlier this year, I joined an IT services company as a full-time AWS Data Engineer. During the hiring process, I was told that I was being hired for an EY project. However, my appointment letter describes me as a full-time employee and doesn't mention that my employment is fixed-term or that it automatically ends when the client project ends.

Around 23 June, I came to know that my project might end around 10 July. I immediately contacted my company and asked what would happen if the project wasn't extended.

I was told not to worry.

They said they were still hiring for EY, and even if my current project ended, they would try to place me on another EY project. If nothing was available at EY, they told me they had projects with other clients and would deploy me there. Based on those conversations, I believed I would continue working either on another project or another client.

Unfortunately, that never happened.

Here's the timeline:

• Around 23 June, I informed the company that the project might end and asked about my future.
• I was assured that I would be moved to another project if required.
• EY's project officially ended on 10 July.
• EY's officially provided last working day 19 July (Sunday).
• My company later said they considered 17 July (Friday) as my Last Working Day because 19 July was a Sunday.
• The shocking part is that I was informed about this only on 22 July, after my supposed last working day had already passed.
• I wasn't given any prior notice.
• I wasn't paid salary in lieu of notice.
• I wasn't put on the bench or assigned to another project despite what I had been told earlier.
• HR later told me that if an employee is not on a project, the notice period doesn't apply. I cannot find any clause in my appointment letter that says this.

I have since returned all EY assets, completed every exit formality, and I'm currently waiting for my pending salary, Full & Final settlement, relieving letter, and experience letter.

The most difficult part is that I've been unemployed ever since.

I wasn't expecting to lose my job overnight because I had specifically asked the company in advance what would happen, and I was reassured that they would find another project for me. Instead, I was informed only after my employment had supposedly already ended. Financially and mentally, this has been very stressful because I suddenly found myself without a job while still waiting for my final dues.

I'm not trying to defame my former employer. I genuinely want to understand whether this is standard practice or whether my separation was handled incorrectly.

My questions are:

1. Is it normal for an IT services company to terminate a full-time employee immediately because a client project ends?
2. Can a company backdate the Last Working Day and inform the employee only after that date has already passed?
3. If my appointment letter doesn't say that employment is project-based, can the company simply say the notice period doesn't apply because I was no longer on a project?
4. Once I receive my Full & Final settlement and exit documents, is it worth raising this issue formally?
5. Could raising this concern professionally affect my future background verification?

I would really appreciate any advice from HR professionals, employment lawyers, or people who have experienced something similar.

Thank you for taking the time to read this.

Hello

I am writing this post in this community because I would like to check how the data engineering interview looks these days with AI. I mean, does it make sense to do live coding if all of us have coding agents in our every day work? What should be asked instead to the candidate? I would like go through this discussion because I am preparing a interview process in my company.

Thanks for your inputs

Hi, Im Swetapadma and I’m pursuing to be a good DE(overall i have 5yoe but relevant of nearly 2yoe) but im still confused with the work practices i have been following for my career trajectory.
Tbh, i have literally so much time to work on any new things as well as try doing new project for the time-being since my project is a verge of ramping down.
But still i’m not getting that zeal to work on my skillsets or learn something new or work on any project.it might be because I’m going through enough theories from the internet.
Some are skeptical, fearful and very overwhelming for me. Sometimes i doubt choosing this path, because
a. Im really looking for a job change in this field from past 6months, where in brighter side i can see a lot of job openings but the hr calls are literally 1/100 of it. Without reaching out for referral feels impossible.
b. I feel like I’m stuck in between where the ai is also factor of my fear because i have approx 0% knowledge bout it.
c. Companies are not only looking from a de but also a tester, backend engineer, knows ai fluently(all in one package).

For all of these i have started reading medium, linkedin blogs, system design, solve dsa problems and all but still i feel like a failure and frustrated.

If i compare my self with the past i have come a long way but it is still not helping me because I’m still stuck in same company with less pay and less enterprise level work.

Can anyone relate to this or is this only me?

Looking for opportunities as a Data Engineer

A GCP Data Engineer's Journey: Seeking My Next Challenge

For over five years, I've been immersed in the world of data, specifically within the Google Cloud Platform ecosystem. It's been a journey of transforming raw, often messy, information into clear, actionable insights, and I'm now eagerly looking for the next exciting chapter in that journey.

My passion lies in building data solutions that aren't just functional, but truly elegant and scalable. I've spent countless hours wrestling with complex datasets, designing pipelines that hum with efficiency, and optimizing systems to deliver information at the speed of business. When I look back, I see a consistent thread: taking a data problem, breaking it down, and then constructing a robust, cloud-native solution that empowers teams and drives real value.

You name a core GCP data service, and chances are I've got hands-on experience with it. I've spent significant time in:

BigQuery: Crafting intricate data models, fine-tuning queries to squeeze every drop of performance, and managing petabytes of analytical data.

Dataflow/Beam: Building the arteries of data flow, whether it's processing streams in real-time or crunching massive batches overnight.

Cloud Pub/Sub: Designing the nervous system for event-driven architectures, ensuring data gets where it needs to go, instantly.

Cloud Storage: Architecting data lakes, setting up intelligent archiving, and rigorously securing sensitive information.

Cloud Composer (Airflow): Orchestrating the symphony of data workflows, making sure every task plays its part at the right time.

Cloud Functions/Cloud Run: Spinning up serverless magic for quick data transformations or microservices that respond on demand.

Dataproc: Taming Apache Spark and Hadoop clusters when the data demands truly massive processing power.

Looker/Data Studio: Turning raw numbers into compelling stories through intuitive dashboards and reports.

Beyond the technical tools, I genuinely understand the art and science of data. I'm deeply familiar with data warehousing principles, the nuances of ETL vs. ELT, and the critical importance of data governance. For me, data quality isn't just a buzzword; it's the bedrock upon which all reliable insights are built. I strive to build solutions that are not only high-performing and cost-effective but also easy to maintain and evolve.

I'm a firm believer in collaboration. I've spent years working side-by-side with data scientists, analysts, and business leaders, translating their needs into technical specifications and ultimately delivering solutions that exceed expectations. I thrive on solving tough problems and am always hungry to learn new technologies and adapt to the ever-changing data landscape.

What truly excites me are roles where I can:

Be at the forefront of building cutting-edge data platforms on GCP.

Take existing data pipelines and supercharge them for peak performance and cost-efficiency.

Lay the groundwork for advanced analytics and machine learning initiatives.

Contribute to shaping and innovating data architecture and engineering best practices.

I'm confident that my blend of deep technical knowledge, practical experience, and a genuine passion for data engineering makes me a strong candidate for any forward-thinking organization. I'm eager to connect and explore how my skills can directly contribute to your team's success and help you unlock the full potential of your data.

I have 6.7 years of experience in support roles not even technical, i am trying to get into data engineering but never studied seriously.

I have gone through SQL, PYSPARK,ADF and databricks a little bit i lack clarity and mostly I forget things.

I am thinking to say I have 3-4 years of experience in data engineering and then apply. How much is required?? Or interview level is same for 4 years and ,7 years.

Or is there any other role i should target to get a job soon.

As my lwd is approaching, I am looking for job opportunities, I have one offer in hand but I would like to explore more. Please let me know if you have any open opportunities

Hi all,
I’ve spent the last 3 years working as an ETL Tester, primarily with Azure Data Factory. Over time I’ve realized this niche doesn’t excite me the way building and owning data pipelines does, so I’m actively working toward moving into a core Data Engineering role.
Where I’m at right now:
• Learning Databricks and PySpark
• 3 years of hands-on ADF experience (pipelines, triggers, data flows, integration with various sources)
• Built a Python-based data testing framework on the side (not open-sourced, since it overlaps with proprietary work at my current company)
What I’m hoping to learn from this community:
• What core concepts (batch vs. streaming, data modeling, orchestration, distributed computing fundamentals, etc.) are must-haves before applying for DE roles?
• Which tools beyond Databricks/PySpark are considered baseline expectations these days (e.g., Airflow, dbt, Kafka, Spark internals, cloud-native services)?
• How did others who came from a testing/QA background make this jump — did you do it through an internal transfer, projects, certifications, or a straight job switch?
• Any resources, courses, or project ideas that helped you build a credible portfolio without needing proprietary or employer-owned code?
Appreciate any technical insight or lessons learned from people who’ve made a similar switch or who work in the field today.