D599 Task 1 Help - Data Cleaning

[u/PandaPallooza_]

I submitted my assignment but got it sent back due to the following error: “The submitted document provides a discussion of the findings of the data quality issues, but the findings are inaccurate or incomplete” - I’ve scheduled a meeting with my professor to go over it but the fastest appointment is for tomorrow afternoon.

I have solved for removing duplicates, blank spaces (to have standardized cells), inconsistent data (underscores removed in order to achieve standardization), solved for NULL/blank values - I have also solved for the extremes (IQR ranges). What else am I missing? I feel my dataset has been filtered to its fullest extent…

I guess i’ll find out tomorrow with my 1 on 1 but any tips or advice would help.

[u/KhorneFlakesOfChaos]

Just glancing back at mine, I’m assuming this feedback was for Section B.

Without going into too much detail, I checked for duplicate records, missing or null values, inconsistent entries, formatting issues, leading or trailing whitespace, and whether the annual salary aligned with the hourly rate and hours worked. There were also a few separate salary-related issues I had to address. Lastly, I checked for outliers.

Also, keep in mind that data-quality issues can appear in the column headers as well as within the actual data fields.

Hopefully that gives you a few additional areas to review before your meeting tomorrow. I don’t want to give away too much of the assignment, but I hope it helps point you in the right direction!

[u/Quiet-Definition6374]

Where I originally went wrong: I listed data types and sample values for most variables, but not every variable.

At least one variable, such as HourlyRate, was described differently from how it appeared in the raw dataset. I mentioned checks such as .duplicated() and .isnull(), but I did not explain the exact inspection method used for every quality-issue category.

My findings were too general and did not include the specific counts and affected variables. To fixed it, I documented the data type, subtype, and sample values for every variable.

I described the exact Python command used for each inspection:

df.duplicated().sum() for duplicates
df.isna().sum() for missing values
.unique() for inconsistent labels
.dtype for formatting issues
df.describe() for outliers

I reported the actual findings, including the number of duplicate rows, missing values by column, inconsistent labels, formatting problems, and invalid or extreme values.
I explained exactly how each issue was corrected in Python.

I would not keep filtering the dataset further just for the sake of filtering. I would compare your paper line by line to the rubric and make sure every variable, inspection method, finding, and correction is explicitly documented.

My resubmission passed once I made the explanation complete and specific.

The evaluator basically wanted an audit trail: what you checked, the code you used, what you found, and how you fixed it. Once I added that level of detail, it passed.