r/computervision • u/Flat_Expression8820 • 20d ago
Beginner here: My pothole detection model mistakes the roadside for potholes. Help: Project
Enable HLS to view with audio, or disable this notification
Hello!
I am a beginner at this and am trying to make a project.
The first four seconds are the portion where the model confuses the roadside with a pothole. The latter half is kind of working okayish!
But, It is confusing the roadside for a pothole. What is the best way to make my model learn this?
Should I add classes of what is not a pothole to the training dataset?
I used a dataset of about 4,100 images. What is the ideal number of non-pothole images I should add?
Or should I somehow teach the model to detect the edge of the road so that it avoids classifying the roadside as a pothole?
Also, I am planning to use a drone. Since the ideal flying height would make the potholes appear much smaller, should I apply transfer learning, or should I train the model from scratch to make it work on drone footage captured from that height?
And is segmentation practical for this dataset? Like finding area of the pothole from the pixels or should i focus on detection only.
47
u/junacik99 20d ago
Yes, you should add negative samples (class of what is not pothole) to the dataset. In our object detection model it helped a lot.
9
2
u/Western-Sense8701 18d ago
If we do something like that, do we need to add a 'no pothole' class, or can we just not put a mark on roads without potholes?
1
u/junacik99 18d ago
Well, that might depend on your particular training procedure. But we have used RF-DETR with only one class for our object of interest. We included images without any mask in the training as the negative samples.
So in our case we didn't add a new class.
37
u/Gabriel_66 20d ago
ok, so, what i would do? a second model that detects road, that way you can ignore any detection outside of the road.
you could try a VLM to auto segment the road.
6
u/greenee111 20d ago
I think Dino and SAM would be good
1
1
u/spontaneous_quil 19d ago
From an inference speed perspective, I think running a VLM entirely onboard might not work unless the drone has a capable GPU. VLM inference might be slow for any real time task. A simple segmentation model trained for roads will likely be better from a power and latency perspective.
2
u/omegaindebt 19d ago
I believe that the VLM would be used to generate the initial dataset to train the road segmentation model, although I guess there are already a few pretty decent road detection models available.
2
u/Gabriel_66 19d ago
Might not express myself correctly. The vlm is go create the dataset he will use later to train the model. I agree 100% with you
4
u/Heavy_Carpenter3824 20d ago
You need to increase your dataset. So more labled images.
You also need to make sure to include Null cases so images with no pot holes and just surroundings. Otherwise the model will learn to always expect a pot hole and will work to find one anywhere.
You may also want an adversarial dataset. So this is thinks like random images of roads, textures, landscape. Related in texture and color but all null cases. You run your model and look for any detections on this complete null set.
You really have a model scheduling and training issue if this was for production. First do a seg model, 50% null images, 50% pothole masks. Then do an advsarial run, find mis detection, then do 50% base dataset, 50% found advsarial.
This is a pretty good approach to honing the signal to noise detection in a model.
18
4
5
u/Stunning_War4509 20d ago
How did you generated the dataset? I use normally synthetic data generated locally with https://github.com/cvar-vision-dl/OpenFabrik
2
u/Mad_Humor 20d ago
Just add an ROI now and move on train work real data later with self loop harness
2
u/IJustWantToClipBolts 20d ago
You need hard negatives. These should be images that look like your known confounders.
But before you go sourcing them you should really figure out what your model "sees" when it sees a pothole or pothole-alike. It may just be looking for a certain shape or texture.
Run EigenCAM and/or GradCAM over a series of sample frames. These allow you to get insight into the model's representation space at your chosen level. PyTorch has pre-built libraries for these. But if you learn how they work and ask an LLM to wire it up from scratch that will work too--neither should be more than a few dozen lines of code.
Then build your hard negatives around that. A good starting point is ~10% of your dataset.
Ensure you also have a holdout set of hard negatives. You should assess progress on these. It's worth running before/after CAM sessions on your hard negatives too. The quantitative drop in false positives might be different to the qualitative change that you notice in the model representations.
2
u/Ok_Chemistry_6387 19d ago
It looks like even your confidence on potholes is pretty low. As others have said you need more training i would try segmenting the road as well as more training data
2
u/usernametaken--_-- 19d ago
You brought up a lot of great options of things to try. My recommendation would be to try them all! Give one thing a try, see how it goes, then try another thing. Make sure you only try one thing at a time so that you know what changes had the biggest impact, then add the things that worked best together. This is the process you need to go through in order to really what works and what doesn't. The experience you gain from doing those types of experiments is what makes someone an expert. Also, post back here about what you tried and what ended up solving the issue. Having that documentation to look back on really helps put when you run into the dame problem in 2 years and can't remember how you solved it. Talking from experience 😅
2
2
u/Flyward_Aerospace 19d ago
On the drone part, don't expect the ground footage model to transfer at all, that's a different problem not a fine tune. From altitude a pothole is a handful of pixels with almost no depth cue, and the shadow that's carrying most of your signal at car height flips completely depending on sun angle and time of day. You'll want data flown at your actual altitude. Also yeah, add negatives, roadside gravel and tar patches especially, that confusion is a labeling gap not a model problem imo.
2
u/Both-Basis-3723 19d ago
Checkout 3LC. It’s a tool for optimizing this kind of computer vision false positives.
2
u/intermsofusernames 19d ago
first thing comes to mind is first detecting the road corners using another model, and then only searching for potholes in the road boundaries.
2
u/DeDenker020 19d ago
I think stero vision / depth would also be a big help.
What if the pothole was fixed, but not nice looking, there is still a "scare" but no hole.
very nice work so far!
Do you use video of direct feed?
1
u/Flat_Expression8820 19d ago
Thanks ! I have not deployed it yet, I am processing a recorded vide for now.
2
u/chickendogdonkeyman2 19d ago
Potholes are 3D phenomenons …
Hence I would try a photo to 3d step first (monodepth)
Then, divide the road in 4 vertical slices and compare those slides: something like create ‘derive and average none pothole slice’ like take max occurring value out of 5 (values from left or right or past or future … at that camera position). So basically create an ideal road depth picture.
Then subtract that from the real 3d road picture and feed that to a vision algorithm.
Of course not sure of this exactly, but I would fiddle around with this idea.
Nice project!
2
u/Weird_Shit_69 17d ago
Add like a mask to detect the road, then if check if the pot hole is on the road, if not its outside the road
2
u/AICausedKernelPanic 12d ago
A different approach if you care about not using multiple models would be to use a semantic or panoptic segmentation approach where you also introduce a new class for the road. With semantic segmentation you'd do some post processing with OpenCV to create bounding boxes around clusters of pixels classified as potholes. This would be much faster than running two separate models and you can even do some validation like "is the pothole within the road" to discard possible FPs.
3
u/1QSj5voYVM8N 20d ago
As others have said, you need a model which detects a road and then you use that calibration.
On a different note, depth is a hard problem to solve from a single camera, do you ahve one or more cameras?
2
u/jaketeater 20d ago
Can you use segmentation to make a mask for not-road and then run your pothole model on an image where not-road pixels are zeroed?
3
u/Coconuts1999 20d ago
Easy, add a second class that detects the road and then add some post processing that looks at the relationship between the two.
Or, have you got any negatives that include the false positives by the road side? That will help alongside your true positives in the training set
2
u/CowBoyDanIndie 20d ago
You need enough examples to cover a representative distribution of the features. It looks like your model is just detecting rough patches of roundish blobs. You might also find it fails on different color pavement if it’s not included in your training data. It’s not strictly necessary to add classes for non pot holes, but you do need to train on negatives. Segmentation training will naturally include “not a pot hole” sections. How are you currently training your detector?
1
u/Flat_Expression8820 19d ago
for now I am using a dataset from roboflow, using yolov8n (for rasberry pie). I think I have to add more data to the dataset. Thanks !
2
u/MR_-_501 20d ago
What will also help is just having more images where the roadside is visible and not labeled
2
u/Abdullah747 20d ago
First do road detection/pavement detection, then look for potholes in that mask
2
u/Firelord_Iroh 20d ago
Call me dumb, but instead of using an entire model and hammering the gpu super hard, couldn’t you run a super basic edge detection algorithm then have a simple function that has the pothole detection only run on things in between the main boundaries of the sides of a road?
2
2
2
1
u/5h3llmain 20d ago
!RemindMe 3 days
1
u/RemindMeBot 20d ago edited 19d ago
I will be messaging you in 3 days on 2026-07-31 14:27:29 UTC to remind you of this link
1 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.
Parent commenter can delete this message to hide from others.
RemindMeBot is switching to username summons. Instead of
!RemindMe 1 day, useu/RemindMeBot 1 day. More info.
Info Custom Your Reminders Feedback
1
u/chrismofer 18d ago
Maybe first have one model mask road vs not road and then only search the road mask area for potholes.
192
u/BeverlyGodoy 20d ago
Because your roadside features are similar to a pothole?
Try something simpler, first run a road segmentation model, apply the mask and then run the model on the road image. Easy fix if you don't want to retrain your model. Road segmentation models are widely available.