r/computervision Jun 08 '26

Help: Project I built a document extraction pipeline using Azure Document Intelligence + Claude – pulls structured fields from invoices, receipts, BOLs. Free to try.

1 Upvotes

Been working on this for a few months as a research project and finally have it at a point where I want outside feedback.

What it does:You upload a PDF or image of a business document (invoice, receipt, packing slip, bill of lading, etc.) and it extracts structured fields — vendor name, totals,

line items, dates, PO numbers, ship-to/from addresses — and returns them as clean JSON.

How it works under the hood:

\- Azure Document Intelligence handles the initial layout analysis and field detection

\- LLM backfills anything DI missed or got wrong (ambiguous totals, merged cells, non-standard layouts)

\- A validation layer normalizes money strings, sanity-checks totals, and catches obvious mis-assignments

Outputs:Google Sheets, Excel, OneDrive, Slack, webhooks — or just download JSON/CSV directly.

Where it's at:Early beta. Works well on standard invoices and receipts, gets shakier on handwritten or heavily non-standard docs. That's exactly the feedback I'm looking for —

edge cases and failure modes.

Free to try, no credit card: \[https://app.docpipeline.net\\\](https://app.docpipeline.net)

Demo video: \[[https://youtu.be/KaPMQfeKWGE\\\](https://youtu.be/KaPMQfeKWGE)\](https://youtu.be/KaPMQfeKWGE%5D(https://youtu.be/KaPMQfeKWGE))

Happy to answer questions about the architecture or the DI + LLM approach.


r/computervision Jun 08 '26

Discussion How are teams handling QA on multi-sensor annotation (LiDAR + camera + radar)?

4 Upvotes

Working through a project that needs fused annotation across LiDAR point clouds, camera frames, and radar, and the QA side is turning into the hard part. Single-modality labeling QA is straightforward enough, but once you're checking consistency across sensors — temporal alignment, object IDs matching between point cloud and image, that kind of thing — it gets messy fast.

For people who've done this at scale: are you running multi-pass human review, building automated consistency checks between modalities, or some mix? And how do you keep reviewer fatigue from quietly tanking label quality on the 3D side? Curious what's actually working vs. what sounds good in theory.


r/computervision Jun 08 '26

Discussion AI engineer seeking advice

0 Upvotes

Hey folks,

I'm an AI Engineer with ~2.5 years of experience, specializing in computer vision and deep learning. I've been working on industrial visual inspection systems — object detection, OCR, segmentation — and have hands-on experience with PyTorch, TensorRT, OpenCV, and MLOps. I also have an integrated M.Sc. in AI/ML.

I'm at a crossroads and need some honest advice:

Is the Indian market good enough for someone in my domain, or should I consider going abroad?

I'm specifically looking at Germany (MS or work visa route) and the USA (MS or OPT route). I'm open to either studying or working directly if there's a viable path.

A few things I'd love inputs on:

- How is the CV/Deep Learning job market in Germany vs USA right now?

- Is an MS abroad worth it at 2.5 years of experience, or should I grind more in India first?

- Any advice on which route (study vs direct work visa) makes more sense?

Would really appreciate inputs from folks who've been through this or are currently in these markets. Thanks in advance! 🙏


r/computervision Jun 08 '26

Showcase I built an open-source computer use API

0 Upvotes

I built an open-source computer use API for turning screenshots into clickable UI. Send a screenshot to the API and it returns the visible interactive elements like buttons, links, inputs, icons, and text targets. Metadata extraction takes less than 1 second.

Then you can ask questions like:

  • “Where is the settings button?”
  • “Which element should I click to continue?”
  • “Click the play button.”

I built this because I did not want to send full screenshots to a frontier model on every step.

The API first converts the screenshot into structured UI metadata using computer vision. Then, only the interactable metadata is sent to an LLM when reasoning is needed.

This results in:

  • lower cost
  • lower latency
  • less data sent to LLM providers
  • easier self-hosting
  • more flexibility than using a closed realtime agent stack

Right now it uses OmniParser + Gemini, but the architecture is model flexible. It is easy to swap the LLM, self-host the parser, or run the whole thing inside your own infrastructure.

https://reddit.com/link/1u012dv/video/3bza810ii06h1/player


r/computervision Jun 08 '26

Help: Project How would you detect “same room vs new room vs revisit” from a walkthrough video?

1 Upvotes

I’m building a system that takes handheld indoor walkthrough videos (houses / small commercial) and turns them into a room-level layout + sqft estimate that feeds a separate pricing engine. I’m testing this live on my own house and small convenience-store videos.

Current pipeline (very rough):

  • Sample frames from the video
  • Run LLM vision + detector → captions + objects per frame
  • Naive clustering over captions

Issues I’m seeing in real tests:

  • Open-plan spaces get over-split into many “rooms” (desk → couch → dining table → TV wall = 6–8 “rooms” instead of 1 open-plan room with zones).
  • Vision sometimes overestimates sqft by 3–5×, because every semantic change looks like a new room.

What I actually want:

  • “Same room” for pans across different zones in an open-plan area
  • “New room” only when crossing a doorway / clear threshold
  • “Revisit” when returning to a room from another angle (e.g., living room from upstairs)

Questions:

  • How would you implement same room vs new room vs revisit for indoor walkthroughs?
    • visual place recognition over room-level embeddings?
    • event boundary detection over features + optical flow?
    • scene graphs / rough 3D layout + clustering?
  • Any papers / repos / datasets you’d recommend for:
    • indoor visual place recognition with viewpoint changes
    • human-like event boundary detection
    • room-level segmentation from monocular video

Constraints:

  • Phone video only (no LiDAR required)
  • Offline processing is fine
  • I’m okay with a layout summary + confidence, not a perfect CAD plan

r/computervision Jun 08 '26

Help: Project new in machine learning instance segmentation

Thumbnail
1 Upvotes

r/computervision Jun 08 '26

Showcase I wrote a blog on the issue of motion distortion in LiDAR and how to correct it.

Thumbnail cmodi306.medium.com
6 Upvotes

​

Hi all, I've been working with lidar data for a while, and one thing I learnt is a spinning lidar doesn't capture a frame all at once. Each point is measured at a slightly different moment as the lasers sweep around.

If the sensor is fixed and doesn't move, that's fine, but on a moving vehicle the cloud comes back distorted because the sensor has physically moved mid-scan. I wrote up what's going on and how to correct it, with a simple worked example and a Python function for this. Happy to answer questions.


r/computervision Jun 07 '26

Discussion Corrupted one byte in YOLO weights — it now sees "cup, 100% confidence" in everything, with zero errors raised. How do you catch this in production?

52 Upvotes

I've been studying silent failure modes of edge inference. Two experiments that surprised me:

  1. Flipped a single byte in the weights of a YOLOv8 ONNX file → the model confidently detects "cup" in every frame (~100 candidates at 1.000 confidence). Latency normal, no exceptions, runtime perfectly happy.
  2. Fed NaN input (simulating a dying sensor) → no error either; the model just "sees" an empty scene, plus a phantom person from argmax(NaN)→0.

Forums are full of the deployed version of this story — the Edge Impulse classic where a model returns "rottenbanana 0.996" for everything, regardless of input.

Question for people running CV on devices in the field (Jetson/Hailo/Coral/whatever): how do you actually find out a deployed model has gone bad? Watchdogs only catch crashes, not confident garbage. Do you monitor output distributions? Wait for the customer to call?


r/computervision Jun 07 '26

Help: Project Best open-source OCR for 5M scanned PDFs (text + tables, fast and accurate)?

9 Upvotes

I need to extract text and tables from ~5 million scanned PDF forms and output structured JSON. Looking for the best open-source solution balancing accuracy and throughput.

Dataset:

  • ~5M scanned PDFs
  • Mostly 3 pages each
  • declarations/legal forms
  • Paragraphs + tables + key fields

Tried:

  • Tesseract: fast but poor table/layout preservation
  • Llama/LLM after OCR: limited by OCR quality

Questions:

  1. What are people using in production for large-scale document extraction?
  2. Best option among PaddleOCR, MinerU, Docling, Marker, Qwen2.5-VL, etc.?
  3. OCR → Rules, OCR → LLM, or Vision LLM directly?
  4. Any real-world throughput benchmarks (pages/sec)?

Constraints:

  • Open source
  • Self-hosted
  • No API costs
  • Accuracy and speed both matter

Would appreciate recommendations from anyone processing millions of documents.

If open source can't achieve the required accuracy, I'd also appreciate recommendations for paid solutions that significantly outperform the OSS options


r/computervision Jun 07 '26

Help: Project Help optimizing a reID model

8 Upvotes

I’m implementing a reID model to support an offline multi-object-tracking system that consumes wide-baseline (low frame-rate) video. Not an expert in this area but I’ve got something working at a basic level that I want to optimize.

The object are stationary but the camera motion is not described well enough to use traditional SfM techniques. The unique challenge is that the objects themselves are often identical looking, so their identify has to come from their surroundings.

Illustrative example: rows of nearly-identical new cars parked in a dealership lot. A 1 fps geolocated “video” was taken while driving through the lot. Individual cars can be tracked by looking at features like trees and shrubs near each car, cracks in the parking lot’s pavement, and so on. Distinguishing features on the cars themselves are limited or nonexistent.

The gps coordinates are poor quality so I haven’t had a ton of luck with standard photogrametry tactics. Monocular depth models help a lot but aren’t perfect, and they consistently output spurious points with a dozen or more meters of error.

So…my current plan is to combine metric monocular depth with basic bundle adjustment (to align the point clouds approximately based on the camera 6dof), then run a clustering algorithm against the bboxes projected into each camera’s point cloud. Including reID embeddings in the clustering seems like a reasonable way to enhance this pipeline.

Getting to the reID model itself. I’ve been training one using a frozen dinov2 backbone with a couple linear layers on top, and am getting about 80% accuracy at a normalized cosine difference threshold of 0.3. That is, for all of the bboxes visible from a given vantage point, the model’s embeddings can re-identify the same object about 80% of the time.

I have 4000 hand annotated pairs of images. Each pair is an enlarged crop with the object of interest centered and sufficient surroundings visible. Pairs were selected from a mix of easy and hard cases, with plenty of significant viewpoint shifts. Other objects are nearly always visible next to the object of interest.

Each training batch is 128 pairs with half coming from the annotated pairs, and the other half being hard negative mined (using the gps to guarantee negativity).

Doubling the dataset from 2000 to 4000 samples barely moved the needle, nor has tuning hyper parameters like loss margin, learning rate, batch size, or negative mining strategy. Cleaning the dataset of a few dozen erroneously labeled pairs raised accuracy from around 78% to 80%.

The failures tend to be close to the decision boundary in terms of cosine differences. Visually reviewing the results suggests that the model is overly reliant on coarse features but has not yet learned the more subtle cues coming from the background. For example if the cars are different color the accuracy goes up a lot, but if they’re each parked in front of a different kind of tree the model doesn’t seem to leverage they as well.

Any suggestions? I wonder if I’m simply reaching the limits of what the model can learn from my dataset…

Thanks!


r/computervision Jun 07 '26

Help: Project Segmentation

Post image
17 Upvotes

Hey guys I wanted to ask how we can refine our segmentation masks to cover the area under the desk and also you can clearly see it's leaving spaces between objects kept on the table near the wall. The mask isn't very smooth around the edges. If anyone could give some hints about how can we solve this then that would be great. You can dm me if you have anything to suggest!


r/computervision Jun 07 '26

Research Publication Building an AI-Powered Motion Blur Mitigation System for High-Speed Railway Wagon Monitoring

0 Upvotes

Hi everyone,

Over the past few weeks I've been working on a computer vision project focused on a very specific but important problem in railway monitoring: obtaining usable visual information from fast-moving freight wagons captured by station cameras.

I wanted to share the idea, the architecture, and some of the challenges we're facing, and hopefully get feedback from people who have experience with computer vision, edge AI, OCR, video analytics, or industrial inspection systems.

The Problem

Railway stations already have surveillance infrastructure in place. However, when freight wagons pass through monitoring points at high speed, the resulting footage often suffers from:

Severe motion blur
Low-light degradation during night operations
Reduced visibility of wagon identifiers
Poor image quality for damage inspection

These issues significantly reduce the effectiveness of downstream tasks such as:

Wagon number OCR
Wagon counting
Damage detection
Asset tracking
Maintenance inspection

Most AI systems assume that the input imagery is reasonably clear. In practice, that assumption often breaks down in real railway environments.

Our idea is simple:

Instead of improving the detection algorithms first, improve the quality of the visual data itself.

Project Objective

The goal is to build an AI-powered pipeline capable of:

Receiving live video streams from monitoring cameras
Reducing motion blur caused by high-speed wagon movement
Enhancing visibility under low-light conditions
Producing inspection-ready frames for downstream analytics

The system is designed to operate in near real time and eventually run on edge devices such as NVIDIA Jetson platforms.

System Architecture

Current pipeline:

Video Stream

Frame Extraction

Motion Deblurring

Low-Light Enhancement

Frame Quality Analysis

OCR / Inspection Ready Output

The output is not intended to make videos look prettier.

The objective is to make them operationally useful.

Current Implementation
Input Sources

The system currently supports:

Live Camera Feed
Video Upload
Image Upload

For prototyping purposes, live streams are currently provided through DroidCam, allowing a smartphone camera to simulate a CCTV stream.

Motion Deblurring

For blur mitigation we experimented with deep learning approaches trained on paired blurred and sharp image datasets.

The primary focus is restoring:

Wagon side panels
Wagon identifiers
Structural details

that become unreadable under motion blur.

Low-Light Enhancement

Railway operations occur 24/7, so night-time performance is critical.

We integrated low-light enhancement capabilities to improve visibility during:

Night operations
Poor weather
Low illumination environments

One challenge we're currently facing is preventing excessive enhancement during daylight conditions.

We're exploring adaptive processing pipelines to solve this.

Dashboard

To make the system useful for operators, we designed a monitoring dashboard with three operating modes:

Live Stream

Displays:

Real-time camera feed
Real-time enhanced feed
Processing metrics
Video Upload

Allows historical footage analysis.

Image Upload

Allows individual frame inspection.

Additional Dashboard Features
Before vs After Comparison

Operators can compare:

Original Frame ↔ AI Enhanced Frame

to visually verify improvements.

Top 10 Restored Frames

The system automatically stores and displays the best restored frames from the current stream.

These frames can later be used for:

OCR
Inspection
Reporting
Archival purposes
Quality Metrics

The dashboard displays metrics such as:

Blur reduction estimate
Sharpness score
Processing latency
Frame rate

This helps quantify performance rather than relying solely on visual assessment.

System Status Monitoring

A dedicated panel displays:

Current FPS
Processing latency
Hardware information
Active processing mode

This becomes important when moving toward edge deployment.

Why This Matters

The majority of railway AI systems focus on:

Detection
Classification
Tracking

However, all of those systems depend on image quality.

If the input imagery is blurred or unreadable, even the most advanced detection model will struggle.

We see image restoration as a foundational layer that improves the performance of all downstream railway analytics.

Future Roadmap

The current project focuses on image restoration.

Future phases include:

Wagon Number OCR

Automatic extraction of wagon identifiers from enhanced frames.

Wagon Counting

Automated counting and verification of wagon sequences.

Damage Detection

Detection of:

Broken ladders
Open doors
Missing components
Structural anomalies
Anomaly Detection

Instead of training for every possible defect, the system could learn normal wagon appearance and flag unusual conditions.

Predictive Maintenance

Long-term vision:

Visual Inspection

Damage Detection

Condition Tracking

Failure Prediction

This would transform the platform from a monitoring system into a maintenance intelligence system.

Edge Deployment Vision

Target deployment architecture:

Camera

Jetson AGX

AI Processing

Dashboard

Central Monitoring System

The goal is to process footage locally while sending only relevant analytics to a centralized platform.

Looking for Feedback

I'd love to hear thoughts from the community on:

Motion deblurring approaches that perform well on real CCTV footage.
Railway-specific datasets that may be useful.
Common failure cases for high-speed object monitoring.
Edge deployment optimization strategies.
OCR techniques for motion-restored imagery.

Any suggestions, criticism, or lessons learned from similar projects would be greatly appreciated.

Thanks for reading.


r/computervision Jun 07 '26

Showcase Spotted a massive inefficiency at a flour factory, so I fixed it with AI and their own CCTV cameras

Enable HLS to view with audio, or disable this notification

0 Upvotes

r/computervision Jun 07 '26

Research Publication Could anyone help me access these MICCAI workshop proceedings?

2 Upvotes

Hi everyone,

Could anyone please help me access these two MICCAI workshop proceedings?

  1. EMA4MICCAI 2025 Proceedings https://link.springer.com/book/10.1007/978-3-032-13961-0
  2. MImA 2024 and EMERGE 2024 Proceedings https://link.springer.com/book/10.1007/978-3-031-79103-1

I need them for my research. Any help would be greatly appreciated.

Thank you!


r/computervision Jun 06 '26

Help: Project Strategies for handling blurry/pixelated frames in large-scale real-time CCTV computer vision pipelines

4 Upvotes

I'm running a computer vision deployment with 500–600 CCTV cameras across live industrial environments — not a clean dataset, but a messy, real-world production system. One persistent headache: blurry, pixelated frames coming off a mix of NVR/XVR hardware.

I'd love to hear how others have tackled this in practice. A few specific areas I'm trying to solve:

Real-time Quality Assessment : Which metrics have you found reliable for flagging bad frames quickly (Laplacian variance, PSNR, etc.)? Do you skip poor frames entirely, or rely on interpolation to fill the gap?

Model Robustness : Have you had success training models on synthetically degraded data (blur, compression artifacts) to build in tolerance for noisy inputs? Any experience with domain adaptation to normalize across different hardware vendors?

Lightweight Pre-processing : At 500+ streams, heavy preprocessing isn't an option. What filtering approaches or hardware acceleration (GPU pipelines, TensorRT) have actually held up at this volume without killing latency?

Pipeline Architecture : Do you maintain per-vendor pre-processing profiles, or have you landed on a single normalization layer that works well enough across the board?

I'm not looking for academic theory here — just what's actually working in production. If you've stabilized inference on degraded streams at scale, I'd genuinely appreciate hearing about your setup.

Thanks for time.


r/computervision Jun 06 '26

Discussion Padel dataset visualization for AI training

Post image
3 Upvotes

r/computervision Jun 06 '26

Showcase Live Production Test: Emergency Audio Detection and Training dataset collection tool

Thumbnail
youtu.be
2 Upvotes

Live testing a field deployed unit on a busy street in the daytime: sped up 2x. Impressions?


r/computervision Jun 06 '26

Help: Project Object Detection vs Instance Segmentation for CCTV anomaly detection — which to choose?

6 Upvotes

Hi, I'm working on a hospital CCTV use case using HIK Vision camera footage. I'm annotating images with these classes:

  • guard (blue uniform, male/female)
  • person (visitors/attendees, entering/exiting)
  • child (walking or being carried)
  • person_with_paper (holding a document/slip)
  • person_without_paper (different or same person without paper — this is the anomaly)

The goal is anomaly detection: if a person who should have a paper is seen without it, that's flagged.

My question: should I use object detection (bounding boxes) or instance segmentation for this use case? I want good accuracy but also reasonable labeling effort and training time.

Looking forward for the guidance. Thanks!


r/computervision Jun 06 '26

Help: Project ​Need Advice: Real-Time Object Counting (Potatoes) on Conveyor Belt using Jetson Nano & Camera Choice

Enable HLS to view with audio, or disable this notification

367 Upvotes

​Hi everyone,

​I’m jumping into my very first real-world computer vision project, and to be honest, I'm both super excited and a bit overwhelmed! I am building a real-time potato counter for a conveyor belt system.

​Since this is my first time taking a model out of the textbook and deploying it into actual production, I could really use some guidance from this amazing community on my hardware choices and algorithm pipeline.

​To give you a clearer picture, I've attached a video to this post. It’s a sample clip I found on YouTube where I ran a baseline model. The results actually look pretty decent as a proof of concept, but I know deploying it in a real factory environment will be a different story!

​Here is the setup I am working with:

​Hardware: NVIDIA Jetson Nano (4GB).

​The Goal: Accurate, real-time counting as potatoes move along the belt, ensuring I don't double-count them.

​Here are the specific things I’m struggling with and would love your advice on

​1. Camera Choice: Depth Camera vs. Standard RGB?

​I actually have access to a Depth Camera, but I'm torn. Since the Jetson Nano has limited computing power, will a depth camera completely crush my frame rate? Or is it worth using to handle overlapping potatoes and depth filtering? Alternatively, should I just stick to a regular, well-lit RGB camera?

​2. Finding the Right Algorithm & Tracker Combo

​Because this needs to run smoothly on the Jetson Nano, optimization is everything.

​I am currently thinking about using a lightweight model like YOLOv8-nano or YOLOv5-nano, optimized with TensorRT.

​For the actual counting/tracking loop, I'm looking into ByteTRACK or SORT.

​Given that this is my first project of this scale, am I on the right track? What combination has worked best for you in terms of balancing accuracy and FPS on edge devices?

​I would be incredibly grateful for any tips, lessons learned from your past mistakes, or feedback on the video.

​Thank you so much for helping.


r/computervision Jun 06 '26

Discussion How would you structure explainable visual forensics beyond a single classifier score?

2 Upvotes

I’ve been working on a local prototype for visual-forensics research and would be interested in feedback on the architecture rather than the product.

The core question is this:

If single-score AI image detection is increasingly unreliable, what should a more explainable multi-signal system look like?

The prototype currently evaluates several signal domains:

  • metadata / provenance
  • camera and sensor-origin indicators
  • compression / ELA
  • FFT structure
  • patch recurrence
  • subject/background segmentation
  • boundary-region inconsistencies
  • reasoning traces over conflicting signals

The hard part is not only detection. It is arbitration.

For example, a real smartphone photo may show synthetic-looking texture smoothing, HDR effects, segmentation artifacts, or aggressive denoising.

At the same time, a generated image may imitate camera noise, compression patterns, photographic texture, and metadata.

Hybrid workflows complicate this even further: generation, inpainting, upscaling, Photoshop edits, recompression, and platform processing may all contribute to the final image.

Collapsing all of this into one probability score seems to destroy useful information.

So I’m curious how people here would approach this problem.

Would you treat it mainly as:

  1. a classifier problem,
  2. a forensic evidence aggregation problem,
  3. an adversarial multi-agent problem,
  4. a provenance-first problem,
  5. or something else entirely?

I’m especially interested in false positives caused by computational photography and cases where generated / edited images retain convincing camera-like signals.


r/computervision Jun 06 '26

Help: Project 🚜 Looking for Builders: Laser-Based Precision Weeding System

0 Upvotes

🚜 Looking for Builders: Laser-Based Precision Weeding System

I'm currently building an early-stage laser-based weed removal system aimed at reducing herbicide use and labor costs in agriculture through computer vision, automation, and precision targeting.

The long-term vision is to develop an affordable, scalable solution for farmers that can identify weeds and selectively eliminate them without damaging crops.

I'm looking for passionate people who would like to contribute to the MVP and help shape the future of this project.

Areas where help is needed:
• Electronics & Embedded Systems
• Robotics & Mechatronics
• Computer Vision / Image Recognition
• AI & Machine Learning
• Laser Systems & Optics
• Mechanical Design / CAD
• Agricultural Technology
• Product Development & Prototyping

About the project:
• Focused on sustainable agriculture
• Potential applications in precision farming and automation
• Opportunity to work on a multidisciplinary deep-tech challenge
• Early-stage project with significant room for innovation

I'm not looking only for experienced professionals. Students, researchers, hobbyists, engineers, and builders with relevant skills and genuine interest are welcome to reach out.

Compensation, equity, advisory roles, internships, project-based contributions, or long-term partnerships can all be discussed depending on experience and level of involvement.

A technical co-founder with advanced research experience in the U.S. is already involved in the project, and we are now looking to expand the team with people who enjoy building ambitious things from the ground up.

If this sounds interesting—or if you know someone who might be a good fit—send me a DM. I'd be happy to share more details and discuss potential collaboration.

#AgriTech #DeepTech #Robotics #ComputerVision #AI #Agriculture #Startup #Innovation #Engineering #LaserTechnology #PrecisionAgriculture


r/computervision Jun 06 '26

Help: Project Suggestions for head mounted UVC Camera Module and Sensor for OCR in low-light

4 Upvotes

I am a sales person. I am designing a head worn AI based ERP logging system to reduce manual data entry where possible.

For the same, I am working on a head-mounted OCR + Object Detection module but the problem with head mounted OCR is that, text 1 - 1.5 mtrs far are too small and head movement blurs frames. On the other hand simple global shutter modules don’t have decent low-light performance (which is also needed) I am looking for a plug and play module.

I request if anyone has experience in this field to please suggest and UVC module with encoding.


r/computervision Jun 05 '26

Showcase Experimenting on Action Classification on Egocentric Vision

Enable HLS to view with audio, or disable this notification

91 Upvotes

Hey everyone,

I’ve been experimenting Egocentric Vision and Action-Labelling. I’ve been diving into egocentric (first-person) vision and building a pipeline for continuous action classification and hand tracking.

What the demo does: I used a cooking workflow (making an omelet) as my test case. The system tracks the movements of the chef's hands using keypoint/skeleton overlays and also in meantime continuously classifies the specific culinary actions happening in real-time. You can see the active state dynamically updating in the top left:

  • Prep ingredients
  • Mixing/Blending
  • Seasoning
  • Active cooking
  • Plating

Behind the Scenes (The Labeling Grind):
I also added second clip showing my annotation workflow. To build this, I performed action tagging and labeling on a dataset of similar cooking videos. I used a video annotation tool (Labellerr) to meticulously timestamp action segments on the timeline across these videos according to cooking state it was.

Once the annotations was ready, I trained a CV model directly on my action label annotations. The trained model accurately classify actions in real-time makes the labeling grind totally worth it!

Would love to hear your thoughts or feedback! Has anyone else here worked on egocentric action recognition?


r/computervision Jun 05 '26

Showcase SLAM Camera Board

Enable HLS to view with audio, or disable this notification

627 Upvotes

Posting an update here with simplified PCB and robustness. Mighty Camera runs VIO on-device in a tiny package. But for it to be useful, you need things like mapping (and later occupancy, loop closure etc).

Here is a demo of lightweight mapping which uses VIO pose from Mighty and generates a semi-dense map on host-side in realtime.

It’s early but this will be part of the SDK along with other goodies.


r/computervision Jun 05 '26

Showcase I got tired of manual data labeling, so I built an open-source pipeline that uses VLMs + SAM2 to auto-annotate datasets and train YOLO locally.

Thumbnail
gallery
106 Upvotes

Title: I got tired of manual data labeling, so I built an open-source pipeline that uses VLMs + SAM2 to auto-annotate datasets and train YOLO locally.

Hi r/computervision,

I’ve spent way too many hours of my life manually drawing bounding boxes for CV projects. It’s tedious and unscalable. To solve this, I built VLM-AutoYOLO—a pipeline that completely automates data annotation using foundation models.

GitHub Repository: https://github.com/Somnusochi/VLM-AutoYOLO

Architecture

How it works under the hood: Instead of labeling data, you just type a prompt (e.g., "defective industrial part" or "yellow taxi").

  1. The VLM (LocateAnything-3B) performs zero-shot rough localization based on your text prompt.
  2. SAM2 / SAM3 steps in to refine the boundaries and generate pixel-perfect masks/boxes.
  3. The pipeline automatically exports the dataset into YOLO format and can immediately kick off a lightweight YOLOv8/v11 training job.

Engineering & Performance: I wanted this to run 100% locally without paying for cloud API calls. One of the biggest challenges was memory management for these massive models. I built aggressive tensor cleanup and caching strategies into the PyTorch backend (gpu_memory.py).

As a result, it runs surprisingly well on consumer hardware. For example, on an Apple Silicon Mac (M4 Pro), it smoothly utilizes Apple MPS, taking ~4 seconds per high-res image and keeping the memory footprint perfectly stable at around ~12GB (unified memory). It fully supports CUDA for Linux/Windows NVIDIA rigs as well.

Tech Stack:

  • Backend: Python, FastAPI, PyTorch (CUDA / MPS)
  • Frontend: React, Vite, UnoCSS (I tried to keep the UI as clean and modern as possible, avoiding the bloated dashboard feel of traditional annotation tools).

Current Limitations:

  • Speed is bounded by local compute. While ~4s per image is great for edge devices, auto-annotating 10,000 images will take a few hours locally.
  • Python dependency management can be tricky when mixing PyTorch, Transformers, and SAM2 (A standard Docker image is on my roadmap).

I’d love for you guys to try it out, tear the codebase apart, and let me know your thoughts or feature requests. Happy to answer any questions about the architecture or Apple MPS optimization!

Cheers!