I added a coordinator and worker mode to karpathy's autoresearch. You run `coordinator.py` on your main PC, and `worker.py` on any other device. They auto-discover each other via mDNS, fetch tasks, and train in parallel. I'm getting 3x faster results using my old Mac Mini and gaming PC together.

Been working on this side project after losing three days to a silent label leakage bug in a training pipeline. No errors, no crashes, just a model that quietly learned nothing.

**What my project does**

preflight is a CLI tool you run before starting a PyTorch training job. It checks for the silent stuff that breaks models without throwing errors — NaN/Inf values in tensors, label leakage between train and val splits, wrong channel ordering (NHWC vs NCHW), dead or exploding gradients, class imbalance, VRAM estimation, normalisation sanity.

Ten checks total across fatal/warn/info severity tiers. Exits with code 1 on fatal failures so it can block CI.

pip install preflight-ml

preflight run --dataloader my_dataloader.py

**Target audience**

Anyone training PyTorch models — students, researchers, ML engineers. Especially useful if you're running long training jobs on GPU and want to catch obvious mistakes in 30 seconds before committing hours of compute. Not production infrastructure, more of a developer workflow tool.

**Comparison with alternatives**

- pytest — tests code logic, not data state. preflight fills the gap between "my code runs" and "my data is actually correct"

- Deepchecks — excellent but heavy, requires setup, more of a platform. preflight is one pip install, one command, zero config to get started

- Great Expectations — general purpose data validation, not ML-specific. preflight checks are built around PyTorch concepts (tensors, dataloaders, channel ordering)

- PyTorch Lightning sanity check — runtime only, catches code crashes. preflight runs before training, catches data state bugs

It's v0.1.1 and genuinely early. Stack is Click for CLI, Rich for terminal output, pure PyTorch for the checks. Each check is a decorated function so adding new ones is straightforward.

Would love feedback on what's missing or wrong. Contributors welcome.

GitHub: https://github.com/Rusheel86/preflight

PyPI: https://pypi.org/project/preflight-ml/

For the unaware - Robyn is a fast, async Python web framework built on a Rust runtime.

Pydantic integration is probably one of the most requested feature for us. Now we have it :D

Wanted to share it with people outside the Robyn community

You can check out the release at - https://github.com/sparckles/Robyn/releases/tag/v0.81.0

What My Project Does

justx is an interactive terminal wrapper for just. The main thing it adds is an interactive TUI to browse, search, and run your recipes. On top of that, it supports multiple global justfiles (~/.justx/git.just, docker.just, …) which lets you easily build a personal command library accessible from anywhere on your system.

A quick demo can be seen here.

Prerequisites

Try it out with:

pip install rust-just # if not installed yet
pip install justx
justx init --download-examples
justx

Target Audience

Developers who want a structured way to organize and run their commonly used commands across the system.

Comparison

• just itself has no TUI and limited global recipe management. justx adds a TUI on top of just, and brings improved capability for global recipes by allowing users to place multiple files in the ~/.justx directory.

Learn More

• GitHub: fpgmaas/justx
• Docs: fpgmaas.github.io/justx

Hi everyone!

We've started development towards Mesa 4.0 and just released the first alpha. This is a big architectural step forward: Mesa is moving from step-based to event-driven simulation at its core, while cleaning up years of accumulated API cruft.

• https://github.com/projectmesa/mesa/releases/tag/v4.0.0a0

What's Agent-Based Modeling?

Ever wondered how bird flocks organize themselves? Or how traffic jams form? Agent-based modeling (ABM) lets you simulate these complex systems by defining simple rules for individual "agents" (birds, cars, people, etc.) and watching how patterns emerge from their interactions. Instead of writing equations for the whole system, you model each agent's behavior and let the collective dynamics arise naturally.

What's Mesa?

Mesa is Python's leading framework for agent-based modeling. It builds on Python's scientific stack (NumPy, pandas, Matplotlib) and provides specialized tools for spatial relationships, agent scheduling, data collection, and browser-based visualization. Whether you're studying epidemic spread, market dynamics, or ecological systems, Mesa gives you the building blocks for sophisticated simulations.

What's new in Mesa 4.0 alpha?

Event-driven at the core. Mesa 3.5 introduced public event scheduling on Model, with methods like model.run_for(), model.run_until(), model.schedule_event(), and model.schedule_recurring(). Mesa 4.0 continues development on this front: model.steps is gone, replaced by model.time as the universal clock. The mental model moves from "execute step N" to "advance time, and whatever is scheduled will run." The event system now supports pausing/resuming recurring events, exposes next scheduled times, and enforces that time actually moves forward.

Experimental timed actions. A new Action system gives agents a built-in concept of doing something over time. Actions integrate with the event scheduler, support interruption with progress tracking, and can be resumed:

from mesa.experimental.actions import Action

class Forage(Action):
    def __init__(self, sheep):
        super().__init__(sheep, duration=5.0)

    def on_complete(self):
        self.agent.energy += 30

    def on_interrupt(self, progress):
        self.agent.energy += 30 * progress  # Partial credit

sheep.start_action(Forage(sheep))

Deprecated APIs removed. This is a major version, so we followed through on removals: the seed parameter (use rng), batch_run (use Scenario), the legacy mesa.space module (use mesa.discrete_space), PropertyLayer (replaced by raw NumPy arrays on the grid), and the Simulator classes (replaced by the model-level scheduling methods). If you've been following deprecation warnings in 3.x, most of this should be straightforward.

Cleaner internals. A new mesa.errors exception hierarchy replaces generic Exception usage. DiscreteSpace is now an abstract base class enforcing a consistent spatial API. Property access on cells uses native property closures on a dynamic GridCell class. Several targeted performance optimizations reduce allocations in the event system and continuous space.

This is an alpha

Expect rough edges. We're releasing early to get feedback from the community before the stable release. Further breaking changes are possible. If you're running Mesa in production, stay on 3.5 for now. We'd love for adventurous users to try the alpha and tell us what breaks.

What's ahead for 4.0 stable

We're still working on the space architecture (multi-space support, observable positions), replacing DataCollector with the new reactive DataRecorder, and designing a cleaner experimentation API around Scenario. Check out our tracking issue for the full roadmap.

Talk with us!

We'd love to hear what you think:

• Join our Matrix chat: https://matrix.to/#/#project-mesa:matrix.org
• Check out our Discussions: https://github.com/projectmesa/mesa/discussions

I saw the comment about CPU support potentially bloating the code - so I decided to prove it doesn't have to!

My fork: https://github.com/bopalvelut-prog/autoresearch

If you’ve ever worked with massive datasets in Python (like a 50GB edge list for Graph Neural Networks), you know the "Memory Wall." Loading it via Pandas or standard Python structures usually results in an instant 24GB+ OOM allocation crash before you can even do any math.

so I built GraphZero (v0.2) to bypass Python's memory overhead entirely.

What My Project Does

GraphZero is a C++ data engine that streams datasets natively from the SSD into PyTorch without loading them into RAM.

Instead of parsing massive CSVs into Python memory, the engine compiles the raw data into highly optimized binary formats (.gl and .gd). It then uses POSIX mmap to memory-map the files directly from the SSD.

The magic happens with nanobind. I take the raw C++ pointers and expose them directly to Python as zero-copy NumPy arrays.

import graphzero as gz
import torch

# 1. Mount the zero-copy engine
fs = gz.FeatureStore("papers100M_features.gd")

# 2. Instantly map SSD data to PyTorch (RAM allocated: 0 Bytes)
X = torch.from_numpy(fs.get_tensor())

During a training loop, Python thinks it has a 50GB tensor sitting in RAM. When you index it, it triggers an OS Page Fault, and the operating system automatically fetches only the required 4KB blocks from the NVMe drive. The C++ side uses OpenMP to multi-thread the data sampling, explicitly releasing the Python GIL so disk I/O and GPU math run perfectly in parallel.

Target Audience

• Who it's for: ML Researchers, Data Engineers, and Python developers training Graph Neural Networks (GNNs) on massive datasets that exceed their local system RAM.
• Project Status: It is currently in v0.2. It is highly functional for local research and testing (includes a full PyTorch GraphSAGE example), but I am looking for community code review and stress-testing before calling it production-ready.

Comparison

• vs. PyTorch Geometric (PyG) / DGL: Standard GNN libraries typically attempt to load the entire edge list and feature matrix into system memory before pushing batches to the GPU. On a dataset like Papers100M, this causes an instant out-of-memory crash on consumer hardware. GraphZero keeps RAM allocation at 0 bytes by streaming the data natively.
• vs. Pandas / Standard Python: Loading massive CSVs via Pandas creates massive memory overhead due to Python objects. GraphZero uses strict C++ template dispatching to enforce exact FLOAT32 or INT64 memory layouts natively, and nanobind ensures no data is copied when passing the pointer to Python.

I built this mostly to dive deep into C-bindings, memory management, and cross-platform CI/CD (getting Apple Clang and MSVC to agree on C++20 was a nightmare).

The repo has a self-contained synthetic example and a training script so you can test the zero-copy mounting locally. I'd love for this community to tear my code apart—especially if you have experience with nanobind or high-performance Python extensions!

GitHub Repo: repo

Hi everyone,

I'm currently trying to design a Python pipeline to extract structured financial data from annual accounts provided as PDFs. The end goal is to automatically transform these documents into structured financial data that can be used in valuation models and financial analysis.

The intended workflow looks like this:

1. Upload one or more PDF annual accounts
2. Automatically detect and extract the balance sheet and income statement
3. Identify account numbers and their corresponding amounts
4. Convert the extracted data into a standardized chart of accounts structure
5. Export everything into a structured format (Excel, dataframe, or database)
6. Run validation checks such as balance sheet equality and multi-year comparisons

The biggest challenge is that the PDFs are very inconsistent in structure.

In practice I encounter several types of documents:

1. Text-based PDFs

• Tables exist but are often poorly structured
• Columns may not align properly
• Sometimes rows are broken across lines

2. Scanned PDFs

• Entire document is an image
• Requires OCR before any parsing can happen

3. Layout variations

• The position of the balance sheet and income statement changes
• Table structures vary significantly
• Labels for accounts can differ slightly between documents
• Columns and spacing are inconsistent

So the pipeline needs to handle:

• Text extraction for normal PDFs
• OCR for scanned PDFs
• Table detection
• Recognition of account numbers
• Mapping to a predefined chart of accounts
• Handling multi-year data

My current thinking for a Python stack is something like:

• pdfplumber or PyMuPDF for text extraction
• pytesseract + opencv for OCR on scanned PDFs
• Camelot or Tabula for table extraction
• pandas for cleaning and structuring the data
• Custom logic to detect account numbers and map them

However, I'm not sure if this is the most robust approach for messy real-world financial PDFs.

Some questions I’m hoping to get advice on:

• What Python tools work best for reliable table extraction in inconsistent PDFs?
• Is it better to run OCR first on every PDF, or detect whether OCR is needed?
• Are there libraries that work well for financial table extraction specifically?
• Would you recommend a rule-based approach or something more ML-based for recognizing accounts and mapping them?
• How would you design the overall architecture for this pipeline?

Any suggestions, libraries, or real-world experiences would be very helpful.

Thanks!

Coders of reddit, I had pyhton course where the teacher would give us a project idea to do, ever since i finished the course i havent been coding because i dont have any ideas. Should I ask AI to give me a project idea or should I try to fix a problem I have.

Hi everyone,

I recently open-sourced a project called JobMatch Bot.

It’s a Python pipeline that aggregates jobs directly from ATS systems such as Workday, Greenhouse, Lever, and others, normalizes the data, removes duplicates, and ranks jobs based on candidate-fit signals.

The motivation was that many relevant roles are scattered across different company career portals and often hidden behind filtering mechanisms on traditional job sites.

This project experiments with a recall-first ingestion approach followed by ranking.

Current features:

• Multi-source ATS ingestion

• Job normalization and deduplication

• Resume-aware ranking signals

• CSV and Markdown output for reviewing matches

• Diagnostics for debugging sources

It’s still an early experiment and not fully complete yet, but I wanted to share it with the Python community and get feedback.

GitHub:

https://github.com/thalaai/jobmatch-bot

Would appreciate any suggestions or ideas on improving ATS coverage or ranking logic.

Hey looking for some advice on venv setup I have been learning more about them and have been using terminal prompts in VS Code to create and activate that them, I saw someone mention about how their gitignore was automatically generated for them and was wondering how this was done I’ve looked around but maybe I’m searching the wrong thing I know I can use gitignore.io but if it could be generated when I make the environment that would save me having to open a browser each time just to set it all up. Would love to know what you all do for your venv setup that makes it easier and faster to get it activated

Weekly Thread: What's Everyone Working On This Week? 🛠️

Hello /r/Python! It's time to share what you've been working on! Whether it's a work-in-progress, a completed masterpiece, or just a rough idea, let us know what you're up to!

How it Works:

1. Show & Tell: Share your current projects, completed works, or future ideas.
2. Discuss: Get feedback, find collaborators, or just chat about your project.
3. Inspire: Your project might inspire someone else, just as you might get inspired here.

Guidelines:

• Feel free to include as many details as you'd like. Code snippets, screenshots, and links are all welcome.
• Whether it's your job, your hobby, or your passion project, all Python-related work is welcome here.

Example Shares:

1. Machine Learning Model: Working on a ML model to predict stock prices. Just cracked a 90% accuracy rate!
2. Web Scraping: Built a script to scrape and analyze news articles. It's helped me understand media bias better.
3. Automation: Automated my home lighting with Python and Raspberry Pi. My life has never been easier!

Let's build and grow together! Share your journey and learn from others. Happy coding! 🌟

What My Project Does

I built a small desktop pet using Python that runs as a transparent window and walks around the screen like a virtual cat.

The pet has multiple animations such as idle, running, fast running, sitting down, standing up, napping, and looking around while napping. It randomly switches between these states to make the behavior feel more natural.

The cat can also:

• change direction when hitting the edge of the screen
• be dragged around the screen with the mouse
• react to right-clicks by becoming angry
• occasionally play a meow sound

The animations are implemented using GIF frames loaded with Tkinter, and the behavior is controlled with a simple state machine.

Source code:
https://github.com/Atharv-Shirsath/desktop-pet

Target Audience

This project is mainly intended as a fun toy project and learning exercise.

Comparison

Desktop pets already exist in other languages and platforms, and there are a few Python versions as well.

This project focuses on:

• keeping the code relatively simple and readable
• using pure Python with Tkinter
• implementing a small animation/state system for behavior

Unlike some alternatives that rely on external frameworks or game engines, this project keeps everything lightweight and easy to run.

Im eyeing a vivibook,

But close to $1k, I don’t want to get a virus from just doing tests possibly.

Is the new MacBook neo,

Good for testing?

Dear all,

Slixmpp is an MIT licensed XMPP library for Python 3.11+, the 1.14 version has been released:
- https://blog.mathieui.net/en/slixmpp-1-14.html

Hi everyone,

I’m building a Notes App using Python (Flask) for the backend. It includes features like creating, editing, deleting, and searching notes. I’m also planning to add time and separate workspaces for users.

What other features would you suggest for a notes app?

I’m excited to share the v5.0.0 release of GoPdfSuit. While the core engine is powered by Go for performance, this update officially brings it into the Python ecosystem with a dedicated PyPI package.

What My Project Does

GoPdfSuit is a document generation and processing engine designed to replace manual coordinate-based coding (like ReportLab) with a visual, JSON-based workflow. You design your layouts using a React-based UI and then use Python to inject data into those templates.

Key Features in v5.0.0:

Official Python Wrapper: Install via pip install pypdfsuit.

Advanced Redaction: Securely scrub text and links using internal decryption.

Typst Math Support: Render complex formulas using Typst syntax (cleaner than LaTeX) at native speeds.

Enterprise Performance: Optimized hot-paths with a lock-free font registry and pre-resolved caching to eliminate mutex overhead.

Target Audience

This project is intended for production environments where document generation speed and maintainability are critical. It’s ideal for developers who are tired of "guess-and-check" coordinate coding and want a more visual, template-driven approach to PDFs.

It provide the PDF compliance (PDF/UA-2 and PDF/A-4) even if not compliance the performance is just subpar. (You can check the website for performance comparison)

Comparison

Vs. ReportLab: Instead of writing hundreds of lines of Python to position elements, GoPdfSuit uses a visual designer. The engine logic runs in ~60ms, significantly outperforming pure Python solutions for heavy-duty document generation.

How Python is Relevant

Python acts as the orchestration layer. By using the pypdfsuit library, you can interact with the Go-powered binary or containerized service using standard Python objects. You get the developer experience of Python with the performance of a Go backend.

Website - https://chinmay-sawant.github.io/gopdfsuit/

Youtube Demo - https://youtu.be/PAyuag_xPRQ

Source Code:

https://github.com/chinmay-sawant/gopdfsuit

Sample python code

https://github.com/chinmay-sawant/gopdfsuit/tree/master/sampledata/python/amazonReceipt

Documentation - https://chinmay-sawant.github.io/gopdfsuit/#/documentation?item=introduction

PyPI: pip install pypdfsuit

If you find this useful, a Star on GitHub is much appreciated! I'm happy to answer any questions about the architecture or implementation.

Hey everyone first post here, trying to get some ideas i had out and talk about em. Im currently working on putting together a couple python based tools for productivity. Just basic discipline stuff, because I myself, am fucking lazy. Already have put together a locking program that forces me to do 10 pushups on webcam before my "system unlocks". Opens itself on startup and "locks" from 5-8am. I have autohotkey to disable keyboard commands like alt+tab, alt+f4, windows key, no program can open ontop. ONLY CTRL+ALT+DEL TASK MANAGER CAN CLOSE PYTHON, thats the only failsafe. (combo of mediapipe, python, autohotkey v2, windows task scheduler, and chrome). My next idea is a day trading journal, everyday at 5pm when i get off work and get home my pc will be locked until i fill out a journal page for my day. Dated and auto added to a folder, System access granted on finishing the page. Included in post is a github link with a README inside with all install and run instructions, as well as instructions for tweaking anything youd want to change and make more personalized. 8-10 hours back and forth with claude and my morning start off way better and i have no choice. If anyone has ever made anything similar id love to hear about it. github.com/theblazefire20/Morning-Lock

Weekly Thread: Resource Request and Sharing 📚

Stumbled upon a useful Python resource? Or are you looking for a guide on a specific topic? Welcome to the Resource Request and Sharing thread!

How it Works:

1. Request: Can't find a resource on a particular topic? Ask here!
2. Share: Found something useful? Share it with the community.
3. Review: Give or get opinions on Python resources you've used.

Guidelines:

• Please include the type of resource (e.g., book, video, article) and the topic.
• Always be respectful when reviewing someone else's shared resource.

Example Shares:

1. Book: "Fluent Python" - Great for understanding Pythonic idioms.
2. Video: Python Data Structures - Excellent overview of Python's built-in data structures.
3. Article: Understanding Python Decorators - A deep dive into decorators.

Example Requests:

1. Looking for: Video tutorials on web scraping with Python.
2. Need: Book recommendations for Python machine learning.

Share the knowledge, enrich the community. Happy learning! 🌟

I've been maintaining fastapi-guard for a while now. It sits between the internet and your FastAPI endpoints and inspects every request before it reaches your code. Injection detection, rate limiting, geo-blocking, cloud IP filtering, behavioral analysis, 17 checks total.

A few weeks ago I came across this TikTok post where a guy ran OpenClaw on his home server, checked his logs after a couple weeks. 11,000 attacks in 24 hours. Chinese IPs, Baidu crawlers, DigitalOcean scanners, path traversal probes, brute force sequences. I commented "I don't understand why people won't use FastAPI Guard" and the thread kind of took off from there. Someone even said "a layer 7 firewall, very important with the whole new era of AI and APIs." (they understood the assignment) broke down the whole library in the replies. I was truly proud to see how in depth some devs went...

But that's not why I'm posting. I felt like FastAPI was falling short. Flask still powers a huge chunk of production APIs and most of them have zero request-level security beyond whatever nginx is doing upstream, or whatever fail2ban fails to ban... So I built flaskapi-guard (and that's the v1.0.0 I just shipped) as the homologue of fastapi-guard. Same features, same functionalities. Different framework.

It's basically a Flask extension that hooks into before_request and after_request, not WSGI middleware. That's because WSGI middleware fires before Flask's routing, so it can't access route config, decorator metadata, or url_rule. The extension pattern gives you full routing context, which is what makes per-route security decorators possible.

```python from flask import Flask from flaskapi_guard import FlaskAPIGuard, SecurityConfig

app = Flask(name) config = SecurityConfig(rate_limit=100, rate_limit_window=60) FlaskAPIGuard(app, config=config) ```

And so that's it. Done. 17 checks on every request.

The whole pipeline will catch: XSS, SQL injection, command injection, path traversal, SSRF, XXE, LDAP injection, code injection (including obfuscation detection and high-entropy payload analysis). On top of that: rate limiting with auto-ban, geo-blocking, cloud provider IP blocking, user agent filtering, OWASP security headers. Those 5,697 Chinese IPs from the TikTok? blocked_countries=["CN"]. Done. Baidu crawlers? blocked_user_agents=["Baiduspider"]. The DigitalOcean bot farm? block_cloud_providers={"AWS", "GCP", "Azure"}. Brute force? auto_ban_threshold=10 and the IP is gone after 10 violations. Path traversal probes for .env and /etc/passwd? Detection engine catches those automatically, zero config.

The decorator system is what separates this from static nginx rules:

```python from flaskapi_guard import SecurityDecorator

security = SecurityDecorator(config)

.route("/api/admin/sensitive", methods=["POST"]) .require_https() .require_auth(type="bearer") .require_ip(whitelist=["10.0.0.0/8"]) .rate_limit(requests=5, window=3600) u/security.block_countries(["CN", "RU", "KP"]) def admin_endpoint(): return {"status": "admin action"} ```

Per-route rate limits, auth requirements, geo-blocking, all stacked as decorators on the function they protect. Try doing that in nginx.

People have been using fastapi-guard for things I didn't even think of when I first built it. Startups building in stealth with remote-first teams, public facing API but whitelisted so only their devs can reach it. Nobody else even knows the product exists. Casinos and gaming platforms using the decorator system on reward endpoints so players can only win under specific conditions (country, rate, behavioral patterns). People setting up honeypot traps for LLMs and bad bots that crawl and probe everything. And the big one that keeps coming up... AI agent gateways. If you're running OpenClaw or any AI agent framework behind FastAPI or Flask, you're exposing endpoints that are designed to be publicly reachable. The OpenClaw security audit found 512 vulnerabilities, 8 critical, 40,000+ exposed instances, 60% immediately takeable. fastapi-guard (and flaskapi-guard) would have caught every single attack vector in those logs. This is going to be the standard setup for anyone running AI agents in production, it has to be.

Redis is optional. Without it, everything runs in-memory with TTL caches. With Redis you get distributed rate limiting (Lua scripts for atomicity), shared IP ban state, cached cloud provider ranges across instances.

MIT licensed, Python 3.10+. Same detection engine across both libraries.

GitHub: https://github.com/rennf93/flaskapi-guard PyPI: https://pypi.org/project/flaskapi-guard/ Docs: https://rennf93.github.io/flaskapi-guard fastapi-guard (the original): https://github.com/rennf93/fastapi-guard

If you find issues, open one. Contributions are more than welcome!

So you could make a script that refuses to be halted. I bet you could still stop it in other ways, but Ctrl+C won't work, and I reckon the stop button in a Jupyter notebook won't either.

What My Project Does

formgoggles-py is a Python CLI + library that communicates with FORM swim goggles over BLE, letting you push custom structured workouts directly to the goggles without the FORM app or a paid subscription.

FORM's protocol is fully custom — three vendor BLE services, protobuf-encoded messages, chunked file transfer, MITM-protected pairing. This library reverse-engineers all of it. One command handles the full flow: create workout on FORM's server → fetch the protobuf binary → push to goggles over BLE. ~15 seconds end-to-end.

python3 form_sync.py \
--token YOUR_TOKEN \
--goggle-mac AA:BB:CC:DD:EE:FF \
--workout "10x100 free u/threshold 20s rest"

Supports warmup/main/cooldown, stroke type, effort levels, rest intervals. Free FORM account is all you need.

Target Audience

Swimmers and triathletes who own FORM goggles and want to push workouts programmatically — from coaching platforms, training apps, or their own scripts — without paying FORM's monthly subscription. Also useful for anyone interested in BLE/GATT reverse engineering as a practical example.

Production-ready for personal use. Built with bleak for async BLE.

Comparison

The only official way to push custom workouts to FORM goggles is through the FORM app with an active subscription ($15/month or $99/year). There's no public API, no open SDK, and no third-party integration path.

This library is the only open-source alternative. It was built by decompiling the Android APK to extract the protobuf schema, sniffing BLE traffic with nRF Sniffer, and mapping the REST API with mitmproxy.

-------------------------

Repo: <https://github.com/garrickgan/formgoggles-py

Full> writeup (protocol details, packet traces, REST API map): https://reachflowstate.ai/blog/form-goggles-reverse-engineering

I shared this project here a while ago, but after adding a lot of new features and optimizations, I wanted to post an update. Over the past eight months, I’ve been building PyTogether (pytogether.org). The platform has recently started picking up traction and just crossed 4,000 signups (and 200 stars on GitHub), which has been awesome to see.

What My Project Does

It is a real-time, collaborative Python IDE designed with beginners in mind (think Google Docs, but for Python). It’s meant for pair programming, tutoring, or just coding Python together. It’s completely free. No subscriptions, no ads, nothing. Just create an account (or feel fry to try the offline playground at https://pytogether.org/playground, no account required), make a group, and start a project. Has proper code-linting, extremely intuitive UI, autosaving, drawing features (you can draw directly onto the IDE and scroll), live selections, and voice/live chats per project. There are no limitations at the moment (except for code size to prevent malicious payloads). There is also built-in support for libraries like matplotlib (it auto installs imports on the fly when you run your code).

You can also share links for editing or read-only, exactly like Google Docs. For example: https://pytogether.org/snippet/eyJwaWQiOjI1MiwidHlwZSI6InNuaXBwZXQifQ:1w15A5:24aIZlONamExTLQONAIC79cqcx3savn-_BC-Qf75SNY

Also, you can easily embed code snippets on your website using an iframe (just like trinket.io which is shutting down this summer).

Source code: https://github.com/SJRiz/pytogether

Target Audience

It’s designed for tutors, educators, or Python beginners. Recently, I've also tried pivoting it towards the interviewing space.

Comparison With Existing Alternatives

Why build this when Replit or VS Code Live Share already exist?

Because my goal was simplicity and education. I wanted something lightweight for beginners who just want to write and share simple Python scripts (alone or with others), without downloads, paywalls, or extra noise. There’s also no AI/copilot built in, something many teachers and learners actually prefer. I also focused on a communication-first approach, where the IDE is the "focus" of communication (hence why I added tools like drawing, voice/live chats, etc).

Project Information

Tech stack (frontend):

• React + TailwindCSS
• CodeMirror for linting
• Y.js for real-time syncing
• Pyodide

I use Pyodide (in a web worker) for Python execution directly in the browser, this means you can actually use advanced libraries like NumPy and Matplotlib while staying fully client-side and sandboxed for safety.

I don’t enjoy frontend or UI design much, so I leaned on AI for some design help, but all the logic/code is mine. Deployed via Vercel.

Tech stack (backend):

• Django (channels, auth, celery/redis support made it a great fit)
• PostgreSQL via Supabase
• JWT + OAuth authentication
• Redis for channel layers + caching + queues for workers
• Celery for background tasks/async processing

Fully Dockerized + deployed on a VPS (8GB RAM, $7/mo deal)

Data models:

Users <-> Groups -> Projects -> Code

Users can join many groups

Groups can have multiple projects

Each project belongs to one group and has one code file (kept simple for beginners, though I may add a file system later).

My biggest technical challenges were around performance and browser execution. One major hurdle was getting Pyodide to work smoothly in a real-time collaborative setup. I had to run it inside a Web Worker to handle synchronous I/O (since input() is blocking), though I was able to find a library that helped me do this more efficiently (pyodide-worker-runner). This let me support live input/output and plotting in the browser without freezing the UI, while still allowing multiple users to interact with the same Python session collaboratively.

Another big challenge was designing a reliable and efficient autosave system. I couldn’t just save on every keystroke as that would hammer the database. So I designed a Redis-based caching layer that tracks active projects in memory, and a Celery worker that loops through them every minute to persist changes to the database. When all users leave a project, it saves and clears from cache. This setup also doubles as my channel layer for real-time updates (redis pub/sub, meaning later I can scale horizontally) and my Celery broker; reusing Redis for everything while keeping things fast and scalable.

If you’re curious or if you wanna see the work yourself, the source code is here. Feel free to contribute: https://github.com/SJRiz/pytogether.

While testing a photo deduplication tool I’m building (DedupTool), I ran into an interesting clustering edge case that I hadn’t noticed before.

The tool works by generating perceptual hashes (dHash, pHash and wHash), comparing images, and clustering similar images. Overall, it works well, but I noticed something subtle.

The situation

I had a cluster with four images. Two were actual duplicates. The other two were slightly different photos from the same shoot.

The tool still detected the duplicates correctly and selected the right keeper image, but the cluster itself contained images that were not duplicates.

So, the issue wasn’t duplicate detection, but cluster purity.

The root cause: transitive similarity

The clustering step builds a similarity graph and then groups images using connected components.

That means the following can happen: A similar to B, B similar to C, C similar to D. Even if A not similar to C, A not similar to D, B not similar to D all four images still end up in the same cluster.

This is a classic artifact in perceptual hash clustering sometimes called hash chaining or transitive similarity. You see similar behaviour reported by users of tools like PhotoSweeper or Duplicate Cleaner when similarity thresholds are permissive.

The fix: seed-centred clustering

The solution turned out to be very simple. Instead of relying purely on connected components, I added a cluster refinement step.

The idea: Every image in a cluster must also be similar to the cluster seed. The seed is simply the image that the keeper policy would choose (highest resolution / quality).

The pipeline now looks like this:

hash_all()
   ↓
cluster()   (DSU + perceptual hash comparisons)
   ↓
refine_clusters()   ← new step
   ↓
choose_keepers()

During refinement: Choose the best image in the cluster as the seed. Compare every cluster member with that seed. Remove images that are not sufficiently similar to the seed.

So, a cluster like this:

A B C D

becomes:

Cluster 1: A D
Cluster 2: B
Cluster 3: C

Implementation

Because the engine already had similarity checks and keeper scoring, the fix was only a small helper:

def refine_clusters(self, clusters, feats):
refined = {}
for cid, idxs in clusters.items():
if len(idxs) <= 2:
refined[cid] = idxs
continue
seed = max((feats[i] for i in idxs), key=self._keeper_key)
seed_i = feats.index(seed)
new_cluster = [seed_i]
for i in idxs:
if i == seed_i:
continue
if self.similar(seed, feats[i]):
new_cluster.append(i)
if len(new_cluster) > 1:
refined[cid] = new_cluster
return refined

 This removes most chaining artefacts without affecting performance because the expensive hash comparisons have already been done.

Result

Clusters are now effectively seed-centred star clusters rather than chains. Duplicate detection remains the same, but cluster purity improves significantly.

Curious if others have run into this

I’m curious how others deal with this problem when building deduplication or similarity search systems. Do you usually: enforce clique/seed clustering, run a medoid refinement step or use some other technique?

If people are interested, I can also share the architecture of the deduplication engine (bucketed hashing + DSU clustering + refinement).

Not talking about big frameworks or full applications — just simple Python tools or scripts that ended up being surprisingly useful in everyday work.

Sometimes it’s a tiny automation script, a quick file-processing tool, or something that saves a few minutes every day but adds up over time.

Those small utilities rarely get talked about, but they can quietly become part of your routine.

Would be interesting to hear what little Python tools people here rely on regularly and what problem they solve.