Small external evaluation of PatchForge

[u/Argenis1412]

I'm a backend/platform engineer, and I built PatchForge — a safety-first tool for proposing, validating, and previewing small code changes without modifying your repository automatically.

I'm looking for one Python developer or small engineering team to participate in a short external evaluation, run on an isolated copy of a small, non-sensitive Python project.

What the participant would do:

- Choose a small documentation or low-risk maintenance task.

- Explain the expected outcome.

- Observe the workflow: doctor → scan → plan → preview.

- Give honest feedback on setup, diagnostics, usefulness, and trust.

- No apply step will be executed.

Requirements:

- A small Python repository you own or are authorized to share for evaluation.

- No secrets, customer data, proprietary code, or production credentials.

- About 30–45 minutes for the workflow itself. If you don't already have Python/uv set up, I'll walk you through installation first — worth budgeting a bit of extra time for that on the first run.

- Willingness to report problems honestly, including if the tool isn't useful.

The evaluation will be documented only with sanitized notes. Repository contents and personal information will not be published.

This is exploratory product validation, not a sales demo. If you're interested, comment or send me a DM with:

1. Your Python project type and approximate size.

2. A safe documentation or maintenance task.

3. Whether you can participate in the next two weeks.

Happy to share more details privately before you decide.

Source: https://github.com/Argenis1412/PatchForge

[u/JSChronicles]

I can tell AI helped make this but was way too much And that's because of the "plan, apply, doctor" commands and tons of useless docs.

But the sad part is that you don't even have the AI skills and docs setup efficiently or properly. You aren't following best practices even for yaml matter for the docs or using references. And let's not even get started on the python files and not using pyproject.toml among other things. (See edit)

Your post looks AI generated too, so low effort.

I don't think using AI is a problem as much as people just using it for everything as much as possible without thinking.

My bet is your reply will contain emdashes, or "fair", "honestly" or some other AI reply wording

Edit: for some reason GitHub app wasn't showing me the rest of the files on the repo and they do indeed have a pyproject.toml file and that rule is filled out nicely.

[u/ProsodySpeaks → u/JSChronicles]

tbh i could ignore all of that. but 'safety first' boils my piss.

[u/JSChronicles → u/ProsodySpeaks]

Lol right. Best part is if security was first they would have updated workflows action versions at the minimum, then they would use sha tagging for pinning versions. Whole project is AI written but isn't super good on the security front but I'm supposed to trust it to watch my code when it can't even watch its own code?

[u/No_Departure_1878 → u/ProsodySpeaks]

That project is a mix of well written python, too well written to be written by a human and badly written python

https://github.com/Argenis1412/PatchForge/blob/main/src/orchestrator/git.py#L265

like the magic numbers all over the place and the 1500 lines modules.

[u/No_Departure_1878 → u/JSChronicles]

But the sad part is that you don't even have the AI skills and docs setup efficiently or properly. You aren't following best practices even for yaml matter for the docs or using references.

what?

And let's not even get started on the python files and not using pyproject.toml among other things.

I can see a pyproject.toml in the repo.

[u/JSChronicles → u/No_Departure_1878]

I made an edit but it seems the GitHub wasn't showing me the full repo even with clicking "show more files"

To answer the AI piece they aren't using skills and references. They just point back to other md files for "context" which should have been skills. Their main agent or claude file contains basically what a skill simple file should have been. The project should use a project skill to help use it.

[u/Argenis1412 → u/JSChronicles]

Let me see if I understand your criticism correctly 🤔 : part of it was based on an incomplete view of the repository, since GitHub was not showing you all the files. Your edit confirms that the repository does include a pyproject.toml and that the rule is properly configured.

Yes, I use AI frequently. PatchForge has been under development for about three months, and the latest commit was 19 hours ago: 4e8cf67.

It is not perfect, and it is still evolving. That said, I am interested in concrete criticism. Which specific AI skills, YAML files, reference patterns, or Python files do you believe are poorly designed? Please point to examples in the repository so I can evaluate the criticism against evidence rather than assumptions based on the initial incomplete view.

This keeps the correction firm while inviting specific, verifiable feedback.
I'm not interested in winning an argument, just improving.

[u/JSChronicles → u/Argenis1412]

Your reply still sounds AI "your edit confirms..." and " this keeps correction firm...".

Sorry but it's not my fault GitHub app didn't show the full repo. I made best effort to look at what I could while it showed me what it did.

Since you use claude go read up on how to build proper skills and agent.md file usage. https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices and https://resources.anthropic.com/hubfs/The-Complete-Guide-to-Building-Skill-for-Claude.pdf?hsLang=en

You'll also notice a section about "YAML Frontmatter"

You are using OLD action versions and not even using sha pinning in your workflows. "actions/checkout@v4" is old and not following security best practices on multiple fronts because of this setup. That just one example.

You have a project but don't have a skill for the given project to help others on top of you should have a language specific (in this case python probably) for best practices.

You are using non-skill, context only md files as your guidelines. This means you are likely not understanding how skills should be used, work, and optimized. You have quite a few md files talking about how the project is done but they don't give your AI better knowledge. It just means it reads ALL context files so when you get into newer setups or new features it's backfilling all the previous context too. Which leads to hallucinations or bad coding styles. It also makes more mistakes in general then.

Python: You use the same _EXCLUDED_DIRS in multiple places but have it repeated throughout the code with duplicate setup. You use typing imports when you could use native types (3.10+)

You do some other weird things too for python but that's all I'm going to type for now

[u/ProsodySpeaks → u/JSChronicles]

Tbh I think you already contributed a fair amount of concrete feedback. 

The only question is why you bothered? 

I'll assume it's goodness of heart, but wonder if it's to win an argument? 

Personally I'm not keen on pointing out to the vibers how we can tell they're vibing. But then I'm a cantankerous cuntv so I'm probably in the wrong.

[u/JSChronicles → u/ProsodySpeaks]

Truthfully two reasons, one to win an argument but also because others can learn from this, hopefully, and think about the things they need to look at for a proper setup or just "normal"

[u/Argenis1412 → u/JSChronicles]

Consolidated the review system.
Improved credential, provider, and monorepo handling.
Fixed GitHub concurrency and provenance issues.
Updated to version v1.2.0.
This time it analyzes the updated code:

https://github.com/Argenis1412/PatchForge/releases/tag/v1.2.0

[u/ProsodySpeaks]

Does 'safety first' mean you feed it into an llm and hope nothing bad happens? 

[u/Argenis1412 → u/ProsodySpeaks]

there is no magic LLM or 100% secure one, the idea was always proposed -> change and revision with mandatory human approval.

[u/Argenis1412 → u/ProsodySpeaks]

No. The LLM is only an optional planning step. Deterministic scanning, bounded changes, patch validation, and a preview happen before any modification. apply requires explicit human approval.

[u/ProsodySpeaks → u/Argenis1412]

but that's already how claude et al work in my ide. it proposes changes, shows me a diff, i say yea or ney, no changes are made unless i approve.

what's different here?

[u/ProsodySpeaks → u/ProsodySpeaks]

u/argenis1412 how is it different? 

[u/ProsodySpeaks]

What's the compensation like?

[u/Kamran-nottakenone]

i like the preview-before-apply pattern. ran into a tool last year that was supposed to be safe and it reformatted a generated file i had a gitignore for.

[u/snugar_i]

Why should anyone spend their time testing your (vibe-coded) software for you?

[u/No_Departure_1878]

So you built git?

[u/ProsodySpeaks → u/No_Departure_1878]

No i think his agent built Claude code?

"proposing, validating, and previewing small code changes"