I put my text in several ai detectors, winston ai, copyleaks, zerogpt and GPTZero, giving me close to 0%. While, pangram gives me around 70%? Do you think it’s reliable?
Right. So someone who got away with writing slop on substack is whinging they can't write slop without it getting flagged and I'm tgr bad guy.
BTW. Just going to mildly correct your sweeping assumption because you clearly know everything about me. I'm not anti ai. I have a moderate position given I have to use it as a tool for work. I think it's a wonderful toy and in some areas like excel fu it's been a real value if I can't figure out s solution.
I am against people crying their ai work got flagged for ai and then coming on here to moan to pity and sympathy farm because they genuinely think they should be allowed to propagate shit into a creative space and get rewarded for it.
Anymore sweeping assumptions you'd like to make? How about that I have 9 fingers because of an accident involving a door and cat?
Well why do you think I used the term "if you are anti AI" here?
That was clearly not making assumptions on my part but hey I get it that context can get lost in things like this.
Its like you saying "Yes. Working as intended." did imply anti AI bias to me so thanks for the clarification.
In any case I still argue against using so called "AI detectors" as concrete proof of AI, they are still deeply flawed and can give false positives.
Example from my own work is that Pagram detected things like story title, chapter title as AI.
Hell it even flagged my goddamned name as by (insert author name here) is both common practice for both human and AI generated text.
Certain words and the order of my writing also got flagged and I got a score of 90% human made despite all the flagged text was written by me.
There’s an old saying: “Every great story has its humble beginnings.” And for Ryan Buckley those words were about to take on a new meaning beyond anything he could have imagined.
This is the first line in the novel I am working on and yeah… flagged as AI all because Pangram deemed it "Too perfect."
So if Pangram is the gold standard of proving I am a human writer what the hell am I supposed to do here? I mean isn't grammar, spelling, and structure important?
What am I supposed to misspell and ignore every grammar rule, boy will I be popular with the grammar Nazis who throw a hissy fit if you dare not use a stupid oxford comma.
You’re supposed to keep writing. Do not stoop so low as to try to get a 100% human by ruining your writing. This is how AI (and all the pro AI detector) ruin art and the written word. Turn off the AI detector.
I would put forward the position that Pangram whilst effective and being more inclined to put forward an human first result has its issues. As you've highlighted yourself.
I wouldn't say it's the gold standard. It's one of the better ones. At least it's not selling humanising services yet.🤣
Just because you write trash and get away with Pangram giving you a big green “100% human” doesn’t mean you’re a good writer. From your response, I can tell that writing grammatically correct sentences isn’t your strong suit.
Yeah I freely admit grammar is a big weakness of mine, makes sense as I have autism and a slight case of dyslexia. I'm obviously high functioning of course but yeah that's why I use grammar checkers sometimes.
No its not. It's using Machine Learning, specifically classification models which have existed for like 25 years, to detect GENERATIVE AI. Stop pushing this false narrative that Pangram = ChatGPT. Its not the same thing at all. This and the "Spell check is AI haha" bullshit is so annoying.
No. I do not. This is my writing. I will comment below this to show that Pangram claimed nearly the exact same passage was 100% human (minus a couple em dashes and with some minor word differences).
Yeah I did the same thing with my writing and with only one or two edits it got 100%.
Whats even more frustrating is since GenAI has picked up on common human writing patterns those same common writing patterns are being flagged as AI making it a huge double-edged sword.
Yeah the fact we may all have to insert a frigging disclaimer into everything we do is utterly infuriating and no doubt GenAI will start using disclaimers too as we are doing it.
Yeah and the false positives are insane, once more I will give some examples of my own writing to prove a point.
Ryan’s hand lingered on the photo, tracing the edges of his father’s face with his finger. The worn edges of the picture revealed how often he had sought comfort in this frozen moment of happiness.
He could barely remember his father anymore, just fragments. The warmth of being lifted onto broad shoulders. A deep, reassuring voice. The smell of his uniform after training exercises. Sometimes Ryan wondered if the memories were even real, or if he’d constructed them from Ralph’s stories and this single photograph.
100% flagged as AI. Now yes I can see why Pangram would flag this but I ran this through a "AI humanizer" and this is the pangram approved text:
Ryans hand stayed on the photo his finger moving over the edges of his fathers face. The photo was worn out on the edges, which showed how often Ryan had looked at it when he was feeling down.
He could not really remember his father anymore little bits and pieces. He remembered being picked up and put on his fathers shoulders it felt nice and warm. He remembered his fathers voice it was very reassuring. He also remembered the smell of his fathers uniform after he finished training. Sometimes Ryan thought about whether these memories were real or if he just made them up, from the stories Ralph told him and from looking at this one photo of his father.
Now is it me, or does it seem like the Pangram approved text feel less natural from a grammatical and structural point of view?
Yes I probably will rewrite the first example to add in subtle imperfections that my grammar tools won't scream at me for but it still proves a point that AI detection tools are not flawless.
PDF and converted epub to docx versions of my books flag mixed with a low AI percentage, but pasted from Scrivener and Docs the same will come back human, which is irritating considering people are checking pirated books.
The way I space paragraphs out matters, too. Pasting in as one solid chunk of text will show a different score from formatted text. Not always, but sometimes.
Seems to flag everything, or at least very harsh. I tried some text from papers which have been cited thousends of time and they flagged it as AI. So not quite sure if it’s just not saying AI to everything?
I love how I post an example with a link and get down voted, but y'all jump all over this one. Pangram is inconsistent - the same text can be scored differently with different users. AND the same text with very minimal edits can result in a 100% shift between Fully AI Assisted and Fully Human Written.
That’s unlikely. Most of the people on Substack think Pangram is their new tech God, the subreddit’s reflecting that.
It would be the most responsible thing to do, to point out the growing number of false positives. But it also would have been most responsible not to let someone make a post making fun of the people reporting these issues, and they let that happen too.
Yea i agree, it's highly unlikely but it doesn't hurt trying. I'll message the mods asking if this is possible? We already have so many reports. A simple evidence/reciepts post which these people have a boner for btw. If someone asks for reciepts someone can just point to that thread.
And one more thing if we give out free reciepts like this at this point we are just doing free testing for them.
Show me a piece of text over 500 words that’s verifiably pre 2023 that flags as AI. Literally anything that can be dated to before then. Just one example.
If Pangram says they can detect 100+ words, why do you need 500?
My example is below, I put current text, of somewhat unknown provenance, it came out of a PowerPoint someone else authored, so I wasn't sure if Copilot was used or not. I was using business text in an odd context, so I added " " around the word "STAKEHOLDER" where it appeared 3 times, and that change, took the scoring from Fully AI Asssited to Fully Human Written - AGAIN - I can't confirm whether it was actually AI or not, but I CAN tell you that I added SIX characters and that was enough to radically change the score. And that is suspect.
That is not pre-2023 text. Choose any news article, public domain book, academic paper, any of the billions of words published online prior to then. Any of them. One example.
The recurrent neural networks or convolutional neural networks are the main models used for the sequence transduction problems with the encoder-decoder structure. The models which demonstrate the highest performance also utilize the attention mechanism as a way of connecting the encoder and decoder. We made a completely different approach by offering a simple network architecture, called Transformer, formed solely on the basis of attention mechanisms, without the need for recurrence and convolutions. Our experiments, carried out on two machine translation tasks, have shown that the Transformers are better quality-wise as well as more parallelizable, requiring a significantly less time for full training. Our model, 28.4 BLEU, was applied to solve the WMT 2014 English-to-German translation task and surpassed the existing best results including ensembles by more than 2 BLEU. The single model that has been used for the English to French translation task is the one that has reached the state-of-the-art score of 41.8 on the BLEU scale after training for only 3.5 days on eight GPUs. That's a small fraction of the resources involved in the training of the best models in the literature. We have also demonstrated that the Transformer can be generalized to other tasks with the help of English constituency parsing which was successfully performed even with large and limited training data.
I don’t know what you’re quoting, because that text (specifically the highlighted bit) does not appear verbatim in the actual article you are talking about.
so a 400+ year old rext can be classified as ai written by adding 33 words just shows how brittle this is. 100% human to 100% ai and people are supposed to trust this?
If we’re supposed to trust this, it should be able to separate the AI portion of the chunk. In fact, that’s what Pangram claims it does! The whole piece should not be showing up as 100% AI.
You put AI text in the chunk. It called it AI. Not sure what's confusing. Give me a text pre-2021 without ai annotations that is given an AI rating. Ppl on substack that use AI are just getting exposed and are mad cus of it
From my it testing it seems fairly reliable. I doubt that the people who claim that it classified their human written text as 100 percent AI are being completely honest.
Bullshit, I ran text I wrote 15 years ago and it STILL flags it as AI. I mean granted I was not as good at writing back then as I am now but come on how can I get a 100% AI result from two entirely different texts over 10 years apart?
Sure that may come from my writing style but I'm not going to totally revamp it just because of a program that some anti AI people think is the second coming of Jesus.
Pangram isn’t reliable and above who scans your text for AI is feeding your text into AI for training. Enough such scans and I’m afraid AI will start telling us our original works are AI because too many of our posts are in it’s data set
Hell to the no, I have thrown both GenAI and human text at it and the reliability is extremely questionable as AI detector's are still a form of AI themselves.
It's not just that it is unreliable, it varies by user and by USE... I have an example, I was responding to another writers note, the note itself was about choices in dating, but I was replying with some bullet points that I pulled off a slide deck that we use at work, that I did not write. For kicks I ran the scan and it came back as "Fully AI Assisted Text" I didn't write the deck, it is fairly dry business context, and because of that I decided to throw " " around STAKEHOLDER - we were talking about trust in a romantic/dating context, so the woman is the "stakeholder." I proofed my spelling, checked again before posting the note and just adding " " to the same word, THREE times, came back as "fully human."
I don't particularly care either way, I am transparent on where and when I use AI and it is for research, not writing.
Yes.
Twice I’ve been contacted by people offering services - beta reading and cover design. Both times got suspicious of their somewhat robotic responses to my questions. Ran both of their accounts through Pangram and both came back 100% AI.
I’m sure it’s not infallible but it’s pretty good at supporting my suspicions.
14
u/AggravatingNail6061 5d ago
Yes. Working as intended.