AI and plagiarism detector

[enjay]

Can anyone recommend a good AI detector which is used in their school? Also, what "score" do you consider as evidence something has been AI-written? I put a few different paragraphs from ChatGPT through a checker, and it generally scored them around 55-60% AI, which I could get down to 15% using a "humaniser" tool. Some random paragraphs of my own writing were 2-3%.

[sigma]

Probably not what you want to hear:

 

https://www.theedadvocate.org/the-ai-detection-debacle-why-uk-universities-are-ditching-flawed-tools/

 

Quote

Across the UK, a significant number of universities are now abandoning or severely restricting the use of these very same AI detection tools. This isn’t just a minor tweak; it’s a major retreat, driven by a growing, uncomfortable realization: these tools just aren’t reliable enough. The core issue? A surprisingly high rate of false positives, leading to innocent students being wrongly accused. It’s a messy situation, and it’s forcing institutions to rethink their entire strategy for dealing with AI in the classroom, pushing them towards more nuanced, and frankly, more human-centric approaches.

 

My niece had problems with some school work she handed in and was falsely accused of having produced it using AI.   

[machy]

7 minutes ago, sigma said:

Probably not what you want to hear:

 

https://www.theedadvocate.org/the-ai-detection-debacle-why-uk-universities-are-ditching-flawed-tools/

 

 

My niece had problems with some school work she handed in and was falsely accused of having produced it using AI.   

the irony in the bit you quoted sounding like its come from AI

 

 

 

[enjay]

50 minutes ago, machy said:

the irony in the bit you quoted sounding like its come from AI

 

55% human, according to the tool I was using earlier! EDIT - I've just tried a couple of others, and they all flagged that passage as being heavily AI-generated.

 

To address @sigma's comment though, the teacher isn't proposing using the tool in isolation, but as part of the evidence to support their professional judgement based on the child's previous body of work. The article makes exactly that point further down:

 

Quote

The implication was clear: using these scores as the sole basis for an accusation of plagiarism is fundamentally unfair and, as it turns out, legally risky

 

[dmj]

For this to work, you'd have to feed all the children's work to the (AI) plagiarism detector before it could 'get to know' the students and what their writing style is like. 

Besides; LLM's all speak in different ways, and can easily be told to change their style - mine speaks like a caveman to save me output tokens. 

Sounds like a horribly expensive way to replace a teacher, who could do this with their eyes closed. 

[psydii]

2 hours ago, dmj said:

...a teacher, who could [read the kids' work] with their eyes closed. 

They can do many amazing things better than AI, but I've never seen one quite that good.

[Primus]

There's also an argument under the Equality Act since lots of AI writes in a similar fashion to neurodivergent people so people with neurodivergence are flagged more often and unfairly.

 

Ultimately the advice going back to them is that it's not part of evidence supporting their professional judgement in any way and the use of it isn't acceptable.

[pete]

We also push the "if you've been teaching Geoff for 3 years and you think that Geoff didn't write this, you're going to have a higher degree of accuracy than an AI tool" line. With an "unless it's an EPQ (or other independent project) and Geoff has picked his specialist subject of Pokemon" caveat.

[BKGarry]

We use this - https://plagiarismcheck.org/

 

Was quite cheap, support is good and teachers seem to be able to use with with Google classroom and office docs without any issues

[enjay]

15 hours ago, Primus said:

There's also an argument under the Equality Act since lots of AI writes in a similar fashion to neurodivergent people so people with neurodivergence are flagged more often and unfairly.

 

Which again is why this wouldn't be used in isolation. If the teacher knows the child to be neurodivergent, they can accommodate that in the assessment from the tool. Also, if the teacher were to compare several pieces of work from a child, they should get a similar score; if one document rates significantly higher for AI than others, that also suggests the child might not have written it.

 

That said, I'm not particularly confident of the accuracy of these tools! I used several different passages generated by ChatGPT and they typically scored around 50%. That's a lot higher than text I know to be human-written, but still not a brilliantly accurate score considering 100% of the text was AI. I also found several AI checkers also have a "humaniser" feature, which brought that score down to less than 5%.

[enjay]

16 hours ago, psydii said:

They can do many amazing things better than AI, but I've never seen one quite that good [reading with eyes closed].

 

Maybe the student submitted the work in Braille 🙂

I know of a teacher who gets their screen-reader to read out student assignments sometimes, so actually they could do it with their eyes closed.

[psydii]

50 minutes ago, enjay said:

 

Which again is why this wouldn't be used in isolation. If the teacher knows the child to be neurodivergent, they can accommodate that in the assessment from the tool. Al

AI tends to write like newspaper/magazine journalists, or those practiced at communicated with inattentive senior executives. Which is why those groups hyped it so much in the early years - it wrote just like/speaks to them. The key is that their prose followed a set pattern to deliver information and keeping the reader hooked.  Basically its  "writing frame",  a tool heavily used in support of ND students. So "writes like ND students" is actually "optimised information delivery for those who aren't giving it their full attention"!

[Primus]

6 hours ago, enjay said:

 

Which again is why this wouldn't be used in isolation. If the teacher knows the child to be neurodivergent, they can accommodate that in the assessment from the tool. Also, if the teacher were to compare several pieces of work from a child, they should get a similar score; if one document rates significantly higher for AI than others, that also suggests the child might not have written it.

 

That said, I'm not particularly confident of the accuracy of these tools! I used several different passages generated by ChatGPT and they typically scored around 50%. That's a lot higher than text I know to be human-written, but still not a brilliantly accurate score considering 100% of the text was AI. I also found several AI checkers also have a "humaniser" feature, which brought that score down to less than 5%.

They are not suitable for any use whatsoever. They are worthless whether used in isolation or otherwise.

 

Using them in any way in your process or approving them in any way is liable to lead to a claim under the Equality Act, they offer absolutely no insight and will only cause issues.

[enjay]

1 minute ago, Primus said:

They are not suitable for any use whatsoever. They are worthless whether used in isolation or otherwise.

 

Using them in any way in your process or approving them in any way is liable to lead to a claim under the Equality Act, they offer absolutely no insight and will only cause issues.

 

That's the conclusion I've come to as well!

[itskdog]

There is a potential future of these tools working, as hidden watermarks are being applied to the text generated by these models with slight alterations to the probabilities of the next token as the text is generated, a bit like OpenAI's SynthID for images.

 

I could see a future where you can subscribe to a tool similar to how Turnitin is already widely used for plagiarism checking, that just reaches out to various APIs for different companies to check against their watermarks.

[sigma]

There's a problem with Turnitin:

 

https://www.bbc.co.uk/news/articles/c920v288gjd7o

Quote

 

University students have raised concerns over the software used to submit assignments and check them for plagiarism, amid fears their work could be used to train tools which use artificial intelligence.

The University of York Students' Union said planned changes to the licence agreement for the Turnitin system could see students' work used to teach AI models.

The University of York has said it would review any changes to the software's terms and conditions, while the University of Southampton has already started to phase out the system.

 

Quote

 

Sam Dickinson, union affairs officer at York Students' Union, said students' concerns were raised in August before Turnitin announced that any changes would not take place at least until September next year.

"The key issue here is that students' intellectual property is something that should belong to them," he explained.

"I think there's a fear when AI policies are introduced like this that their work is going to be whisked away and used for something other than student progress at university."