[Regex] Trying to make an obfuscation character regex

[Garacesh]

Essentially, I'm trying to create a rule that flags any email with obfuscation characters. You know the ones. The We have naughty videos of you now pay us bitcoin plz ones where they replace random letters with șìɱίḽȃṛ ƚȯοḳίňģ οηѐṡ to bypass word filters.

So my regex is, essentially*, (?i)[ậḅĉḏęḟǵĥіʝḳḷɱրợῥգŕşեữѵŵϰўʑ] and I'm removing characters as-and-when that have legitimate use (used commonly in French, for example ô or ç) when they get flagged. This'll never be perfect, but it'll be better than nothing I guess?

Weirdly, though, I'm getting emails trip the filter that don't match the rule. They get redirected to the right inbox, they get the [Obfuscation] tag applied to the header.. But they don't match?

 

Regex101 says they don't match.

Google's own policy tester says they don't match (both copying-and-pasting the email as-is, and dumping the entire raw contents)

 

Has anybody ever encountered this before? This is happening with quite a few services, and I just can't figure out why.

Only thing I can think of is maybe there are certain characters that PCRE just really doesn't like?

 

I can't dump the raw email(s) here because of EduGeek's 15k character limit

 

* A looot of characters have been removed to make it a bit more palatable, but you get the picture.

Edited November 29, 2021 by Garacesh

[Bedders]

I know that my string for detecting Bitcoin Wallets sometimes gets matched by BASE64 encoding (I think that's the one anyway) when images get sent. Maybe it's something similar?

 

Edit to add: Sorry for the necro, just seen the post date - I was searching for something regex and came across this

[Garacesh]

Eh it's not that much of a necro, I'm still facing the issue lol.

[mbedford]

can you drop the email into a pastebin?

[Rooty]

Rather than try to filter for what you don't want, have you tried doing the inverse and filtering for only what you do need?

 

Email regex is notoriously tricky, but the following regex from Stack Overflow is good (stackoverflow.com/questions/201323/how-can-i-validate-an-email-address-using-a-regular-expression):

 

(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(??:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(??:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])

Edited October 23, 2022 by Rooty

[Garacesh]

The above wasn't meant to be filtering based off domain, it was meant to be filtering off message body, bесаսѕе ⅿаⅼісіоսѕ асtοrѕ ԝіⅼⅼ реrⅿսtаtе tḣеіr tехt ԝіtḣ οḃfսѕсаtеḋ сḣаrасtеrѕ tο ḃеаt ѕtаոḋаrḋ rеģех tḣаt ԝоսⅼḋ ḃе рսt іո рⅼасе tο саtсḣ сοⅿⅿοոⅼу-սѕеḋ ԝоrḋѕ, ѕսсḣ аѕ 'ḃіtсоіո', 'рοrոοǵrарḣу', 'ⅿаѕtսrḃаtіոģ' еtс with characters that are functionally similar enough to standard text to be human-readable (your eye probably even skipped over some of them in this sentence without noticing!), but distinct enough that they would fail to match regex rules if not taken into account.

 

At the time, the school I was at was fighting a flood of sextortion emails.

Edited October 24, 2022 by Garacesh

[Rooty]

Ahh from the message body, that's an interesting one and very difficult to protect against perfectly. Really you need your spam software to be catching this because you need something to normalise the diacritics before checking for censored words/content.

 

What are you using as your mail and anti-spam solution?

[Garacesh]

I'm not at the school now, I've changed employers since then.

It was G-Suite, which we're not using at my new place, so this thread is kind of moot now unless anybody is looking to do similar, but I find Defender's spam filtering to be much more robust having seen them both in action now.

 

To be fair, Google was still blocking the emails. I just wanted to have a filter going so I could keep on top of what was coming in, keep an eye on the latest trends, any new tactics etc. But Google's quarantine is (was?) a bit rubbish. Used to be kind of sluggish and take a while to load new messages, I assume it would run a filter/search every time you opened it.

 

The main issue, tl;dr, is that it would trip the rule for obfuscation, move the email to the inbox I had set up for it, but if you actually ran the regex against Google's own policy tester, it said it didn't match. It would catch plenty of legit stuff too.. The polish ż, the french ç, etc, but it would also dump random emails that had no obfuscation at all.