Will File Server Resource Manager's content classifier crash with a bad PDF?

[pete]

In Server 2019 running a content classification search in a particular folder tree, the following happens:

 

.Net reports an unhandled exception in fsdmhost.exe due to a System.Access.ViolationException

fsdmhost.exe points the finger at Windows.Data.Pdf.dll

Content Classifier then fails

Report generation fails

 

There are probably 108K PDFs in the folder tree and subfolders can contain up to 3000.

 

The same search pointed at a different folder tree works without issue and parses PDFs within the tree.

 

I'd hope it skips an unparseable PDF in the same way it does an overly large zip file, but.....

 

Any ideas?

Edited December 12, 2024 by pete

[dhicks]

I'd hope it skips an unparseable PDF in the same way it does an overly large zip file, but.....

 

Can you pre-process the folder(s) with a PowerShell script that calls the same function in the Windows.Data.Pdf.dll DLL but properly handles the exception, marking / recording the PDFs in some way for you to remove / deal with?

[pete]

Can you pre-process the folder(s) with a PowerShell script that calls the same function in the Windows.Data.Pdf.dll DLL but properly handles the exception, marking / recording the PDFs in some way for you to remove / deal with?

 

I'm going to have to. Looking around at a couple of Github projects and commercial tools as an alternate means of looking for the text we're interested in (SAR)). But we also have to find the "bad" PDFs because at the moment there's a terabyte or so of data that can't be effectively content-searched/classified.

[dhicks]

Looking around at a couple of Github projects and commercial tools as an alternate means of looking for the text we're interested in (SAR)).

 

If it's any help, I've recently been doing some work to process (OCR) a directory tree of PDFs, and found this Python library quite decent:

 

https://github.com/ocrmypdf/OCRmyPDF

 

That was for a Python script, obviously, running on Linux (Debian), which proved to be an easy enough install - I've not tried it on Windows, and it might be more useful for you to stick with something that calls the same DLL function that's causing the issue. My guess is that the Python library would also be tripped up by the same un-parsable PDFs, although that is just a guess.

[pete]

Update for future searchers:

 

File Locator Pro (aka Agent Ransack) does a better job than native tools for the content search. It also uses Tesseract to process PDFs. Scope the search appropriately though (it will try to find your text string in .mp4 files if you don't).

 

It looks like the exception log it produces may also be useful for tracking down bad PDFs too.

Edited December 13, 2024 by pete
typo

[dmj]

Can you pre-process the folder(s) with a PowerShell script that calls the same function in the Windows.Data.Pdf.dll DLL but properly handles the exception, marking / recording the PDFs in some way for you to remove / deal with?

 

I'm usually very supportive of @dhicks posts, but on this occasion I would disagree. Writing a wrapper script to fix an upstream issue isn't the way to resolve it. Unfortunately with closed sourced systems, the only way to fix is to raise a bug with the provider. Ultimately a wrapper script is just going to add complexity to the environment and make solving issues harder.

 

I would advocate pushing back to ms to get it fixed, or perhaps, if the issues are genuine pdf corruptions then identifying and fixing those files.

[pete]

Second follow-up. We're reasonably certain the PDFs belong to terrible lesson resource packs that someone's helpfully shared with their colleagues.

 

So the suspected culprits will be moved to a single area and scanned in chunks by FSRM. Anything that doesn't cause it to fall over gets to go back into circulation, anything that causes it to crash gets the file hash added to FSRM restrictions.