SAR Slack Request, Python Script

[DalekSec]

Hi All,

 

We've had a Subject Access Request and as part of that means i have to export all of our slack messages and search though them.. 8 years worth, 400 folders, 36k files. I don't know how to code so taken to ChatGPT to help...

 

This is the script I ended up with (below), it runs but comes back with a result way to fast and no results, when i know there is files with the name in.

 

What am i missing?

 

import os
import json
import concurrent.futures

search_terms = ["[email protected]", "Mr Placeholder"]
folder_path = r"c:\sar\sar"  # Use 'r' before the string to treat it as a raw string

def search_json_file(file_path, search_terms):
   try:
       with open(file_path, "rb") as json_file:
           data = json_file.read().decode("utf-8")
           json_data = json.loads(data)
           if "email" in json_data and any(term in json_data["email"] for term in search_terms):
               return file_path
           elif "name" in json_data and any(term in json_data["name"] for term in search_terms):
               return file_path
   except (UnicodeDecodeError, json.JSONDecodeError):
       pass
   return None

def search_json_files_parallel(folder_path, search_terms):
   found_in_files = []

   with concurrent.futures.ProcessPoolExecutor() as executor:
       file_paths = []
       for root, _, filenames in os.walk(folder_path):
           for filename in filenames:
               if filename.endswith(".json"):
                   file_path = os.path.join(root, filename)
                   file_paths.append(file_path)

       results = executor.map(search_json_file, file_paths, [search_terms] * len(file_paths))
       for result, file_path in zip(results, file_paths):
           if result:
               found_in_files.append(file_path)

   return found_in_files

result = search_json_files_parallel(folder_path, search_terms)
if result:
   print(f"Found in the following files:")
   for file_path in result:
       print(file_path)
else:

   print("Search terms not found in any files.")

[DalekSec]

Have put in: print("Processing:", file_path)

 

Which shows it going through the files but unsure if searching in the files or just file name

[ThomL]

I'm no good with python, asked chatgpt to make a powershell script and this looks pretty good, might be worth a go?

# Prompt the user for the search term
$searchTerm = Read-Host "Enter the search term"

# Specify the root folder to start the search
$rootFolder = "C:\Path\To\Root\Folder"

# Function to search for JSON files and check for the search term
function SearchJsonFiles($path) {
   $jsonFiles = Get-ChildItem -Path $path -Filter *.json -Recurse

   foreach ($jsonFile in $jsonFiles) {
       $filenameContainsTerm = $jsonFile.Name -like "*$searchTerm*"
       $contentContainsTerm = Get-Content $jsonFile.FullName | Out-String -Raw | Select-String -Pattern $searchTerm

       if ($filenameContainsTerm -or $contentContainsTerm) {
           Write-Host "Match found in $($jsonFile.FullName)"
       }
   }
}

# Start the search
SearchJsonFiles $rootFolder

 

 

This is apparently a multi-thread version of the PowerShell above:

# Prompt the user for the search term
$searchTerm = Read-Host "Enter the search term"

# Specify the root folder to start the search
$rootFolder = "C:\Path\To\Root\Folder"

# Function to search for JSON files and check for the search term
function SearchJsonFiles($path) {
   $jsonFiles = Get-ChildItem -Path $path -Filter *.json -Recurse

   foreach ($jsonFile in $jsonFiles) {
       $filenameContainsTerm = $jsonFile.Name -like "*$searchTerm*"
       $contentContainsTerm = Get-Content $jsonFile.FullName | Out-String -Raw | Select-String -Pattern $searchTerm

       if ($filenameContainsTerm -or $contentContainsTerm) {
           Write-Host "Match found in $($jsonFile.FullName)"
       }
   }
}

# Create jobs for parallel processing
$jobs = @()

$folders = Get-ChildItem -Path $rootFolder -Directory
foreach ($folder in $folders) {
   $job = Start-Job -ScriptBlock { param($folderPath) SearchJsonFiles $folderPath } -ArgumentList $folder.FullName
   $jobs += $job
}

# Wait for all jobs to complete
$jobs | Wait-Job

# Retrieve job results and remove the jobs
$jobs | ForEach-Object {
   Receive-Job $_ | ForEach-Object { Write-Host $_ }
   Remove-Job $_
}

 

 

I also asked ChatGPT to convert the PowerShell to Python and multi-thread it, this is the result:

import os
import json
import concurrent.futures

# Prompt the user for the search term
search_term = input("Enter the search term: ")

# Specify the root folder to start the search
root_folder = "C:\\Path\\To\\Root\\Folder"

# Function to search for JSON files and check for the search term
def search_json_files(path):
   for foldername, subfolders, filenames in os.walk(path):
       for filename in filenames:
           if filename.endswith(".json"):
               file_path = os.path.join(foldername, filename)
               with open(file_path, "r", encoding="utf-8") as json_file:
                   try:
                       json_data = json.load(json_file)
                       if search_term in filename or search_term in json.dumps(json_data):
                           print(f"Match found in {file_path}")
                   except json.JSONDecodeError:
                       pass

# Create a thread pool for parallel processing
with concurrent.futures.ThreadPoolExecutor() as executor:
   folders = [os.path.join(root_folder, folder) for folder in os.listdir(root_folder) if os.path.isdir(os.path.join(root_folder, folder))]
   results = list(executor.map(search_json_files, folders))

# Note: In this example, we're using ThreadPoolExecutor. You can also use ProcessPoolExecutor for parallel processing using separate processes.

Edited August 16, 2023 by ThomL

[DalekSec]

I've been using ChatGPT but it keeps coming back with 0 results, in Python and Powershell.

[ThomL]

You tried the scripts I posted and still got the same 0 results?

[DalekSec]

Yupp :/ Have tested it with a string i know is in some of the files.

[ThomL]

Well that's annoying! Guessing you've also updated the parts of the scripts as needed? e.g. in the PowerShell there is a line that need you put the root folder path:

[color=#333333]# Specify the root folder to start the search
[/color][color=#333333]$rootFolder = "C:\Path\To\Root\Folder"[/color]

[DalekSec]

Yeah have changed to right folder path

[ThomL]

Well that sucks - hard to help further without files to test with, which can't be shared because of the nature of the files!

 

Maybe it's an issue with the way the scripts are trying to find the data in the JSON, or maybe the way they are accessing the JSON... I dunno! I'll try to find some demo JSON files from the web to play with...

[ThomL]

This worked for me with some example and test JSON files downloaded from random sites, original non multi-threaded PowerShell had a bad parameter on the out-string cmdlet, so I removed it and things look good:

# Prompt the user for the search term
$searchTerm = Read-Host "Enter the search term"

# Specify the root folder to start the search
$rootFolder = "C:\path\here\"

# Function to search for JSON files and check for the search term
function SearchJsonFiles($path) {
   $jsonFiles = Get-ChildItem -Path $path -Filter *.json -Recurse

   foreach ($jsonFile in $jsonFiles) {
       $filenameContainsTerm = $jsonFile.Name -like "*$searchTerm*"
       $contentContainsTerm = Get-Content $jsonFile.FullName | Select-String -Pattern $searchTerm

       if ($filenameContainsTerm -or $contentContainsTerm) {
           Write-Host "Match found in $($jsonFile.FullName)"
       }
   }
}

# Start the search
SearchJsonFiles $rootFolder

[DalekSec]

This worked for me with some example and test JSON files downloaded from random sites, original non multi-threaded PowerShell had a bad parameter on the out-string cmdlet, so I removed it and things look good:

# Prompt the user for the search term
$searchTerm = Read-Host "Enter the search term"

# Specify the root folder to start the search
$rootFolder = "C:\path\here\"

# Function to search for JSON files and check for the search term
function SearchJsonFiles($path) {
   $jsonFiles = Get-ChildItem -Path $path -Filter *.json -Recurse

   foreach ($jsonFile in $jsonFiles) {
       $filenameContainsTerm = $jsonFile.Name -like "*$searchTerm*"
       $contentContainsTerm = Get-Content $jsonFile.FullName | Select-String -Pattern $searchTerm

       if ($filenameContainsTerm -or $contentContainsTerm) {
           Write-Host "Match found in $($jsonFile.FullName)"
       }
   }
}

# Start the search
SearchJsonFiles $rootFolder

 

IT'S WORKING!!! I could kiss you!

[ThomL]

Not sure how performant it is, hopefully good enough?

 

I did try to make the script multi-threaded, not something I've done much of before and this seems to work for me, for each JSON file found it should launch a new job to search the file (I think​):

# Prompt the user for the search term
$searchTerm = Read-Host "Enter the search term"

# Specify the root folder to start the search
$rootFolder = "C:\Users\Thom\Downloads\"

# Get json files
$jsonFiles = Get-ChildItem -Path $rootFolder -Filter *.json -Recurse

# Function to search JSON filenames and content for the search term
$functions = {
   function SearchJsonFiles($json, $searchTerm) {    
       $filenameContainsTerm = $json.Name -like "*$searchTerm*"
       $contentContainsTerm = Get-Content $json.FullName | Select-String -Pattern $searchTerm

       if ($filenameContainsTerm -or $contentContainsTerm) {
           Write-Output "Match found in $($json.FullName)"
       }
   }
}

# Create jobs for parallel processing
$jobs = @()

# Foreach Json file found spawn a new job to search the file
foreach ($json in $jsonFiles) {   
   $job = Start-Job -InitializationScript $functions -ScriptBlock {SearchJsonFiles $using:json $using:searchTerm}
   $jobs += $job
}

# Wait for all jobs to complete
$jobs | Wait-Job | Out-Null# Retrieve job results and remove the jobs
$jobs | ForEach-Object {
   Receive-Job $_
   Remove-Job $_
}

 

I did try another version that starts a job per directory found, but this seems far slower... maybe there's something wrong in the code. It did complete with the expected result eventually:

# Prompt the user for the search term
$searchTerm = Read-Host "Enter the search term"

# Specify the root folder to start the search
$rootFolder = "C:\Users\Thom\Downloads\"

# Function to search JSON filenames and content for the search term
$functions = {
   function SearchJsonFiles($Folder, $searchTerm) {        
       $jsonFiles = Get-ChildItem -Path $Folder.FullName -Filter *.json
       foreach ($json in $jsonFiles) {
           $filenameContainsTerm = $json.Name -like "*$searchTerm*"
           $contentContainsTerm = Get-Content $json.FullName | Select-String -Pattern $searchTerm

           if ($filenameContainsTerm -or $contentContainsTerm) {
               Write-Output "Match found in $($json.FullName)"
           }
       }
   }
}

# Create jobs for parallel processing
$jobs = @()

# Get root folder as directory object
$rootDir = Get-Item $rootFolder

# Get all subfolders
$folders = Get-ChildItem -Path $rootFolder -Directory -Recurse

# spawn job for root folder
$jobs += Start-Job -InitializationScript $functions -ScriptBlock {SearchJsonFiles $using:rootDir $using:searchTerm}

# Foreach subfolder spawn a new job searching for json files and searching the files
foreach ($folder in $folders) {   
   $job = Start-Job -InitializationScript $functions -ScriptBlock {SearchJsonFiles $using:folder $using:searchTerm}
   $jobs += $job
}

# Wait for all jobs to complete
$jobs | Wait-Job | Out-Null

# Retrieve job results and remove the jobs
$jobs | ForEach-Object {
   Receive-Job $_
   Remove-Job $_
}

 

I've been messing about with this far too long and give up!

[DalekSec]

Didn't take too long to go though, 5mins or so. So didn't need the multi-threading

[dmj]

Just incase anyone is searching for this thread and needs to do something like this bash, it's a oneliner and multithreaded. (windows SFL, Linux, mac)

 

find /path/to/source -name "*.json" | xargs grep -il "search term" | xargs -I{} cp --parents  {}  /path/to_dest

[andy_b]

Agent Ransack (https://www.mythicsoft.com/agentransack/) is my goto for searches that require a little more customisation than default.

[DalekSec]

Sorry to bring up this thread again, having to do this again and dug out the script, but im now getting an "Cannot bind argument to parameter 'Pattern' because it is an empty string." error?

 

Update: used Agent Ransack which worked perfectly, tested with a few i knew would appear and came up. Did it with SAR subject and 0 has come up! (Have tons in microsoft)

Edited April 18, 2024 by DalekSec