Scrape content from the web

[HPlum78]

The better 1/2 had been trying to search for information that had been released by the ESFA on the gov.uk site the only problem is that the search capability of the site is lets say a little user intensive! She was essentially opening each document! and then saving ....

 

So I wrote this to help her out:-

 

$links = ((Invoke-WebRequest -uri 'https://www.gov.uk/government/collections/skills-funding-agency-update').links | ? {$_.href -like "/government/publications/esfa*"}).href

$links | % {
   $BaseFilter = $_
   #write-host -fore Yellow $BaseFilter

   $WebUri = $_.replace("/government/publications/", "https://www.gov.uk/government/publications/")
   #write-host -fore green $WebUri
   
   $lnk = ((Invoke-WebRequest -uri $WebUri).links | ? {$_.href -like "*$($BaseFilter)*"}).href

       $lnk  | % {
           
          if ($_ -Like "/government/publications/*"){
              $PubUri = "https://www.gov.uk"+$_
              $PubUri = $PubUri | select -Unique
              $PageContent = Invoke-WebRequest -Uri $PubUri
              $flName = $PubUri.Split("/") | select-object -Last 1

              Write-host -Fore green "Got update from web address - copying content to $("$env:HOMEDRIVE\Desktop files\EsfaUpdate\$($flName).htm")" 
              $PageContent.Content | Out-File "$env:HOMEDRIVE\Desktop Files\EsfaUpdate\$($flName).htm"
          
          }
          Else{
               Write-host -fore Magenta $($_) "No valid content at this link"
          }

       }
}

 

It essentially grabs the content out of each document and saves it to the desktop in a folder called EsfaUpdate (you will need to make sure that folder is created!), you can then either use Windows search or as she has done import the files in to OneNote this has allowed her to search and annotate.

 

Hopefully will help someone else out.

Edited July 5, 2019 by HPlum78

[FishCustard]

Excellent! I'm always offering to automate things for my partner. Unfortunately her job doesn't really lend itself to that much automation, even though it's the kind of role (operations) that you assume would.

[sted]

there is (or was its been a while) a firefox plugin download them all and you could set it to download all/specific file types from a website

[HPlum78]

@sted you are indeed correct but has limited access on her laptop to add extensions to the browsers.... Also I write PowerShell! :-p

[bald_pig]

@sted you are indeed correct but has limited access on her laptop to add extensions to the browsers.... Also I write PowerShell! :-p

 

What bass-ackwards corporation locks down their browsers to restrict extensions, but lets the users run powershell scripts?!?

[Arthur]

there is (or was its been a while) a Firefox plugin download them all and you could set it to download all/specific file types from a website

The WebExtension version of DownThemAll isn't available for Firefox 57+.

 

www.downthemall.org/category/news/

https://github.com/downthemall/downthemall/issues/270

 

There are a few alternative extensions, but they aren't as good.

[HPlum78]

I don't make the rules I just write the code.....