Link Scraping

[Hightower]

Hey all,

 

I have setup a link scraper (using some tutorials I found from my trusty friend Google) in a PHP script and the code looks like this:

 

(I have modified the code a little from tutorials so that it only returns links that start with http - excuse the sloppy code as I am just messing about with this)

 

if(isset($_POST['target_url'])){
$target_url = $_POST['target_url'];
$userAgent = 'Googlebot/2.1 (http://www.googlebot.com/bot.html)';

// make the cURL request to $target_url
$ch = curl_init();
curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
curl_setopt($ch, CURLOPT_URL,$target_url);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
$html= curl_exec($ch);
if (!$html) {
   echo '
An error occurred. Please make sure you have typed the correct address and try again.';
}

// parse the html into a DOMDocument
$dom = new DOMDocument();
@$dom->loadHTML($html);

// grab all the on the page
$xpath = new DOMXPath($dom);
$hrefs = $xpath->evaluate("/html/body//a");
}
?>

</pre><form action="" method="POST" name="frmurl">
   
   
</form><br><br>if(isset($_POST['target_url'])){<br>   $num = 0;<br>   for ($i = 0; $i < $hrefs->length; $i++) {<br>       $href = $hrefs->item($i);<br>       $url = $href->getAttribute('href');<br>       <br>       //See if site starts with http://<br>       $tmp = strpos($url, "http");<br>       //If it does then continue<br>       if($tmp !== 0){<br>       }<br>       else{<br>           $fullurl = $url;<br>           <br>           //Filter it so only the www.address.com is returned<br>           $url = parse_url($url, PHP_URL_HOST);<br>           <br>           //Trim the site to make sure it has no blanks at either end (verification)<br>           $site_raw_address = trim($site_full_address);<br>           <br>           //Check if site begins with WWW. - if it does, remove it<br>           $www_exists = strpos($url, "www.");<br>           if($www_exists !== FALSE){<br>               $url = substr($url,4);<br>           }<br>           <br>           echo $url . '<br>';<br>       }<br>   }<br><br>}<br>?&g

 

The format of the returned sites is just as I want, and looks like this (using BBC - Homepage as an example):

 

newsvote.bbc.co.uk
bbc.co.uk
bbc.co.uk
bbc.co.uk
bbc.co.uk
bbc.co.uk
bbc.co.uk
bbc.co.uk
bbc.co.uk
bbc.co.uk
bbc.co.uk

 

The problem I have got is, as you can see, 'bbc.co.uk' is returned more than once (obviously more than 1 link on the page for it) but I don't want it to be.

 

Can anybody help me modify this code so it does not display duplicates like this - so it should only be returning:

 

newsvote.bbc.co.uk
bbc.co.uk

 

Thanks!

[contink]

Here you go...

 

Find:

if(isset($_POST['target_url'])){
   $num = 0;
   for ($i = 0; $i < $hrefs->length; $i++) {
       $href = $hrefs->item($i);
       $url = $href->getAttribute('href');

 

Replace with:

$visited = array();

if(isset($_POST['target_url'])){
   $num = 0;
   for ($i = 0; $i < $hrefs->length; $i++) {
       $href = $hrefs->item($i);
       $url = $href->getAttribute('href');
       if($visited[$url]){
	continue;
} else {
	$visited[$url] = true;
}

 

 

That's untested but I think it'll work.

[tmcd35]

put the url's into an array and use a loop to check against previous array enteries.

 

$url = $href->getAttribute('href');

 

becomes

 

$newurl = $href->getAttribute('href');
$accepturl=1;
while ($loop != $urlcount + 1)
{
  if ($url[$loop] == $newurl) {$accepturl = 0;}
}
if ($accepturl != 0)
{
 $urlcount = $urlcount + 1;
 $url[$urlcount] = $newurl;
}

 

Then refer to the URL by it's array entry in the rest of the code - $url[$urlcount] rather than $url, if that make sense

[CESIL]

array_unique() takes input array and returns a new array without duplicate values

[ZeroHour]

Could you not use array_unique() at the end or something?

There are many ways to do this including the in_array function.

[Hightower]

That's untested but I think it'll work.

 

Just tried that one - didn't work

[Hightower]

Thanks for the help guys - the array_unique has worked a treat and only needed to change a couple of lines.

[Hightower]

Ok, I thought it worked a treat, but the two address are left in 81 & 141 of the array. How do I get them to move to the start of the array?

[Hightower]

Doesn't matter - got it by using

 

array_keys(array_flip($array));

 

Instead of

 

array_unique($array);