Twitter API

[danIT]

Firstly Im not a programmer but have recently started trying to pickup bits and pieces from here and there.

 

But now I have actually found something (a project) to get stuck in with.

 

Basically I want to create an archive tool for Twitter for searches so that anyone can search for a term in Twitter and save the results offline.

 

I know tools exist to store your own tweets in CSV format but there isnt anything for a search, so im going to create one.

 

Below is what I have got so far:

 

$q=$_GET['q'];

if($_GET['q']==''){

$q = '%23mmuk09';}

$limit = "100"; #Set by Twitter API

$search = "http://search.twitter.com/search.atom?q=".$q."&rpp=". $limit;"";

$tw = curl_init();

curl_setopt($tw, CURLOPT_URL, $search);
curl_setopt($tw, CURLOPT_RETURNTRANSFER, TRUE);
$twi = curl_exec($tw);
$search_res = new SimpleXMLElement($twi);


echo "Twitter search results for '".$q."'";

## Echo the Search Data

foreach ($search_res->entry as $twit1) {

$description = $twit1->content;

$description = preg_replace("#(^|[\n ])@([^ \"\t\n\r<]*)#ise", "'\\1@\\2'", $description);  
$description = preg_replace("#(^|[\n ])([\w]+?://[\w]+[^ \"\n\r\t<]*)#ise", "'\\1\\2'", $description);
$description = preg_replace("#(^|[\n ])((www|ftp)\.[^ \"\t\n\r<]*)#ise", "'\\1\\2'", $description);

$retweet = strip_tags($description);


echo "author->uri,"\" target=\"_blank\">\"",$twit1-link[1]->attributes()->href,"\" title=\"", $twit1->author->name, "\" />\n";
echo "".$description."From: ", $twit1->author->name," Retweet!";

}


curl_close($tw);

?>

 

Now at the moment it only collects 100 Tweets, this is a limit determinded by the Twitter API. Maximum of 100 tweets per requests.

 

Now for the above query searching on '%23mmuk09' there are about 7 pages of tweets if the limit is set at 100. What I want to do is add a loop of some kind to cycle through the all 7 pages and output the results to screen. I need to create 7 seperate requests to get the 7 pages.

 

Any help appreciated.

[powdarrmonkey]

Hmm, I probably can't be much help with Twitter, but a few things you could consider (mostly style):

 

$q=$_GET['q'];

if($_GET['q']==''){

$q = '%23mmuk09';}

 

Any reason you assign $_GET['q'] to $q but then don't use it in your test? It would be clearer like this:

 

if ( $_GET['q'] != '' )
   $q = $_GET['q'];
else
   $q = '%23mmuk09';

 

You can also use the ternary operator to make it even leaner:

 

$q = ($_GET['q'] != '') ? $_GET['q'] : '%23mmuk98';

 

This evaluates the expression in the brackets, and if true assigns the expression after the question mark to your variable, and if false assigns the expression after the colon.

 

$search = "http://search.twitter.com/search.atom?q=".$q."&rpp=". $limit;"";

 

Any reason for the ""; after the line has been terminated ($limit; )?

 

In double quotes, PHP does variable substitution inline, so you don't need to concatenate them with '.':

 

$search = "http://search.twitter.com/search.atom?q=$q&rpp=$limit";

 

But to make it clearer, and specify what is a variable without making it have to guess, you can wrap them in braces:

 

$search = "http://search.twitter.com/search.atom?q={$q}&rpp={$limit}";

 

I prefer this, because it makes it clear what's supposed to be a variable when reading it back, it allows you to use arrays inline, and it helps avoid those really hard to find bugs when PHP is too greedy in its matching and substitutes a different variable than you were expecting.

 

$description = preg_replace("#(^|[\n ])@([^ \"\t\n\r<]*)#ise", "'\\1@\\2'", $description);  
$description = preg_replace("#(^|[\n ])([\w]+?://[\w]+[^ \"\n\r\t<]*)#ise", "'\\1\\2'", $description);
$description = preg_replace("#(^|[\n ])((www|ftp)\.[^ \"\t\n\r<]*)#ise", "'\\1\\2'", $description);

 

Sweet monkey tuesdays, I'm not even going to try and read those expressions My regex-fu is weak today, what do they do? can you simplify them? They'll only bite you in six months when you need to change them!

 

preg_match() and preg_replace() are also quite expensive functions to call, so if you're using them on large datasets you might take a bit of a hit. Finding a PHP way to do them might save you a bit of lag.

 

 

Now for the above query searching on '%23mmuk09' there are about 7 pages of tweets if the limit is set at 100. What I want to do is add a loop of some kind to cycle through the all 7 pages and output the results to screen. I need to create 7 seperate requests to get the 7 pages.

 

Any help appreciated.

 

You probably want to decide how many pages are available and then use a for() loop to make multiple curl requests. But it sounds like you're screen-scraping HTML, isn't there an XML api for Twitter? Or is it that which is giving you paged results (eugh)?

 

These are really nitpicky things, it's pretty good stuff as it is They're habits worth getting into though, else in six months time you'll come back to it and go 'why did I do that?'.

 

 

Edit: something else I just thought of: you could use $_REQUEST to get your input. It's a merge of your GET and POST variables, with GET taking precedence, so it doesn't matter how they are sent to you. Obviously for something like an address or password field you'll want to use POST

Edited April 12, 2009 by powdarrmonkey

[danIT]

powdarrmonkey thanks for the extensive breakdown!!! Think im slowly realising that im out of my depth lol

 

I think scraping the t'internet for a few hours yesturday and copying lines from here there and everywhere isnt the safest/cleanest option.

 

I think the regex stuff takes various pieces of text to create the links to profiles and "retweet" etc...

 

Demo here

 

I think i might also be going to wrong route trying to use RSS to get the data, I want pages of data but the RSS feed doesnt really recognise that and the only way the RSS feed could would be if I asked for specific Tweets to be included in the feed - messy

 

The XML idea might work better, but seen as im totaly bewildered by all this it might have to be put on hold.

 

You'd think it would be easy to:

 

Twitter - Give me all the posts that match "something" - dont limit me to 100.

 

I understand that there servers would eventually break if people like me were let loose with the API but essentially I only want it once lol

[powdarrmonkey]

I think scraping the t'internet for a few hours yesturday and copying lines from here there and everywhere isnt the safest/cleanest option.

 

You mean: "scraping t'internet", don't you?

 

Twitter API Wiki / REST API Documentation is the full documentation for the REST API, which you can get XML, JSON, RSS and Atom data from. That's probably a good place to start.

[Jackd]

Just to let other people know incase they want to use this i made a small script that will get the info from tags and save it as a csv

 

here

 

HTH Jack.

[danIT]

Yesturday I found a big long list of all the variables for PHP:

 

E.g. author->name

author->uri

description

published

 

But cannot for the life of me find the site I was using, im busily trying to search the API documentation WIKI on Twitter but dont seem to be able to find this, its probably becuaee im not using the right terminology.

 

Can anyone help?

[Jackd]

Twitter doesnt have variables, they're just the XML paths.

 

Jack

[powdarrmonkey]

Twitter doesnt have variables, they're just the XML paths.

 

Well, strictly what he's referring to are the PHP variables generated from the XML elements.