+ Post New Thread
Results 1 to 8 of 8
Coding Thread, Twitter API in Coding and Web Development; Firstly Im not a programmer but have recently started trying to pickup bits and pieces from here and there. But ...
  1. #1

    Join Date
    Apr 2006
    Location
    UK
    Posts
    939
    Thank Post
    39
    Thanked 70 Times in 54 Posts
    Rep Power
    30

    Twitter API

    Firstly Im not a programmer but have recently started trying to pickup bits and pieces from here and there.

    But now I have actually found something (a project) to get stuck in with.

    Basically I want to create an archive tool for Twitter for searches so that anyone can search for a term in Twitter and save the results offline.

    I know tools exist to store your own tweets in CSV format but there isnt anything for a search, so im going to create one.

    Below is what I have got so far:

    Code:
    <?php
    
    $q=$_GET['q'];
    
    if($_GET['q']==''){
    
    $q = '%23mmuk09';}
    
    $limit = "100"; #Set by Twitter API
    
    $search = "http://search.twitter.com/search.atom?q=".$q."&rpp=". $limit;"";
    
    $tw = curl_init();
    
    curl_setopt($tw, CURLOPT_URL, $search);
    curl_setopt($tw, CURLOPT_RETURNTRANSFER, TRUE);
    $twi = curl_exec($tw);
    $search_res = new SimpleXMLElement($twi);
    
    	
    echo "<h3>Twitter search results for '".$q."'</h3>";
    
    ## Echo the Search Data
    
    foreach ($search_res->entry as $twit1) {
    
    $description = $twit1->content;
    
    $description = preg_replace("#(^|[\n ])@([^ \"\t\n\r<]*)#ise", "'\\1<a href=\"http://www.twitter.com/\\2\" >@\\2</a>'", $description);  
    $description = preg_replace("#(^|[\n ])([\w]+?://[\w]+[^ \"\n\r\t<]*)#ise", "'\\1<a href=\"\\2\" >\\2</a>'", $description);
    $description = preg_replace("#(^|[\n ])((www|ftp)\.[^ \"\t\n\r<]*)#ise", "'\\1<a href=\"http://\\2\" >\\2</a>'", $description);
    
    $retweet = strip_tags($description);
    
    
    echo "<div class='user'><a href=\"",$twit1->author->uri,"\" target=\"_blank\"><img border=\"0\" width=\"48\" class=\"twitter_thumb\" src=\"",$twit1->link[1]->attributes()->href,"\" title=\"", $twit1->author->name, "\" /></a>\n";
    echo "<div class='text'>".$description."<div class='description'>From: ", $twit1->author->name," <a href='http://twitter.com/home?status=RT: ".$retweet."' target='_blank'>Retweet!</a></div></div><div class='clear'></div></div>";
    
    }
    
    
    curl_close($tw);
    
    ?>
    Now at the moment it only collects 100 Tweets, this is a limit determinded by the Twitter API. Maximum of 100 tweets per requests.

    Now for the above query searching on '%23mmuk09' there are about 7 pages of tweets if the limit is set at 100. What I want to do is add a loop of some kind to cycle through the all 7 pages and output the results to screen. I need to create 7 seperate requests to get the 7 pages.

    Any help appreciated.

  2. #2

    powdarrmonkey's Avatar
    Join Date
    Feb 2008
    Location
    Alcester, Warwickshire
    Posts
    4,867
    Thank Post
    412
    Thanked 778 Times in 651 Posts
    Rep Power
    182
    Hmm, I probably can't be much help with Twitter, but a few things you could consider (mostly style):

    Quote Originally Posted by danIT View Post
    Code:
    <?php
    
    $q=$_GET['q'];
    
    if($_GET['q']==''){
    
    $q = '%23mmuk09';}
    Any reason you assign $_GET['q'] to $q but then don't use it in your test? It would be clearer like this:

    PHP Code:
    if ( $_GET['q'] != '' )
        
    $q $_GET['q'];
    else
        
    $q '%23mmuk09'
    You can also use the ternary operator to make it even leaner:

    PHP Code:
    $q = ($_GET['q'] != '') ? $_GET['q'] : '%23mmuk98'
    This evaluates the expression in the brackets, and if true assigns the expression after the question mark to your variable, and if false assigns the expression after the colon.

    Code:
    $search = "http://search.twitter.com/search.atom?q=".$q."&rpp=". $limit;"";
    Any reason for the ""; after the line has been terminated ($limit; )?

    In double quotes, PHP does variable substitution inline, so you don't need to concatenate them with '.':

    PHP Code:
    $search "http://search.twitter.com/search.atom?q=$q&rpp=$limit"
    But to make it clearer, and specify what is a variable without making it have to guess, you can wrap them in braces:

    PHP Code:
    $search "http://search.twitter.com/search.atom?q={$q}&rpp={$limit}"
    I prefer this, because it makes it clear what's supposed to be a variable when reading it back, it allows you to use arrays inline, and it helps avoid those really hard to find bugs when PHP is too greedy in its matching and substitutes a different variable than you were expecting.

    Code:
    $description = preg_replace("#(^|[\n ])@([^ \"\t\n\r<]*)#ise", "'\\1<a href=\"http://www.twitter.com/\\2\" >@\\2</a>'", $description);  
    $description = preg_replace("#(^|[\n ])([\w]+?://[\w]+[^ \"\n\r\t<]*)#ise", "'\\1<a href=\"\\2\" >\\2</a>'", $description);
    $description = preg_replace("#(^|[\n ])((www|ftp)\.[^ \"\t\n\r<]*)#ise", "'\\1<a href=\"http://\\2\" >\\2</a>'", $description);
    Sweet monkey tuesdays, I'm not even going to try and read those expressions My regex-fu is weak today, what do they do? can you simplify them? They'll only bite you in six months when you need to change them!

    preg_match() and preg_replace() are also quite expensive functions to call, so if you're using them on large datasets you might take a bit of a hit. Finding a PHP way to do them might save you a bit of lag.


    Now for the above query searching on '%23mmuk09' there are about 7 pages of tweets if the limit is set at 100. What I want to do is add a loop of some kind to cycle through the all 7 pages and output the results to screen. I need to create 7 seperate requests to get the 7 pages.

    Any help appreciated.
    You probably want to decide how many pages are available and then use a for() loop to make multiple curl requests. But it sounds like you're screen-scraping HTML, isn't there an XML api for Twitter? Or is it that which is giving you paged results (eugh)?

    These are really nitpicky things, it's pretty good stuff as it is They're habits worth getting into though, else in six months time you'll come back to it and go 'why did I do that?'.


    Edit: something else I just thought of: you could use $_REQUEST to get your input. It's a merge of your GET and POST variables, with GET taking precedence, so it doesn't matter how they are sent to you. Obviously for something like an address or password field you'll want to use POST
    Last edited by powdarrmonkey; 12th April 2009 at 04:56 PM.

  3. #3

    Join Date
    Apr 2006
    Location
    UK
    Posts
    939
    Thank Post
    39
    Thanked 70 Times in 54 Posts
    Rep Power
    30
    powdarrmonkey thanks for the extensive breakdown!!! Think im slowly realising that im out of my depth lol

    I think scraping the t'internet for a few hours yesturday and copying lines from here there and everywhere isnt the safest/cleanest option.

    I think the regex stuff takes various pieces of text to create the links to profiles and "retweet" etc...

    Demo here

    I think i might also be going to wrong route trying to use RSS to get the data, I want pages of data but the RSS feed doesnt really recognise that and the only way the RSS feed could would be if I asked for specific Tweets to be included in the feed - messy

    The XML idea might work better, but seen as im totaly bewildered by all this it might have to be put on hold.

    You'd think it would be easy to:

    Twitter - Give me all the posts that match "something" - dont limit me to 100.

    I understand that there servers would eventually break if people like me were let loose with the API but essentially I only want it once lol

  4. #4

    powdarrmonkey's Avatar
    Join Date
    Feb 2008
    Location
    Alcester, Warwickshire
    Posts
    4,867
    Thank Post
    412
    Thanked 778 Times in 651 Posts
    Rep Power
    182
    Quote Originally Posted by danIT View Post
    I think scraping the t'internet for a few hours yesturday and copying lines from here there and everywhere isnt the safest/cleanest option.
    <pet-hate>
    You mean: "scraping t'internet", don't you?
    </pet-hate>

    Twitter API Wiki / REST API Documentation is the full documentation for the REST API, which you can get XML, JSON, RSS and Atom data from. That's probably a good place to start.

  5. #5

    Join Date
    Jan 2007
    Location
    Durham, UK
    Posts
    328
    Thank Post
    33
    Thanked 17 Times in 12 Posts
    Rep Power
    21
    Just to let other people know incase they want to use this i made a small script that will get the info from tags and save it as a csv

    here

    HTH Jack.

  6. #6

    Join Date
    Apr 2006
    Location
    UK
    Posts
    939
    Thank Post
    39
    Thanked 70 Times in 54 Posts
    Rep Power
    30
    Yesturday I found a big long list of all the variables for PHP:

    E.g. author->name
    author->uri
    description
    published

    But cannot for the life of me find the site I was using, im busily trying to search the API documentation WIKI on Twitter but dont seem to be able to find this, its probably becuaee im not using the right terminology.

    Can anyone help?

  7. #7

    Join Date
    Jan 2007
    Location
    Durham, UK
    Posts
    328
    Thank Post
    33
    Thanked 17 Times in 12 Posts
    Rep Power
    21
    Twitter doesnt have variables, they're just the XML paths.

    Jack

  8. #8

    powdarrmonkey's Avatar
    Join Date
    Feb 2008
    Location
    Alcester, Warwickshire
    Posts
    4,867
    Thank Post
    412
    Thanked 778 Times in 651 Posts
    Rep Power
    182
    Quote Originally Posted by Jackd View Post
    Twitter doesnt have variables, they're just the XML paths.
    Well, strictly what he's referring to are the PHP variables generated from the XML elements.



SHARE:
+ Post New Thread

Similar Threads

  1. Twitter Poll How Many people use twitter
    By russdev in forum Jokes/Interweb Things
    Replies: 11
    Last Post: 27th May 2009, 05:19 PM
  2. Twitter
    By Edu-IT in forum Other Stuff
    Replies: 42
    Last Post: 26th May 2009, 03:16 PM
  3. SIMS on Twitter
    By ajbritton in forum MIS Systems
    Replies: 3
    Last Post: 3rd April 2009, 11:17 AM
  4. Google map API
    By mossj in forum EduGeek Joomla 1.5 Package
    Replies: 0
    Last Post: 30th January 2009, 05:16 PM
  5. Alternative to google maps API
    By _Bat_ in forum Web Development
    Replies: 48
    Last Post: 2nd March 2007, 03:18 PM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •