+ Post New Thread
Results 1 to 9 of 9
Web Development Thread, Screen Scraping.. Ha! Success in Coding and Web Development; After literally a whole 10 hours of willing myself not to throw the computer (or myself for that matter) out ...
  1. #1
    contink's Avatar
    Join Date
    Jul 2006
    Location
    South Yorkshire
    Posts
    3,791
    Thank Post
    303
    Thanked 327 Times in 233 Posts
    Rep Power
    118

    Screen Scraping.. Ha! Success

    After literally a whole 10 hours of willing myself not to throw the computer (or myself for that matter) out the window I have finally understood how the PHP Simple HTML DOM class works and bent it to my needs.

    For anyone else intending to use this, there's a very important nugget of intel I'm going to offer you that took my 8 hours to discern.

    PHP Code:
    $html = new simple_html_dom();

    foreach(
    $html->find($pattern) AS $found) {
        echo 
    $found;

    $found is still an object!!

    If you attempt to put $found into an array you will end up in hell... Pure, unadulterated hell! (I just got back! )

    $data[foo] = $found;
    VERY BAD... FAIL... LOSE MIND!


    $data[foo] = $found->innertext;
    GOOD.. VERY GOOD.. GO DIRECTLY TO PUB!


    The correct example uses one of the attributes like innertext, tag, outertext, etc... to get the string you wanted.
    PHP Code:
    $html = new simple_html_dom();

    $data = array();
    foreach(
    $html->find($pattern) AS $found) {
        echo 
    $found;

        
    // Get the value we want between the tags and put it in the $data array;
        
    $data[] = $found->innertext;
    }

    print_r($data); 
    Hope this saves some poor sap from the "fun" I had today

  2. #2

    webman's Avatar
    Join Date
    Nov 2005
    Location
    North East England
    Posts
    8,406
    Thank Post
    639
    Thanked 961 Times in 661 Posts
    Blog Entries
    2
    Rep Power
    324
    And if you find yourself in the same sort of situation again, you can use var_export()/var_dump() functions

    Or read the documentation for the find() function

  3. #3
    contink's Avatar
    Join Date
    Jul 2006
    Location
    South Yorkshire
    Posts
    3,791
    Thank Post
    303
    Thanked 327 Times in 233 Posts
    Rep Power
    118
    Quote Originally Posted by webman View Post
    And if you find yourself in the same sort of situation again, you can use var_export()/var_dump() functions
    Trust me I went there... it went into a horrendous loop even with the var_dump()... One of the key reasons it went wrong.

    Or read the documentation for the find() function
    ... I'm sorry, what is this doc-you-men-tashun you speak of??

    I did actually... still didn't make a huge amount of sense... perhaps I was having one of those too close to see the tree as I hit it moments.. Either way, painful day

  4. #4

    webman's Avatar
    Join Date
    Nov 2005
    Location
    North East England
    Posts
    8,406
    Thank Post
    639
    Thanked 961 Times in 661 Posts
    Blog Entries
    2
    Rep Power
    324
    We've all been there

    Sometimes best idea is to walk away, clear your head, having something to eat/drink and come back and look at it from a different perspective.

    Ask yourself why isn't it giving you the result you expect? What result do you expect? Check all return values etc etc. Please you got there in the end though

  5. #5
    contink's Avatar
    Join Date
    Jul 2006
    Location
    South Yorkshire
    Posts
    3,791
    Thank Post
    303
    Thanked 327 Times in 233 Posts
    Rep Power
    118
    Quote Originally Posted by webman View Post
    We've all been there
    We work in education and with some of the most "educated" computer users ever... I thought we'd moved there permanently

    Sometimes best idea is to walk away, clear your head, having something to eat/drink and come back and look at it from a different perspective.
    Definitely what was required... unfortunately it was a bit like the routine of pretending not to look at a place where you thought you left something in the hope it will magically re-appear.. I left it 5 times... Not a glimmer, till 12 midnight

    Ask yourself why isn't it giving you the result you expect? What result do you expect? Check all return values etc etc. Please you got there in the end though
    That was the biggest problem of all... total lack of output that made any sense at all... The only data I was getting out was a very long loop of which indicated an unterminated loop.

    It was only when I realised that the object pulled out another object and it was that which was borking things that I realised were I was going fubar. The key confusion was the assumption that something couldn't output a string whilst containing methods, variables, etc.. I now know better

  6. #6

    powdarrmonkey's Avatar
    Join Date
    Feb 2008
    Location
    Alcester, Warwickshire
    Posts
    4,859
    Thank Post
    412
    Thanked 777 Times in 650 Posts
    Rep Power
    182
    Quote Originally Posted by contink View Post
    Definitely what was required... unfortunately it was a bit like the routine of pretending not to look at a place where you thought you left something in the hope it will magically re-appear.. I left it 5 times... Not a glimmer, till 12 midnight
    I think the difference is going away and forgetting totally about it, instead of going away to do something else with the intention of coming back in a few minutes. It has to be something else so totally absorbing that our subconscious can get to work without distraction. That's my theory, anyway.

  7. #7
    contink's Avatar
    Join Date
    Jul 2006
    Location
    South Yorkshire
    Posts
    3,791
    Thank Post
    303
    Thanked 327 Times in 233 Posts
    Rep Power
    118
    Quote Originally Posted by powdarrmonkey View Post
    I think the difference is going away and forgetting totally about it, instead of going away to do something else with the intention of coming back in a few minutes. It has to be something else so totally absorbing that our subconscious can get to work without distraction. That's my theory, anyway.
    Well, I went off to grab some food, watch Made of Honour and sit with my hand on my wife's bump waiting for the baby to move.. The latter was quite challenging as the baby is still playing hide and seek with me.

    Take your point though... I yield

  8. #8

    powdarrmonkey's Avatar
    Join Date
    Feb 2008
    Location
    Alcester, Warwickshire
    Posts
    4,859
    Thank Post
    412
    Thanked 777 Times in 650 Posts
    Rep Power
    182
    I bet it was still bugging you though, right?

  9. #9
    contink's Avatar
    Join Date
    Jul 2006
    Location
    South Yorkshire
    Posts
    3,791
    Thank Post
    303
    Thanked 327 Times in 233 Posts
    Rep Power
    118
    Quote Originally Posted by powdarrmonkey View Post
    I bet it was still bugging you though, right?
    Actually I think the extended break that fixed it for me... My wife has taken to falling asleep if I talk work during "us" time so I'm getting better at clearing my head. It's probably how I was able to fix it...

    Playing hunt the baby does tend to focus the mind elsewhere too...

SHARE:
+ Post New Thread

Similar Threads

  1. Link Scraping
    By Hightower in forum Web Development
    Replies: 8
    Last Post: 3rd March 2009, 01:38 PM
  2. Scraping info from apps
    By localzuk in forum Coding
    Replies: 4
    Last Post: 7th April 2008, 01:58 PM
  3. Success!!!
    By contink in forum General Chat
    Replies: 28
    Last Post: 22nd January 2007, 09:50 PM
  4. Success Maker
    By Gatt in forum Windows
    Replies: 11
    Last Post: 27th April 2006, 07:46 AM

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •