Lighthouse has a new layout. Prefer the old one? Return to the old layout, and switch back any time from the link at the top of each page.

There was a problem

You must be a member of this account.

Suggestion

#1

Dear Peter,

you asked the users to submit suggestions for examples of the new scrubyt release. So here is mine: I was trying to scrape the search suggestions on the german yahoo site (unfortunately there is no api like for the us version). Since I didn't got it working it would be cool to see a solution. Pretty selfish I know :-)

Reported by crisb · November 5th, 2008 @ 07:38 PM

State: resolved
Milestone: 0.4.1
Assigned to: Peter Szinek Peter Szinek

Activity

  1. Peter Szinek
    Peter Szinek
    • State changed from new to open
    • Assigned user set to Peter Szinek

    OK - let me know where the site is located and I'll look into it!

    November 24th, 2008 @ 09:44 AM

  2. crisb
    crisb

    The site is located here: http://de.yahoo.com/

    Actually it works the same as http://yahoo.com/, but since the german site doesn't offer an api, it would be really cool if you could take a look at it.

    November 24th, 2008 @ 09:51 AM

  3. Peter Szinek
  4. crisb
  5. antel
    antel

    Scrubyt doesn't support javascript, the drop down menu with urls start only by an javascript input, it ask the server and return html, test it with firebug and webdeveloper inspecting objects with javascript enabled/disabled, at now it will work only with firewatir.

    November 24th, 2008 @ 03:37 PM

  6. crisb
    crisb

    I know, that's why I didn't got it working. But I thought that Firewatir is included in the new scrubyt and therefore it would be able to scrape the suggestions.

    Or am I wrong? :-)

    November 24th, 2008 @ 03:51 PM

  7. antel
    antel

    If you activate the firewatir agent, it will work!

    November 24th, 2008 @ 04:32 PM

  8. Peter Szinek
    Peter Szinek

    Yeah, it should (at least that's the idea :-)) however, currently it's throwing

    
    textfield name and //input["@id=p"] is read only. (Watir::Exception::ObjectReadOnlyException)
    

    WTF?! how can it be read only??!? Need to dig around a bit more and/or ask the Watir guys.

    November 24th, 2008 @ 05:28 PM

  9. Peter Szinek
    Peter Szinek

    Ok, I found the problem... will need to push out one more release ;-)

    November 25th, 2008 @ 12:12 AM

  10. crisb
    crisb

    Fantastic. Looking forward to it.

    November 25th, 2008 @ 10:50 AM

  11. Peter Szinek
  12. Peter Szinek
    Peter Szinek

    OK, done - now I just need to find some time to crank the release out - should happen today, or in the worst case tomorrow!

    December 1st, 2008 @ 01:07 PM

  13. crisb
    crisb

    can you drop me a line as soon as you release it?

    December 4th, 2008 @ 11:13 PM

  14. Peter Szinek
  15. Peter Szinek
    Peter Szinek
    • State changed from open to resolved

    Done, phew was not easy...

    http://pastie.org/332793

    (maybe to wait 5 seconds is too much - try it out and play around with it).

    Let me know if it doesn't work (well, for some time it won't - the gem needs to be mirrored @ rubyforge)

    December 6th, 2008 @ 09:14 PM

  16. crisb
    crisb

    cool, thanks.

    but could it be possible that the gem still isn't mirrored at rubyforge?

    December 10th, 2008 @ 03:11 PM

  17. Peter Szinek
    Peter Szinek

    hmmm.... wtf it seems it's not yet there. it should be 0.4.1 but for me it installs just 0.4.06... checking it out.

    December 10th, 2008 @ 03:18 PM

  18. crisb
    crisb

    i now installed it manually and tried to get it working with de.yahoo.com. but i don't even get the script to fill out the textfield.

    yahoo_data = Scrubyt::Extractor.define :agent => :firefox do
      fetch 'http://de.yahoo.com/' 
      fill_textfield_and_wait 'p', 'ruby', 5
    end
    

    December 11th, 2008 @ 10:44 AM

  19. Peter Szinek
    Peter Szinek

    Running out for lunch now, but will check it out after I am back!

    December 11th, 2008 @ 11:00 AM

  20. Peter Szinek
    Peter Szinek

    Does it work for you with yahoo.com (without the de?)

    December 11th, 2008 @ 12:52 PM

  21. crisb
    crisb

    I checked it with yahoo.com He doesn't seem to fill in the textfield their too.

    December 16th, 2008 @ 12:31 PM

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.