Suggestion
Reported by crisb | November 5th, 2008 @ 07:38 PM | in 0.4.1
Dear Peter,
you asked the users to submit suggestions for examples of the new scrubyt release. So here is mine: I was trying to scrape the search suggestions on the german yahoo site (unfortunately there is no api like for the us version). Since I didn't got it working it would be cool to see a solution. Pretty selfish I know :-)
Comments and changes to this ticket
-
Peter Szinek November 24th, 2008 @ 09:44 AM
- State changed from “new” to “open”
- Assigned user set to “Peter Szinek”
OK - let me know where the site is located and I'll look into it!
-
crisb November 24th, 2008 @ 09:51 AM
The site is located here: http://de.yahoo.com/
Actually it works the same as http://yahoo.com/, but since the german site doesn't offer an api, it would be really cool if you could take a look at it.
-
-
-
antel November 24th, 2008 @ 03:37 PM
Scrubyt doesn't support javascript, the drop down menu with urls start only by an javascript input, it ask the server and return html, test it with firebug and webdeveloper inspecting objects with javascript enabled/disabled, at now it will work only with firewatir.
-
crisb November 24th, 2008 @ 03:51 PM
I know, that's why I didn't got it working. But I thought that Firewatir is included in the new scrubyt and therefore it would be able to scrape the suggestions.
Or am I wrong? :-)
-
-
Peter Szinek November 24th, 2008 @ 05:28 PM
Yeah, it should (at least that's the idea :-)) however, currently it's throwing
textfield name and //input["@id=p"] is read only. (Watir::Exception::ObjectReadOnlyException)WTF?! how can it be read only??!? Need to dig around a bit more and/or ask the Watir guys.
-
Peter Szinek November 25th, 2008 @ 12:12 AM
Ok, I found the problem... will need to push out one more release ;-)
-
-
Peter Szinek November 29th, 2008 @ 08:29 PM
- Milestone set to “0.4.1”
-
Peter Szinek December 1st, 2008 @ 01:07 PM
OK, done - now I just need to find some time to crank the release out - should happen today, or in the worst case tomorrow!
-
-
-
Peter Szinek December 6th, 2008 @ 09:14 PM
- State changed from “open” to “resolved”
Done, phew was not easy...
(maybe to wait 5 seconds is too much - try it out and play around with it).
Let me know if it doesn't work (well, for some time it won't - the gem needs to be mirrored @ rubyforge)
-
crisb December 10th, 2008 @ 03:11 PM
cool, thanks.
but could it be possible that the gem still isn't mirrored at rubyforge?
-
Peter Szinek December 10th, 2008 @ 03:18 PM
hmmm.... wtf it seems it's not yet there. it should be 0.4.1 but for me it installs just 0.4.06... checking it out.
-
crisb December 11th, 2008 @ 10:44 AM
i now installed it manually and tried to get it working with de.yahoo.com. but i don't even get the script to fill out the textfield.
yahoo_data = Scrubyt::Extractor.define :agent => :firefox do fetch 'http://de.yahoo.com/' fill_textfield_and_wait 'p', 'ruby', 5 end
-
Peter Szinek December 11th, 2008 @ 11:00 AM
Running out for lunch now, but will check it out after I am back!
-
-
crisb December 16th, 2008 @ 12:31 PM
I checked it with yahoo.com He doesn't seem to fill in the textfield their too.
Please Sign in or create a free account to add a new ticket.
With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.
Create new ticket
Create your profile
Help contribute to this project by taking a few moments to create your personal profile. Create your profile ยป
A simple to learn and use, yet powerful web scraping toolkit written in Ruby.
antel
crisb
