Often getting execution expired
Reported by Matthew Williams | February 20th, 2009 @ 11:15 PM
I'm using Scrubyt to discover favicon and feed URL's from various websites. However, many sites I'm hitting seem to timeout with a trace like the one below. But then many other sites function just fine... Any thoughts?
/usr/local/lib/ruby/1.8/timeout.rb:60:in `rbuf_fill'
/usr/local/lib/ruby/1.8/timeout.rb:62:in `timeout'
/usr/local/lib/ruby/1.8/timeout.rb:93:in `timeout'
vendor/gems/right_http_connection-1.2.4/lib/net_fix.rb:51:in `rbuf_fill'
/usr/local/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
/usr/local/lib/ruby/1.8/net/protocol.rb:126:in `readline'
/usr/local/lib/ruby/1.8/net/http.rb:2020:in `read_status_line'
/usr/local/lib/ruby/1.8/net/http.rb:2009:in `read_new'
vendor/gems/right_http_connection-1.2.4/lib/net_fix.rb:146:in `request'
/usr/local/lib/ruby/gems/1.8/gems/mechanize-0.9.0/lib/www/mechanize.rb:467:in `fetch_page'
/usr/local/lib/ruby/gems/1.8/gems/mechanize-0.9.0/lib/www/mechanize.rb:227:in `get'
/usr/local/lib/ruby/gems/1.8/gems/scrubyt-0.4.06/lib/scrubyt/core/navigation/agents/mechanize.rb:52:in `fetch'
/usr/local/lib/ruby/gems/1.8/gems/scrubyt-0.4.06/lib/scrubyt/core/navigation/navigation_actions.rb:53:in `fetch'
lib/scrapers/feed/feed.rb:9:in `get'
lib/scrapers/feed/feed.rb:8:in `get'
app/models/service_models/feed.rb:37:in `discover'
app/models/service_models/feed.rb:31:in `discover'
app/models/service_models/feed.rb:30:in `discover'
app/controllers/service_subscriptions_controller.rb:702:in `create_feed'
app/controllers/service_subscriptions_controller.rb:109:in `send'
app/controllers/service_subscriptions_controller.rb:109:in `create'
Comments and changes to this ticket
-
Matthew Williams February 23rd, 2009 @ 05:08 PM
You can dismiss this ticket. My scraping was happening inside another HTTP call that had a short timeout period. I modified the timeout and I'm in business.
Please Sign in or create a free account to add a new ticket.
With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.
Create new ticket
Create your profile
Help contribute to this project by taking a few moments to create your personal profile. Create your profile ยป
A simple to learn and use, yet powerful web scraping toolkit written in Ruby.
People watching this ticket
- Nobody is watching this ticket.