#19 new
Matthew Williams

Often getting execution expired

Reported by Matthew Williams | February 20th, 2009 @ 11:15 PM

I'm using Scrubyt to discover favicon and feed URL's from various websites. However, many sites I'm hitting seem to timeout with a trace like the one below. But then many other sites function just fine... Any thoughts?

/usr/local/lib/ruby/1.8/timeout.rb:60:in `rbuf_fill'
/usr/local/lib/ruby/1.8/timeout.rb:62:in `timeout'
/usr/local/lib/ruby/1.8/timeout.rb:93:in `timeout'
vendor/gems/right_http_connection-1.2.4/lib/net_fix.rb:51:in `rbuf_fill'
/usr/local/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
/usr/local/lib/ruby/1.8/net/protocol.rb:126:in `readline'
/usr/local/lib/ruby/1.8/net/http.rb:2020:in `read_status_line'
/usr/local/lib/ruby/1.8/net/http.rb:2009:in `read_new'
vendor/gems/right_http_connection-1.2.4/lib/net_fix.rb:146:in `request'
/usr/local/lib/ruby/gems/1.8/gems/mechanize-0.9.0/lib/www/mechanize.rb:467:in `fetch_page'
/usr/local/lib/ruby/gems/1.8/gems/mechanize-0.9.0/lib/www/mechanize.rb:227:in `get'
/usr/local/lib/ruby/gems/1.8/gems/scrubyt-0.4.06/lib/scrubyt/core/navigation/agents/mechanize.rb:52:in `fetch'
/usr/local/lib/ruby/gems/1.8/gems/scrubyt-0.4.06/lib/scrubyt/core/navigation/navigation_actions.rb:53:in `fetch'
lib/scrapers/feed/feed.rb:9:in `get'
lib/scrapers/feed/feed.rb:8:in `get'
app/models/service_models/feed.rb:37:in `discover'
app/models/service_models/feed.rb:31:in `discover'
app/models/service_models/feed.rb:30:in `discover'
app/controllers/service_subscriptions_controller.rb:702:in `create_feed'
app/controllers/service_subscriptions_controller.rb:109:in `send'
app/controllers/service_subscriptions_controller.rb:109:in `create'

Comments and changes to this ticket

  • Matthew Williams

    Matthew Williams February 23rd, 2009 @ 05:08 PM

    You can dismiss this ticket. My scraping was happening inside another HTTP call that had a short timeout period. I modified the timeout and I'm in business.

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.

New-ticket Create new ticket

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile ยป

A simple to learn and use, yet powerful web scraping toolkit written in Ruby.

People watching this ticket