Lighthouse has a new layout. Prefer the old one? Return to the old layout, and switch back any time from the link at the top of each page.

Often getting execution expired

#19

I'm using Scrubyt to discover favicon and feed URL's from various websites. However, many sites I'm hitting seem to timeout with a trace like the one below. But then many other sites function just fine... Any thoughts?


/usr/local/lib/ruby/1.8/timeout.rb:60:in `rbuf_fill'
/usr/local/lib/ruby/1.8/timeout.rb:62:in `timeout'
/usr/local/lib/ruby/1.8/timeout.rb:93:in `timeout'
vendor/gems/right_http_connection-1.2.4/lib/net_fix.rb:51:in `rbuf_fill'
/usr/local/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
/usr/local/lib/ruby/1.8/net/protocol.rb:126:in `readline'
/usr/local/lib/ruby/1.8/net/http.rb:2020:in `read_status_line'
/usr/local/lib/ruby/1.8/net/http.rb:2009:in `read_new'
vendor/gems/right_http_connection-1.2.4/lib/net_fix.rb:146:in `request'
/usr/local/lib/ruby/gems/1.8/gems/mechanize-0.9.0/lib/www/mechanize.rb:467:in `fetch_page'
/usr/local/lib/ruby/gems/1.8/gems/mechanize-0.9.0/lib/www/mechanize.rb:227:in `get'
/usr/local/lib/ruby/gems/1.8/gems/scrubyt-0.4.06/lib/scrubyt/core/navigation/agents/mechanize.rb:52:in `fetch'
/usr/local/lib/ruby/gems/1.8/gems/scrubyt-0.4.06/lib/scrubyt/core/navigation/navigation_actions.rb:53:in `fetch'
lib/scrapers/feed/feed.rb:9:in `get'
lib/scrapers/feed/feed.rb:8:in `get'
app/models/service_models/feed.rb:37:in `discover'
app/models/service_models/feed.rb:31:in `discover'
app/models/service_models/feed.rb:30:in `discover'
app/controllers/service_subscriptions_controller.rb:702:in `create_feed'
app/controllers/service_subscriptions_controller.rb:109:in `send'
app/controllers/service_subscriptions_controller.rb:109:in `create'

Reported by Matthew Williams · February 20th, 2009 @ 11:15 PM

State: new
Milestone: none
Assigned to: nobody

Activity

  1. Matthew Williams
    Matthew Williams

    You can dismiss this ticket. My scraping was happening inside another HTTP call that had a short timeout period. I modified the timeout and I'm in business.

    February 23rd, 2009 @ 05:08 PM

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.