1

I'm relatively new to Ruby and Mechanize but i'm having some difficulty with an asp website

URL: http://www.adecco.co.uk/careercentre/job-search-results.aspx?kws=&pstc=&cty=&prvnm=&pdx=1 agent = Mechanize.new page = agent.get('http://www.adecco.co.uk/careercentre/job-search-results.aspx?kws=&pstc=&cty=&prvnm=&pdx=1') puts page.body

I was working through the examples from the Mechanize website the thing is the HTML I get back via Mechanize is very different from what I get using view source in my browser, I need the finished HTML?

Update

I'm not quite sure what to do here since the problem is actually with the fact that the page renders the content using jquery afterwards so i've ended up using Selenium to grab the page with the correct html, neither of these answers are actually wrong so i've upvoted both, but neither actually solve the problem?

Thanks

Marc

Improve this question

2 Answers 2

Reset to default
3

Try the following code,

require 'mechanize' require 'nokogiri' agent = Mechanize.new page = agent.get('http://www.adecco.co.uk/careercentre/job-search-results.aspx?kws=&pstc=&cty=&prvnm=&pdx=1') document = Nokogiri::HTML(page.content) puts document
Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

You don't need to include nokogiri separately or do anything with Nokogiri::HTML. Mechanize does that for you.
2

I think this is because the website handle different user agents differently, you can set the user agent same as your browser as follows

a = Mechanize.new a.user_agent_alias = 'Mac Safari'

you can any value of these

AGENT_ALIASES = { 'Windows IE 6' => 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)', 'Windows IE 7' => 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)', 'Windows Mozilla' => 'Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4b) Gecko/20030516 Mozilla Firebird/0.6', 'Mac Safari' => 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_2; de-at) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10', 'Mac FireFox' => 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6', 'Mac Mozilla' => 'Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.4a) Gecko/20030401', 'Linux Mozilla' => 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) Gecko/20030624', 'Linux Firefox' => 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.1) Gecko/20100122 firefox/3.6.1', 'Linux Konqueror' => 'Mozilla/5.0 (compatible; Konqueror/3; Linux)', 'iPhone' => 'Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1C28 Safari/419.3', 'Mechanize' => "WWW-Mechanize/#{VERSION} (http://rubyforge.org/projects/mechanize/)" }

the above list exists here https://github.com/sparklemotion/mechanize/blob/master/lib/mechanize.rb#L115

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.