How to scrape .aspx site

Question

I'm trying to scrape: http://search.bgc-group.com/Default.aspx

When I'm running my code, I'm always getting the first page.

My code:

var casper = require('casper').create({ verbose: true, logLevel: "debug", type: 'xpath', }); casper.start('http://search.bgc-group.com/Default.aspx'); casper.thenOpen('http://search.bgc-group.com/Default.aspx', { method: 'post', data: { '__EVENTARGUMENT': 'Page$4', '__EVENTTARGET': 'ctl00$body$ctlJobListing1$gvJobListing', '__EVENTVALIDATION': '', '__PREVIOUSPAGE': '', '__VIEWSTATE': '' } }); casper.then(function() { this.echo('POSTED it.'); this.capture("test.png") }); casper.run();

I've not left the __EVENTVALIDATION, __PREVIOUSPAGE and __VIEWSTATE fields blank in the actual code. Just skipped here as they were very long string.

Where is the post data coming from? Why not use CasperJS for what it is intended for by filling that form and submitting it? — Artjom B.
– Artjom B., Commented May 5, 2014 at 8:41

Artjom B. · Accepted Answer · 2014-05-05 08:57:12Z

You can do the following instead of the casper.thenOpen block:

var x = require('casper').selectXPath; casper.thenClick("a[href*='Page$4']"); casper.waitForSelector(x("//td/span[text()='4']")); // might be unnecessary

Then you need specify it in your question, but it was already discussed at length see here or here

Collectives™ on Stack Overflow

How to scrape .aspx site

1 Answer 1

1 Comment

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Linked

Related