We currently monitor our webapps using cURL. More and more of our webapps use the GWT framework, which uses tons of JavaScript, and we can't rely on our cURL system to monitor anymore. Therefore, we search the right tool to monitor, but it seems difficult to find a crawler which is light (no Selenium please) but handles JavaScript correctly.
P.S. : we host our webapps as well as the probes, we don't want any Internet monitoring service.
I would highly recommend investing time into PhantomJS or CasperJS.
They are browser simulations that interact with HTTP and DOM level JS. We use it at Top Hat to test our app and it's a very heavy JS client.
PhantomJS isn't for the feint of heart as getting it to work exactly like you want it to can be challenging however, it's worth the time.
Good luck, let me know how it goes @kentf.
Create a PHP page that downloads whatever is passed to $_GET["URL"] and saves it to temp.html. Then create an <iframe> with the source temp.html. Then use JavaScript to get the iframe's innerHTML. Pass that innerHTML to another PHP page that saves it as domain.html and have your monitor check out domain.html to get a look at the JavaScript compiled version of the site. You can just add a line to open the browser that starts this whole process and then check domain.html 30 seconds later.
