Ive tried using the js load function but as the external site does not allow CORS requests, my original GET request gets blocked.
<div id="test"></div> <script> $(document).ready(function () { $("#test").load("https://mywebsite.com"); }); </script> So it seems that my only approach is to use iframes?! Is there a way to only crawl a specific div with iframes? I dont want to display the whole website.
EDIT: Since I am using Django I was able to crawl the website with python in a view and then push the crawled and cleaned up code snippet in the html template. Nevertheless to answer my question -> There is no correct way of doing it as long as the website you are trying to access is blocking the content.
load()does, it just filters the response before adding it to the DOM). If you absolutely have to do this, then you would be best off proxying the request by making an AJAX request to your server, which then makes a request to third party, receives the response, filters the HTML you need then returns it back to you.load(), which won't work here due to CORS, and the accepted answer suggest accessing the iframe content, which won't work here either as it's a cross domain origin so security policies in the browser will deny access.