How to scrape data from a table with dynamic values?

Question

This may be challenging since the builders of the website seem to trying to block this.

I am trying to build a simple app to view some data from a website table. The table is located here:

http://www.cepteteb.com.tr/doviz-kurlari

However, the table data seems to load after page is loaded, so when I try to get the HTML of table, it comes empty. How do I get the table with data?

I am using JSOUP to scrape the table.

private class GetData extends AsyncTask<String, Void, Element> { @Override protected Element doInBackground(String... params) { try { Document document = Jsoup.connect(params[0]).get(); Log.e("Yiit",document+""); Element table = document.getElementById("dovizTablo"); return table; } catch (Exception e) { e.printStackTrace(); } return null; } @Override protected void onPostExecute(Element element) { super.onPostExecute(element); Log.e("Yiit",element+""); tvMain.setText(element+""); } }

Result:

<table class="prices prices2" id="dovizTablo"> <thead> <tr> <th>D&ouml;viz Adı</th> <th>CEPTETEB Alış</th> <th>CEPTETEB Satış</th> </tr> </thead> <tbody> </tbody> </table>

Expected Behaviour:

<table class="prices prices2" id="dovizTablo"> <thead> <tr> <th>Döviz Adı</th> <th>CEPTETEB Alış</th> <th>CEPTETEB Satış</th> </tr> </thead> <tbody> <tr><td>USD</td><td>2.9096 TL</td><td>2.9908 TL</td></tr><tr><td>EUR</td><td>3.1555 TL</td><td>3.2435 TL</td></tr><tr><td>GBP</td><td>4.0558 TL</td><td>4.1688 TL</td></tr></tbody> </table>

try using Xpath -- stackoverflow.com/questions/7085539/does-jsoup-support-xpath -- eg (//*[@id="dovizTablo"]/tbody/tr[1]/td[1]) -- how to get Xpath -- youtube.com/watch?v=vCNLPHP3E_U — Tasos
– Tasos, Commented Mar 1, 2016 at 23:55
So if I understand correctly, xpath is used to easily access the required element. However, I have no problems navigating to table, it just comes empty. So xpath probably wont be helpful anyway. — yigitserin
– yigitserin, Commented Mar 2, 2016 at 0:01
you probably right as it sounds like a timing issue. Xpath gets whats inside an element. Have you tried putting a 2 - 3 second delay before you scrape? — Tasos
– Tasos, Commented Mar 2, 2016 at 0:05

Community · Accepted Answer · 2017-05-23 10:33:56Z

0

This site you provided initializes the data with javascript. You can't scrape it with Jsoup.

I can think 2 ways to scrape this page

Use a WebView to visit the page and the run some js to parse what you want and return in to main app. Read this how to achieve it.
Create a web service that can parse and return the data from this site.