7
$\begingroup$

I have Mathematica code that uses Import[] to download URLs. I noticed that first time I use the Import[] command it works. Next time it fails. I'm using Windows 7, Mathematica 8.0.4.

Here is an example. I start new sesssion of Mathematica, and then type

Import["http://groups.google.com/group/comp.soft-sys.math.mathematica/about"]

No problem. Now I issue the same command again on the same URL (or different one)

Import["http://groups.google.com/group/comp.soft-sys.math.mathematica/about"]

and now this gives me an error

FetchURL::conopen: The connection to URL http://groups.google.com/group/comp.soft-sys.math.mathematica/about cannot be opened. If the URL is correct, you might need to configure your firewall program, or you might need to set a proxy in the Internet connectivity tab of the Preferences dialog (or by calling SetInternetProxy). For HTTPS connections, you might need to inspect the authenticity of the server's SSL certificate and choose to accept it.

On the same Windows 7 PC, I start MATLAB, and type

data=urlread('http://groups.google.com/group/comp.soft-sys.math.mathematica/about') data=urlread('http://groups.google.com/group/comp.soft-sys.math.mathematica/about')

and both work. I get the page fine the second time around.

I open the browser, and type the URL, and I can see the page. This is the same PC. No proxy. home PC. Nothing special.

P.S. This code used to work fine with Mathematica on an earlier version (forgot which one now), becuase I used Mathematica to generate this web page and all the Import commands I used are listed above and you can see them. I have list of many URL's that I loop over calling Import[] on them one by one.

Again, the problem is that I now get an error on a URL using Import[] and the problem seems to show up the second time I use the Import[]. Any suggestions how to fix this? Is this a known issue?

update I still have Mathematica 7 and 6 installed. So I tried this on them. And I get the same problem there. This is strange, since this code worked before. I think now it is a google groups/server problem, becuase when I try different domain (i.e. not google groups), I do not get this problem

data = Import["http://www.yahoo.com", "Data"]; data = Import["http://www.yahoo.com", "Data"];

No error in the second time.

I do not understand why this problem (i.e. using the Import command again) gives an error on some URLs and not on others.

update

Thanks for everyone's help and for WReach solution. I have used WReach nice solution to bypass the problem and have just updated the currrent Mathematica code I was using to update the usenet programming languages trends charts. The updated charts are on this page if you are interested in seeing the result.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Perhaps it's Google that is clamping automated calls to its services. I know the StackExchange API calls can be throttled too if you use them too often. Why it works with MATLAB? Who knows, perhaps just luck, or MATLAB might mimic its calls as an actual browser by setting the user agent to mozilla or so? Or perhaps it just caches the call so that Google never sees it a second time? $\endgroup$ Commented Oct 30, 2012 at 12:13

2 Answers 2

Reset to default
11
$\begingroup$

Watching the network traffic with WireShark reveals that the second and subsequent accesses to the URL fail due to an internal server error.

The initial request looks like this:

 GET /group/comp.soft-sys.math.mathematica/about HTTP/1.1 User-agent: Mathematica/8.0.1.0.0 PM/1.3.1 Host: groups.google.com

and yields a successful response with the following headers:

 HTTP/1.1 200 OK Pragma: no-cache Expires: Fri, 01 Jan 1990 00:00:00 GMT Cache-Control: no-cache, must-revalidate Set-Cookie: NID=65=tsm2...ijos;Domain=.google.com;Path=/;Expires=Sun, 05-May-2013 17:41:45 GMT;HttpOnly P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info." Content-Type: text/html; charset=ISO-8859-1 Set-Cookie: PREF=ID=59d3...eBf7; expires=Mon, 03-Nov-2014 17:41:45 GMT; path=/; domain=.google.com X-Content-Type-Options: nosniff Date: Sat, 03 Nov 2012 17:41:45 GMT Server: GWS-GRFE/0.50 X-XSS-Protection: 1; mode=block X-Frame-Options: SAMEORIGIN Transfer-Encoding: chunked

The second and subsequent requests include the returned cookies:

 GET /group/comp.soft-sys.math.mathematica/about HTTP/1.1 User-agent: Mathematica/8.0.1.0.0 PM/1.3.1 Host: groups.google.com Cookie: $Version=0; NID=65=tsm2...ijos; $Path=/; $Domain=.google.com Cookie: $Version=0; PREF=ID=59d3...eBf7; $Path=/; $Domain=.google.com

This elicits a response that indicates a server error:

 HTTP/1.1 500 Internal Server Error Pragma: no-cache Expires: Fri, 01 Jan 1990 00:00:00 GMT Cache-Control: no-cache, must-revalidate Content-Type: text/html; charset=UTF-8 X-Content-Type-Options: nosniff Date: Sat, 03 Nov 2012 17:39:13 GMT Server: GWS-GRFE/0.50 Content-Length: 1100 X-XSS-Protection: 1; mode=block X-Frame-Options: SAMEORIGIN

The body of the response is a generic "server error" message that offers no additional clues:

The server encountered an error and could not complete your request.

If the problem persists, please report your problem and mention this error message and the query that caused it. That...s all we know.

After the Mathematica kernel has been restarted, the Import command once again generates an HTTP request that has no cookies -- which succeeds. So it would appear that the presence of the cookies in the request confuses the server. It is possible that this is an intentional behaviour intended to discourage site access by robots. Perhaps an intermediate cache is struggling with the no-cache, must-revalidate header. Or maybe it is just a programming error in the server.

There is no way to control request cookies through Import or FetchURL. Restarting the kernel is a pretty severe way to work around this problem. A slightly less severe action would be to RestartPacletManager[], but that is still pretty severe.

Workaround

We could just bypass Import altogether and go directly to JLink:

Needs["JLink`"] httpGet[url_String] := JavaBlock@Module[{http, get} , http = JavaNew["org.apache.commons.httpclient.HttpClient"] ; get = JavaNew["org.apache.commons.httpclient.methods.GetMethod", url] ; http@executeMethod[get] ; get@getResponseBodyAsString[] ]

Unlike the Paclet Manager, this code uses a fresh HTTPClient each time, so cookies are never retained and this request can be issued successfully any number of times.

httpGet["http://groups.google.com/group/comp.soft-sys.math.mathematica/about"] (* <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" ... <title>comp.soft-sys.math.mathematica | Google Groups</title> ... *) httpGet["http://groups.google.com/group/comp.soft-sys.math.mathematica/about"] (* <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" .... *)

ImportString can be applied to the result of httpGet to interpret the results:

ImportString[ httpGet["http://groups.google.com/group/comp.soft-sys.math.mathematica/about"] , {"HTML", "Data"} ] (* {{{Search,Images,Maps,Play,YouTube,News,Gmail,Documents, {Calendar,Translate,Mobile,Books,Offers,Wallet,Shopping,Blogger,Reader,Finance,...}}, {Sign in,Help}}, ... *) ImportString[ httpGet["http://groups.google.com/group/comp.soft-sys.math.mathematica/about"] , {"HTML", "XMLObject"} ] (* XMLObject[Document][...] *)
Improve this answer
$\endgroup$
0
4
$\begingroup$

I actually tried FetchURL[url] and gives same error. Unfortunately FetchURL is not documented. I tried some of the Options[FetchURL], like 

FetchURL["http://groups.google.com", "RequestHeaderFields" -> {"Cache-control" -> "no-cache", "Connection" -> "close"}, "RequestParameters" -> {"x" -> ToString[AbsoluteTime[]]}, "FileFilters" -> {}]

but couldn't trick the kernel into something else. My (wild) guess is that it is related to the response being cached/blocked by the kernel -including resources!- along with the fact that the above mentioned page contains a broken resource https://groups.google.com/intl/en/images/logos/groups_logo_sm.gif returning 502 (which, by the way, produces the same error on its own)

Improve this answer
$\endgroup$

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.