Simulating Fiddler requests in buffering mode in C#

Question

I am building a web scraping or crawler C# .NET application that keeps sending requests to a server to collect some information. The problem is that for certain web pages for this specific server that web response is always a 404 not found. However surprisingly enough I've discovered that as long as "Fiddler" is working the problem seems to vanish and the request returns with a successful response. I've been searching the web since seeking an answer but found none. On a brighter side, after searching the web and analysing Fiddler's timeline feature I have came to some conclusions.

1.Fiddler loads these web pages using Buffered mode while my application uses Stream mode. 2.It also appears that Fiddler reuses the connection or in other word Keep-Alive is set to be true.

And now the question is how can I mimic or simulate the way Fiddler loads the web response in Buffered mode, and whether Fiddler actually does some trick (i.e modifies the response) to get the correct response. I am using HttpWebRequest and HttpWebResponse to request my pages. I need a way to buffer httpwebresponse completely before returning data to client(which is my server).

public static String getCookie(String username, String password) { HttpWebRequest request = (HttpWebRequest)WebRequest.Create("certain link");

 request.UserAgent = "Mozilla/5.0 (Windows NT 6.0; rv:6.0.2) Gecko/20100101 Firefox/6.0.2"; request.Credentials = new NetworkCredential(username, password); HttpWebResponse wr = (HttpWebResponse)request.GetResponse(); String y = wr.Headers["Set-Cookie"].ToString(); return y.Replace("; path=/", ""); } /// <summary> /// Requests the html source of a given web page, using the request credentials given. /// </summary> /// <param name="username"></param> /// <param name="password"></param> /// <param name="webPageLink"></param> /// <returns></returns> public static String requestSource(String username,String password,String webPageLink){ String source = ""; HttpWebRequest request = (HttpWebRequest)WebRequest.Create(webPageLink); if (username != null && password != null) { request.Headers["Cookie"] = getCookie(username, password); request.UserAgent = "Mozilla/5.0 (Windows NT 6.0; rv:6.0.2) Gecko/20100101 Firefox/6.0.2"; request.Credentials = new NetworkCredential(username, password); } StreamReader sr; using (HttpWebResponse wr = (HttpWebResponse)request.GetResponse()) { sr = new StreamReader(wr.GetResponseStream()); source = sr.ReadToEnd(); } return source; }

FWIW, Buffering isn't what was causing the change in behavior; there's something else going on. FWIW, you really need to call .Close() on the object returned from GetResponseStream. That trips many people up. — EricLaw
– EricLaw, Commented Dec 11, 2012 at 23:20

Wiktor Zychla · Accepted Answer · 2011-09-25 20:32:13Z

0

Did you try to take a look at the HttpWebRequest's AllowWriteStreamBuffering property? Also you could try to append all the Fiddler's headers to your request to be as close to Fiddler as you can.

answered Sep 25, 2011 at 20:32

Wiktor Zychla

48.5k6 gold badges83 silver badges114 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

user963991 Over a year ago

Actually I've tried these two possible solutions but without any result.....what made me detect that the problem is that httpwebrequest is requesting in streaming mode while fiddler is off is that when fiddler is open by default the request succeeds(because fiddler uses buffering mode by default).....but when I switch on streaming mode option in fiddler while it is open the request fails exactly like it fails when fiddler is off!....

Wiktor Zychla Over a year ago

And did you try to connect on a lower level, using the TcpClient?

Wiktor Zychla Over a year ago

One of my old examples (somewhere at the bottom): netpl.blogspot.com/2007/05/klient-webservices.html

Wiktor Zychla Over a year ago

Be patient, experiment. Web requests are always done on TCP sockets so there's no way this doesn't work.

Widor · Accepted Answer · 2011-12-21 17:25:17Z

0

Could it be that your scraper is being detected and shut down and Fiddler slows it enough so it doesn't get detected? http://google-scraper.squabbel.com/

edited Dec 21, 2011 at 17:25

Widor

13.3k7 gold badges47 silver badges67 bronze badges

answered Sep 25, 2011 at 21:16

kenny

22.6k8 gold badges53 silver badges89 bronze badges

1 Comment

user963991 Over a year ago

Well other pages from the same host works when I request them.....and other don't!!

Collectives™ on Stack Overflow

Simulating Fiddler requests in buffering mode in C#

2 Answers 2

4 Comments

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

1 Comment

Related