12

We have a multilingual website in which a bug was discovered some days ago. It was displaying other language data in other language and also the mixture of data like English language was selected but it was displaying other language data as well in the page and vice-versa. It is doing it infrequently but is present in the website. Going through the code also doesn't help because this is not always occurring.

Any suggestion in finding the issue in a timely manner? I am asking for strategies here.

Improve this question
1
  • 4
    start probing the code for situations which will allow this bug to happen ( instead of doing it the other way around ) Commented Apr 11, 2011 at 9:28

6 Answers 6

Reset to default
20

The first step is to try and characterize what can cause this type of problem. Since this is related to selecting the correct language for sections of the code, start by considering the following:

  • How is the language detected? Is it based on information from the HTTP request? Is it based on session information?, or is it based on database fields? In essence, can this be a problem related to how you app selects the language for each section?
  • How is the language displayed? Are you pulling from a properties file, or a database? Is it possible the reference to the correct language is getting lost some how? Is the mixed in language you see always the default for the site?
  • Is there a correlation to the client environment? This is related to the first bullet, but goes a bit further. I've had strange rendering problems due to downstream caching proxies. Typically those types of problems are a whole page that is stale or serving one person's page to other users (that was embarrassing).
  • Are you using a Thread Local value? If a request is handled my more than one thread, the thread local value will have different information based on the thread that is working at the time. In a web server environment, you can't assume that the thread that you started processing on will be the same thread you complete processing on--unless that is part of the spec for your platform. Server writers have found that if they reuse a small pool of threads and multiplex work to them in chunks, they can handle more requests simultaneously. Even if you have one thread from the start to the finish of a request, the server may be multiplexing other requests on to that thread at the same time. Instead of thread locals, consider binding that value to the request or session attributes.

Now, once you've characterized the possibilities of what can go wrong, it's time to make sure you have the data you need to try and find out what did go wrong.

  • Use profuse logging around the problem areas. This is a place where a tool like Log4J or Log4Net can really shine. That logging framework, and others like it, allows you to turn up the logging for certain categories while keeping down the noise for everything else--all by changing a configuration file. You want to introduce new logging statements to figure out if what you suspect could possibly be the problem. Also make sure your HTTP access logs have all the information you want about each request (cookies, http header parameters, etc.)
  • Attempt to simulate the problem. Since this happens sporadically, what is the load like on the server at the time it does occur? Are you getting hit with a number of simultaneous requests from a mix of languages? If so, attempt to simulate that kind of load in your test environment. A tool similar to JMeter might be what you need. You'll also want to be able to spoof IP addresses for your fake clients. Remember that IP addresses are portioned out so that you can figure out what country/region the IP is based on the first two segments of the address.
  • The problem will be just as sporadic in your test environment, but as you narrow down into your real cause you can skew the results to make it happen more often than it does in the wild. Additionally, you can more easily review the log files and try to learn from them.
  • It's an iterative process, so be patient. You have to induce the type of load you think will reproduce the bug, check the logs, and refine your tests based on what you find. The important thing is to identify the problem, so resist the urge to make some simple fixes that might only make the real problem happen less often.

Finally, once you've narrowed down the problem to the point where you know how to reproduce it, and what causes it, write the smallest automated test you can to force the issue in code. If you've narrowed the problem down to one class, or a pair of classes not working together correctly, reproduce it at that level. You shouldn't have to spawn 100 threads to do it, just do the smallest test that can cause the issue to happen 100% of the time.

Now you can fix it, and be reasonably confident that it won't come back to bite you again.

Improve this answer
10

The bug isn't irreproductable. You just haven't found out how to reproduce it yet.

No bug is random unless you are throwing an exception based on the return value of some Random() statement.

I know this may seem like semantics but it's reassuring mentally to tell this to yourself.

It's very hard and frustrating to find out how to repro a bug that only happens because of complex race conditions or such.

As for how to find it, I would turn on/add some logging to the application in places that could give you more information.

Next tell the people who are seeing the bug (wether they be Devs, QA, end users) to report as soon as they see it with the time it happened and then consult your logs. Ask them for other information as well as the bug may only happen due to the interaction of several different systems or because of a race condition

Hopefully you will be able to find a lead.

Improve this answer
7
  • even Random() calls are not truly random unless they are derived from a hardware white noise generator. They are psuedo-random, which means the numbers are mathematically distributed in as random an order as possible. But if you start from the same "seed" value, you will get the same answer every time. Commented Apr 11, 2011 at 12:35
  • 1
    @Berin: I know. Commented Apr 11, 2011 at 12:51
  • +1 for "you just haven't found out how to reproduce it yet." All bugs have a root cause or else they wouldn't happen. Commented May 4, 2011 at 21:11
  • 1
    It doesn't have to be off a Random(), things which are timing dependent, especially those things involving improper access to a shared resource can be very hard to reproduce. Commented Feb 13, 2012 at 18:30
  • 2
    @Gilles: Except they may not be deterministic on anything you can reasonably measure. (Say, exactly when some other task released it's time slice.) Commented Feb 14, 2012 at 0:22
5

You can try to find places in your code where you can recognise that the problem occurred (inconsistent parameters in a method for instance), add the checks to your code and let them add extra information to the debug log (like a stack trace, objects added to the session, etc.)

Doing this you with a bit of luck you can capture information about the occurances and deduce your way back to the problem.

Improve this answer
2

Automation should help, if it's the same steps to reproduce that sometimes fail, automate that and put it in a loop. Run in 50,000 times and it's very likely to occur.

Improve this answer
6
  • The event isn't random, it just seems random. Doing this may get it to appear, but will give you very little information about why it appeared. Commented Apr 11, 2011 at 14:01
  • 1
    @Josh - If he's not able to reproduce it, this may be a good way of doing that and obtaining a stack trace with debug symbols, for example. I figure that's a great first step - seeing it first-hand Commented Apr 11, 2011 at 14:09
  • You're assuming that there is a stack and that it is obtainable. He hasn't given us any technical information about the application or how accessible it is for debugging under this kind of load. This is not a debugging strategy, this is hitting it with a hammer trying to catch the exact moment it breaks. Commented Apr 11, 2011 at 14:25
  • @Josh - my real-world experience tells me the single most valuable thing in investigating/fixing a bug is seeing it first hand. Whether its something with timing you can see, a stack trace, something in the logs, or anything else. Where possible, having seemingly randomly-occurring issues tested in a loop has gotten me there very quickly indeed. If you have a different idea, post it as an answer for christ's sake - this is a valid method and a valid answer. Commented Apr 11, 2011 at 14:34
  • I disagree, and I believe Berin's answer is the correct way to go about solving this. Commented Apr 11, 2011 at 15:09
1

try to find patterns to pin down the conditions that cause this problem to manifest itself. That should point you towards the sections of your code that fail (or behave inconsistently).

Improve this answer
0
0

Can you detect when the problem is occurring? If so, can you reliably dump information about the state of the system at that point?

If the answer to both of these questions is yes, instrument your code to log as much information as it can when the error actually occurs, then wait.

This isn't a replacement for what others have suggested (you're still going to need to reason through how the code can get into the state you're seeing), but as long as you can't reproduce the bug at will, it's a good idea to not waste the occasions where it does appear.

Improve this answer

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.