1

I am trying to help my uncle out who is in a bit of a jam. An application he had a consultant come in on has stopped working and the consultant has moved on from the company and no one can fix the issue. He's asked me to try and help by isolating where it is going wrong.

I figure writing an integration test would be a good start as we would be programmatically testing on their 'test' instance. Here's the setup:

They're pulling in streams of data every 10 minutes and sending these orders to an AMQ instance they have running. (we'll call this module 'Streamer'). On the other side of the AMQ instance we have the 'Puller' that takes those messages off AMQ and processes them into a database. Somewhere along the line the messages are being manipulated or lost. He's asked me to find out where.

Does anyone know of a simple way in which I can test the entire system and yet have the ability to verify all the information as it's being sent around or updated? (I would like to do this with live data, rather than mocked up data).

Any and all help is appreciated.

Improve this question

2 Answers 2

Reset to default
1

You may need the combination of logging and a post-processing program that reads the logs and checks their consistency.

Streamer sends a message. That should generate a log record describing the message, with a time stamp.

There should be another log record showing that Puller received the message.

A log processing program could produce a histogram of how long it takes for Puller to receive what Streamer sent, and detailed accounting for any messages that Streamer sent and Puller did not receive, or that Puller thought it had received but that had not been sent. You can also extract statistics about any long gaps in activity.

The objective of your logging should be to make sure that the raw data gets into the logs. The objective of the log processing should be to tell you what you need to know about the system's activity, at the right level of detail.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

That would be awesome if this was a performance problem that needed visualizing. Just to get it working again can't he just open the logs and read them?
It depends on the volume of data. If one million messages are getting through successfully, but one million and ten messages were sent, finding the discrepancy by "just open the logs and read them" would take much longer than writing a suitable script and running it.
If you have each file log one line per message, that discrepancy is as easy to spot as looking at the line counts. Why don't we let Dan clear up what the real problem is?
0

That's not strictly speaking an integration test. An integration test is testing that new code plays well with other code before you let it go live.

What you need is logging. Each thing that handles these messages needs to have it's logging turned on, at least until you're sure it's working.

I've no idea if your consultant added logging code to this thing but if s/he didn't I suggest adding some.

Keep in mind that logging can be very granular. You can focus on one method and even one variable. Start about halfway between where you know it's working and where you know it's broken and find out if it's working there. Repeat until you know exactly where it stopped working. It's called a binary search. It can be a very powerful troubleshooting technique.

Another technique is to send a heartbeat message from your streamer to the puller. You could send a very simple message quite often so you wouldn't need to wait 10 minutes for activity. Just need to be sure the heartbeat message doesn't get confused with real data.

Improve this answer

2 Comments

I need to make sure the 'Streamer' and 'Puller' are playing well with one another. As I make changes to the code, I need to ensure that they keep sending and receiving messages. This is why I would like to do this as an integration test. I have logging in place currently, but it is still not telling me enough.
You need to define "enough" then because the fact that they are sending and receiving messages is exactly what I'd have you log.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.