7

I've reported to a third party what I consider a very serious performance bug in a widely used product. Older versions worked fine. Newer versions are practically unusable except for what I consider trivial situations. I have provided test cases showing older version performance versus newer version performance.

After waiting a few weeks with no response, I was finally told this:

We appreciate you taking the time to report this problem. We are currently prioritizing problems that are impacting a broad set of our customers, so we may not be able to investigate this one immediately. We know this problem is important to you, so we will continue to monitor it.

Due to some recent upgrades to our development environment, it's very difficult to continue using the older version of this library. We use this library in many places, so completely ripping it out and replacing it would be hard to do. Plus, there aren't many options for another product.

My concerns are as follows:

  1. Am I understanding how to use the new library correctly? Is my reproducible test case actually correct, or am I missing something?

  2. What (if anything) can I do to convince this vendor to put more consideration into the problem?

  3. If I can't get a resolution, what are my options? Revert to (really old) previous versions of the product? Replace this library with another product and end up rewriting a ton of battle hardened code?

So my thought on #1 is to get code reviews of my test case. I'm currently the senior software engineer at work. I supervise two other engineers, but only one is really qualified to review. I believe he has ran my test case and has not reported any issues with it. We've both been trying to understand the causes of the performance issues, though we have came to different conclusions. I thought about posting the test harness on here asking for a code review, but I'm not sure if that's the appropriate thing to do.

I should also note that I have two other bug reports with this vendor. Also with reproducible test cases. They also seem to be getting the same lackadaisical treatment, though this was the only one where they explicitly said it wasn't a priority.

So I'm just looking for guidance. I don't really see any great option as it is.

It was requested to make this more specific, so I will attempt to describe the problem itself...

Until recently, we had been using Visual Studio 2008 and ReportViewer 2008. Upgrading our environment has been planned for awhile. As a result of an unrelated (and as yet unfixed) bug in ReportViewer, we decided to go ahead with our upgrade, hoping that would fix the bug. It did not.

We kept ReportViewer 2008. Everything there works fine. However, newer versions of Visual Studio won't allow you to design reports for 2008. You either have to upgrade or you have to develop your reports in raw xml.

So the obvious choice was to just upgrade ReportViewer, all of our RDLC files, and everything should work. However, what I found was even the simplest of reports took twice as long to run. Other relatively short reports took 10x as long. Larger reports (though still what I consider reasonable) went from 30 seconds in ReportViewer 2008 to "never finishes" in every newer version I tried. I tried with 2012, 2015, and 2017, though I suspect 2010 has the same issue. I suppose it's more than just a performance bug, since a large enough report never finishes. And it also doesn't seem to give any kind of errors.

Other people have had similar issues with it taking 10x as long to run their reports. Suggestions include changing CAS security settings or changing dynamic grouping to static. I could not find any CAS settings that fixed the issue. As for grouping, my test case was using RowNumber as a group. I'm not sure whether that is considered dynamic since it takes a group name. As for the actual data in my report, everything else was static.

As for determining the problem, I have ran a profiler and determined newer ReportViewer versions do a massive amount of binary serialization. My other programmer seems to think vb expressions are causing an issue, though he hasn't explained why he thinks this. Not to say they aren't, but I think we need to know why.

My test harness was developed to run every combination of ReportViewer 2008, 2012, 2015, and 2017 against every .NET version of 3.5, 4.0, 4.5, 4.5.1, 4.5.2, 4.6 and 4.6.1. ReportViewer 2008 used a 2008 version of the RDLC. Everything else used the newer version.

The test harness showed the problem was connected to ReportViewer version, not .NET version. However, my other programmer keeps thinking it's .NET that is the issue.

As I see it, the options to weigh are:

  1. Hope the vendor eventually looks at this, verifies there is an issue, and fixes it. Downside is in the mean time we have to either edit our reports directly in XML or work with them in VS2008, then pull them over to our current VS version (which is 2017)

  2. Switch from ReportViewer to something else. The obvious one is Crystal. I have no experience with that product. This would require a massive amount of rewriting in our app, which I'm not fond of.

  3. Revert back to VS2008/ReportViewer 2008. I'm ok with this, though that means converting some of our more recent code back to working with .NET 3.5. What bothers me about this route is future support from MS. Will ReportViewer 2008 continue to work on newer versions of Windows? Will .NET 3.5 continue to work?

First and foremost on my mind is making sure I'm not making a mistake. Maybe there's something I don't understand about newer ReportViewer versions. Maybe my understanding of the workarounds I've found is incorrect. Getting feedback from either my bug report to Microsoft or on the MSDN forums has been fruitless. I feel like I must be missing something, as this is such a widely used product that this should have been addressed a long time ago.

For those who are curious:

MSDN Thread

Bug Report

Improve this question
6
  • As for #2: could you pay the vendor for your specific needs? This would easily drive answer to #3: compare money needed to have a guaranteed reply from this vendor or the money needed to switch to something else and rewrite what needs to be rewritten. Also, try to find other users of the same library and see if they have the same problem. Commented Dec 18, 2017 at 20:40
  • I believe the Vendor has (had?) some kind of extra support plan. Vendor in this case is Microsoft. It's a bit confusing what they offer now for support. They seem to have two different sites to report bugs and it's a toss-up which site gives better response. Commented Dec 18, 2017 at 20:49
  • The test cases I coded to reproduce the issue have been posted to places where other users could run them and see the issues. I did the same thing with one of the other bugs and someone was nice enough to run my tests and verify there was a problem. So far, nothing with this one. Commented Dec 18, 2017 at 20:50
  • I should clarify I want to make sure I've generated an appropriate reproducible test case. I think I have. My other programmer has not pointed out any issues he sees. The vendor response suggests to me that nobody has even looked at it. I've done research on the issue and have tried every workaround I've found without success. According to my testing, every version since at least 2012 has an issue. However, I think this is a relatively popular product, so it's hard for me to believe problems haven't been reported yet. Commented Dec 18, 2017 at 21:30
  • My IT department had trouble with encoding of accented characters in PDF with report viewer. MS frontline support wanted an upgrade to even consider the bug, and I finally convinced IT to take it to a forum. More and more people piled on and the bug got fixed. It took over a year. The fact that you have got the word out will increase the chance that if the bug is affecting many, they'll find where to +1 with their voice. Commented Dec 19, 2017 at 0:48

2 Answers 2

Reset to default
1

IMO, you shouldn't have upgraded your development framework if the newer versions drastically impact your project's performance.

In the first place, you should have tested newer versions of the framework before start implementing new code in this new development environment. In terms of process, I think this would be the correct way to go.

But, since you already upgraded and already have code written using the new framework, I suggest one of the following options (if necessary, you could provide more details/effort needed for each option so we could help evaluate which one is more feasible):

  • For now, downgrade only the Reporting feature into a separate project, and deploy it separately, in order to clients be able so use the feature with good performance;
  • Downgrade entirely to your previously working development environment, rewriting new code to work with older versions of .NET framework;
  • Try to use some conversion tool to convert your reports to another reporting engine;
  • Write a conversion tool to convert your reports to something else (in another engine);
  • Investigate harder to check why your reports are lacking performance in this new version;
  • Create a fallback report, so your clients still have something to use during your main Reporting feature is under maintenance;

You could combine options, e.g. use the first option while you plan to migrate your reports to another engine.

Improve this answer
4
  • Can't you rollback the reports to be back in 2008 version, and then you use VS2008 just to edit the .rdlc files? This is a workaround, but I'm suggesting this because the lack of support from the product vendor, and efforts needed to migrate to something else. Then, rolling back you are able to edit your report files in another IDE, you get rid of performance issues and remain only with your original bug, about which you will have more time to investigate. Is anything here in my answer helpful for your problem? Commented Dec 19, 2017 at 17:15
  • The answer is helpful, but my preference is still towards having the vendor address the issue, even if it is just a verification that there is or isn't a problem. We never have released them as anything but 2008 version, so it's not a rolling back issue. And you're right, VS2008. That's a path I was considering. But then you also have to have your datasources available. Some of my datasources are business objects. If those business objects use later language features I may have issues. Though I suppose I could either comment out that code or perhaps not even worry at all about compile issues. Commented Dec 19, 2017 at 18:50
  • In this case you would need to rewrite part of the business code in order to work with older 2008 version. Unfortunately I'm out of better ideas here. Other things you might wanna verify: is the report itself buggy, or the integration between report and DB? You might want try setting a hardcoded set of data and evaluating how the report in newer versions behave. Commented Dec 19, 2017 at 18:58
  • The test harness I created and sent to MS was hard coded data. It allowed for some variables that basically end up just changing how many pages of this hard-coded data get generated. The report file was a copy of a production report that has had no issues with ReportViewer 2008. I picked this report because it was not very complicated. If it has performance issues, most or all of our reports will. Commented Dec 19, 2017 at 21:54
11

This is one of the drawbacks of using third-party libraries: as you get more and more dependent of them, you're at the mercy of the company which develops those libraries. Not only they can set the priorities as they want, but they can even decide to stop maintaining this library, or simply go bankrupt, leaving you with no much choices.

The choices made by another company are their choices. Some factors, such as being a very important customer or suggesting to pay a lot for them to work on what is important to you may help the negotiation. However, this is too specific to a given case, and is outside the scope of this site.

Back to technical aspects, there is a bunch of things you can work on to mitigate those risks:

Interfaces

The most important aspect consists of putting an interface between third-party libraries and your business logic. This would help switching to a different library if a competitor happens to develop a better one (or to your own implementation if you're able to develop your own).

Sometimes, this is extremely straightforward. Usually, small libraries which are used to perform a very specific and narrow task are just like that: abstracting them is very easy, and one can swap a library for another with nearly zero effort. Example: a library which generates QR code images from URIs.

In other cases, it is extremely complex. Frameworks or libraries on which practically all your code relies tend to become very interleaved with your code. Even if abstractions are used, it may not be that easy to swap one library/framework for another. Example: SharePoint.

Not only is it important to reduce the coupling between your code and third-parties, but it is also essential to be able to measure how difficult is it to swap a library if you're required to do it in the future. Separating business code and interface code into several packages gives you that hint, by showing how much code belongs to the business logic, and how much provides the interface with the third-party library. The more code you get in the interface package, the more you're in trouble.

Automated deployment

A specific problem where a new version of a library breaks what worked before can be avoided by following a bunch of practices related to the deployment of software.

In badly maintained environments, one of the major risks is that by upgrading a library to a new version by hand, one could find himself in a situation where it is hard to impossible to go back to a previous version. Problems could range from the downtime during the downgrade operation, to situations such as when the retailer doesn't distribute the old version of the library any longer, and you never kept any backup of the binaries. Or it may simply be impossible to reproduce all the steps which were performed on the server to install the old version in the first place, because nobody noted those steps.

In a well-maintained environment, with automated deployment, proper versioning of dependencies and an artifacts repository, switching from the newly deployed version to the older one could be as simple as a click on “Deploy” on one of the previous revisions of your app, the one which used the older version of the third-party library.

Migration

Finally, an important aspect is that before you migrate to another version of the library (or from a library to the competitor's one), you need to ensure that it (1) works correctly and (2) fits your needs.

When it comes to checking that it works correctly, you have to play with the library to see how it behaves, but also run the tests, which includes stress and load tests. I'm not talking here about the tests of your product (since you would be able to run them only after you migrate), but the tests written with the sole purpose of seeing how the third-party library performs the major scenarios you'll use.

Example 1: when migrating to a different library which generates QR codes, one can write a test which generates ten thousand images to see how if the new library outperforms the old one (or at least is not much slower).

Example 2: when migrating a site to a new version of SharePoint, if the major usage of the site is to store Word documents, one could write a test which would compare how two versions of SharePoint deal, in terms of performance and disk space, with this specific storage (for instance trying to study how the introduction of chunks for Office documents helps saving disk space, and what are its consequences on the CPU usage and overall performance).

Reading your question, it seems that having a few stress tests may have prevented you from upgrading all your assets to the new version of the library, and only then finding the performance issues.

Improve this answer
16
  • Good suggestions, though I think it would be hard to apply here. If it helps, the product is Microsoft ReportViewer. I don't think that's an easy switch. Commented Dec 18, 2017 at 20:47
  • It also depends of your company. If you're a major multinational corporation, you may have more leverage than if you're a small company. Commented Dec 18, 2017 at 20:49
  • We're very small, so likely not much leverage. Commented Dec 18, 2017 at 20:50
  • I'm totally agreed here (wanna upvote but I have to wait 2 hours, I'll do it later!). The biggest thing that got me was when you said you depend on the library in so many places. In my opinion a third party lib should never be used directly in more than an isolated area of your codebase, or else it's generally going to save so much more time and grief upfront to wrap it and provide your own interface sandwiched between your codebase and the library. This way you have that central control... Commented Dec 18, 2017 at 21:52
  • 1
    This is a pretty good answer. In a nutshell, there is a cost to dependencies. Some of this cost is lowered/eliminated by use of open-source instead of proprietary libs but wrapping 3rd party functionality in your own code is super important either way. If an piece of software lives for more than a few years, you are almost guaranteed to either need to remove a 3rd party component or deal with a major, non-backward compatible change. These wrappers help you understand what functionality you are using from these and greatly easy the transition. The relative upfront cost is negligible. Commented Dec 18, 2017 at 22:15

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.