2

We are using a Software as a Service platform that allows to create custom code which integrates in the platform and all its features (dialogues for common objects like Account, Customer, Address, and so on) and a GUI designer to create dialogues and reports. The platform uses a proprietary strongly typed, object-oriented programming language and a proprietary database.

We are facing multiple issues

  • the current implementation has bugs
  • the specification (requirements, expected behavior) may be incomplete and is distributed over around 1250 jira requirement issues (the former mentioned number 2500 does include tasks, bugs, tests, etc.)
  • the implementation (the code written) may already deviate from the requirement (incomplete implementation, misunderstanding) or is just buggy
  • the external team is new and took over the code from the previous external team
  • the platform has restrictions regarding mass data handling (creating 15 million rows (15 000 000) during an import proves to be challenging
  • budget for the external team size is tight
  • internal employees (apprentices) are available but are inexperienced
  • internal organizational politics are playing a role as well

Is this a duplicate?

I do not think so

Transpiler or rewrite?

  • My very basic understanding of ANTLR is that with the help of a grammar file code can be translated / transpiled from one language to another
  • But this does not solve that the translated code calls classes and methods for the proprietary framework that do not exist in the new target framework (maybe Asp.Net)
  • There is a possibliy inactive open source library that may convert some calls to the proprietary library to and outdated Asp.Net version

If we go the transpiler route we have to tackle

  • inexperienced with ANTLR and the lexer, parser, codegenerating process
  • rewriting the transpiled code to use the most recent classes of Asp.Net 8
  • creating test cases for the transpiled code to verify that it does behave as expected
  • deal with the design decions made (object names, object relations, which object / class takes care of what)
  • live with the database schema where some decisions could be based on the restriction the proprietary database has

If we rewrite from scratch (that one should never do and without losing sanity)

  • we could start creating test first
  • create a new database schema
  • take care of data migration after the new system can take over (big bang switch)

What are your thoughts and experiences?

Edit

Thanks for all the input. The comments and answers are immensly helpfull.

The question if transpiling the buggy code and fix it then might be still better than starting from scratch has been answered. The advise was that trying to transpile code requires a lot of Know-How and is only a minor part of the effort. It seems most advise against a rewrite but recommend to fix the current application.

The more important aspect than the technical issue is what the expectation of management is and what can be achieved given the current conditions (small external team and small budget).

Improve this question
15
  • 1
    I second steves comment. you list the problems, but none of them are really problems that force you to change anything technical. Put in work arounds for the bugs, renegotiate the external team contract, train the internal employees etc etc Commented Feb 2 at 13:41
  • 7
    An external "outsourcing" team that is switched and still underfunded and only apprentices for internal support. Have you though about moving on to a company that wants to be successful? There is only so much you can do with code. You cannot fix your management by programming. Commented Feb 2 at 14:42
  • 6
    Investigating a transpiler would be a going down a rabbit hole. Lexing/parsing is not simple for any real language, and, that's not even half of the job. Commented Feb 2 at 19:12
  • @Ewan: i am uncertain if the platform really has such strict constraints that working around them seems to be quite the effort or if this is a skill issue of the external team. Regarding train internal employees this is were organizational politics plays a role. Since the inexperienced team is from another departement and the responsible departement states that training inexperienced people is inefficient and will not solve the issues but put an additional burden on the external dev team. Commented Feb 2 at 19:36
  • @nvoigt This Have you thought about moving on to a company that wants to be successful put a smile on my face. This you cannot fix your management by programming is spot on. Commented Feb 2 at 19:42

5 Answers 5

Reset to default
15

Let's me take on your issues one-by-one:

the current implementation has bugs

Yep, and when you transpile such a code, what makes you think those bugs will not be transpiled as well? With a rewrite, your new system will also have bugs, but other ones, and not necessarily less.

the specification (requirements, expected behaviour) may be incomplete and is distributed over around 2500 jira issues (~ 1250 requirement issues)

Neither a transpilation nor a rewrite will fix that. Btw, when porting a program to a different platform, I would not use resolved issues from an issue tracker as a replacement for a spec, I would use the existing system and it's current behaviour as a replacement for a spec.

the implementation (the code written) may already deviate from the requirement (incomplete implementation, misunderstanding) or just buggy

A transpilation will not change this. A rewrite by people who take the old system as a template will probably copy these issues.

the external team is new and took over the code from the previous external team

You told us about an external team and and internal team, but forgot to mention who has the responsibility for maintaining what. However, I don't think any of your two approaches will change here much.

the platform has restrictions regarding mass data handling (creating 15 million rows (15 000 000) during an import proves to be challenging

Maybe going to a different platform might fix that. But usually solving an isolated issue like that in an isolated way is much cheaper.

budget for the external team size is tight

internal employees (apprentices) are available but are inexperienced

internal organisational politics are playing a role as well

These are all indicators for better not changing the platform, neither by a rewrite nor by transpilation.

What I miss completely in your post, however, is any kind of reason why you want to move away from the current platform. Are there hard technical or legal reasons why you need to leave the current platform? Or do you expect to make development cheaper or quicker on a new platform? How critical is the system for your business?

Hence, my best recommendation here is: first clarify what you want to achieve (and why), then start looking for solutions - and don't be fixated on "rewrite" vs. "transpilation". Think also about organizational solutions (like training unexperienced developers), or the option of keeping the old system as it is (and solve your major issues within the bounds of the platform).

Improve this answer
15
  • 1
    Before even trying to clarify what you personally want to achieve, its probably worth clarifying what the systems stakeholders want to achieve. If those who control the purse strings don't want to pay for a complex project and take on the short term risk, then you may be far more limited in what you personally can achieve. Commented Feb 2 at 15:53
  • 4
    @user1937198: I agree. From what the OP currently wrote, I think it is hard to identify any real goals or why they want to change something, so it is unclear what comes from themselves - just as a personal idea - or what comes from their stakeholders and which contextual restrictions they have. Commented Feb 2 at 17:05
  • 1
    @surfmuggle 'No one dares to state that the current budget is too little and switching the external team did not help much.' This is your number 1 problem. If you can't resolve this, then there isn't much hope. Changing the system will require budget. It will make features slower in the short to medium term. Stakeholders have to make the choice to invest in building a system that enables faster feature delivery, or they will force you to continue the short term trajectory, by requiring more features than you can deliver, leaving no capacity for long term product improvement work. Commented Feb 3 at 1:07
  • 4
    @surfmuggle: comments here are likely to be missed by other readers. If you want to clarify things, better edit your question. Moreover, to be honest, the things you mentioned sound more like wishful thinking than based on facts. You will probably get much better answers when you disclose the specific technical platforms you have in mind - the fact that you did not mention it seems you believe it does not matter, but I believe that's wrong - the whole transpilation approach will only make sense for certain technologies. Commented Feb 3 at 6:31
  • 3
    @surfmuggle: so what exactly are you trying to do - to convert Salesforce-Apex code into C# code (using the Salesforce SDK for C#), and keeping Salesforce? By "wishful thinking", I meant that there is no factual indication that such a transition will solve the problems you listed. Commented Feb 3 at 9:15
6

Obviously you ask "rewrite or transpile?", but I'm not clear what the underlying issue is.

You mention the existing implementation is riddled with bugs. You mention that there is limited budget even to maintain the status quo. And you mention that you have nothing but apprentice labour available internally.

I'm afraid no special technique will solve the problem of a business that isn't living in the real world regards budget and staff skills.

Another thing that businesses are only slowly fathoming, is that virtually all kinds of technology has some kind of lock-in, and the point of SAAS (rather than periodic capital investment in permanent software licences) is to facilitate future price-gouging and private taxation of your business by the vendor once you are locked in.

Most people have forgotten when IBM used to rent the mainframe to you, and you weren't going anywhere else than IBM because you'd spent a decade with an army of staff writing the bespoke software for whatever IBM platform you were using.

That all said, work on your own bespoke transpiler will not pay off (I gather you're talking about making one, not just applying a COTS transpiler), and it's not obvious you even have the internal resources and know-how to perform either a transpile or a rewrite.

I would recommend avoiding do-something-itis, and just stick with what you can afford which is the status quo.

Improve this answer
7
  • 1
    "Most people have forgotten when IBM used to rent the mainframe to you"... I remember one company being surprised and annoyed that when they upgraded their server to dual processors their bill doubled, because the licence was "per cpu" Commented Feb 2 at 13:32
  • 1
    @Ewan, indeed. The problem is that businesses usually find it easier to stand still and hold out when faced with extortionate costs of change, and the need to upgrade the number of CPUs usually doesn't happen without increased business growth (and profitability) putting more strain on yesteryear's hardware. SAAS allows them to extort you even when you're standing still, because if you hold out then you're given notice and the service is simply withdrawn. Commented Feb 2 at 13:39
  • 1
    @surfmuggle, I think transpilation is completely out of the question then. It sounds like you're considering it as an option for quite naive reasons. If you'd just spent several years at university studying programming languages, compilers, and transpilers, and had written a transpiler before - so you had man-years invested in these skills already, as well as the advice of a network of full-time academic experts available when learning - then maybe you'd now write your own bespoke transpiler in a reasonable time. Otherwise you've no hope. (1/2) Commented Feb 3 at 8:20
  • 2
    @surfmuggle Re "wonder how far this helps in transpilation", it takes care of the easy 10% of the job for you. There's another easy 10% part left for you to do, which leaves just the hard 80% entirely on your shoulders. Commented Feb 3 at 8:21
  • 2
    I think if you're not a financial decision-maker at your firm, nor in a position to seriously influence anyone who is, then the situation is probably hopeless and your best solution would be to find different employment. It sounds like the main constraint is gross underfunding and the absence of enough competent technical staff to permanently oversee a bespoke in-house development (indeed it sounds like the development was supposed to be out-house, but your management have tired of the actual costs), and there's no trick that will circumvent the effects of these problems. (2/2) Commented Feb 3 at 8:22
2

Why the whole hog?

You have listed numerous issues that make wholesale change risky:

  • Novice Staff
  • Tight Budgets
  • Third Party Supporters (misaligned goals)
  • Buggy/Weird behaviors
  • Lack of tests

From the sounds of it you also have a management that wants a well oiled machine but aren't willing to pay for one. You may want to investigate why they are unwilling. Is it actual financial constraints, were they burnt, is there a Hippo casting shade at you and your team?

Divide and Conquer

If you keep breaking the problem apart you will eventually find a problem so small that you can just solve it.

(Addendum: Except there are situations where you can't solve a problem in pieces - in which case this won't work look for something else.)

I can see several vectors where you can do this.

  • Write Specification Tests
  • Strangle off modules
  • Prototyping

Writing Specification Tests is probably the most obvious. Grab Gherkin and Cucumber. Write business language tests and write the steps to execute each statement against the system. Get your business stakeholders across how to write these, and how to read these. They will love them. Also teach your second/third tier support to use these to figure out customer issues. The more teams in the business write and contribute the better defined these system is. It also allows those business units to get clearer about explaining to you new functionality.

The best thing about these tests is that they are a visible win, a feedback mechanism that lets you talk to the business about what is wrong, what is needed, and how stable everything is. The first step on the road to money is getting the people that have it to understand what it is doing for them.

Another approach you could do is strangle off modules. Find a way to host the program and divert people from it to another program for particular activities. Start with a small area and slowly move functionality across. I can't speak for your situation but there are many ways to do this. The win here is that you can demonstrate greater reliability in the new modules, they can have the testing from day dot, and be in a platform that is useful.

Strangling off modules is itself a win that can be used to demonstrate to the business where their money is going, and what effect it is having. You can also show case what differences the testing/platform/language have, and also demonstrate staff training.

Prototyping is closest to what you have in mind. Throw up as quickly as possible a replacement for the entire system. Throw it up in the platform/language of choice. Hit the most important functionality. The point is not to replace the current system. Its to demonstrate the technology, development patterns (like unit tests/pipeline/etc), and train the developers. The whole point of the prototype is to be deleted after demonstration.

What it achieves is a counter example that is usable enough that the business can decide if its better. They can see the costs as they can extrapolate from the prototype, and they will see what it gets them. The best part is that it can be payed off as a team collaboration/training exercise.

None of the above solve the problem with the current system but they work to increase the social capital with the business and to alleviate one of the core problem you currently have (Lack of specification, tight budgets, Team Experience).

Improve this answer
6
  • Regarding Find a way to host the program and divert people from it to another program for particular activities Do you mean that part of the UI comes from the new system? AFAIR Google did this (for their CRM or AdWords. Commented Feb 3 at 7:51
  • 2
    "If you keep breaking the problem apart you will eventually find a problem so small that you can just solve it." - that's a serious and dangerous fallacy that mires many a fool. Many problems in software require an all-round integrity to the workings which are difficult to grasp as a whole, and you can't break it down, otherwise you'll lose the ability to appreciate the overall qualities which it must have in order to work properly. God knows I've dealt with people whose favourite activity seems to be the whack-a-mole that ensues from their "break it down" approach. Commented Feb 3 at 9:21
  • @surfmuggle Yes, you've got it. That or a back-end, as long as you can draw a boundary around it and have the new and old system pass off to each other. This way you can bring the new world in slowly a piece at a time. Commented Feb 5 at 6:24
  • @Steve Yes, there are problems that resist divide and conquer. I don't think either it or a holistic approach is superior. I find you have to delve down into details and bring the learning up to the top to look at the big picture, then dive back into the details many many times over to arrive at a solution that works in the details and overall. What I was trying to get at here is that surfmuggle is solving the whole problem at once with a single hell mary solution and that might benefit from some divide and conquer to find a problem that can be solved and to use it to learn what works or not. Commented Feb 5 at 6:36
  • @Kain0_0, I think you were far more sweeping in how you phrased it the first time. The delving-and-reviewing process you're describing now is merely the process of learning about the problem and assembling an understanding in your mind (and which does not automatically lead to any solution to the problem, and the learning process itself does not even necessarily succeed if the problem is more complicated than what the learner can grasp). It's not divide-and-conquer (where you said a solution must eventually emerge from looking at smaller and smaller parts in isolation). Commented Feb 5 at 10:32
1

Transpiler or rewrite?

Just convert it.

Easily 80% of my entire career has been doing this. You get one requirement: make it do what it did before on this new system.

It isn't a rewrite and you don't have to use a transpiler (these can cause more problems then they solve). All you need is coders who are familiar enough with both languages and systems to convert from one to the other.

We are facing multiple issues

Every single issue you mentioned is an issue even if you don't covert it. Thinking those are impediments to converting it just shows you have the wrong attitude about converting it.

Converting software doesn't make it perfect. But it is absolutely the most thorough form of code review. If you do this, even if you don't use the new version, your knowledge of the code will be better for it.

Be very reluctant to make any improvements to how this works. Your goal is the conversion. You will notice things that need fixing. Don't fix them yet. Convert first.

If you were hoping things would be better with the converted code, if that was your main reason for doing this, forget it. Converting wont make the mess better. It just moves it into a new stack.

we could start creating test first

This is an option whenever you write code. I've done it in shops that won't support unit tests. I just keep my tests to myself. If you want to do this there is never a reason not to.

create a new database schema

Dear god why?

take care of data migration after the new system can take over (big bang switch)

Well, if you insist on changing the schema in the middle of this (seriously why?) make sure you automate the migration. Test the hell out of it before you take it live. Have a plan for when this fails.

Improve this answer
4
  • 2
    Although OP didn't mention how large the current codebase is, "2500 JIRA issues [describing the requirements]" plus having a whole team struggling with maintaining it makes me feel that we're talking about many hundreds of thousands of LOC if not more. In such a situation, "a coder" won't cut it, you need a team of many such coders to accomplish the conversion in any reasonable time frame (the longer it takes, the harder it gets as the source codebase evolves under your hands). Commented Feb 3 at 8:17
  • @TooTea better now? Commented Feb 3 at 9:17
  • @candied_orange the current database scheme seems to be a mediocre fit. Some columns are unused. a few duplicate other columns. Some tables could be likely refactored so that the records can be stored in a single table. For others i believe a history table or adding a time span (valid_from, valid_until) would be helpfull. Commented Feb 3 at 17:00
  • 1
    @surfmuggle I recommend making the conversion about conversion before improvements to the DB. Makes it much easier to see if you did what you meant to do. Improve the DB when you're sure the conversion is complete (and correct). Just being complete doesn't mean you have to go live. Commented Feb 3 at 17:23
0

My Experience is that "lock in" is a real thing and that this problem is unsolvable. It's the "should I rewrite or refactor" problem, which is already a no win situation, but refactoring is even harder than normal and a bunch of extra costs and contracts have been thrown in to rewriting.

My thought is, you may have found a problem that AI is A solution for. "AI" by which I mean the current Chat GPT style offerings, seems to be passably good at problems like "explain this code" or "write this in language X"

I wouldn't expect 100% automation, but it might take away the "configure ANTLR" problems and leave you with just the "finish and test this code" problems. It might be worth a try.

The normal solution I would suggest is not "rewrite" but "write v2". Write a new system which isn't a direct replacement, but competes with the old to solve the same problem.

This allows you to pick a smaller scope or problem to work on, address fundamental problems or changes in problem statement and business case that might have occured over the years and crucially move forward as a business while working on this issue.

Another solution that just occurred to me is that you could suggest migrating to a different SAAS product/supplier. SAAS companies usually promise that they can migrate in existing processes and offer consultant help to do the work.

You might be able to get a good deal and reduce costs by switching, as the new company will be incentivised to aquire you as a customer. Indeed even the threat of moving might elicit more help from your current supplier.

Improve this answer
5
  • 2
    "Write a new system which isn't a direct replacement, but competes..." - it sounds like basic ERP software that coordinates the whole business - customers, accounts, delivery addresses, etc. - so he probably cannot produce a competing yet simple product, without a complicated integration between the new and old, and without massively deteriorating the simplicity of the system as a whole and undermining the ability of anyone to oversee the whole lot and keep it in working order. Commented Feb 2 at 13:33
  • Yes, you are probably right. The reason this is my normal "answer" to this type of question is because that is what other companies will be doing rather than making a like for like copy of your business. So you know it can be done and it gives you a way to try and stay relevant Commented Feb 2 at 13:37
  • To comment on basic ERP software that coordinates the whole business fortunately it only covers a part of the business. @Ewan Have you tried to create a fully functionaly program with an LLM? I only tried toy applications (< 400 LOC) with the free (non subscription) versions. That a self hosted deepseek delivers only ~ 4 Tokens per second. unsloth selectively quantized layers to get around 10 to 14 TPS generation speed with 2 nvidia H100 GPUs (GPU cost 60000 to 90000 $ USD) Commented Feb 2 at 21:33
  • 1
    I'm not suggesting it will do the whole thing for you in one go! I assume you have multiple short scripts and forms in their custom language and want to change each to some other less custom language. Rather than writing a custom transpiler you could do this manually, where step one is ask the AI to convert it. and steps 2 to 10 are fix the code manually Commented Feb 3 at 10:36
  • 1
    this would be as an alternative to steps 1-99 write a transpiler, step 100 run the transpiler Commented Feb 3 at 10:36

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.