How to scale a monolith's release process?

Reliance on manual testing is almost always a bottleneck. If you were to only use automated testing for regression testing, you would be able to run your regression tests far more frequently. This means you won't be waiting until your release process to find potential issues as early as commit time. If you incorporate the creation and maintenance of automated testing as part of your development work, you can ease the integration burden as well. By automating these tests, people can focus on exploratory testing to think through edge cases and how end users will interact with the system, which could result in other interesting test cases to automate.

I would focus on test automation before thinking about decomposition into microservices. If you were to split your monolith into separate buildable and deployable modules, you would lose opportunities to find errors at build time. Plus, having robust automated testing at all levels would allow you to perform any refactoring necessary to split functionality into services.

Depending on what your release process is, there may be other opportunities for improvement. One option to consider would be to create a process map or value stream map of your development process, perhaps focused on your release/deployment activities, and see where your time is spent. Understanding steps that take a long time or segments where people are waiting can help you find places to improve.

automating the system testing by qa engineers?

Yes, using a tool like Selenium or SmartBear would certainly help you detect regressions. It's no substitute for manual system testing and acceptance testing, but it can definitely reduce the time needed.

You didn't mention your release pain points. What defects manage to get through {part of, all of} your process, and how many {dollars, man hours} are you willing to spend to improve those KPIs? Improve them by how much?

A relatively low cost technique you can use is Feature Flags. Then new feature code can be released in production in an "inert" state. Individual testers can enable it as needed. You can do a canary release, and rapidly back it out if things don't go well.

I will appeal to the last comment

I want each team to be able to release their features independently(as much as possible) without having to queue up too much(waiting to do manual testing on the last version) without adding risk to the process

Ok, manual testing is lagging for sure, but whether these they are the real bottleneck or not, I'm not so sure.

I think @ThomasOwens hit the nail

Depending on what your release process is, there may be other opportunities for improvement. One option to consider would be to create a process map or value stream map of your development process.

They become a bottleneck when we turn them into a gateway. Turning a stage of the release process into a gateway is rather an organizational issue.

To the said above, I would add an evaluation of the teams (its composition) and the distribution of the work. These are factors that potentially generate bottlenecks due to dependencies. Siloed teams, teams by stack, and homogeneous setups are often very dependent and not very autonomous. However, when work is distributed by verticals (complete business or functional units) and teams are cross-functional, the work happens with higher levels of autonomy.

When you enable the previous, then it's time to look for techniques to mitigate the unwanted but possible dependencies and bottlenecks. For example, the well-known feature flags, modularization, plugin & microkernel architectures, microservices, etc.

Now you have several teams, working on "decoupled" business units, with few or no dependencies. It's time for you to provide them with resources to run automated, manual, exploratory or monkey tests in parallel too. For example, think about sandbox test environments deployed in IaC fashion.

Bear in mind that the technique (how) serves the strategy (what) not the other way around.

Is there a way, other than microservices, to speed this up?

I think a lot of people get stung thinking that because they call things microservices and adopt its form, they must therefore be gaining something.

Like all forms of componentisation, it only really works if firstly the software design itself is amenable to some kind of componentisation (that is, independent staffers or teams can proceed with the design of their component, without unreasonable overheads of communication or ongoing coordination with other teams), and secondly if the whole is just so big that you need to allocate different full-time teams to different components (you can't do it monolithically).

If the software isn't amenable to componentisation, then you end up with a kind of faux-componentisation where the design itself is still monolithic but the implementation has all the overheads and trappings of multiple components.

And if the same team handles all components, then clearly the design process is still monolithic, but now your team have extra overheads for nothing - there's no second team existing to justify those overheads.

Even if the componentisation occurs because a future second team is anticipated, without the actual contemporaneous existence of another team, it's very unlikely the design will be done in a way that is sensible and actually allows another independent team to adopt the component later.

The definition of a monolith, in my view, is that it is designed by one mind - whether an individual or a team. It isn't specifically about any particular pattern manifest in the source code or arrangement of machinery.

A monolith may have internal modularity for good housekeeping - what it doesn't have, is the capability of more than one team working on it at a time.

Monolithic design is invariably the most efficient, until the scale and complexity exceeds the capability of one individual or team mind to work with it.

You only need to consider non-monolithic designs at the point where you have at least two independent development teams who do not (and would not be expected to) work on any component besides their own, because by definition the design is so complicated it needs two full-time teams just to marshal enough intellect.

Many systems with microservices end up as a poorly integrated, non-transactional menagerie, with an unfathomable number of implied constraints that operate across interface boundaries. Not only are these designs often flawed from the outset, there is often no hope later of modifying anything in the zoo independently of anything else.

Even just simple things like you can't temporarily turn one service off for maintenance independently of any other, and expect things to catch up later. So you don't even have independence of maintenance and deployment. It's faux-componentisation; it's still monolithic in design and operation.