Best practice of merging the branch to Dev, Stag and Prod environment

Welcome to Raju Gandhi,

I just want to git merging technic for the scenario which will be there in almost all the projects.
In a project, there are many different kinds of tasks and bugs assigned to the different team members. Which needs to gradually merge and deploy to Dev, Stag and Production environment/branch in a sequence.

Some tasks are very small and some are like an Epic level. For everything, if it is a 1-day task or 30 days epic, I prefere to create a branch so that it can be easily merged to the release branches.

Consider There are two major epic kinds of tasks which are developed simultaniously. feature/epic1 and feature/epic2. And there are 3 release branches release/dev, release/stag and release/prod.

As and when few of the work complete under epic1 or epic2 it is merged to the release/dev branch. So, it has well integration testing. But epic1 needs to be moved to prod first then epic2. What should be the best way to get all tested code updates of feature/epic1 from release/dev to release/stag?

Hello Ketan,

Thanks for the question. Let me start by saying that I am not a fan of long running branches in Git, for the exact reason that you posed this question—it makes dealing with these kinds of scenarios a nightmare. And when I mean long running branches, I mean specifically branches like release/stag and release/prod.

I have always advocated (assuming you don't have a very unique scenario, and truth be told, very few orgs have such scenarios) to only have one integration branch—that is your development branch (you can call this main, develop, master—I'll refer to it as main hereon).

Everything is created off this branch, and disappears as soon as a release is done (I am assuming some variant of the GitFlow model here, and not trunk based development). How would this work?

- You are ready to release—create the "release" branch from the commit on main that is going to production. This is the code that represents the next release
- Deploy it to whatever environment, test it, and make sure all is well. If you find bugs, fix them on the release branch
- Once you are done, the tip of the release branch represents the version that you will take to production
- Deploy that to production
- Tag the commit on the release branch (more on this in a minute)
- Merge the release branch back to main - This is important b/c you want all the fixes you detected on testing and the tag you just created to be part of the main history
- Delete the release branch

The reason for the tag is so if you have a production bug, you can check out the tag, create a branch, make a hotfix, deploy that to production, and then merge that "hotfix" branch back to main.

This works 95% of the time. There is no reason to have release branches live longer than a day or two (however long it takes for you to validate the code in the release).

With this model, your question is a no-op. There is no where else to merge epic1 or epic2 except main!

Now I realize I haven't answered your question, so here is my answer—very carefully

The reason, and again, it's the same reason I don't like long-running branches is this—unless you are super careful, you have NO idea if the release/stag and release/prod branches have diverged from the main branch! So even if you "catch up" with the main, and you test it, and then merge it to say release/stag—well if release/stag has diverged away from main, you aren't seeing the same final result!

In conclusion, and I realize I never answered your question, but honestly, it's hard for me to give any more specifics since I don't know how you manage your long running branches. And of course, if I completely misread your question, I apologize. Feel free to ping me again with any specifics.

Regards,

Hello Raju,

Thanks for your quick and descriptive answer. And sorry for the question which is not properly written.
Actually, I have no detailed understanding of the Git flow or Trunk based development. So, what we manage is based on our requirements.
like, there is 3 environments of the project.

Dev

- Dev environment where integration testing will be done. all team completes their small or big feature development. Unit test in their local environment and merge it into the release/dev branch for integration testing.
- This release dev branch is initially created from the master branch only.
- when any new feature development started, a new branch is created from the release/dev branch.  

Stag

- Stag environment where all the features will be merged which are to be released on the next production release. We don't have any concept of setting a TAG. So, whenever it is decided that in the next release feature1, feature 3, and feature 5 will be released, we merge those branches into the release/stag branch. resolve merge issues if any.
- This release stag branch is also initially created from the master branch only.

Prod

- Production environment where all last tested stage branch code will be used. This is a master branch. Stag code is merged to the master branch when required to release.

I also read one post https://www.toptal.com/software/trunk-based-development-git-flow If I compare the above commit workflow then it is not a pure git flow or pure trunk based.  

I have always advocated (assuming you don't have a very unique scenario, and truth be told, very few orgs have such scenarios) to only have one integration branch—that is your development branch (you can call this main, develop, master—I'll refer to it as main hereon).

Everything is created off this branch, and disappears as soon as a release is done (I am assuming some variant of the GitFlow model here, and not trunk based development). How would this work?

- You are ready to release—create the "release" branch from the commit on main that is going to production. This is the code that represents the next release
- Deploy it to whatever environment, test it, and make sure all is well. If you find bugs, fix them on the release branch
- Once you are done, the tip of the release branch represents the version that you will take to production
- Deploy that to production
- Tag the commit on the release branch (more on this in a minute)
- Merge the release branch back to main - This is important b/c you want all the fixes you detected on testing and the tag you just created to be part of the main history
- Delete the release branch

The reason for the tag is so if you have a production bug, you can check out the tag, create a branch, make a hotfix, deploy that to production, and then merge that "hotfix" branch back to main.

I think this is really a very good way if it is rapid development and rapid release. But what if we have already merged 3-4 features in the main branch and also resolve merge integration issues on the main branch too (That is the same dev branch). But after that, it is decided that only 1 feature should be taken out of it and released on production?

Highly effective teams practice Trunk Based Development (TBD) but this requires a high degree of discipline and collaboration and test automation to detect and resolve change conflicts quickly. You'll also need to look at feature flags to manage the actual release of a feature, which is separated from deployments. That is, you can deploy as frequently as you want but releasing depends on whether or not a deployment meets quality standards and acceptance. In your case, each of the 4 or 5 features will have their own flag and can be deployed at any time but if you choose to only release one feature to all your users, then you'd only turn on that particular feature flag while all the others are not turned on or set so that only developers and testers can access them.

Also, the notion of release/dev, release/stag, release/prod branches seems strange and unnecessarily complicated to me.

If you are going to have a release branch, then there would be only one of those. A single deployment artifact would be created which would then go through a promotion process through various environments.

That is, if you create a build and jar it up, then it might have some metadata in it that designates it as the deployment artifact for Build 2022.0322001, for example. This would then be deployed to some preproduction environment, say dev. Integration and system tests can be run against it in that environment and when the development team is satisfied that it works as expected, that same jar file is "promoted" to the next environment, say stage. The testers will do UI testing and user acceptance testing. When that's done, then maybe you're ready to promote to the production environment.

The way you describe your process, it seems like you create a different deployment artifact for each environment that you deploy to. This makes managing code versions much more complicated and there's less confidence that what you verified and approved in dev is the same thing that you're looking at in stage or prod since they are entirely different builds.

Hi Ketan,

Thank you for elaborating upon your workflow—I think it sheds more light on how your team does things. I will start by saying that @Junilu Lacar's posts are very insightful.

I don't think I am going to be able to add much outside of what has already been discussed in this thread, but I do want to make sure I understand your workflow:

- It seems to me that your master is what many teams called production—in other words, it's the branch that reflects the code in production. This is fine, I just want to be sure I am using the right terminology
- The "release" branch is created from the master branch (so it has everything production does) and then developers create feature branches from the release branch
- The "release" branch is also an integration branch, where developers merge back after they are done with a feature
- The "staging" branch is where you are picking which of the features are going to go into the next "production" release—you merge the branches that have those features in there

If I have this correct, I feel it's the last step that I disagree with. Based on your description, you are taking feature branches from developers (created from the "release" branch) and merging them in two different places

- They merge them back into "release" to do integration testing
- They merge _some_ of them into the "staging" branch depending on which of the features are going in the next release to production

Let's say you have 3 developers working on three different features, which we'll refer to as features 1, 2 and 3. You merge them into the "release" branch and you test. So what you are effectively testing is how these three features work with the existing feature set, and with each other.

However, let's say we decide on features 1 and 2 are going to the next production release. So you only take 1 and 2 and merge them into the staging branch. The question is—where are you testing this? B/c what you tested on the release branch is 1+2+3, but what you have on the staging branch is just 1+2. So:

- Do you test again? If no, then you really haven't tested anything b/c you are changing the combination of features
- And if you are testing again, then doesn't that seem like a waste of time and effort?

Junilu highlighted this very important point—If you'll allow me to quote them

A single deployment artifact would be created which would then go through a promotion process through various environments.

I have seen this happen one too many times—many teams treat branches as a way to manage multiple versions of code without realizing that

a) it's not necessary
b) it unnecessarily complicates the workflow

You are using the "release" and the "staging" branch to manage two versions of the code—one that has ALL the features, and one that has only the features that you are ready to take to production. So consequently you have all this merging shenanigans all over the place to manage all this complexity.

The thing to really understand is this—how you branch, integration, tag, and build code in Git affects your entire release engineering process. This is far too often overlooked in many teams. Get the Git workflow right and suddenly the release engineering process comes into sharp focus.

FWIW, I don't understand why you did what you are doing today. I am certain you have very good reasons, and I don't mean to come across as dismissive of those. You are being astute in asking the questions you are asking, and I do hope this thread has shed some light on it.

I am always available to chat, so feel free to reach out. And many thanks to Junilu for their thoughts on this as well.

Regards,

Thank you Raju and Junilu,

You are correct. It increases the testing time. Also, difficult to manage the merging as well.
This thread is really helpful for me to clear the concept. Thanks to both of you.

I really need to think of the promotion process. Based on my understanding, if I put it in different words, a single branch will have all the features all team members are working on it. It is some configuration that will make it active/inactive. And based on the decision, making it active on pre-poroduction environment. Which will be tested and the same artifact will move forward to production. Please let me know if I misinterpreted.  

Thanks a lot.

Hi Raju. I'm not Junilu, but I'd like to put some of my thoughts as well. And really had a very similar discussion the other day with multiple our teams about the same thing.

I have worked with both, so called Git Flow and Trunk Based models. And actually I start finding Git Flow similar to what you'd know about Agile vs Waterfall in a sense, that one is more modern way of thinking, while another is more stubborn and dated.

I find Git Flow model overly complicated and hard to follow along with challenges to automated it. Extra complexity gets added when multiple repositories contribute to a single project/product, now you have 2 or even 3 parallel complicated release processes. I had an experience with that as well.

Raju Gandhi wrote:It seems to me that your master is what many teams called production—in other words, it's the branch that reflects the code in production.

Not exactly. Each merge to master produces an artifact in an artifactory, think of it as a candidate release. That artifact you can use to deploy it to your dev, staging, demo or any other environments you'd wish. Now what's next if you are happy with it? You promote it, which in simple words we could think of as a simply copy/paste to a production meant artifactory, where only production environment has an access to. And that all it is, you just released it. And you can do that very quickly multiple times a day!

Raju Gandhi wrote:The "release" branch is created from the master branch (so it has everything production does) and then developers create feature branches from the release branch

That's not necessary, because every merge to master (with version defined in pom for instance) is a potential release. You may want to tag particular commit though after you promoted to a release, but that's it.

=Raju Gandhi wrote:- The "release" branch is also an integration branch, where developers merge back after they are done with a feature
- The "staging" branch is where you are picking which of the features are going to go into the next "production" release—you merge the branches that have those features in there

That's not needed. There are only Master or Main (some teams prefer to call it that way) and Feature branches - that is it. That means that Master branch always should be relatively stable, that requires having quite a good test coverage (integration tests where applicable), what Junilu might alluded to as a required discipline.

If developers are nervous about their changes going to master so quick, an optional step could be done so Feature branches would produce an artifact as well, that way they could test their changes first in i.e. dev environment and only then merge.

Having said all that, I trust potentially there are some fairly complicated products which may require a fairly complicated process of release as well, however, I in a lot of teams I believe that's not the case, so Trunk Based development model should suffice, and I see it most developers find it much easier to follow, and see a release process as a light task as opposed to a something big.

Hi Ketan,

Based on my understanding, if I put it in different words, a single branch will have all the features all team members are working on it. It is some configuration that will make it active/inactive. And based on the decision, making it active on pre-poroduction environment.

What you are referring to are "feature toggles"—a mechanism that allows you turn features on and off in a system. This, IMO, is a great way to develop software, but also requires rather disciplined engineering practices.

Which will be tested and the same artifact will move forward to production.

This is absolutely on point! Once you create an artifact[1] from a branch (say "release") that is the artifact that gets promoted from QA to Staging to Production. In other words, you do not rebuild the artifact once QA signs off. You deploy the very same artifact everywhere—maybe you have a pre-prod or even production because that is the only artifact that has been verified. Rebuilding can introduce changes (if you are building off different branches you might inadvertently introduce new and potentially untested features), but that's not it—simply rebuiding may change the dependency graph.

Hope this helps. Feel free to reach out if you have any other questions.

[1]: Some ecosystems like frontend applications in Vue or Angular make it harder to make one build that can be easily used between different environments. I am just putting this out there in case you are working with one of those.

I might have confused who with whom were talking to, so I potentially incorrectly annotated replies to author. Pardon here, please feel free to ignore all that post to avoid more confusion

Hi Liutauras,

Thanks for your response. I just wanted to clarify a few things:

- You quoted some of my statements, and responded to those. Those were not my suggestions—rather, they were my making sure I understand the OP (@Ketan's workflow). In other words, I wasn't describing GitFlow nor was I describing Trunk based, but elaborating on what Ketan said in their initial question so I could respond appropriately.

And actually I start finding Git Flow similar to what you'd know about Agile vs Waterfall in a sense, that one is more modern way of thinking, while another is more stubborn and dated.

I know many teams reach for GitFlow b/c it works, so I am hesitant to dismiss it as stubborn and dated. That's not to say you are wrong, but I am reticent to suggest one over the other till I understand the constraints that teams operate in.

That means that Master branch always should be relatively stable, that requires having quite a good test coverage (integration tests where applicable), what Junilu might alluded to as a required discipline.

In trunk based development I don't think that the master needs to be stable, especially if you reach for tools like feature toggles.

Other than that I agree with you, and you can find my thoughts (which align with yours) here—https://coderanch.com/t/750134/ide/Git-workflows#3483540

Hope this helps. Feel free to reach out if you have any other questions.

Ah! Damn! @Liutauras posted their response, so feel free to ignore mine as well. Apologies to all!

Regards,

Cows on me, and apologies once again!

Hi Everyone, this post is a great one indeed.
I have a question on best practices when multiple developers are working on same feature. Is it a goos practice to work on the same feature branch , or seperate feature branch?

Thanks
Biswajit

Biswajit Banerjee wrote:
I have a question on best practices when multiple developers are working on same feature. Is it a goos practice to work on the same feature branch , or seperate feature branch?

It depends on a lot of things. First thing to consider is how well developers communicate with each other. How is design done? How is testing done? How is integration done? There are many disciplines that come with working on the same line in source control. Everyone working on the same line/branch requires far more discipline and collaboration than working in separate feature branches. On the other hand, working in feature branches also has trade-offs that some teams might be better off choosing under certain conditions.

Thanks Junilu,
Let say 2 developers work on a feature. One works on UI and the other on backend. At the end of the day, both need to check-in their respective codes and preferably do a sqaush commit in 'master branch'. We then cherrypick individual  squash commits for each feature into a shortlived release branch for a releases.
The problem we face is , when we have multiple MRs (from multiple feature branches ) for same feature, sometimes it gets complicated to do cherry-pick into release branch. It seems easier if all work on same feature branch and submit just one MR when the work is ready to be merged into the master branch.
Would love you hear your experience on this.

Biswajit Banerjee wrote:
The problem we face is , when we have multiple MRs (from multiple feature branches ) for same feature, sometimes it gets complicated to do cherry-pick into release branch. It seems easier if all work on same feature branch and submit just one MR when the work is ready to be merged into the master branch.

Have you tried working this way? If so, how did it go? Was it, in fact, easier? If you haven't tried, I suggest you do and see if the experience lives up to your expectations.

@Biswajit

@Junilu brings up some very valuable points—workflows vary because how teams communicate, integrate and test their work vary. Leaving aside the vagaries of each individual team, I'd say that multiple team members should never work on the same branch. The confusion starts because we think of master, main etc as "special integration" branches—but to me, ANY branch that "integrates" the work of multiple developers is an integration branch.

Suppose you and I are working on a big feature—we create a branch off master—call it feat-a You and I then create our own feature branches off feat-a—we work, we create PRs, and merge. Occasionally we merge (or rebase) the master branch into feat-a (our "integration" branch) so we don't fall too behind, and when we are done with our feat, we issue a PR from feat-a into master. Done.

At the end of the day, both need to check-in their respective codes and preferably do a sqaush commit in 'master branch'. We then cherrypick individual  squash commits for each feature into a shortlived release branch for a releases.
The problem we face is , when we have multiple MRs (from multiple feature branches ) for same feature, sometimes it gets complicated to do cherry-pick into release branch. It seems easier if all work on same feature branch and submit just one MR when the work is ready to be merged into the master branch.

I wrote a book on Git, and I am confused about your workflow . Not to get on a soapbox, but I really don't understand this fascination that developers have grown for squashing, and cherry-picking. To me, those are the two smells in any workflow. In my book, I mentioned cherry-picking—b/c I feel the minute developers find out about it, there we are—looking at cherry-picking as a viable alternative to a reasonable flow involving merging branches.

Your problem, at least from the way you describe it, does not arise from how you are integrating work into feature (or integration) branches. Your problem arises from your choosing cherry-picking as a way to merge code. Any change you make to a reasonable integration workflow will most likely fail b/c cherry-picking is, IMO, a hack, not a solution.

I have said this in this thread, and I have said it in other threads—the point of an integration branch is much more than "bring work together". Its a way to verify if the work that just came together works together (via testing). Its a way to produce an an artifact that goes from DEV to PROD.

And to further highlight how much I feel thinking through a good workflow is, I am going to quote myself from this thread

The thing to really understand is this—how you branch, integration, tag, and build code in Git affects your entire release engineering process. This is far too often overlooked in many teams. Get the Git workflow right and suddenly the release engineering process comes into sharp focus.

If I were you, I'd ask myself, and the team—what can we do to not using cherry-picking as a way to get code in the release branch? B/c in essence—that's your problem—multiple folks issuing multiple MRs against master, and you aren't sure if you are getting ALL the ones you really want to the release branch. Ergo, your question.

Hope this helps. Feel free to reach out if you have any other questions.

Update: Edited for formatting

I too am very wary of cherry-picks. If you are cherry-picking a lot, you are doing something wrong.

The only times when I have thought of cherry-picking as useful, was when I needed to port a bugfix to different versions of the application that were released to different customers. This should not be a common occurrence.

Squashing to me is not necessarily a smell. I agree that many people do it too often and too habitually. You need to think about what you want your history to look like, so following a policy of "squash everything that goes to master" blindly is just nonsense. But squashing commits is a very useful tool for making the project history easier to read and understand.

If I'm working on a very small bugfix or change that contains multiple commits (some with messy commit messages), I will often squash all the commits together into a single commit and give that commit a descriptive name of the bugfix or change.

If I'm working on or overseeing a larger feature, I perform an interactive rebase before I make a pull request to the main branch. During the rebase, I will squash strongly related commits together, and I will give all resulting commits descriptive names for the changes that they introduce. Finally, the feature branch containing the squashed and rebased commits is merged to main.

Personally, I like having a long-lived branches for each major version of my application. Those branches always point to the latest stable release of that major version. Sometimes you need to write a hotfix for software that you already released, and then it's easy to just branch off of the stable branch to write the fix. That way you can easily and quickly release the hotfix without having to test all the features that were merged to the main branch in the mean time.

Now, if I've read Raju's answers in this topic correctly, his point is that you don't need branches for this, you just branch off of a tagged commit. This is true, but I find it convenient that you can just check out a stable release branch and know that you have the latest stable version to branch off of, without having to look up the tag name. It also helps with setting up your continuous deployment, where you can tell the pipeline to deploy the build to an Acceptance Testing server whenever the release branch is updated.

Of course, it's important to instruct the developers that minor fixes and new features are always based off of the main branch, and not off of a stable branch. Besides using them to create a new release, hotfixes that were based on a stable branch must be merged into the main branch when they're finished.

...so following a policy of "squash everything that goes to master" blindly is just nonsense.

If I'm working on or overseeing a larger feature, I perform an interactive rebase before I make a pull request to the main branch. During the rebase, I will squash strongly related commits together, and I will give all resulting commits descriptive names for the changes that they introduce. Finally, the feature branch containing the squashed and rebased commits is merged to main.

Great points. FWIW, when I say I disagree with "squashing", I should have been explicit—which is the use of the "squash" button that GitLab and GitHub offer. I am completely on-board with doing interactive rebases, and if need be, squashing commits to make atomic commits.

The bottom line is—make your history clean, explicit and decipherable. And interactive rebasing buys you the best of all worlds—you can make ad-hoc, WIP commits as you are working, thereby not forcing you to break flow just to organize commits, while at the same time, making sure you have good reset points in case things go awry or a branch point if you want to try something different.

Now, if I've read Raju's answers in this topic correctly, his point is that you don't need branches for this, you just branch off of a tagged commit.

Thank you for the credit, and that is usually my stance. My reasoning is simple—if you put a bug fix on a "stable" branch, you have to bring that back everywhere else—you choose cherry-picking to get that. I prefer to branch off the tag, make the hotfix, tag again, deploy, then merge that branch back into the mainline and throw the branch away.

I have said this before, and I'll say it again—I am not one to dismiss a workflow b/c if you've conceived it, you did so b/c you had good reasons. If your environment forces you to manage multiple versions simultaneously, perhaps having long running branches pointing to "stable versions" in production makes sense.

My biggest fear with long-lived branches is that it's super easy to switch to it, and accidentally commit. And usually, I've observed that with most long running branches, there is always something that is in one those branches that isn't in the integration branch—in other words, diffing a "stable" branch with the main development pipeline shows differences on both sides, when technically, it should be so that only the development pipeline has "more", or is "ahead" of the stable branches. However, if you can keep up the discipline, then so be it.

Appreciate the thoughts @Stephan, and the shout out.

Raju Gandhi wrote:My biggest fear with long-lived branches is that it's super easy to switch to it, and accidentally commit. And usually, I've observed that with most long running branches, there is always something that is in one those branches that isn't in the integration branch—in other words, diffing a "stable" branch with the main development pipeline shows differences on both sides, when technically, it should be so that only the development pipeline has "more", or is "ahead" of the stable branches. However, if you can keep up the discipline, then so be it.

Strongly agree that long lived* branches should never be ahead of the main branch. In practice I've rarely had the problem that people commit to stable branches by accident, because we usually set up branch policies that prevent committing directly, and only allow PRs that have a certain amount of approvals by senior team members.

*longer lived than two to four weeks, for larger features.

Appreciate the thoughts @Stephan, and the shout out.

I enjoyed the insights in this topic. I actually sat here staring at my screen for a few minutes and thinking about whether I *really* needed a stable/v2 branch. My conclusion was no, I don't need it, but it's convenient sometimes.

Thanks @Raju, @Stefan, @junilu,
I am, too, a not big fan of Cherry-picking. Our previous model was to create a temp release branch from master after receiving QA sign-off on a bunch of features/bugs. 'master' branch would have only those features which were planned for next release and we would merge only those MRs into master. However, slowly our QA could not keep pace with the development, and we ended up merging too many features into master. For example, as I write, we have around 40 MRs(features) merged into master, but only 10 features( in randon order) are QA approved for release.So, we CAN NOT  create a release out of master, and hence we had to take the cherry-picking approach.