12

While I only have a github repository that I'm pushing to (alone), I often forget to run tests, or forget to commit all relevant files, or rely on objects residing on my local machine. These result in build breaks, but they are only detected by Travis-CI after the erroneous commit. I know TeamCity has a pre-commit testing facility (which relies on the IDE in use), but my question is with regards to the current use of continuous integration as opposed to any one implementation. My question is

Why aren't changes tested on a clean build machine - such as those which Travis-CI uses for post-commit tesing - before those changes are committed?

Such a process would mean that there would never be build breaks, meaning that a fresh environment could pull any commit from the repository and be sure of its success; as such, I don't understand why CI isn't implemented using post-commit testing.

Improve this question
1
  • 1
    This is the question people were asking A LOT - and now at least Jenkins and TeamCity do support "pre-tested commits". :-) Commented Sep 9, 2017 at 5:56

3 Answers 3

Reset to default
6

I preface my answer with the details that I am running on GitHub and Jenkins.

Why should a developer have to run all tests locally before committing. Especially in the Git paradigm that is not a requirement. What if, for instance, it takes 15-30 minutes to run all of the tests. Do you really want your developers or you personally sitting around waiting for the tests to run locally before your commit and push your changes?

Our process usually goes like this:

  1. Make changes in local branch.
  2. Run any new tests that you have created.
  3. Commit changes to local branch.
  4. Push local changes remotely to GitHub and create pull request.
  5. Have build process pick up changes and run unit tests.
  6. If tests fail, then fix them in local branch and push them locally.
  7. Get changes code reviewed in pull request.
  8. After approval and all checks have passed, push to master.
  9. Rerun all unit tests.
  10. Push artifact to repository.
  11. Push changes to an environment (ie DEV, QA) and run any integration/functional tests that rely on a full environment.
    • If you have a cloud then you can push your changes to a new node and only after all environment tests pass reroute the VIP to the new node(s)
  12. Repeat 11 until you have pushed through all pre-prod environments.
  13. If you are practicing continuous deployment then push your changes all the way to PROD if all testing, checks, etc pass.

My point is that it is not a good use of a developers time to run tests locally impeding their progress when you can off-load that work onto a Continuous Integration server and be notified of issues that you need to fix later. Also, some tests simply can't be run until you commit them and deploy the artifact to an environment. If an environment is broken because you don't have a cloud and maybe you only have one server, then fix it locally and push the changes quickly to stabilize the environment.

You can run tests locally if you have to, but this should not be the norm.

As to the multiple developer issue, open source projects have been dealing with that for a long time now. They use forks in GitHub to allow contributors the chance to suggest new fixes and functionality, but this is not really that different from a developer on the team creating a local branch, pushing it remotely, and getting team buy-in via code review before pushing. If someone pushes changes that break your changes then you try to fix them yourself first and then ask for their help. You should be following the principle of "merging early and often" as well as merging in updates from master to your branch periodically.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Reading my question years later I do feel it is phrased poorly, but the essence of the question is not "why aren't tests run locally before pushing them to the remote", but rather "why doesn't the CI server test the changes pushed to the remote before they are 'merged'". I would liken your structure to the "Not Rocket Science" method, where the merge is performed manually instead of automatically. My question centers more around an architecture where pushes are made directly to a central repository, though I think it's still relevant for others.
@SeanKelleher You could always configure a hosting environment to automatically run tests and merge to master for you. You could even, with GitHub or similar, fully automate it through API integrations and web hooks. The main point is that if you don’t want developers to push untested code to master, you should have them push to something that isn’t master and only push it onto master after tests pass—and that last push can be automated.
3

The assumption that if you write code and it compiles and tests are passed locally, no builds could be broken is wrong. It is only so, if you are the only developer working on that code. But let's say I change the interface you are using, my code will compile and pass tests as long as I don't get your updated code That uses my interface. Your code will compile and pass tests as long as you don't get my update in the interface. And when we both check in our code, the build machine explodes...

So CI is a process which basically say: put your changes in as soon as possible and test them in the CI server (it should be of course compiled and tested locally first). If all developers follow those rules, the build will still break, but we will know about it sooner rather than later.

Improve this answer

6 Comments

But consider your first example - in a CI environment, the first change will be checked in and the build will pass. Then the second change (submitted at the "same" time) will be checked in and the build will fail. With pre-commit testing, the changes will be queued, and the second change will be rejected because it breaks the build before it's committed. This allows the second developer to update his code to work with the newly committed changes and resubmit, without ever breaking the build.
Who said he is the one who broke the build? how should he have known that the first developer changed the interface? maybe it was really the first developer who broke the build and he should fix it EVEN though his check ins passed?
And by the way, now think about 10 developers working in 3 teams, each one claiming that on his computer the code works fine. and in the build machine the build is failing. Why is the last poor soul that checked in responsible for all changes made by the other 9 developers?
True, the blame shouldn't necessarily be put on the developer whose code couldn't be committed, but pre-commit testing will allow the developer whose code couldn't be committed to realise that erroneous, but non-breaking, code was committed, and who it was committed by, allowing instant queries to be carried out. Meanwhile, anyone retrieving the code at this time will be able to build successfully. I realise that this does have the problem that they have an invalid interface that they are working against, but I see this as a problem regardless of the approach used.
Hence CI is not the only paradigm. but it is my favourite as it syas: if there is a problem everyone should know about it and can not continue to work until it is resolved. from my experience the costs are well worth the fact that problems are solved quickly by the team and are not hidden. this will decrease the overall time spent solving those problems.
|
3

The CI server is not the same as the version control system. The CI server, too, checks the code out of the repository. And therefore the code has already been committed when it gets tested on the CI server.

More extensive tests may be run periodically, rather than at time of checking in, on whatever is the current version of the code at the time of testing. Think of multi-platform tests or load tests.

Generally, of course, you'll unit test your code on your development machine before checking it in.

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.