Assuming that everything is legal and above board, and that we are talking about a product that has a kosher open source license before the process starts:
How does the transition happen ...
Basically, the company makes a new release of the software with the non-open source license.
and what can be done to prevent it beyond choosing a difference license?
In general nothing. The only case where it can be prevented (or delayed) is if there are multiple copyright owners, and some of them object to the re-licensing. But if the company is serious, they could decide to solve that problem by rewriting the relevant parts of the code-base, etcetera.
Attempting to apply "moral pressure" before the fact probably are unlikely to work. The company is likely to do this kind of thing without any prior notification.
Having said that, there are examples where attempts to "wind back" open source has backfired on the company doing it. Consider the cases of OpenOffice, Hudson and MySQL, where the actions of Oracle have lead to a fork, a mass exodus of the developer community and (for OO and MySQL) distros increasingly dumping the original product in favour of the fork (LibreOffice and MariaDB).
What are the (ethical or social) responsibilities for the company?
To be frank (and somewhat cynical), it is not relevant what you and I think the ethical and social responsibilities of a company should be.
From a legal perspective perspective, the company directors' and executives' sole responsibilities are 1) to maximize value for the share-holders / owners, and 2) ensure that the law is obeyed. You could argue that as individuals, these people have social and ethical responsibilities, but ... unfortunately ... many of them would disagree.
But either way, ethical and social responsibilities are only relevant in a practical sense if the company and its officers take them on board and take them seriously.
If the open source version and closed source version are both available, how does the competition affect either product?
I don't think there is an answer to this. It depends on the circumstances.
Are there any examples of companies or products that have done this (either successfully or unsuccessfully) in the past? What was the community attitude toward those projects?
Yes there are examples. Just Google for the phrase "goes closed source" and apply a bogosity filter. Reading through the (non bogus) hits will give you a feel for community attitude, and if you research further into the products / companies you can make up your mind whether they were successful or not. (I'm not going to do this for you because "successful" is a value judgement.)