My Party of Three Saw Phandelver and Returned Victorious
I have nothing but respect for the other answers, but the specific question seems to be at least partly about the ease or necessity of adjusting not a basic adventure but the basic adventure from the starter set, namely the Lost Mine of Phandelver.
The particular advantage of this module is that it contains all creature and magic weapon stats in the back and generally does not expect one to own the core books. It also seems to go out of its way to cover the classic tropes of what new players tend to imagine the game is like (goblins, orcs, bandits, a dragon).
So, to address my experience with this adventure specifically:
Playing the Starter Set
My very first DM experience and D&D experience was for a group of three new players in this module. Although I adjusted it extensively, not much of that had anything to do with party make up, it was all to deal with the particular direction they took things and to change things that bothered me (a level three Necromancer can't animate those pet zombies he has, so to level 5 he goes!). In any case I spent very little effort on adjusting. Here is how I and we approached it:
My party recruited an NPC party member
Some people shy away from these because they can easily become a DM player character, and the near omniscient DM controlling a party member is potential trouble. However once the players got a sense for how he (a rogue) played I let them start making rolls for him and most of the combat decisions, and once the party got a good sense for his fairly simple personality they basically got to collectively control him so long as they did not abuse the privilege and kept him true to the character (which was never a problem with this group). Eventually we let him leave to his own devices when his arc came to a natural conclusion and the PCs were on the cusp of level 5 and no longer squishy.
My Party Often Avoided Combat
One nice thing about this module is that there are almost no encounters until the final dungeon where the party can not potentially get around a straight up fight through stealth, disguise, surprise, persuasion, magical shenanigans, or simply leaving an enemy on a side quest alone. Coming up with creative alternatives to combat is one of the joys of the game over a computer rpg. The module often seeds these, and I occasionally came up stuff on my own, but mostly the players just thought for themselves. Just make sure to give them roughly equivalent XP at least some of the time for dealing with enemies in alternative ways so that they don't fall behind on levels.
This is less effective with players of the murder hobo persuasion, but since they will get a lot of XP shared few ways they will be above intended level for many encounters, which compensates for lack of party members. Maybe just occasionally signal to them when a given foe is indisputably beyond them (which is rare in this module unless they get way out of order).
I adjusted monster stats on the fly, where it seemed logical or necessary.
I follow the basic principle that not every bandit or orc will be awake, sober, wearing their armor, and holding their sword at every moment of their lives, especially when they are in their hideouts. Just remember that a drunk enemy has disadvantage, a creature humanoid out of their armor has an AC of ten +their dex modifier, that some creatures might only have a dagger or whatever handy while off guard duty or whatever, and that you can let the party slit the occasional sleeping throat. This sort of thing makes for a flavorful if very inexact way to adjust difficulty, and when the enemy is in a room they haven't got to yet and the party hasn't done anything to alert them, do so on the fly.
Much easier, unlike in a movie most most realistic creatures, intelligent or otherwise, would rather run for it than die to "take all of you with them". Even a winning enemy force may have the odd member book it if they are looking rough, which may in turn distract their allies who have to stop them, etc. Bandits aren't paid enough to lay down their lives for the cause of crime.
Easier, in a situation where the party isn't doing so hot, maybe the enemies in one of the next encounters before they get rest simply aren't there. This is also the easiest time to throw in those unarmored, unarmed, etc. enemies I suggested above. When this is not appropriate you can just have the enemies use suboptimal tactics. Other than one very avoidable encounter with Hobgoblins they never run into anyone who is necessarily going to be a particularly skilled and sage tactician.
And finally, and easiest to always implement an easier battle on the fly, is to remember that the HP of all enemies is a die roll based range. They give you the average result in the stat block, but there is nothing unfair about just retroactively decreasing it to be at the low end of the range it could have been. In any case once they get a weak enemy to the single digits or a strong one into the teens, you can always just let that hit kill them. Not the most honest approach, but had you rolled out their HP and rolled low for them it would have been enough to kill them so it is clearly within the range of what was intended to be balanced for the encounter. This is very much an "if you feel comfortable with it" sort of thing, but in an emergency it can save you.
I Sometimes Came Up with Defeat Contingency Plans
One of the encounters, if not the encounter that the most groups die on is the very first encounter with four goblins. Whether this is inherent difficulty or just new players thinking they are in some sort of tutorial mode I can not say. It also may be that since the goblins are in good position to use tree cover and/or get a surprise round a strategically minded DM is not going to play them as stupid as the module intends. On the other end of the spectrum, they're goblins; if it starts looking deadly for them they should run away because they are a race of cowards.
In any case, this encounter is actually written with a specific recommendation for if the party goes down (goblins rob them and leave them for dead). Which leads me to the point of this anecdote, which is that sometimes even when the party fails because it is too difficult you can let them survive somehow. You might plan it ahead sometimes, but it is also a reasonable time to just say "I wasn't expecting this, let's take a break while I think through what happens next".
Of the four "dungeons" involved in this module three (goblin cave, goblin castle, bandit hideout). would be places where the enemies might imprison them for questioning or whatever when they wake up. The biggest potential fight is with a dragon while there is a friendly druid of undetermined power nearby, so guess who might just intervene. And failing that it is a green dragon, a type that would perhaps rather force the party to serve it than kill them.
On those occasions when you have do a contingency in mind you really don't need to worry much about the balance of the fight at all.
I Just Didn't Worry Too Much About It
From a balance perspective it is really hard to ever get things quite right, and there is never a perfect answer. Life is short, and this module provides few places where you don't have a way to make it easier on the fly or otherwise rescue a party, and few dangers where there isn't some way around conflict. Places where you get the feeling that players are likely to quit in frustration just aren't many. Phandelver lets the players do more or less all the level 1-4 things out of order, which means that it is impossible to balance the encounters for any party of any size with much certainty unless you are doing it to some degree on the fly. This is the nature of basically any non-railroading module.
So while you could spend hours upon hours stressing over CRs (and the CR system is not really that reliable anyway), it's really not worth it. My party was defeated twice, once in that first little encounter because they didn't really know the game, and once because of an elaborate series of player driven decisions and bad die rolls the module included no plans for. I had to come up with a reason for them to be kept alive in the later case. It was the worst thing that ever happened, and it actually ended up making things more fun in the end.
Of course this was all just my experience with the module; your mileage may vary.
