The short answer: forget it, because of 5.

1. The loop is closed. If the encoder has less resolution, it cannot distinguish between two positions of the motor. If the motor has less resolution, it cannot make that move to correct what the encoder measured as position error.
2. The stepper motor can only reach (and hold) the positions according to its (micro)steps. Those are the positions it can reach accurately. throwing a higher resolution encoder at it will not change that.
3. What positions a stepepr motor can reach, depends on two numbers \$\frac{steps}{revolution}\$, which is a property of the motor itself and \$\frac{microsteps}{step}\$ which depends on the electronics that drive the motor. $$\frac{360°}{\frac{steps}{revolution} \times \frac{microsteps}{step}}=angle~of~one~microstep$$
4. Depends on your definition what a problem is for you (your system) if never reaching a position is a problem, then yes, there's your problem.
5. As pointed out above, use a different actuator or open loop
6. If you want to treat this like a synchronous motor, why don't you simply use a synchronous motor? I have seen it a lot that people come up with stepper motors for applications where they are simply not applicable. You stumbled upon some of the problems one can encounter. This is not an optional question. this is the mandatory first question of using the right tools for the right job. If you feel that a stepper motor is not the right tool for this job, don't try to force it into that role but pick a different actuator or control scheme.

The short answer

forget it, because of 5. Edit: open loop.

Regarding your edit

The main question here is how to choose the resolution of the encoder [...]

Asking for a better encoder is simply not the right question to ask. A higher resolution encoder will bring the two states of the flip-flop-control (that you are essentially creating here) closer together, but it will remain a flip-flop.

[...] and the motor

As stated above, if you need accurate positions that are not arbitrary, just use open loop. It will be able to reach the positions as calculated in 3. and do so accurately. Choosing a higher resolution for the motor means smaller steps between positions. If this is necessary depends on what positions you want to reach.

Does the encoder increase the accuracy in a microstep setup?

No. A microstep is a microstep and reaching a position in between is still not accurately possible. The microsteps represent the smallest interpolation possible. If you need smaller steps, try to find electronics that do more microsteps per step.

but a global motor and mechanical tools manufactor offered the closed loop system as it would be standard

I worked for a company producing positioning systems like the one you describe. It doesn't matter if it is a standard or not. What matters is if it is the right tool for the job or not.

Maybe it is a standard product. But it's not the right product for your application.

almost every motor driver has an encoder input

Modern motor drivers are often able to control different kinds of actuators. For other actuators it is very common to be controlled in close loop, but that doesn't mean you have to use that input. Depending on the driver, the encoder input can be combined with the limit switches. Using limit switches is common for a lot of actuators (including stepper motors) to find the limits, because encoders usually do not give absolute position values.

And now for something completely different: Why do male humans have nipples?

So I thought I'd be a simple question

It is, just like the answer I provided above.

I have the feeling that you bought this positioning system, realised that it doesn't work in your application as intended and you think buying a newer/better/whatever encoder will solve your problems somehow.

It's very strange to me that you list all the problems that are intrinsic to a closed loop controlled stepper motor yourself, yet refuse to let go of your idea of a better encoder solving all your problems.

I don't know what kind of answer you expect. Maybe something like "Yes go ahead, throwing more money at it will make it work". But that's not the one I can give you, sorry.

The original answer

1. The loop is closed. If the encoder has less resolution, it cannot distinguish between two positions of the motor. If the motor has less resolution, it cannot make that move to correct what the encoder measured as position error.
2. The stepper motor can only reach (and hold) the positions according to its (micro)steps. Those are the positions it can reach accurately. throwing a higher resolution encoder at it will not change that.
3. What positions a stepper motor can reach, depends on two numbers \$\frac{steps}{revolution}\$, which is a property of the motor itself and \$\frac{microsteps}{step}\$ which depends on the electronics that drive the motor. $$\frac{360°}{\frac{steps}{revolution} \times \frac{microsteps}{step}}=angle~of~one~microstep$$
4. Depends on your definition what a problem is for you (your system) if never reaching a position is a problem, then yes, there's your problem.
5. As pointed out above, use a different actuator or open loop
6. If you want to treat this like a synchronous motor, why don't you simply use a synchronous motor? I have seen it a lot that people come up with stepper motors for applications where they are simply not applicable. You stumbled upon some of the problems one can encounter. This is not an optional question. this is the mandatory first question of using the right tools for the right job. If you feel that a stepper motor is not the right tool for this job, don't try to force it into that role but pick a different actuator or control scheme.

The short answer: forget it, because of 5.

In the end it will always be an on-off control between two microsteps. There's no way around it. They are called stepmotors for a reason.

If you are fine with being able to only reach those positions dictated by the microsteps of the stepper motor, just run it in open loop.

If you want smooth, accurate movement to arbitrary positions, use a regular motor.

From my experience with such systems, you need special requirements to justify a closed loop stepper motor and live with the drawback mentioned above.

Trying to address some of your questions:

1. The loop is closed. If the encoder has less resolution, it cannot distinguish between two positions of the motor. If the motor has less resolution, it cannot make that move to correct what the encoder measured as position error.
2. The stepper motor can only reach (and hold) the positions according to its (micro)steps. Those are the positions it can reach accurately. throwing a higher resolution encoder at it will not change that.
3. What positions a stepepr motor can reach, depends on two numbers \$\frac{steps}{revolution}\$, which is a property of the motor itself and \$\frac{microsteps}{step}\$ which depends on the electronics that drive the motor. $$\frac{360°}{\frac{steps}{revolution} \times \frac{microsteps}{step}}=angle~of~one~microstep$$
4. Depends on your definition what a problem is for you (your system) if never reaching a position is a problem, then yes, there's your problem.
5. As pointed out above, use a different actuator or open loop If you want to treat this like a synchronous motor, why don't you simply use a synchronous motor? I have seen it a lot that people come up with stepper motors for applications where they are simply not applicable. You stumbled upon some of the problems one can encounter. This is not an optional question. this is the mandatory first question of using the right tools for the right job. If you feel that a stepper motor is not the right tool for this job, don't try to force it into that role but pick a different actuator or control scheme.

The short answer: forget it, because of 5.

In the end it will always be an on-off control between two microsteps. There's no way around it. They are called stepmotors for a reason.

If you are fine with being able to only reach those positions dictated by the microsteps of the stepper motor, just run it in open loop.

If you want smooth, accurate movement to arbitrary positions, use a regular motor.

From my experience with such systems, you need special requirements to justify a closed loop stepper motor and live with the drawback mentioned above.

Trying to address some of your questions:

1. The loop is closed. If the encoder has less resolution, it cannot distinguish between two positions of the motor. If the motor has less resolution, it cannot make that move to correct what the encoder measured as position error.
2. The stepper motor can only reach (and hold) the positions according to its (micro)steps. Those are the positions it can reach accurately. throwing a higher resolution encoder at it will not change that.
3. What positions a stepepr motor can reach, depends on two numbers \$\frac{steps}{revolution}\$, which is a property of the motor itself and \$\frac{microsteps}{step}\$ which depends on the electronics that drive the motor. $$\frac{360°}{\frac{steps}{revolution} \times \frac{microsteps}{step}}=angle~of~one~microstep$$
4. Depends on your definition what a problem is for you (your system) if never reaching a position is a problem, then yes, there's your problem.
5. As pointed out above, use a different actuator or open loop
6. If you want to treat this like a synchronous motor, why don't you simply use a synchronous motor? I have seen it a lot that people come up with stepper motors for applications where they are simply not applicable. You stumbled upon some of the problems one can encounter. This is not an optional question. this is the mandatory first question of using the right tools for the right job. If you feel that a stepper motor is not the right tool for this job, don't try to force it into that role but pick a different actuator or control scheme.