#25743 caused a performance regression with Series operations (mean, sum, etc). This can be avoided by moving some of the ifs around so that the additional check is not performed for those operations.