Why does type inference not work well with reference types?

I wrote a code similar to below:

fn foo() { let x = &1; let y = 2; let z: u32 = x * y + 42_u32; } 

and I got an error:

error[E0277]: cannot add `u32` to `i32` --> src/main.rs:11:24 | 11 | let z: u32 = x * y + 42_u32; | ^ no implementation for `i32 + u32` | = help: the trait `std::ops::Add<u32>` is not implemented for `i32` 

But when I dereference x, it just works fine.
During some experiments, the following codes were compiled too.

let z: u32 = x + y * 42_u32; 

let z: u32 = 42_u32 + x + y; 

I wonder why Rust's type inference does this.
Is the order of type inference affected by the order of operations?
If so, why does it make type inference impossible in this situation?