This isn't weird looking. It's what normal MCU code actually looks like.
Arduino code is the weird stuff, with functions that access the MCU control registers indirectly. While this is somewhat "nicer" looking, it's also much slower, and uses a lot more program space.
The mysterious constants are all described in great detail in the ATmega328P datasheet, which you really should read if you're interested in doing anything more then occationally toggling pins on an arduino.
Select excerpts from the datasheet linked above:
So, for example, TIMSK1 |= (1 << TOIE1); sets the bit TOIE1 in TIMSK1. This is achieved by shifting binary 1 (0b00000001) to the left by TOIE1 bits, with TOIE1 being defined in a header file as 0. This is then bitwise ORed into the current value of TIMSK1, which effectively set this one bit high.
Looking at the documentation for bit 0 of TIMSK1, we can see it is described as
When this bit is written to one, and the I-flag in the Status Register is set (interrupts globally enabled), the Timer/Counter1 Overflow interrupt is enabled. The corresponding Interrupt Vector (See ”Interrupts” on page 57) is executed when the TOV1 Flag, located in TIFR1, is set.
All the other lines should be interpreted in the same manner.
Some notes:
You may also see things like TIMSK1 |= _BV(TOIE1);. _BV() is a commonly used macro originally from the AVR libc implementation. _BV(TOIE1) is functionally identical to (1 << TOIE1), with the benefit of better readability.
Also, you may also see lines such as: TIMSK1 &= ~(1 << TOIE1); or TIMSK1 &= ~_BV(TOIE1);. This has the opposite function of TIMSK1 |= _BV(TOIE1);, in that it unsets the bit TOIE1 in TIMSK1. This is achieved by taking the bit-mask produced by _BV(TOIE1), performing a bitwise NOT operation on it (~), and then ANDing TIMSK1 by this NOTed value (which is 0b11111110).
Note that in all these cases, the value of things like (1 << TOIE1) or _BV(TOIE1) are fully resolved at compile time, so they functionally reduce to a simple constant, and therefore take no execution time to compute at runtime.