RTS un-stacks the caller address (Program Counter) and resumes at the instruction following the original JSR So, in your interrupt handler returns to an undefined location because it is loading the processor flags (The three registers and the CPU flags) put there by the Interrupt trigger.
You must use RTI for an interrupt handler.
Since the question has changed, here's some more comments.
You'll want to wedge your interrupt handler in front of the existing one. That's what @Tommy is explaining.
Also the C64 architecture overloads the IRQ line with several triggers:
- VIC raster interrupt (what you want to do)
- CIA clock
- CIA timer 1
- CIA timer 2
- CIA serial I/O
- CIA cassette I/O
So an interrupt handler has to determine which of these has been triggered and act accordingly.