When I populate the quickfix window (say with grep or Ag.vim), I can scroll through the entries, and all the positions inside the quickfix window show up in :jumps. But as soon as I navigate away (say I gf on a quickfix entry to go to the file), and then attempt to <c-o>, it doesn't take me back to the quickfix window, and indeed :jumps has been pruned of all my quickfix locations.
I would like to gf into a file, poke around, and then <c-o> back to the quickfix buffer -- is there anyway to get this behavior?
I believe I've seen this behavior before but I'm not sure if there's some option I set to change it.

gfin the quickfix window? Use<CR>to open a quickfix entry. Also, use<C-w>pto put the cursor in the previous window.