What you want can be accomplished by just pressing Esc after or Alt+j or Alt+k but if you want to save that 1 keystroke, then adding the following to your .zshrc can help you.
vi-cmd-up-line-history() { zle vi-cmd-mode zle up-line-or-history } zle -N vi-cmd-up-line-history bindkey -M vicmd '^[k' vi-cmd-up-line-history bindkey -M viins '^[k' vi-cmd-up-line-history vi-cmd-down-line-history() { zle vi-cmd-mode zle down-line-or-history } zle -N vi-cmd-down-line-history bindkey -M vicmd '^[j' vi-cmd-down-line-history bindkey -M viins '^[j' vi-cmd-down-line-history
What it does is bind new widgets called vi-cmd-down-line-history and vi-cmd-up-line-history to Alt+j and Alt+k. These widgets change the mode to cmd by calling the widget vi-cmd-mode and then fire the down-line-or-history and up-line-or-history respectively.