The answer is that there is no such isotopy for each $f\neq g$. The problem changes if one asks only for isotopy between the supports of the graphs without asking for the final $x$-coordinates to correspond. See comments and other answers for this. In fact there is no reason to suspect such a property to be determined just by their intersection points. What one can prove is however that when $f, g$ do not touch then there is such a "weaker" isotopy. Anyway, coming back to OP problem.
Let us consider arbitrary $f, g$ and let $[a, b]\subset \{x| f(x)\neq g(x)\}$ with $a\neq b$ and assume $f>g$ here. Let us call $R$ the path connected component of $x=((a+b)/2, (f((a+b)/2)+g((a+b)/2))/2)$ in the complement of the union of the graphs.
Now consider the loop $\gamma: (a, f(a))\stackrel{f}{\rightarrow} (b, f(b))\stackrel{\text{down}}{\rightarrow} (b, g(b))\stackrel{g}{\rightarrow}(a, g(a))\stackrel{\text{up}}{\rightarrow}(a, f(a))$. This is a simple closed curve and it has winds just one time clockwise around $x$. This is easily seen to be the case because it is homotopic to a rectangle by simply projecting the graph of $f$ and of $g$ on sufficiently close but disjoint horizontal lines near $x$. Suppose a wanted isotopy $H((\bullet, \bullet), t)$ swapping "vertically" the graphs of $f, g$ exists.
Consider the image $H(\gamma, t)$ of this loop as an element of the first homology group of the region $\mathbb{R}^2\setminus H(x, t)$.This is always well defined because $H((\bullet, \bullet), t)$ is bijective for each $t$.
Now compare the loop $\gamma$ and $H(\gamma, 1)+x-H(x, 1)$.
These two loops are homotopic (not necessarily in a pointed sense) as loops in $\mathbb{R}^2\setminus x$ (via $H$, or to be more precise via a translated version of $H$ by the vector $x-H(x, t)$).
However, it can be seen that the the loop $H(\gamma, 1)+x-H(x, 1)$ rotates counterclockwise around $x$.
One way to see this is by using Green's Theorem on an appropriate "winding function around $x$". and realizing that the integrals of the arcs on the graphs of $f, g$ get their $y-$position relative to $x$ swapped.
Anyway, the winding number represents one of the $3$ classes that a simple loop can have in the real homology of a punctured plane. In particular this new loop is not homologous to the starting $\gamma$ and so is not even free homotopic to $\gamma$, which gives us the desired contradiction (again, if $H$ existed it would have witnessed such a free homotopy).