I have two functions with three arguments $f_x(a,b,t)$ and $f_y(a,b,t)$. And i need a nesting like this
fx[A,B,0] (*First step*) fx[fx[A,B,dt],fy[A,B,dt],0] (*Second step*) fx[fx[fx[A,B,dt],fy[A,B,dt],dt],fy[fx[A,B,dt],fy[A,B,dt],dt],0] (*Third step*) Thus, with every step, $A$ is replaced by $f_x(A,B,dt)$ and $B$ by $f_y(A,B,dt)$.
How can I implement this in Mathematica?
Almost the same wuyingdgg's answer but works for general parameters.
g[fx[a_, b_, c_]] := fx[a, b, c] /. {a -> fx[a, b, dt], b -> fy[a, b, dt]} With[{n = 2}, Nest[g, fx[A, B, 0], n]] fx[fx[fx[A, B, dt], fy[A, B, dt], dt], fy[fx[A, B, dt], fy[A, B, dt], dt], 0]
Should it be desired to have control over dt as well, then
g[fx[a_, b_, dt_]] := fx[a, b, dt] /. {a -> fx[a, b, dt], b -> fy[a, b, dt]} ReplacePart[With[{n = 2}, Nest[g, fx[x, y, .1], n]], -1 -> 0], n]], -1 -> 0] fx[fx[fx[x, y, 0.1], fy[x, y, 0.1], 0.1], fy[fx[x, y, 0.1], fy[x, y, 0.1], 0.1], 0]
You can definite a function by Replace, like this
f[x_] := Replace[x, {A -> fx[A, B, dt], B -> fy[A, B, dt]}, -1] And then it is easy to do that by Nest or NestList 
I think your operation is being obfuscated by your thinking of and describing it in reverse, and by the 0 in the outermost head.
Nest repeatedly adds a given head to the outside of an expression; it does not replace the inside of the expression with (some portion of) itself. This may sound like two ways to express the same thing but the latter requires additional control of level. Without that control you not only needlessly traverse the entire expression at each step, you risk replacing the wrong parts of the expression if they happen to match the pattern you are using.
Instead I believe you should be using something like this:
fx[##, 0] & @@ Nest[Through @ {fx, fy}[##, dt] & @@ # &, {A, B}, 2] fx[fx[fx[A, B, dt], fy[A, B, dt], dt], fy[fx[A, B, dt], fy[A, B, dt], dt], 0]
The outer level of the expression does not match the rest, therefore it is applied after the Nest using fx[##, 0] & @@
This method is quite flexible; for example you can have a different number of arguments or functions:
multiNest[fns_List, expr_List, n_Integer, {extra___}] := Nest[Through @ fns[##, extra] & @@ # &, expr, n] Test:
multiNest[{x, y, z}, {Q, R}, 2, {foo, bar, baz}] {x[x[Q, R, foo, bar, baz], y[Q, R, foo, bar, baz], z[Q, R, foo, bar, baz], foo, bar, baz], y[x[Q, R, foo, bar, baz], y[Q, R, foo, bar, baz], z[Q, R, foo, bar, baz], foo, bar, baz], z[x[Q, R, foo, bar, baz], y[Q, R, foo, bar, baz], z[Q, R, foo, bar, baz], foo, bar, baz]}
multiNest[{f, g}, {1, 2, 3}, 3, {z}] {f[f[f[1, 2, 3, z], g[1, 2, 3, z], z], g[f[1, 2, 3, z], g[1, 2, 3, z], z], z], g[f[f[1, 2, 3, z], g[1, 2, 3, z], z], g[f[1, 2, 3, z], g[1, 2, 3, z], z], z]}
And for the original example:
multiNest[{fx, fy}, {A, B}, 2, {dt}] fx[##, 0] & @@ % {fx[fx[A, B, dt], fy[A, B, dt], dt], fy[fx[A, B, dt], fy[A, B, dt], dt]} fx[fx[fx[A, B, dt], fy[A, B, dt], dt], fy[fx[A, B, dt], fy[A, B, dt], dt], 0]
If for whatever reason you really do want to perform the nest operation in reverse you could use this:
Fold[ Replace[#, h_[a_, b_, c_] :> h[fx[a, b, dt], fy[a, b, dt], c], {#2}] &, fx[A, B, 0], Range[0, 1] ] fx[fx[fx[A, B, dt], fy[A, B, dt], dt], fy[fx[A, B, dt], fy[A, B, dt], dt], 0]
The length of the Range determines the depth of the nesting.
