I'm designing a hash function which uses a double-tree construction and a compression function $c(s,A,B,C,a)$ ($s$ and $a$ may be omitted when there's only one of them) where
- $s$ is a set of three key-dependent 8×8 S-boxes
- $A$, $B$, and $C$ are 32-byte blocks; $C$ may be empty, in which case it is omitted
- $a$ is 0, 1, or 2 and tells which S-box to use first.
It is easy
- to compute $H=c(s,A,B,C,a)$ given $s$, $A$, $B$, $C$, and $a$;
- to find some $A$,$B$, and $C$ given $H$, $s$, and $a$.
It should be hard
- to find one or two of $A$, $B$, and $C$ given the rest and $H$;
- to find $A$ and $B$ given $H$ and $H'$ when $H=c(s,A,B,0)$ and $H'=c(s,A,B,1)$ except in the extremely rare case that all three S-boxes are identical;
- to find the blocks in any other case of overlapping compression function inputs;
- to find $s$ given $H$, $A$, $B$, $C$, and $a$.
Linear cryptanalysis is unlikely to be useful because of the large, key-dependent S-boxes. Differential cryptanalysis and (because the S-boxes are permutations) integral cryptanalysis make more sense for this hash, and I've tried differential cryptanalysis on the related cipher. The objective of cryptanalyzing a hash function is to find preimages or collisions, which is different from cryptanalyzing a cipher. How should I attack this compression function?
