View the border hexagon as a projection of box, so that the hexagonal tiles appear to represent somewhat-lumpy spheres stacked in layers.
How does the number of visible spheres (that is, the number of tiles) compare to the area of the three visible faces of the box?
(Image adapted from the figure given on the Codeforces web site.)
To answer your personal questions: I solved this instantly, once I adopted the proper (ahem) perspective; at that point, "the mathematics behind solving" this particular problem became almost-trivial. My advice about such problems in general is to seek out similar insights; reading contest books ---and anything by Martin Gardner--- can help in building your intuition.
By the way, I can't take too much credit for the box insight. Years ago, I came across a fairly classic puzzle about how many unit double-equilateral-triangle rhombi fit into one of these elongated hexagons; the answer becomes clear upon coloring the rhombi based upon which direction (out of three) the long diagonal points: the coloring immediately creates an illusion of stacked cubes, with each color corresponding to the direction in which each cube face ... um ... faces (up, forward, sideways). Ever since then, I've tended to see projected cubes whenever I look at hexagons ... which underscores my point that exposure to mathematical puzzles builds intuition.