- What role the sorting plays.
- How the final area is being calculated without swapping any column as the question demands !
The sorting IS the swapping of columns. If we look at the 3rd row under step 2:
3 3 1 0 0
The sorted row corresponds to swapping the columns so that the column with the highest possible rectangle is placed first, after that comes the column that allows the second highest rectangle and so on. So, in the example there are 2 columns that can form a rectangle of height 3. That makes an area of 3*2=6. If we try to make the rectangle wider the height drops to 1, because there are no columns left that allow a higher rectangle on the 3rd row.
Edit: Avoiding unnecessary sorting
Using a standard O(n*log(n)) sorting algorithm gives us good performance.
It is possible to reduce the sorting needed by avoiding unnecessary sorting. The suggestions that follow won't reduce the O rating, but it will reduce the number of swaps substantially.
To reduce the number of swaps we need to abort the sorting of a row as soon as possible. To be able to do that I recommend using quicksort and always sort the left partition (higher numbers) first. Whenever the pivot times the partition size is smaller than the largest rectangle we have found so far, we know that the right partition (lower numbers) cannot hold the largest rectangle, so we can skip sorting the right partition.
Example:
Assume that the largest rectangle found so far has size 6 and the next row to sort looks like this:
1 3 0 3 0
We take the first element, 1 as pivot. The pivot times the partition size is 1 * 5 = 5, which is less than or equal to the largest size found. This means that we can skip the right partition, since it cannot yield a rectangle larger than 5.
3 3 (keep sorting this partition) - 1 0 0 (skip this partition)
Mergesort only allows us to skip parts of the final merge, so that's why I'd go with quicksort.