There are several issues. First of all Context[] is the same here and can be replaced with $Context to make the example clearer. Now:
DumpSave is HoldRest so in case of symbol = 2 it can save that definition instead of seeing 2.
DumpSave["file", symbol] will save defnitions of symbol. So will DumpSave["file", $Context]. It will save information that $Context = whateverthecontextthereis.
If you want to DumpSave["file", "Global`"] you need to explicitly write it or inject it before DumpSave is evaluated. Those are most common ways to do this:
Function[x, DumpSave["file", x]] @ $Context DumpSave["file", #]& @ $Context DumpSave["file", Evaluate @ $Context] With[{ x = $Context }, DumpSave["file", x] ]
In your recent question you had DumpSave["file", { Evaluate @ $Context }]. It didn't work because Evaluate only works on the first level of an expression that has Hold* attribute. This expression here is DumpSave but on the first level it has a string and a list so evaluator does not care about Evaluate.
$Contextdoes not get evaluated becauseDumpSavehas the attributeHoldRest(see here for details).DumpSave["~/Documents/test.mx", Evaluate@$Context]works. $\endgroup$