A solution that uses Sow and Reap
(a)
Generate a list of the number of values in each required sublist, which can be used by TakeList:
Update
Some slight tweaks of the original posted method (g[list_,limit_]) gives a significantly faster method (but still not as fast as other methods)
takeValues[list_, limit_]:= Module[{i=0,j=Nothing}, (list[[1]]>=limit) || (j=0); (* Prevent '0' at output[[1]] *) Reap[ Scan[ If[(i+=#)<limit, j+=1, i=#; Sow[j]; j=1]&, list ], _, Join[#2,{j}]& ][[2]]//Catenate ]
For alist:
takeValues[alist,20] (* {6, 3, 3, 6, 2} *) TakeList[#,takeValues[#,20]]&@alist (* {{2, 5, 1, 8, 1, 1}, {9, 7, 1}, {5, 2, 9}, {6, 2, 2, 2, 4, 3}, {2, 7}} *) TakeList[#,takeValues[#,10]]&@alist (* {{2, 5, 1}, {8, 1}, {1}, {9}, {7, 1}, {5, 2}, {9}, {6, 2}, {2, 2, 4}, {3, 2}, {7}} *)
(b)
The same as above, but with TakeList as part of the final argument to Reap:
splitLessThan[list_, limit_]:= Module[{i=0,j=Nothing}, (list[[1]]>=limit) || (j=0); (*Prevent '0' at #2[[1]]*) Reap[ Scan[ If[(i+=#)<limit, j+=1, i=#; Sow[j]; j=1]&, list ], _, TakeList[list,Join[#2,{j}]]& ][[2]]//Catenate ]
Example
splitLessThan[alist,20] (* {{2, 5, 1, 8, 1, 1}, {9, 7, 1}, {5, 2, 9}, {6, 2, 2, 2, 4, 3}, {2, 7}} *)
Timing
testlist=RandomInteger[{1, 10}, 1000000]; TakeList[testlist, takeValues[testlist,20]];//AbsoluteTiming (* {2.46277, Null} *)
Takelist adds very little 'overhead':
takeValues[testlist,20];//AbsoluteTiming (* {2.25093, Null} *)
Combined function:
splitLessThan[testlist,20];//AbsoluteTiming//OutputForm (* {2.37236, Null} *)
A comparison with the neat method given by Ben Izd
(Module[{i = 0}, Split[testlist, ((i += #1) + #2 < 20) || (i = 0) &] ]);//AbsoluteTiming//OutputForm (* {2.36251, Null} *)
However, using FoldPairList with TakeDrop is much slower.
Compare:
testlistSmall=RandomInteger[{1, 10}, 100000]; FoldPairList[TakeDrop, testlistSmall, takeValues[testlistSmall,23]];//AbsoluteTiming (* {1.01953, Null} *) TakeList[testlistSmall, takeValues[testlistSmall,21]];//AbsoluteTiming (* {0.275854, Null} *)
A comparison with the originally posted method:
TakeList[testlist, g[testlist,20]];//AbsoluteTiming (* {2.9718, Null} *)
Originally posted function:
g[list_, limit_]:=Module[{i=0,j=0}, Reap[ Scan[If[i+#<limit, i+=#;j+=1, i=#;Sow[j];j=1]&,list], _, DeleteCases[Join[#2,{j}],0]& ][[2]]//Catenate ]
Splitby @BenIzd for being a good balance between fast, short and simple. $\endgroup$