Timeline for Memory problems when fitting large (correlated) datasets
Current License: CC BY-SA 4.0
5 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Nov 9, 2023 at 16:15 | comment | added | JimB | With 18 parameters (and based on your recent question), I would bet that your model is linear in the parameters. That means generalized least squares is what you want (as I mentioned in an answer to your most recent question). That is not an iterative approach which means there is an explicit formula for the estimates and the covariance matrix for the estimates. | |
| May 6, 2022 at 6:36 | comment | added | pablo | Actually, the problem is at computing the logL. The inverse covariance matrix is computed without problems indeed. Apparently, Mathematica precomputes the expression for logL (note we have 336^2 long expressions). That's why I tried to append ?NumericQ to prevent evaluation before the model is given a point and produces a number. However, it turns out that fitting this way is terribly time consuming and couldn't produce results with FindMinimum (I don't have your overall - sign, so it is equivalent) or NMinimize. That was funny... Thanks a lot for your help anyway! | |
| May 4, 2022 at 16:01 | comment | added | JimB | If the objective is to get your function and estimation working on your 4GB laptop, then I would suggest determining where things stop working. Taking the inverse of the covariance matrix? If so, then it seems hopeless. If it's at the NonlinearModelFit stage, then it might also be hopeless. But if it's at the last stage (FindMaximum), then maybe some of your equation is linear in some of the parameters which means you can reduce the number of parameters in the FindMaximum. But without knowing the function you're trying to fit, that's only a guess. | |
| May 4, 2022 at 10:15 | comment | added | pablo | Hi, and sorry for my delay. Indeed, that's the way I proceeded (perhaps I was not quite explicit). Essentially, I created your logL function for my data (appart from the minus sign and the factor of 2, that's what I call the chi square). The problem is that, when I create the logL function, my computer runs out of memory and the kernel is restarted. I guess the problem is the length of my function (containing logs, square roots, etc.. and 18 parameters), as doing an example like yours works easily and better computers (I have 4GB ram on my laptop) proceeds easily. | |
| Apr 26, 2022 at 20:46 | history | answered | JimB | CC BY-SA 4.0 |