How to cut runtimes for life-cycle profiles or 'AllStats' when you don't need all the stats (simoptions.whichstats)

1

You can now use simoptions.whichstats to control which statistics are calculated, and if, e.g., you are just wanting the mean and standard deviation, then this will be much faster
[intended to make it easier to use them in loops/optimization; default remains to compute all the statistics as if you only do it once it is anyway fast enough and is easy to use]

You set: simoptions.whichstats=ones(1,7); % by default, compute all stats
% zero values in this optional input are used to skip some stats and thereby cut runtimes
% 1st element: mean
% 2nd element: median
% 3rd element: std dev and variance
% 4th element: lorenz curve and gini coefficient
% 5th element: min/max
% 6th element: quantiles
% 7th element: More Inequality

Then just call the standard life-cycle profile or AllStats commands, and they will only compute the moments you want. If you are unsure what these all are, just run the command with default and you can look at the output structure to see what they refer to

So for example, set simoptions.whichstats=[1,0,1,0,0,0,0] and you will just get the mean and standard deviation (and variance). The runtime will be much faster as things like the lorenz curve and quantiles take most of the runtime.

For the 4th and 6th stats (Gini+Lorenz and Quantiles) you can set a value of 2, which uses a faster but more memory-intensive way to compute them.

The defaults internally are:
simoptions.whichstats=[1,1,1,1,1,1,1] for AllStats
simoptions.whichstats=[1,1,1,2,1,2,1] for LifeCycleProfiles
(4th entry is Lorenz+Gini, 6th entry is Quantiles)

1 Like
2

That’s great, thanks Robert! On my laptop the running time for life-cycle profiles went down from 55 secs to 7 secs. I have set

simoptions.whichstats=zeros(1,7);
simoptions.whichstats(1) = 1; %compute only the mean

since I need only the mean. One thing to point out is that the non-requested statistics are still present as fields of the structure AgeConditionalStats.VarName but they are only initialized with NaN’s. This is perfectly fine of course: I write this because at first I thought the function had computed them anyways, then I checked :slight_smile:

AgeConditionalStats.search_effort_m

ans =

struct with fields:

           Mean: [1×612 gpuArray]
         Median: [1×612 gpuArray]
       Variance: [1×612 gpuArray]
   StdDeviation: [1×612 gpuArray]
           Gini: [1×612 gpuArray]
        Minimum: [1×612 gpuArray]
        Maximum: [1×612 gpuArray]
 MoreInequality: [1×1 struct]
    LorenzCurve: [100×612 gpuArray]
QuantileCutoffs: [21×612 gpuArray]
  QuantileMeans: [20×612 gpuArray]
3

Cleaned this up (I believe).

4

Correct, now it creates only the required statistics

AgeConditionalStats.search_effort_m

ans =

struct with fields:

Mean: [1×612 gpuArray]
1 Like
5

There is now a simoptions.whichstats(6)=2 which makes calculating the quantiles faster.

Does exactly the same thing as simoptions.whichstats(6)=1, but the code is vectorized rather than loop, so faster. It can cause out of memory errors in larger models with large numbers of quantiles, hence is not the default which remains simoptions.whichstats(6)=1.

Note: simoptions.whichstats(6) controls the decision to calculate quantiles. So now setting it to zero skips calculating quantiles, setting it to one calculates quantiles, setting it to two calculates quantiles faster but using more memory.

1 Like