asked on December 9, 2020
Hello,
We have found a problem with the data display when the "Dataset is too large" error happens on expandAllData function call.
We are getting the following message: "Dataset is too large. Some fields cannot be expanded. Please narrow down the dataset."
The problem here is that the part of the dataset is missing in this case, that is not very clear from the message above. There are only successfully expanded rows displayed, but all the failed to expand rows are missing. At the same time, the grand totals are the same before and after the failed expandAllData action.
Is there any chance we can have more consistent pivot behavior here?
Another problem is that there is a significant difference between the datasets size for this error to happen across Google Chrome and Firefox browsers: in FF we can see this error for multidimension dataset with 30K rows (report_01.csv dataset example is attached) vs in Chrome it happens only for way bigger datasets (400K rows).
Attaching the flexmonster config along with dataset csv file to reproduce the expandAllData error in FF for a relatively small dataset with 30K rows. Could you please check if there is anything can be done to fix this, so this functionality works the same way it does in Google Chrome?
Flexmonster version: 2.8.21;
Firefox version: 83.0;
Google Chrome version: 87.0.4280.88;
Thanks in advance,
Dmitriy
⋅ December 11, 2020
Hi Dmitriy,
Thank you for posting your question.
Please note that this warning is triggered when multiple expanding operations are likely to overload the user's browser.
While Flexmonster does not impose any programmatical limitations on the size of the data set and the number of expands, the component still uses the resources available to the client’s browser to perform calculations. While we're at it, it is also worth mentioning that different browser might manage their resources differently. This explains the difference in acceptable data set sizes between Chrome and Firefox that you've mentioned.
Expansion of cells is a heavy operation, sometimes requiring many browser resources. When performed over large data sets, it can lead to exceptions and page crashes.
This is what the mentioned pop-up is for – to notify the user when Flexmonster aborts the operation in order to avoid crashing the page & triggering exceptions. Naturally, this also explains why some of the fields have been left unexpanded.
With that in mind, we would suggest reducing the number of expands in the report.
Please let us know if this helps or if there is still anything else we can help you with.
Best regards,
Mykhailo
⋅ December 14, 2020
Hello Mykhailo,
Thanks for the details!
The problem with the mentioned pop-up is that it says that some fields cannot be expanded.
But what happens is that some data is missing.
For example, in case we have a dataset with 10 collapsed rows by default (Level1Dimension1...10 and aggregated Metric1..10 values):
Level1Dimension1, Metric1
Level1Dimension2, Metric2
...
Level1Dimension10, Metric10
and when we try to expand all data, we get:
Level1Dimension1, Level2Dimension11, Metric11 (successfully expanded)
Level1Dimension2, Level2Dimension11, Metric12 (successfully expanded)
...
-Level1Dimension10, Metric10- (not displayed any more even in the collapsed form, not like the fields are left unexpanded)
We understand the problem with limited browser resources, but the concern is to have clear pivot behavior in case of exceptions like this, so it doesn't lead to a false data analysis assumptions.
Is there any chance we can have the fields that failed to expand still displayed unexpanded?
Thank you,
Dmitriy
⋅ December 15, 2020
Hi Dmitriy,
Thank you for providing an additional explanation of the issue you've experienced.
We have now managed to reproduce the described behavior and can confirm it is indeed an issue. Our team will take a deeper look at this and we'll return to you with a fix on ETA Jan 11th.
Please let us know if there is anything else we can help you with in the meantime.
Kind regards,
Mykhailo
⋅ January 12, 2021
Hi Dmitry,
Thank you for giving us time to look into the issue in question.
We're glad to inform you that the issue with expanding the data has been fixed – all unexpanded members are still available even if the expanding process is terminated.
This is available in the latest minor release of Flexmonster: https://www.flexmonster.com/release-notes/. You are welcome to update the component.
Also, we've done some tests, and it seems like Firefox is the only popular browser that is incapable of executing the mentioned expansion with your data set (due to the resources allocation we've mentioned earlier).
After thorough research on our side, we've concluded that to optimize the expanding process in a way that it executes within similar time limits regardless of the browser would require significant modifications to the expandAllData() algorithm and overall pivot table architecture. Therefore, we cannot provide you with a precise ETA for this, but the issue has been added to our backlog and will be reviewed once our team can put enough effort into it.
In the meantime, we would suggest considering the expandExecutionTimeout option, which lets you specify the execution timeout for expandAllData(). It is set to 9000 milliseconds by default – you can increase this limit so that Firefox has enough time to fully execute the expansion (note that a large execution timeout can cause an unresponsive script alert).
We hope you find the information above helpful. Please let us know if there is anything else we can help you with.
Best regards,
Mykhailo
⋅ January 13, 2021
Hello Mykhailo,
Thank you for fixing the problem and support.
We will be planning checks and changes within the next two weeks.
Best regards,
Dmitriy
⋅ January 19, 2021
Hello,
Taking over from Dima. We can confirm that the issue with some data not being displayed after the "Some fields cannot be expanded" was indeed solved.
But, we're having some trouble understanding expandExecutionTimeout and the behavior around it.
From what we managed to test, either with this specified or not, we're seeing no difference(in either Chrome or Firefox). After a different amount of time(depending on our tests, for some larger reports 40s on Chrome, about 80s on Firefox), unrelated to the default or what we tried to set (more than before the version update), the "Some fields cannot be expanded" error is shown. With Firefox also displaying the script execution wait/stop warning, in the meantime.
For reference, whether we have the expandExecutionTimeout set, or commented in the below, we're seeing the same behavior.
options: { grid: { type: "classic", showReportFiltersArea: true, showTotals: "off" }, chart: { multipleMeasures: true }, datePattern: "yyyy-MM-dd", showAggregationLabels: false,
expandExecutionTimeout: 9500 }
While playing with the timeout may yield some benefits for some of our users(those more technical that may properly understand what browser setting they may or may not want to change), we would generally like to see that default expandExecutionTimeout value in effect, which we are not currently seeing.
Any idea why the option does not seem to have any effect, not even the default if it's not even mentioned?
Thank you,
Mihaita
⋅ January 20, 2021
Hi Mihaita,
Thank you for your response - it is nice to meet you!
Please note that the expandExecutionTimeout option defines the maximum time limit within which the pivot table tries to execute the expansion (by default equal to 9000 ms, that is 9 seconds). If it doesn't manage to expand all the nodes within this limit, the execution is aborted, and the "Some fields cannot be expanded" pop-up is displayed.
That being said, changing the expandExecutionTimeout value from 9000 to only 9500 (therefore increasing the timeout by half a second) won't make a big difference with large data sets.
We've done some tests with the report_01.csv data set Dmitry sent us earlier – once the execution timeout is set to 20000 (20 seconds), Firefox is able to expand all the nodes, even though it needs more time than Chrome and the "stop / wait" prompt is displayed by Firefox (you can simply ignore it and wait).
Naturally, with even larger data sets it would make sense to play around with higher expandExecutionTimeout values (e.g., 40000, 80000, and beyond) to ensure every browser has enough time to execute the expansion.
Please let us know if you find the information above helpful.
Kind regards,
Mykhailo
⋅ January 20, 2021
Hello,
Bad example on my part.
We didn't test specifically with 9500(other than when trying to see if any value has an effect seeing that default wasn't being applied), the idea is that with the "default" - option not specified, or with any value(40000, 80000, etc.) we see no difference in behavior, or, at least, not according to expected behavior. It does not stop trying to expand after 9s(or whatever value we set for the timeout), but after what seems to be an arbitrary (browser/machine resource related) amount of time.
I just ran a new check on a report we have that we know cannot be fully expanded (Chrome Version 87.0.4280.141 (Official Build) (64-bit)):
This is our concern that we are not actually seeing this option having any effect. So even if we would like to play with it(or tell our tool users to "wait, regardless of browser warnings"), it wouldn't matter as there's no predictability here.
For now we are trying to figure out if there is any point in planning any sort of changes on our side around the expandExecutionTimeout option.
Thank you,
Mihaita
⋅ January 21, 2021
Mihaita,
Thank you for your quick response.
Do you think you could send us the mentioned non-expandable report so that we can test it on our side to understand the issue better?
Looking forward to your reply.
Regards,
Mykhailo
⋅ January 25, 2021
Hello,
As far as I can tell:
For all 3 Chrome scenarios above, it took 30s-31s(this will depend on the client resources, the thing to note is - longer than expected) to see the "Dataset is too large" error.
See attached archive(csv had 25MB) for testing. I am pretty sure I could reproduce this with a smaller data set/csv file, but I do not have the time now to play and get a smaller version. To note, yes, it's a large file, but we're not discussing the size of the data, but whether a config setting works or not on Chrome.
Thank you,
Mihaita
⋅ January 27, 2021
Hi Mihaita,
Thank you for sending over the mentioned data set & report.
We'd like to point out that for expansion Flexmonster utilizes something similar to the depth-first search algorithm. When expanding nodes with many depth levels, once the algorithm has run out of its time limit, it might still need some time to terminate execution properly.
This causes the mentioned situation where with different expandExecutionTimeout values, the error is only displayed after about 30 seconds.
Speaking of benchmarks, after some testing of your report on our side, we've managed to have all nodes expanded in about 12 minutes. This was done on a computer with 16 GB of RAM and the CPU Clockspeed of 1.30GHz, with the expandExecutionTimeout value set to 1000000 (one million milliseconds).
We understand that this expansion speed might not be the most efficient – as we've previously mentioned, this is something that we will probably look into in the future.
We hope our explanation above clarifies the situation regarding how the expandExecutionTimeout works.
Regards,
Mykhailo
⋅ January 27, 2021
Hello,
Thanks for the explanation, at least this accounts for the unexpected behavior. We can probably consider this topic closed.
As a side note - one of the biggest inconveniences on our side (not necessarily taking into account the kind of large report I sent) is that when we are not dealing with large strings, Flexmonster generally acquits itself decently of expanding the data, on the exact same data minus the large string column(same number of rows and data, as the large string column is just additional info).
This is a general problem when explaining the situation to our tool customers(most of them non technical). For now, we will start looking into alternative methods of gradual expansion, and likely into custom exports (as the WYSIWYG makes it problematic if they need all the data exported when expand all doesn't fully work).
Thank you,
Mihaita
⋅ January 28, 2021
Mihaita,
Thank you for sharing your feedback on this topic.
We hope you manage to achieve the desired report configuration that would suit your users best.
As always, please let us know in case there is anything else we can help you with.
Kind regards,
Mykhailo
