asked on September 16, 2019
I have to load FM pivot grid calling a Web API service.
The Web API service call will return 1 M rows of data with 20 columns.
How much time will the FM pivot take to load this data.
What if my client have less RAM or processor will the FM works good at that point also?
will loading time gets impact with the clients computers RAM or processor etc..
⋅ September 17, 2019
Hello, Kranthi,
Thank you for your questions.
Being a client-side component, Flexmonster relies on resources available to the browser, and this affects the loading time and the maximum size of the data that can be handled on every particular computer.
This means that the client computer's RAM determines how much data can be loaded at once and CPU capabilities affect how much time is spent on the data analyzation.
We can't say how much time it will take to load the data because loading time depends on the capabilities of each client's machine.
Our team kindly recommends preparing some sample data and running some tests on your end.
We would like to point out that one million rows and 20 columns may be too much for some weak client computers if loaded into the browser at once.
For this reason, we recommend considering moving some calculations on the server-side. One of the possible approaches is to use Elasticsearch data source.
We would like to provide some explanations regarding the difference between Elasticsearch and other data sources (for example, JSON).
The key point is when the browser loads data:
In the case of JSON, you have to load all data before component starts displaying any report.
In the case of Elasticsearch, Flexmonster can handle much larger data because not all data is loaded into the browser at once. You have the ability to load just part of the data necessary for the specified report.
An important remark is that only the part of the data needed for the current representation in Flexmonster is sent to the client side when using Elasticsearch.
Every time more data needs to be shown, a query is formed and a different set of data is sent to Flexmonster.
This way memory is used in moderation.
Please let us know if this was helpful and if you have further questions.
Best Regards,
Vera
⋅ September 24, 2019
Hi Vera,
Thank you for your response.
I have to show grouping on my data for example.
I have to group by an ID and when I expand ID, all the corresponding data related to that ID will be shown as tree view in classic view.
But if I use Elasticsearch as suggested I may lost the data related to that ID, as we are getting only in chunks with Elasticsearch.
And my data only comes as JSON form WebAPI service.
Can we compress JSON data which we are returning from service?
Thanks,
Kranthi
⋅ September 24, 2019
Hello, Kranthi,
Thank you for your response.
Please see our answers below:
1) First of all, we would like to explain that the data related to a specific ID will not be lost in case of using Elasticsearch.
The result of expanding data (in your case, the ID field) is the same for both JSON and Elasticsearch data sources: all the corresponding data related to the expanded ID will be shown.
The only difference is that on expand Flexmonster sends a request to Elasticsearch for getting the data needed for performing the expand.
Using Elasticsearch enables working with large data sets because a new query is formed on every change in the report (for example, expand, filter, etc). This means that only the part of the data needed for the requested representation (expand, filter, etc) is loaded at a time, therefore, the browser's memory is used in moderation.
Please kindly note that in case of JSON, no query is formed, because the entire data source was already loaded into the browser at the beginning. This means that in case of a large data source, certain machines might not handle that much data at once.
2) As an alternative to the Elasticsearch solution, we recommend using our custom data source API approach. Currently, it is in the beta version available on GitHub: https://github.com/flexmonster/api-data-source .
It allows implementing your own data aggregation service based on your existing infrastructure. Such an approach allows sending already aggregated data to the Flexmonster. Custom data source API gives full control for the data management and provides the widest variety of optimization options.
3) We would like to explain that compressing JSON data may improve data transfer from the server-side to Flexmoster but it will not resolve the problem with the data set size. The reason for this is that the compressed JSON data will still need to be extracted when loaded into Flexmonster. Therefore in case of compression, the data size which is loaded at once into the browser will not be decreased.
Please let us know if you have any questions.
Best Regards,
Vera
⋅ March 18, 2020
Hello, Kranthi,
We are glad to inform you that the custom data source API has been released in the new major release version 2.8 of Flexmonster.
This approach greatly improves the performance and provides full control over how the data is processed. Our team highly recommends considering the Custom Data Source API in case a large dataset is being used.
Please see the following guides for more information:
1) What the Custom Data Source API has to offer.
2) Introduction to the Custom Data Source API.
3) A sample implementation of the Custom Data Source API approach with Node.js.
4) A sample implementation of the Custom Data Source API approach with .Net Core.
5) How to implement your own implementation of the Custom Data Source API approach.
Please feel free to reach out to us if any questions arise.
Kind regards,
Vera
