Problem loading 1 million rows

Dear support team,
 
One of the reasons that convinced us to buy flexmonster where the demos where it promissed to load a large amount of data. However, we are not able to do so. For example, using chrome, it crashes while analysing the information that was retrieved from the database.
 
Please advice in how to handle this situation. since it is causing so much internal noise the fact that we cannot achieve what was promissed initially regarding flexmonster.
 
Kindest regards,
Andre Pinheiro
 

Attachments:
error.png

19 answers

Public
Andre Pinheiro December 10, 2018

Some additional backgroud:
 

  • It crashes after it has loaded the data (Accoding to the on-screen load status), while analysing the data
  • It crashes aways while analysing the 300.000 th row
  • I am loading json data
Public
Andre Pinheiro December 10, 2018

Also, it is taking forever to load the data in Microsoft Edge, while in chrome it is very fast.
 
Sorry if the information is very vague, but I am trying to give as much information as possible so you can have a broad view of the scenario. Fell free to ask for any specific information.
 
Kindest regards,
Andre Pinheiro

Public
Tanya Gryshko Flexmonster December 10, 2018

Hello, Andre,

Thank you for providing all the details.

We would like to mention that Flexmonster relies on resources available to the browser, and this affects the maximum size of the data that can be handled on every particular computer. However, there are some techniques which proved to improve the situation and allowed our clients to load more data.

Could you please share more details about your case? This would help us in suggesting the best approach. The following points have an impact:

  1. How your JSON data is getting to the browser?
  2. Do you use Flexmonster Data Compressor? It reduces the size of the response from the server.
  3. When the browser crashes while analyzing the 300.000-th row, is it happening only for 1000.000 records of data?
  4. Could you please consider sharing your sample data? (any sensitive information can be sent via e-mail)

Also, I would like to inform you that we have released a beta version of Flexmonster Pivot for Elasticsearch. This data source allows loading just the data necessary for each report. As a result, it is possible to handle much larger data because not all data is loaded into the browser at once. 

Waiting for your reply.
Regards,
Tanya

Public
Andre Pinheiro December 10, 2018

Hi Tanya,
thank you very much for your prompt response.
Yes, we are aware that it is not flexmonster that is crashing, but rather the browser. However, it is a little bit difficult to understand why it is failling with an amount of data that supposely should not be a problem, as per our understanding.

– How your JSON data is getting to the browser?
we have a web service that executes a SQL query, translate it into json and sends the response to flexmonster
– When the browser crashes while analyzing the 300.000-th row, is it happening only for 1.000.000 records of data?
When the number of records is above 300.000, it crashes
– Do you use Flexmonster Data Compressor? It reduces the size of the response from the server.
No, since based on some previous tests it seemd to be more slow than the approach we are using now. Going to execute a new set of tests using the data compressor approach again, and get back to you with the results
– Could you please consider sharing your sample data? (any sensitive information can be sent via e-mail)
Sure, no problem. Going to export it and attach to this ticket
– Also, I would like to inform you that we have released a beta version of Flexmonster Pivot for Elasticsearch. This data source allows loading just the data necessary for each report. As a result, it is possible to handle much larger data because not all data is loaded into the browser at once.
Thanks. If everything else fails, we will take a look into it.

Public
Andre Pinheiro December 11, 2018

Hi Tanya,
 
I was not able to attach the file here, since it’s size is around 27 mega (zipped). Are there any alternatives for me to send you the file?
 
Thanks,
Andre 

Public
Andre Pinheiro December 11, 2018

If it works for you, I can publish it on our sharepoint site and send you the link.

Public
Andre Pinheiro December 11, 2018

Hi Tanya,
 
some more feedback:
– Do you use Flexmonster Data Compressor? It reduces the size of the response from the server.
I’ve tested it, and it also fails, unfortunatelly.  

Public
Tanya Gryshko Flexmonster December 11, 2018

Hello, Andre,
Thank you for your quick reply.
Yes, the link to your SharePoint site with the file will work. Please share it so that we can test your data on our side. That would allow us to come up with more detailed suggestions. 
Regards,
Tanya

Public
Andre Pinheiro December 11, 2018

Hi Tanya,
 
thank you very much. Here is the link for the file:
 
https://vmetrix.sharepoint.com/:u:/g/EQIGPF4vpmxFgkTwUrAxpmgBZhfIYZw2jkJezA_ZdS_6og
 
Kindest regards,
Andre Pinheiro

Public
Joffroy Christen December 11, 2018

Hello,
We also have a crash issue with large csv file (316629 rows, 240MB). Here are the details if they can help:
 – Google Chrome 64 bits on a 32 GB ram computer (so more than enough memory)
 – Chrome crashes while “Loading data…” after having loaded about 220MB
 – If developers tools are open, it pauses before crash with message “Paused before potential out-of-memory crash”
 – Sometimes Chrome succeed loading the data (exactly same data) without crashing, but not often.
 – Firefox 64 bits does not crash (and it loads data faster)

Public
Iryna Kulchytska Flexmonster December 12, 2018

Hi Andre,
 
Thank you for providing the JSON file and all the details.
 
We analyzed carefully the JSON file and what is going on in the component when it loads the file. What we see is that the JSON has a lot of unique members and it is simply big (256 MB). We need to mention that it is too big to be loaded to the component entirely.
 
Also, taking into account that you are familiar with our best practices for performance optimization https://www.flexmonster.com/question/understanding-flexmonster-restrictions/ and you tried all of them, the option we would recommend you to consider is Elasticsearch.
 
Recently we have released a beta version of Flexmonster that supports it.
It doesn’t require loading entire dataset and requests data by composing exact queries.
You can find more details on this in our blog post and download this version of the component from our website.
 

Do you think it is an option for you to connect your data source to Elasticsearch?

 
Looking forward to your reply.
 
Kind regards,
Iryna

Public
Iryna Kulchytska Flexmonster December 12, 2018

Hi Joffroy,
 
Thank you very much for the details of your case.
 
My previous answer in this thread could be interesting for you as well.
 
Kind regards,
Iryna

Public
Andre Pinheiro December 12, 2018

Iryna,
 
Thanks for the reply. 
Unfortunately, the given answer is not acceptable. One of the key selling points of flexmonster is being able to load large amounts of data, including having an example of loading 1 million of records in your website, and also I don’t see any size limits described anywhere (maybe I have missed it somewhere?).
Regarding the suggestion:

  • I am not very familiar with elastisearch, but the main feature that we need from flexmonster is its pivot table caracteristics. Elastisearch would retrieve the data by demand (please correct me if I am wrong), and for this reason I cannot see how the pivot table feature would work
  • Even if this suggestion is implemented, in a situation where some user loads the complete data, flexmonster would crash as well, wouldn’t it?
  • As stated before, We do not have much knowledge using elasticsearch, and adding an additional component to the project is not a feasible solution at this point, since it would involve additional development and even possible licenses costs.

 
Kind regards,
Andre Pinheiro

Public
Dmytro Zvazhii Flexmonster December 18, 2018

Hello Andre,
Thank you for your feedback.
We were always interested in increasing the capabilities of our component. Currently, our developers are working on adding some possible optimizations to make your JSON work. 
 
At the same time, we would like to pay your attention to the fact that there is always the browser’s resources limitation and your dataset is very close to it. In the modern world, it is only a matter of time for the data to keep growing. When you load a huge dataset into the browser’s memory it always has the risk to be crashed. Consistent data growth is a way for a consistent browser’s crashing.
 
The usage of modern data storages is the way to be prepared for such sort of situations. Our team has certain expertise when it comes to working with massive datasets. There is a number of our clients that have outgrown the plain data sources such as CSV and JSON. At the same time, their requirements for the performance and loading speed remained at the same level. Such sort of challenges leads to the total change of the paradigm. All of that can be summarized to the following idea: “When the browser can no longer handle the whole data set it is time to delegate some calculations on the server side”. That is the reason why we support such things as Mondrian, SSAS and Elasticsearch. All of them are powerful data engines which allow making some pre-calculations on the server side. Such an approach allows Flexmonster to obtain already aggregated data and successfully displaying it. In such case, the component stores fewer data into the browser’s memory, since it is already aggregated, but at the same time, it can show significantly more. 
The main benefit here is that the user experience remains the same. Our key pivoting features such as filtering, expanding, drilling etc… remains the same for the end user. The only difference is that the user keeps working with the massive dataset in real time the same way he worked with a much smaller one.

Elasticsearch is a free and open-source solution for structured and unstructured data with a huge community. It has good potential for scaling depending on your data needs. We recommend you Elasticsearch since it is one of the most promising storages from all of our supported data sources. All of the pivoting features are available in the real-time. You are welcome to try it in our live demo: http://www.flexmonster.com/demos/connect-elasticsearch/. Also here is the example with the much bigger data source which has almost 500K unique members: http://jsfiddle.net/flexmonster/pq7cu9rs/.
Please note that you can always rely on our assistance in case you decide to switch the data source.
Let us know in case of any other question.
Regards,
Dmytro

Public
Andre Pinheiro December 26, 2018

Hello Dmytro,
 
thanks for your detailed answer. Please find my comments in bold:
 
We were always interested in increasing the capabilities of our component. Currently, our developers are working on adding some possible optimizations to make your JSON work. 
 
Thanks. However, we are also able to reproduce it using Flexmonster’s OCVS format, thus not being a isolated JSON “issue”. If you wish I can send the same JSON content in OCVS format.

At the same time, we would like to pay your attention to the fact that there is always the browser’s resources limitation and your dataset is very close to it. In the modern world, it is only a matter of time for the data to keep growing. When you load a huge dataset into the browser’s memory it always has the risk to be crashed. Consistent data growth is a way for a consistent browser’s crashing.
I understand that. However, based in one of the main flexmonster selling points, we grew our espectatives around the 1 million number, even adding a message to the end user if he tries to load more than this amount of data. So far we are only able to load around 300.000 rows with around 10 fields with values, far away from the promissed 1 million…

 
The usage of modern data storages is the way to be prepared for such sort of situations. Our team has certain expertise when it comes to working with massive datasets. There is a number of our clients that have outgrown the plain data sources such as CSV and JSON. At the same time, their requirements for the performance and loading speed remained at the same level. Such sort of challenges leads to the total change of the paradigm. All of that can be summarized to the following idea: “When the browser can no longer handle the whole data set it is time to delegate some calculations on the server side”. That is the reason why we support such things as Mondrian, SSAS and Elasticsearch. All of them are powerful data engines which allow making some pre-calculations on the server side. Such an approach allows Flexmonster to obtain already aggregated data and successfully displaying it. In such case, the component stores fewer data into the browser’s memory, since it is already aggregated, but at the same time, it can show significantly more. 
The main benefit here is that the user experience remains the same. Our key pivoting features such as filtering, expanding, drilling etc… remains the same for the end user. The only difference is that the user keeps working with the massive dataset in real time the same way he worked with a much smaller one.
But if I delegate it to the server, I would take away from the user the hability to set the aggregation by itself, wouldn’t I? I would have to have another “tool” or graphical interface for the user to create specific aggregation queries. And it also would not make sure that the memory problem would not occur, since I cannot be sure about the amount of data that the user can load without crashing the browser.

Elasticsearch is a free and open-source solution for structured and unstructured data with a huge community. It has good potential for scaling depending on your data needs. We recommend you Elasticsearch since it is one of the most promising storages from all of our supported data sources. All of the pivoting features are available in the real-time. You are welcome to try it in our live demo: http://www.flexmonster.com/demos/connect-elasticsearch/. Also here is the example with the much bigger data source which has almost 500K unique members: http://jsfiddle.net/flexmonster/pq7cu9rs/.
Thanks. But the example have only two fields, pre-aggregated. It agains gives a not accurate feeling that I would be able to load 500.000 rows with more columns. The same occurred with the One Million example. If you take a closer look, it only loads this amount of rows becase not all columns have values.
 
How can I calculate the client machines specs in order to be sure about the amount of data can I load at once, without crashing?
 

Please note that you can always rely on our assistance in case you decide to switch the data source.
Let us know in case of any other question.
Regards,
Dmytro

Public
Dmytro Zvazhii Flexmonster December 28, 2018

Hi Andre!
First of all, I’d like to update you on our development progress. We already have proof of concept that works with your data sample. However, right now it’s in the very unstable state so guys need some time to stabilize it, run all necessary tests and check performance indicators. I hope we will show you prerelease version next week.
Regarding Elasticsearch we already nave a necessary server-side tool which makes all calculations. It’s easy to use and will not take a lot of time to set up.
Regards,
  Dmytro

Public
Andre Pinheiro January 3, 2019

Hello Dmytro,
 
first of all, happy new year!
 
Thanks for the update. Looks very promissing. Looking forward to this.
 
Regarding elastisearch, I will take a look at it more carefully. Thanks.
 
Kindest regards,
Andre Pinheiro

Public
Dmytro Zvazhii Flexmonster January 15, 2019

Hi Andre!
Here is the improved version of the component:
https://s3.amazonaws.com/flexmonster/vmetrix-international-spa/FLEXMONSTER-PIVOT-2019-01-15.ZIP
Please try this build with your data and keep in mind this is a beta version and might have some issues. 
Regards,
  Dmytro

Public
Andre Pinheiro January 17, 2019

Hello Dmytro,
Thank you and your team for your effort. We are going to perform some tests and revert with results soon.
 
Kindest regards,
Andre

Please login or Register to Submit Answer