Get Free Trial
Get Free Trial

How long does it take to load 100,000 data?

Answered
Donggeon Kim asked on November 28, 2019

How long does it take to load 100,000 data?

As I know, for csv type, 300,000 files are loaded in 21 seconds.
Is this 21 seconds when the csv file is uploaded as is, not a db connection?
How long does it take to load data through db?

I’m also curious about the time when it’s in json format rather than csv type.
As a result of our test, json type took 100,000 minutes to display on the screen, but I wonder if it can be done faster.

1 answer

Public
Vera Didenko Flexmonster November 29, 2019

Hello, 
 
Thank you for reaching out to us.
 
Please see our answers and recommendations below:
 
1) Some details regarding performance:
Being a client-side component, Flexmonster relies on resources available to the browser, and this affects the loading time and the maximum size of the data that can be handled on every particular computer.
This means that the client computer’s RAM determines how much data can be loaded at once and CPU capabilities affect how much time is spent on the data analyzation.
The loading time depends on the capabilities of each client’s machine.
 
2) About loading data from a database:
The loading time depends on your server-side script’s implementation.

Flexmonster does not provide a direct connection to the SQL database. The data needs to be fetched from your database on your server. Next, a server script that returns the fetched data in CSV or JSON format needs to be created.
 
After this, the URL to the created server-side script is passed to Flexmonster, for example: 

var pivot = new Flexmonster({
	container: "pivotContainer",
	toolbar: true,
	report: {
		dataSource: {
			type: "csv",
			/* URL to your server-side script */
			filename: "http://localhost:3000/data"
		}
... }, ... });

This is how Flexmonster gets the required data to analyze and display it. 
 
This means that the total loading time will consist of:
2.1) The time it takes the server script to fetch and prepare the data.
2.2) The time it takes to transfer the data to the client-side.
2.3) The time it takes for data analyzation.
 
3) About your JSON loading results:
Are you sure it took 100 000 minutes to display the results on the screen?
If desired, you could share a sample dataset with us and we could test it on our end as well.
 
4) We would like to point out that depending on the data structure, 300 000 JSON data may be too much for some weak client computers if loaded into the browser at once.  
For this reason, we recommend considering moving some calculations on the server-side. One of the possible approaches is to use MS Analysis Server data source (please see: https://www.flexmonster.com/doc/connecting-to-microsoft-analysis-services/ ).
 
We would like to provide some explanations regarding the difference between MS Analysis Server and other data sources (for example, CSV).
 
The key point is when the browser loads data:
 
In the case of CSV, you have to load all data before the component starts displaying any report.
 
In the case of MS Analysis Server, Flexmonster can handle much larger data because not all data is loaded into the browser at once. You have the ability to load just part of the data necessary for the specified report.
 
An important remark is that only the part of the data needed for the current representation in Flexmonster is sent to the client side when using MS Analysis Server.
Every time more data needs to be shown, a query is formed and a different set of data is sent to Flexmonster.
This way memory is used in moderation.
 
For more information about performance, please see the following threads from our forum:
4.1) https://www.flexmonster.com/question/load-field-data-when-it-is-needed/
4.2) https://www.flexmonster.com/question/how-much-will-the-flex-monster-pivot-grid-takes-to-load-the-data-of-1m-rows-with-20-column
 
 
Please let us know if this was helpful and if you have further questions.
 
Best Regards,
Vera

Please login or Register to Submit Answer