asked 2 days ago
Hi Flexmonster Support Team,
We are using Flexmonster in flat table mode within an Angular application to display a large dataset with no visualizations.
Here’s our current setup:
The raw JSON (flat array) is approximately 225MB and It is GZIP-compressed to ~15MB and stored in Amazon S3.
new requirement:
The raw JSON (flat array) is approximately 700MB.
It is GZIP-compressed to ~31MB and stored in Amazon S3.
Our implementation:
We generate a presigned URL with Content-Encoding: gzip and Content-Type: application/json.
The file is fetched using Angular's HttpClient, and the browser auto-decompresses it due to the gzip encoding.
We pass the parsed JSON directly to Flexmonster using dataSource
We are not using pagination, data API, or manual decompression libraries like pako. Our goal is to load the entire flat dataset into Flexmonster.
Questions:
Is this approach supported and scalable for ~700MB of raw JSON via dataSource.data in flat table mode?
Are there memory or row-count limits in Flexmonster for flat table rendering?
Any recommendations or feature flags we can disable to improve performance for such large flat datasets?
Would Flexmonster handle this more efficiently with a different data format (e.g., streaming, binary)?
Thank you for your guidance to ensure stability and optimal performance.
⋅ 2 days ago
Hello, Manoj!
Thank you for reaching out to us.
From our experience, the approach you have described follows all best practices for performance optimization that we can recommend.
Please find the answers to your questions below:
1) Is this approach supported and scalable for ~700MB of raw JSON via dataSource.data in flat table mode?
Yes, the described approach is supported in Flexmonster.
Our team also wanted to mention that Flexmonster supports loading JSON files compressed with gzip directly.
We recommend trying the dataSource.filename property instead of dataSource.data to connect JSON file via URL to Flexmonster. You are welcome to check the example: https://jsfiddle.net/flexmonster/4yw50268/.
2) Are there memory or row-count limits in Flexmonster for flat table rendering?
Flexmonster itself doesn't limit the amount of data you can load. Our component is loading the data into the browser's memory, so the amount of data you can load is only limited by your browser's capacity.
3) Any recommendations or feature flags we can disable to improve performance for such large flat datasets?
Please note that all UI features are available for users after the data has already been loaded, so using any of them won't affect the performance.
4) Would Flexmonster handle this more efficiently with a different data format (e.g., streaming, binary)?
For the described case, we don't think that using any other approach, such as streaming or binary, would result in significant improvements in performance.
From what we've seen, using the JSON array of arrays format is the most efficient way to handle such a large amount of data: https://jsfiddle.net/flexmonster/scro38mr/.
Hope you will find our answer helpful.
Kind regards,
Solomiia
⋅ 4 hours ago
Hi Solomiia,
we still need clarity on best practices for high volume data .
we would prefer short technical support call to get guidance on how to proceed effectively
Warm Regards,
Manoj
