Get Free Trial

Is it possible to optimize json data source size by replacing string labels by ids and providing code lists for labels?

Answered
Mojo asked on December 2, 2021

E.g. instead of data:

[
{
"Category": "Accessories",
"Size": "262 oz",
"Color": "white",
"Destination": "Australia",
"Business Type": "Specialty Bike Shop",
"Country": "Australia",
"Price": 174,
"Quantity": 225,
"Discount": 23
},
{
"Category": "Components",
"Size": "214 oz",
"Color": "yellow",
"Destination": "Canada",
"Business Type": "Specialty Bike Shop",
"Country": "Canada",
"Price": 502,
"Quantity": 90,
"Discount": 17
},
]

the data would be:

[
{
"Category id": 1,
"Size": "262 oz",
"Color id": 1,
"Destination id": 1,
"Business Type id": 1,
"Country id": 1,
"Price": 174,
"Quantity": 225,
"Discount": 23
},
{
"Category id": 2,
"Size": "214 oz",
"Color id": 2,
"Destination id": 2,
"Business Type id": 1,
"Country id": 2,
"Price": 502,
"Quantity": 90,
"Discount": 17
},
]

and following code list to convert ids into labels would be provided:

{
"Category": {
"Accessories": 1,
"Components": 2,
},
"Color": {
"white": 1,
"yellow": 2,
},
"Country and Destination: {
"Australia": 1,
"Canada": 2,
},
"Business Type": {
"Specialty Bike Shop": 1,
}
}

For large datasets it would save not only data size but also the processing time/performance.

Thank you in advance for help.

Mojo

3 answers

Public
Illia Yatsyshyn Illia Yatsyshyn Flexmonster December 2, 2021

Hello, Mojo,

Thank you for raising this support ticket.
 
The described JSON format is not supported by Flexmonster. However, you can significantly reduce the data set size using an array of arrays instead of an array of objects. This format is natively supported by Flexmonster and does not require any additional tables with identifiers. It allows remove all the duplicated field names and significantly reduces the final data set size.
 
You can learn more about supported JSON format in our documentation: Connecting to JSON • Flexmonster.
 
Please let us know if it helps.
 
Kind regards,
Illia

Public
Mojo December 2, 2021

Hello Illia,

thank you for answer.

Yes, I’m actually using the JSON array of arrays.
But if it is not supported trivially like prepared JSON format, is it available / fesasible somehow by Flexmonster JS API? This is my question.

E.g.  the mapping type ‘id’ sounds to me like something which could be used for this purpose (maybe I’m wrong, I’m total newbie in Flexmonster). Imagine that you get ids-only-data (as JSON array of arrays) from server and in separate request you get the code lists (as your custom JSON object) to convert ids into labels. Is there some JS API method / event (some ‘beforedisplaylabels’) to replace ids for corresponding labels? Would it be better solution (faster, less memory requring) than the plain all-labels-data?

Thank you in advance,

Mojo

Public
Illia Yatsyshyn Illia Yatsyshyn Flexmonster December 3, 2021

Hello, Mojo,
 
Thank you for your feedback.
 
Please note that the id type can only be used once for each record in Flexmonster. It is not possible to use it as an identifier for each field present in the record. Flexmonster does not provide the possibility to preprocess data before displaying it as well.
Alternatively, you can load the data and preprocess it in the desired way before passing it to Flexmonster. It implies implementing a custom JSON loader that would accept the dataset with ids and the corresponding associative table on the client. Next, process this data to construct one of the formats supported by Flexmonster. This approach would allow reducing the load on the network but may require some additional processing time in the client’s browser.
 
Finally, we suggest checking out Flexmonster Data Server for connecting to JSON files. This server communicates with the client using the custom data source API – our custom communication protocol allowing you to retrieve already aggregated data from a server to Flexmonster Pivot. Since loading, processing, filtering, and aggregating of the data is made on the server side, Flexmonster Pivot gets the data in a ready-to-show format. Thus, the data is visualized faster, and the browser’s resources are used more efficiently. This approach is especially effective when using pivot forms (compact, classic) since displayed intersections take up significantly less space than raw records would.
 
Please let us know if other questions arise.
 
Regards,
Illia

Please login or Register to Submit Answer