Despite the COVID-19 outbreak, our team continues operating at full speed. We are always here to support and answer all your questions.

Feel free to reach out by filling this quick form.

Fill the form
Get Free Trial

How to improve connection performance when loading large amount of records via elasticsearch?

Answered
Francisco Jose Gomez Vargas asked on April 19, 2021

Im currently using a elasticsearch connection to retrieve the data for my flexmonster integration, the problem is, that i got to much records in my elasticsearch and the request takes a good chunk of time on delivering the information, there is any way of improving this requests? currently gettin 200,000k records just for 1 field.

7 answers

Public
Vera Didenko Vera Didenko Flexmonster April 20, 2021

Hello, Francisco,
 
Thank you for reaching out to us.
 
Our team kindly suggests trying the following:

  1. Prefiltering the data:

    One way to decrease the server’s response is to prefilter the data and return only the part necessary for the specific report.

  2. Using subquery: 

    We would like to mention that the subquery parameter allows setting the server-side filter, which helps decrease the size of the response from the server.
    Please note that for the Elasticsearch data source type, the subquery parameter should be set as a Bool Query Object.

  3. Setting more specific reports:

    Flexmonster provides the possibility to set default report settings, specifying how to display the data initially.
    This way, you can provide the desired starting points for the reports (for example, specifying which fields to display, applying an initial filter to the fields, etc.) and, in return, decrease the size of the response from the server.
    We have a section in our documentation with examples on how to configure the report. Please see https://www.flexmonster.com/doc/configuring-report/.

 
 
Please let us know if this helps.
 
Kind regards, 
Vera

Public
Francisco Jose Gomez Vargas April 20, 2021

Thanks for your reply Vera. Im already using reports to filter the data but the size and performance of the response isn’t changing, for example there is a total of 180,000 companys i need to get, i create a query so i only use 80,000 of these but when its loading, it request the 180,000 companys first and then it filters them. I am doing this wrong? this is my code for the report.
Do i need to use query instead of member?
 
 

Public
bernard dradeou April 20, 2021

Do you use a “subquery” in the datasource ?

Public
Vera Didenko Vera Didenko Flexmonster April 21, 2021

Hello, Francisco,
 
Thank you for your response. 
 
Flexmonster first requests all unique entries of a field to display them in the filter pop-up. Then the query for displaying the specified report is initiated, and the required filters are applied. It seems in your case, there are 180,000 unique company names, and this is why all 180,000 are requested at first. 
 
This can be adjusted by specifying the subquery property in the Data Source object.
Here is an example of how subquery could be used to decrease the server’s response:

report: {
"dataSource": {
"type": "elasticsearch",
"node": "https://olap.flexmonster.com:9200",
"index": "fm-product-sales",
"subquery": {
"bool": {
"filter": [
{
"terms": {
"Category.keyword": ["Bikes", "Accessories"],
}
},
{
"terms": {
"Color.keyword": ["blue", "red"]
}
}
]
}
},
"mapping": {...}
},
"slice": {...}
}

As a result, only the specified members will be loaded as opposed to loading all of them.
We have prepared a JSFiddle example for illustration: https://jsfiddle.net/flexmonster/fo2kj4hw/.
 
Please let us know if this works fine for you.
 
Kind regards, 
Vera

Public
Vera Didenko Vera Didenko Flexmonster April 21, 2021

Hello, Bernard,
 
Thank you for your question. 
 
Yes, subquery is part of the Data Source object.
 
Please let us know if the provided example helps.
 
Kind regards,
Vera

Public
Francisco Jose Gomez Vargas April 21, 2021

Going to test it, Thanks for your response, i will keep you updated

Public
bernard dradeou April 23, 2021

Thank you this is the example i was looking for

Please login or Register to Submit Answer