Need a special offer?Find out if your project fits.
+

Elasticsearch filter object - by value

Answered
Vojtěch asked on August 21, 2020

Hi flexmonster team. We are considering the use of your solution for our purposes, but I would like to ask some questions please. We are trying the elasticsearch connection and it looks very promising. However there are some features that are not included in this data source am I correct? As I have read through some other threads in this forum for example the calculated value is not applicable. Please could you help me find or give me the information on what is and what is not available in elastic data source over for example CSV data source? Our main issues is the filter object. There is no option to filter by value i.e. greater than X. This is an important functionality for us.
Thank you very much for your time.
Best
Vojto

7 answers

Public
Mykhailo Halaida Mykhailo Halaida Flexmonster August 21, 2020

Hi Vojtěch,
 
Thank you for posting your question.
 
As you've suggested, functionality like calculated values & specific types of filtering is currently not available under some data sources, including Elasticsearch – this is caused by the complexity of the Flexmonster-Elasticsearch connector, which does not allow to fully implement particular features like these.
 
For the full comparison of features available under different data sources, feel free to check out our technical specifications page: https://www.flexmonster.com/technical-specifications/
 
Having said the above, in case you are not restricted to only using Elasticsearch as a data source, we would like to suggest another approach, which allows using the mentioned filters & calculated values, while also providing a massive boost in data processing performance.
 
The tool we're talking about is called Flexmonster Data Server. Data Server is an implementation of the custom data source API – our custom communication protocol which allows you to retrieve already aggregated data from the server and pass it to the Flexmonster pivot table. At the core of this approach lies the idea of assigning the heavy lifting of fetching, processing and aggregating the data to the server, after which only the necessary parts of the data are transferred to the client-side. 
 
Feel free to check out the following documentation page for a detailed overview of possibilities provided by the Flexmonster Data Server: https://www.flexmonster.com/doc/getting-started-with-data-server/
 
Please let us know if this helps.
 
Best regards,
Mykhailo

Public
Vojtěch August 25, 2020

Thank you very much for your answer. I have tried some other examples and data sources and I have some issues with the amount of data the pivot table can handle.
First I tried to load data of about 300k lines with elastic. It has successfully loaded the data, however if I tried to put a specific field, having a different value in almost all of the 300k rows (therefore I was expecting to get 300k rows in the table), it gave me an error, with info that it failed to load resource (503 error) from elastic.
Then I tried to load the data from CSV. The 300k rows file (33MB) was fine, however when I tried to load a 980MB file it just stopped loading and nothing happened. It just crashed and the table stopped working.
For the most part we wouldn`t have files as big as 1GB, however in some occasions we will. Therefore we would need to have it capable of loading these files.
Thank you again
Vojto

Public
Mykhailo Halaida Mykhailo Halaida Flexmonster August 26, 2020

Hi Vojtěch,
 
Thank you for sharing your observations.
 
Addressing your first point regarding Elasticsearch, it seems to us that the issue most likely begins on the server-side with your Elasticsearch instance, since the 503 HTTP status means a server error. 
 
We would suggest investigating what's happening on your server that might be causing the issue, as it is unlikely that a response of this type is somehow caused by Flexmonster. Please let us know if you manage to get to the core of the issue – we'd be happy to hear about your solution.
 
Speaking of large CSV data, please note that being a client-side component, Flexmonster relies on resources available to the browser, which affects the loading time and the maximum size of the data that can be handled on every particular machine. This means that the client machine’s RAM determines how much data can be loaded at once, and CPU capabilities affect how much time is spent on the data analysis.
 
With that in mind, taking into account the average user's computer, it is safe to say that most machines won't be able to handle CSV data sets of this size well, since if you connect Flexmonster to a CSV file directly, the whole data set is loaded to the pivot table at once.
 
However, the earlier mentioned Flexmonster Data Server solves this problem very well – it can be easily connected to a CSV file of your choosing, while also providing all of the optimizations we've mentioned in our previous response.
 
Feel free to check out a step-by-step tutorial on how to connect to your CSV data set with the Data Server here: https://www.flexmonster.com/doc/pivot-table-connect-to-csv/.
 
We hope this helps.
 
Best regards,
Mykhailo

Public
Mykhailo Halaida Mykhailo Halaida Flexmonster September 2, 2020

Hi Vojtěch,
 
Hope you're doing well.
 
We were wondering if you've received our response to your question. Was it helpful, or is there still anything we can help you with?
 
We would be happy to hear your feedback.
 
Regards,
Mykhailo

Public
Mykhailo Halaida Mykhailo Halaida Flexmonster September 9, 2020

Hi Vojtěch,
 
How are you?
 
Just checking in to ask whether you've had a chance to check out our response. Did you find it helpful?
 
Please let us know your thoughts.
 
Best regards,
Mykhailo

Public
Vojtěch September 10, 2020

Hi Mykhailo

Thank you very much for your answers and suggestions. As of numerous other tasks, work and other reasons we did not have the time to play around with it.
However we will look into it as soon as possible, because we believe flexmonster is the thing we need 🙂
I will write here our results with trying out the Data server and with troubleshooting the elastic stack.
Thank you
Best regards
Vojto

Public
Mykhailo Halaida Mykhailo Halaida Flexmonster September 14, 2020

Hi Vojtěch,
 
Thank you for the update – feel free to reach out to us if you have any questions in the process, we will be happy to assist you!
 
Regards,
Mykhailo

Please login or Register to Submit Answer