Need a special offer?Find out if your project fits.
+

questions on flexmonster data server

Answered
Eric Malalel asked on October 14, 2022

I have to rewrite the statistics functions of my application. I already use flexmonster but I can see there have been a lot of changes.
I am especially interested by the flexmonster data server and I have a few questions about it.
I have one database per customer, but I will provide the same statistics functions to all customers.
I understand that I can predefine queries in the flexmonster data server.
Question 1
Do I need to redefine queries on each database or is there a way to apply the same queries to a set of databases?
Question 2
Is it possible to add parameters to the queries? 
I am afraid not (https://www.flexmonster.com/question/database-query-parameters/) but still ask the question.
Question 3
Once the data is extracted by the query, is it possible to get and send only a slice of it to the client app?
For example, I have customers with 5 000 users and I track their learning activities on thousands on contents, each content having and id.
Customers require to have statistics on one content. Can I server side filter the data to get only those related to a content id? 
Question 4
Is it possible to get the data extracted by the flexmonster data server and put it somewhere so it can be uploaded from a SFTP server for instance?
Question 5
If yes to question 4, can I use MS Excel to import that data and process it with Excel? 
Some customers have no access to my application, so it would be useful.
That's all :), thanks in advance for your answers.

9 answers

Public
Maksym Diachenko Maksym Diachenko Flexmonster October 18, 2022

Hello, Eric!

Thank you for writing to us.

Please see the responses to your questions below:

  1. Currently, it is not possible to apply the same queries for a set of SQL databases. You will need to define a separate index for each database and query pair.
  2. The only possible solutions for adding parameters to queries are described in the thread you have found on our help center. Our team is ready to provide additional assistance if you consider using them.
  3. Flexmonster sends multiple requests from the client to retrieve only the data specified inside the slice. This means the component loads aggregated data from the server instead of loading the entire dataset on the client's machine. You can see which data is sent from the server by checking responses on field, members, and handshake requests in the Network tab. 

Regarding questions 4 and 5, we would like more details about your use case. This will help us to understand what you are trying to achieve and to provide a better answer.

Looking forwards to hearing from you.

Best Regards,
Maksym

Public
Eric Malalel October 18, 2022

Hello Maksym, thanks for feedback.
Answer to question 1 is not what I hopped but I can deal with it. The ability to define indexes without the connection string and then apply it to a database defined by its connection string would be better for me.
Regarding answer to question 2, is it possible to use the Data Server as a DLL from a Java program, or is it limited to ASP.NET which means I would have to use a custom data source API server.
Before going deeper into this and my other questions, I would like to understand better why the Data Server can improve performance, as well as do I really need it instead of using the compressor from a Java program.
My largest customer has 10 000 users recorded in the database of our app, an LMS. We track user activity on 100 learning plans, each learning plan can contain 20 courses, and each course can contain 20 training sequences.
I want to give the user the ability to access statistics on user activity first at the learning plan level, then at the course level, and then at the training sequence level.
This means drilling in 10 000 (users) * 100 (learning plans) * 20 (courses) * 20 (training sequences) = 400 000 000 rows.
What would be your recommandation, data server or compressor, and how to manage the drill down process?

Public
Maksym Diachenko Maksym Diachenko Flexmonster October 20, 2022

Hello, Eric!

Thank you for your reply.

Answering your question regarding Java, kindly note that you can use the FDS DLL only for .NET Core applications. As a workaround, it is possible to use your Java backend as a proxy to route the requests to Flexmoster Data Server service. For example, you could specify your Java-based backend endpoint when connecting Flexmonster to your data (in the dataSource property of the report object). Then you could configure your Java backend to handle Flexmonster requests, acting as a proxy between Flexmonster and the Flexmonster Data Server:

var pivot = new Flexmonster({
  container: "#pivot-container",
  componentFolder: "https://cdn.flexmonster.com/",
  report: {
    dataSource: {
       type: "api",
       url: "https://your_java_backend_endpoint"
     }
  }
});

Another solution would be to create your own custom data source API implementation. Such an approach would provide the most flexibility as to how the data is processed and won't require using .NET Core. Also, this would mean that you could create your own approach as to how the data is processed (for example defining indexes without the connection string and then applying it to a database defined by its connection string).

The main advantage of using Flexmonster Data Server or implementing your own custom data source API server instead of using the compressor is that these approaches allow loading only the necessary part of your data. Flexmonster Data Server is a server-side tool implementing the custom data source API - our communication protocol responsible for fetching data from a data source, processing, and aggregating it. When you use the type: "api" data source in the component, it sends multiple requests to retrieve the data from the data server - you can see them in the "Network" tab of your browser's console. The server aggregates the data and returns only the data needed to be shown instead of the entire dataset. This results in better performance for large datasets because Flexmonster uses larger server capacities for calculations, while the client is only responsible for rendering.

If your client needs to process 400 million rows, we recommend using Flexmonster Data Server or implementing the custom data source API because, in case of compressor usage, all the data is loaded on the client. If you consider developing the custom data source API server, we recommend checking the implementation guide.

Regarding your question about the drill down, Flexmonster sends a separate /select request for expanding or drilling down the data. There is also a /select request with a different structure to open the drill-through view.

Feel free to contact us if more questions arise.

Best Regards,
Maksym

Public
Maksym Diachenko Maksym Diachenko Flexmonster November 1, 2022

Hello, Eric!

Hope you are doing well.
We would like to know if you have time to check our answer or have tried implementing the Java proxy server between the Flexmonster and Flexmonster Data Server.
Our team would be happy to hear your feedback.

Best Regards,
Maksym

Public
Eric Malalel November 2, 2022

Hello Maksym,
I am digging deeper in FlexMonster Data Server (FDS) documentation and would like to check if my understanding is right.
First, maybe I don't need parameters for queries.
I have a database where is recorded tracking information of users on courses.
So I can write a FDS query which extract information for all users and all courses, where in each row I have :
1/ information about the user: id, name, etc.
2/ information about the course: id, name, etc.
3/ tracking information of this user on this course: time spent, score achieved, status, etc.
When I use this FDS query from FlexMonster client, can I define slices to get only rows for:
1/ a user id
2/ a list of users ids
3/ a course id
4/ a list of courses ids
If answer is yes, then I don't need parameters for the query.
Second, the query will be relatively complex and will probably take some time to complete.
So, is it better to schedule it to run let say every 15 minutes, so that the data is ready when FlexMonster client needs it, and as it just requests the appropriate slices, performance is better.

Public
Maksym Diachenko Maksym Diachenko Flexmonster November 3, 2022

Hello, Eric!

Thank you for your reply. Please find our answers below.

Configuring which data to show
You are right, it seems the query parameters are unnecessary for your use case since Flexmonster allows configuring which specific columns, rows, and measures are shown on the grid inside the slice. However, we recommend filtering unnecessary columns inside the SELECT query to your database, as it will reduce the volume of data loaded into RAM, thereby improving the performance. 

Schedule the refresh
While adding a new index in Flexmonster Data Server, you can specify the "Refresh Time" option - time in minutes after which the index will be refreshed. Except for the time needed to run the SQL query on refresh, additional actions are needed to process the data and create an internal data structure used by FDS. Although for some indexes 15 minutes could be sufficient, we recommend increasing the refresh time if possible for your use case to avoid situations when the refresh starts before the initial loading is finished.

Please let us know if you have more questions about the Flexmonster Data Server.

Best Regards,
Maksym

Public
Eric Malalel November 7, 2022

Hello Maksym,
I finally drop FDS since defining queries for each customer is too heavy to manage.
I have moved to JSON using my own REST API, you can close this ticket.

Public
Maksym Diachenko Maksym Diachenko Flexmonster November 8, 2022

Hello, Eric!

Thank you for sharing this insight with us.
Our team hopes that the JSON approach will work for you. Also, we want to remind you that the browser’s capability limits the size of JSON (on average, up to 100 MB). 

Best Regards,
Maksym

Public
Eric Malalel November 9, 2022

100 MB should be enough, you can close this ticket.

Please login or Register to Submit Answer