Connecting to CSV using Flexmonster Data Server

In the previous article, we described how to connect the pivot table to the CSV data source. To gain even better performance while working with CSV datasets, use Flexmonster Data Server – a special server developed by Flexmonster. This server communicates with the client using the custom data source API – our custom communication protocol allowing you to retrieve already aggregated data from a server to Flexmonster Pivot.

Flexmonster Data Server has the following advantages:

Reduced browser’s memory usage. Since loading, processing, filtering, and aggregating of the data is made on the server side, Flexmonster Pivot gets the data in a ready-to-show format. Thus, the data is visualized faster, and the browser’s resources are used more efficiently.
Larger datasets. Despite Flexmonster Pivot can connect to a CSV file directly, the browser’s capability limits the size of a file (on average, up to 100 MB). Flexmonster Data Server can process files more than 1GB — the maximum file size depends on your RAM capacity and the number of unique members in the dataset.
Built-in security configurations. The Data Server allows establishing authorized access to data and managing security in a convenient way.
Cross-platform solution. The Data Server is available for popular operating systems: Windows, Ubuntu/Linux, and macOS.

Prerequisites

To download the Data Server, you will need Flexmonster CLI — a command-line interface tool for Flexmonster. If needed, install the CLI globally using npm:

npm install -g flexmonster-cli

After that, a new flexmonster command will be available in the console. Learn more about Flexmonster CLI and its commands in our documentation.

Now follow the steps below to connect to CSV using the Data Server.

Step 1. Embed the component into your web page

If Flexmonster is not yet embedded, set up an empty component in your web page:

In pure JavaScript

Complete the Quick start guide. Your code should look similar to the following example:

var pivot = new Flexmonster({
    container: "pivotContainer",
    toolbar: true
});

In Angular

Complete the Integration with Angular guide. Your code should look similar to the following example:

<fm-pivot 
 [toolbar]="true">
</fm-pivot>

In React

Complete the Integration with React guide. Your code should look similar to the following example:

<FlexmonsterReact.Pivot
 toolbar={true}
/>

In Vue

Complete the Integration with Vue guide. Your code should look similar to the following example:

<Pivot
 ref="pivot"
 toolbar>
</Pivot>

Step 2. Download Flexmonster Data Server

The previous step demonstrated how to configure Flexmonster Pivot. Now it’s time to set up Flexmonster Data Server.

Get the Data Server with the following CLI command:

flexmonster add fds

The flexmonster add fds command will download the .zip archive with Flexmonster Data Server and automatically unpack the files in the current folder. As a result, the flexmonster-data-server/ folder will appear in your working directory.

The Data Server can be configured in the flexmonster-data-server/flexmonster-config.json file. To learn more about all the available configurations, see the configurations reference.

To set configurations needed for the CSV data source, follow the steps below.

Step 3. Configure the data source type

Set the "Type" property in flexmonster-config.json to "csv". It should be done as follows:

"DataSources": [
    {
        "Type": "csv"
    }
],

Step 4. Define the dataset

Create an index for the file with your data. "Path" is the path to the file. For example:

"DataSources": [
    {
        "Type": "csv",
        "Indexes": {
            "index_csv": {
                "Path": "./data/data.csv"
            }
        }
    }
],

In a similar way, additional indexes can be specified:

"Indexes": {
    "index_csv": {
        "Path": "./data/data.csv"
    },
    "another_index_csv": {
        "Path": "./data/another_data.csv"
    }
}

"index_csv" and "another_index_csv" are dataset identifiers. They will be used to configure the data source on the client side.

Step 5. (optional) Specify the field delimiter

If СSV fields are separated by anything other than ",", the "Delimiter" parameter should be specified:

"Indexes": {
    "index_csv": {
        "Path": "./data/data.csv",
        "Delimiter": ";"
    }
}

Step 6. (optional) Specify the decimal separator

If decimal parts of numbers in the data are separated by anything other than ".", the "DecimalSeparator" parameter should be specified:

"Indexes": {
    "index_csv": {
        "Path": "./data/data.csv",
        "Delimiter": ";",
        "DecimalSeparator": ","
   }
}

Step 7. (optional) Specify the thousand separator

If the data contains numbers where groups of digits are separated by anything other than ",", the "ThousandSeparator" parameter should be specified:

"Indexes": {
  "index_csv": {
    "Path": "./data/data.csv",
    "Delimiter": ";",
    "DecimalSeparator": ",",
    "ThousandSeparator": "."
  }
}

Step 8. Run the server

Start Flexmonster Data Server by running the following command in the console:

on Windows

flexmonster-data-server.exe

on macOS and Ubuntu/Linux

./flexmonster-data-server

As soon as you start Flexmonster Data Server, it automatically preloads the data specified in the "Indexes" property. Thus, when Flexmonster Pivot requests the data, the server responds with already preloaded data.

The Data Server keeps preloaded data in the server’s RAM, so the number of indexes you can specify is limited by the server’s RAM amount.

Step 9. Configure the report

On the client side, the report should be configured as follows:

var pivot = new Flexmonster({
    container: "pivotContainer",
    toolbar: true,
    report: {
        dataSource: {
            type: "api",
            url: "http://localhost:9500",
            index: "index-csv"
        }
    }
});

index must match the name of the index defined in step 4 (e.g., "index_csv").

When Flexmonster Pivot requests the data, Flexmonster Data Server server sends the response and then caches it. In case the component sends the same request once again, the server responds with the data from its cache. The Data Server clears the cache when restarted.

The Data Server’s cache has a memory limit. When the cache does not have enough memory for a new response, the Data Server server deletes one of the previously cached responses.

You can manage the cache size via the “CacheSizeLimit” property.

What’s next?

You may be interested in the following articles: