Data exploration

Data exploration
Prev	Chapter 4. Data Exploration Tutorial	Next

Exploring data structure

Once we have loaded the data, we can explore its structure. This can be done by using the Database Explorer tool accessible from alias node in the Services component. Using this tool you can:

preview the content of the database (by double-click on the alias - see the movie)
determine the size of the selected tables (see the movie)
preview the structure of tables (by double-clicking on the table name in alias subtree - see the movie)
preview the contents of the data table (see the movie on viewing all data or movie on viewing selected columns).
execute SQL commands (see example 1 , example 2 movies)
delete a table from the database (see the movie).

Exploring data content

Creating data representation - PhysicalData

For further work with the data we need to create an object representing the dataset - PhysicalData. This object contains various information about the analyzed data, such as attribute types, basic statistics and data source location.

The PhysicalData object can be created using the Database Explorer tool (see the movie) or using the Metadata Explorer tool (see the movie). After creating the object, the user can open the PhysicalData window (see the movie) which contain variable statistics (if any statistics have been calculated).

More information about working with metadata objects can be found in the Metdata Repository documentation.

Analyzing data content with the Freq tool

Preliminary data content analysis is an important stage of any analytical project. The Freq tool is designed to support the user in this stage. This tool can be launched from the PhysicalData window (see the movie).

After opening the Freq tool it is necessary to calculate the statistics for selected variables (see the movie). This operation will provide various information about the variables, including:

distribution visualization: categorical mode, interval mode (see the movie), statistics mode (see the movie)
histograms: non-cumulative, cumulative (see the movie), smooth (see the movie)
limiting the view by setting the minimum, maximum, and number of bins (see the movie)
variable statistics (see the movie)
exploring the correlation of variables with the target variable (see the movie on visualizing the target level and the movie about marking the a-priori level).
working with numerous variables (see the movie about the variable name filtering mechanism)
creating virtual variables (see the movie about creating a copy of variable preview)
grouping the values of categorical variables (see the movie)
various histogram viewing options e.g.:
- restricting viewed data range on multiple variables (see the movie)
- zooming in and out of the histogram (see the movie)
- displaying a raw data table in the indicated variable range (see the movie)

Note

To execute any action like calculating statistics in Freq, you can use the F6 key, the 'Execute icon' or choose 'Execute' from context menu (right-clicking in Freq component).

The statistics calculated by the Freq tool are stored in a PhysicalData object. After calculating Freq statistics you can also view them in the PhysicalData window (see the movie).

More information about the Freq tool can be found in Technical Documentation here.

Data correlation analysis

In addition to visualisation offered by the Freq tool, the user can visualize the variables correlation matrix. Both tools (Freq and correlation matrix) are located in the CalculateStatisticsTask MR object, which is automatically created upon opening the Freq tool or using the Metadata Explorer tool (see the movie).

Interactions with the Correlation Matrix are shown in the movie.