Data Exploration

Data exploration seeks to evaluate the data set and determine the most appropriate way to configure it for model training.

Steps in data exploration include the following:

1. Exploratory data analysis (EDA) involves looking at data descriptors such as summary statistics, heat maps, word clouds, and so on. The objectives of EDA include: to Understanding data properties, distributions, and other characteristics.

to Finding patterns or relationships, and evaluating basic questions and hypotheses. to Planning modeling in future steps.

1. Feature selection is a process to select only the needed attributes of the data for ML model training. The higher the number of features selected, the higher the model complexity and training time.
2. Feature engineering is the process of creating new features by transforming (e.g., taking the natural logarithm), decomposing, or combining multiple features. A related term is feature extraction, whereby a feature is created from the data set (e.g., creating a value for age using date of birth data).

Model performance depends heavily on feature selection and engineering, and it is common for the analyst to recursively go back to this step and apply tweaks until model performance is acceptable.

Data Exploration for Structured Data

With exploratory data analysis (EDA), structured data is organized in rows (observations) and columns (features). EDA can be performed for a single feature (one dimension) or multiple features (multidimension). When the number of features is large, dimension reduction models such as principal component analysis (PCA) can facilitate data exploration.

• For a single feature, summary statistics include the mean, standard deviation, skewness, and kurtosis. EDA visualizations include box plots, histograms, density plots, and bar charts. Histograms capture the frequency of observations in a series of equal width bins.
• Density plots are smoothed histograms for continuous data overlaid on top of histograms. Bar charts show frequency distributions of categorical variables (e.g., proportion of the population in each of the five geographical zones of the country). Box plots are used for continuous variables and highlight the median, quartiles, and outliers of a normally distributed feature.
• For multiple features, the summary statistic can be a correlation matrix. Data can be visually graphed using a scatterplot, box plots, stacked bar or a line graph. Multiple box plots can be plotted on the same line with each plot representing a feature.
• Parametric statistical tests include ANOVA tables, correlation tables, and t tests.
• Nonparametric statistical tests include the Spearman rank order correlation and the chi square test.

With feature selection, we try to select only the features that contribute to the out of sample predictive power of the model. A parsimonious model (i.e., a model with fewer features) reduces feature induced noise and improves the model’s prediction accuracy. Feature selection requires a good understanding of the business environment and the interrelationships among the features identified in the EDA.

For structured data, feature selection is an iterative, methodical process. Features can be assigned an importance score using statistical methods and then ranked and selected based on that score. Dimension reduction algorithms may be employed to reduce the number of features needed so as to reduce processing time during model training.

Feature engineering (FE) involves optimizing and improving the selected features. Model training results depend on how the features are presented to the algorithm. Feature engineering involves either decomposing a feature into multiple features or converting an existing feature into a new feature.

One hot encoding (OHE) is a process used to convert a categorical feature into a binary (dummy) variable suitable for machine processing. Feature engineering seeks to make model training faster and easier.

Data Exploration for Unstructured Data

Unstructured text can be tokenized, and summary statistics such as term frequency (number of times the word appears in the text) and co occurrence (where two or more words appear together) can be analyzed. A word cloud is a visual representation of all the words in a BOW, such that words with higher frequency have a larger font size. This allows the analyst to determine which words are contextually more important.

Figure 4.1 shows an example of a word cloud.

Figure 4.1: Word Cloud, Apple (NASDAQ: AAPL) SEC Filing

Feature Selection

Feature selection involves selecting a subset of tokens in the BOW. Reduction in BOW size makes the model more parsimonious and reduces feature induced noise. Noisy features do not contribute to prediction accuracy.

High and low frequency words are often eliminated, resulting in a more concise BOW. High frequency words tend to be stop words (if not removed during the data wrangling phase) or common vocabulary words. Low frequency words may be irrelevant.

Consider a model to predict bankruptcy: to effectively separate defaulters from nondefaulters, tokens that are associated with both categories should be removed. Feature selection methods include:

• Frequency. One of the tools used for feature selection in textual data is the document frequency (DF). The DF of a token is calculated as the number of documents containing that token divided by the total number of documents.
• Chi square. This test is used to rank tokens by their usefulness to a class in text classification problems. Tokens with the highest chi square test statistic occur more frequently with a specific class; hence, they are useful in training ML algorithms for discriminant analysis.
• Mutual information. This is a numerical value indicating the contribution of a token to a class of texts. If the token appears in all classes, it is not considered a useful discriminant, and its mutual information (MI) equals 0.
• Tokens associated with only one or a few classes would have MI approaching 1.

Feature Engineering (FE)

Techniques of FE include:

• Numbers. Tokens with standard lengths are identified and converted into a token such as /numberX/. Four digit numbers may be associated with years and are assigned a value of /number4/.
• N grams. These are multiword patterns, and if they are useful, the order is preserved. For example, the words expansionary monetary policy may be best kept as a sequence rather than broken into three different tokens, and therefore would be replaced by a single token, expansionary_monetary_policy.
• Name entity recognition (NER). NER algorithms search for token values, in the context it was used, against their internal library and assign a NER tag to the token. For example, Microsoft would be assigned a NER tag of ORG and Europe would be assigned a NER tag of Place.
• NER object class assignment is meant to make the selected features more discriminatory.
• Parts of speech (POS). This uses language structure dictionaries to contextually assign tags (POS) to text. For example, Microsoft would be assigned a POS tag of NNP (indicating a proper noun), and the year 1969 would be assigned a POS tag of CD (indicating a cardinal number).