python利用决策树进行特征选择

import numpy as np
import tflearn
from tflearn.layers.core import dropout
from tflearn.layers.normalization import batch_normalization
from tflearn.data_utils import to_categorical
from sklearn.model_selection import train_test_split
import sys
import pandas as pd
from pandas import Series,DataFrame
import matplotlib.pyplot as plt data_train = pd.read_csv("feature_with_dnn_todo2.dat")
print(data_train.info())
import matplotlib.pyplot as plt
print(data_train.columns)"""
for col in data_train.columns[:]:
fig = plt.figure(figsize=(, ), dpi=)
fig.set(alpha=0.2)
plt.figure()
data_train[data_train.label == 0.0][col].plot()
data_train[data_train.label == 1.0][col].plot()
data_train[data_train.label == 2.0][col].plot()
data_train[data_train.label == 3.0][col].plot()
data_train[data_train.label == 4.0][col].plot()
plt.xlabel(u"sample data id")
plt.ylabel(col)
plt.title(col)
plt.legend((u'white', u'cdn',u'tunnel', u"msad", "todo"),loc='best')
plt.show()
""" from sklearn.ensemble import ExtraTreesClassifier
X = data_train.iloc[:,:]
y = data_train['label'].tolist()print(X.columns)X = X.values.tolist()
print(X[-:])
print("-------------")
print(y[-:])# preprocess data
from sklearn.preprocessing import StandardScaler
#X = StandardScaler().fit_transform(X)
from sklearn.preprocessing import MinMaxScaler
X = MinMaxScaler().fit_transform(X)
from sklearn.preprocessing import Normalizer
#X=Normalizer().fit_transform(X)# abnormal data process
"""
for i,n in enumerate(y):
if n == 4.0:
y[i]=
"""import collections
print(collections.Counter(y))print("just change to 2 classify !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")
for i,n in enumerate(y):
if n != 2.0:
y[i]=
else:
y[i]=
print(collections.Counter(y))
#print(X.shape)from imblearn.over_sampling import SMOTE
X, y = SMOTE().fit_sample(X, y)
print(sorted(collections.Counter(y).items()))import sys
clf = ExtraTreesClassifier()
print(dir(clf))
X_new = clf.fit(X, y)
print (clf.feature_importances_ )
names = [u'flow_cnt', u'len(srcip_arr)', u'len(dstip_arr)', u'subdomain_num',
u'uniq_subdomain_ratio', u'np.average(dns_request_len_arr)',
u'np.average(dns_reply_len_arr)', u'np.average(subdomain_tag_num_arr)',
u'np.average(subdomain_len_arr)',
u'np.average(subdomain_weird_len_arr)',
u'np.average(subdomain_entropy_arr)', u'A_rr_type_ratio',
u'incommon_rr_type_rato', u'valid_ipv4_ratio', u'uniq_valid_ipv4_ratio',
u'request_reply_ratio', u'np.max(dns_request_len_arr)',
u'np.max(dns_reply_len_arr)', u'np.max(subdomain_tag_num_arr)',
u'np.max(subdomain_len_arr)', u'np.max(subdomain_weird_len_arr)',
u'np.max(subdomain_entropy_arr)', u'avg_distance', u'std_distance']
print "Features sorted by their score:"
print sorted(zip(clf.feature_importances_, names), reverse=True)

1.13.1. Removing features with low variance

VarianceThreshold is a simple baseline approach to feature selection. It removes all features whose variance doesn’t meet some threshold. By default, it removes all zero-variance features, i.e. features that have the same value in all samples.

As an example, suppose that we have a dataset with boolean features, and we want to remove all features that are either one or zero (on or off) in more than 80% of the samples. Boolean features are Bernoulli random variables, and the variance of such variables is given by

so we can select using the threshold .8 * (1 - .8):

As expected, VarianceThreshold has removed the first column, which has a probability of containing a zero.

1.13.2. Univariate feature selection

Univariate feature selection works by selecting the best features based on univariate statistical tests. It can be seen as a preprocessing step to an estimator. Scikit-learn exposes feature selection routines as objects that implement the transform method:

• SelectKBest removes all but the highest scoring features
• SelectPercentile removes all but a user-specified highest scoring percentage of features
• using common univariate statistical tests for each feature: false positive rate SelectFpr, false discovery rate SelectFdr, or family wise error SelectFwe.
• GenericUnivariateSelect allows to perform univariate feature selection with a configurable strategy. This allows to select the best univariate selection strategy with hyper-parameter search estimator.

For instance, we can perform a test to the samples to retrieve only the two best features as follows:

These objects take as input a scoring function that returns univariate scores and p-values (or only scores for SelectKBest and SelectPercentile):

• For regression: f_regression, mutual_info_regression
• For classification: chi2, f_classif, mutual_info_classif

The methods based on F-test estimate the degree of linear dependency between two random variables. On the other hand, mutual information methods can capture any kind of statistical dependency, but being nonparametric, they require more samples for accurate estimation.

1.13.3. Recursive feature elimination

Given an external estimator that assigns weights to features (e.g., the coefficients of a linear model), recursive feature elimination (RFE) is to select features by recursively considering smaller and smaller sets of features. First, the estimator is trained on the initial set of features and the importance of each feature is obtained either through a coef_ attribute or through a feature_importances_ attribute. Then, the least important features are pruned from current set of features.That procedure is recursively repeated on the pruned set until the desired number of features to select is eventually reached.

RFECV performs RFE in a cross-validation loop to find the optimal number of features.

1.13.4. Feature selection using SelectFromModel

SelectFromModel is a meta-transformer that can be used along with any estimator that has a coef_ or feature_importances_ attribute after fitting. The features are considered unimportant and removed, if the corresponding coef_ or feature_importances_ values are below the provided threshold parameter. Apart from specifying the threshold numerically, there are built-in heuristics for finding a threshold using a string argument. Available heuristics are “mean”, “median” and float multiples of these like “0.1*mean”.

For examples on how it is to be used refer to the sections below.