﻿ 机器学习算法应用中常用技巧-一起大数据

# 机器学习算法应用中常用技巧

### 1. 取样

``````indices = [100,200,300]

# 把sample原来的序号去掉重新分配
samples = pd.DataFrame(data.loc[indices], columns = data.keys()).reset_index(drop = True)
print "Chosen samples:"
display(samples)``````

### 2. Split数据

`sklearn.cross_validation.train_test_split` 将数据分为 train 和 test 集。
sklearn

``````from sklearn import cross_validation
X = new_data
y = data['Milk']
X_train, X_test, y_train, y_test = cross_validation.train_test_split(X, y, test_size = 0.25, random_state = 0)
print len(X_train), len(X_test), len(y_train), len(y_test)``````

### 分离出 Features & Label

``````# Store the 'Survived' feature in a new variable and remove it from the dataset
outcomes = full_data['Survived']
data = full_data.drop('Survived', axis=1)``````

### 3. 用 train 来训练模型，用 test 来检验

sklearn

``````from sklearn import tree
regressor = tree.DecisionTreeRegressor()
regressor = regressor.fit(X_train, y_train)
score = regressor.score(X_test, y_test)``````

### 4. 判断 feature 间的关联程度

``pd.scatter_matrix(data, alpha = 0.3, figsize = (14, 8), diagonal = 'kde');``

### 5. scaling

``pd.scatter_matrix(log_data, alpha = 0.3, figsize = (14,8), diagonal = 'kde');``

scaling前后对比图：

### 6. Outliers

``````for feature in log_data.keys():
Q1 = np.percentile(log_data[feature], 25)
Q3 = np.percentile(log_data[feature], 75)
step = 1.5 * (Q3 - Q1)
print "Outliers for feature '{}':".format(feature)
print Q1, Q3, step
display(log_data[~((log_data[feature]>=Q1-step) & (log_data[feature]<=Q3+step))].sort([feature]))``````

``````plt.figure()
plt.boxplot([log_data.Fresh, log_data.Milk, log_data.Grocery, log_data.Frozen, log_data.Detergents_Paper, log_data.Delicassen], 0, 'gD');``````

### 7. 降维－PCA

n_components为降到多少维，用原数据fit后，再用transform转换成降维后的数据。

``````from sklearn.decomposition import PCA
pca = PCA(n_components = 6)
pca.fit(good_data)
reduced_data = pca.transform(good_data)
reduced_data = pd.DataFrame(reduced_data, columns = ['Dimension 1', 'Dimension 2'])``````

### 8. 聚类－选择类别数

``````from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
cluster = KMeans(n_clusters=2, random_state=0).fit(reduced_data)
preds = cluster.predict(reduced_data)
score = silhouette_score(reduced_data, preds)``````

### 9. 恢复维度

``````log_centers = pca.inverse_transform(centers)
true_centers = np.exp(log_centers)
true_centers = pd.DataFrame(np.round(true_centers), columns = data.keys())``````

### 10. 自定义accuracy

``````def accuracy_score(truth, pred):
""" Return accuracy score for input truth and prediction"""

if len(truth)==len(pred):
return "Accuracy for prediction: {:.2f}%.".format((truth==pred).mean()*100)
else:
return "Numbers do not match!"``````