pandas 之特征工程

来源：恒创科技编辑：恒创科技编辑部

2022-08-22 14:40:45

数据映射, 离散化, 异常值, 重采样, one-hot coding....

import numpy as np
import pandas as pd

So far(到目前为止) in this chapter we've been concerned with rearranging data. Filterng, cleaning, and other transformations are another class of important oprations.

数据去重

Duplicate rows may be found in a DataFrame for any number of reasons. Here is an example:

data = pd.DataFrame({
    'k1': ['one', 'two']*3 + ['two'],
    'k2': [1, 1, 2, 3, 3, 4, 4]
})

data

one

two

one

two

one

two

The DataFrame method duplicated returns a boolean Series indcating whether each rows is a duplicate (has been observed in a previous row) or not:

"df.duplicated() 对每一行数据进行重复判断"
data.duplicated()

'df.duplicated() 对每一行数据进行重复判断'






0    False
1    False
2    False
3    False
4    False
5    False
6     True
dtype: bool

Relatedly, drop_dumplicates returns a DataFrame where the duplicated array is False.

"df.drop_duplicates() 删除重复行"

data.drop_duplicates()

'df.drop_duplicates() 删除重复行'

one

two

one

two

one

two

Both of these methods by default consider of the columns; alternatively(非此即彼), you can specify any subset of them to detect(察觉) duplicates. Suppose we had an additional column of values and wanted to filter duplicates only base on the 'k1' columns:

data['v1'] = range(7) 

"指定子集, 去判断, 删除整行 "
data.drop_duplicates(['k1'])

'指定子集, 去判断, 删除整行 '

one

two

duplicated and drop_duplicates by default keep the first observed value combination. Passing keep='last' will return the last one:

保留第一个观察值组合。传递keep='last'将返回最后一个

data

one

two

one

two

one

two

"[k1, k2], 重复的为: [two, 4] 5; [two, 4] 6, 传入last, 保留最后一个做代表"
data.drop_duplicates(['k1', 'k2'], keep='last')

'[k1, k2], 重复的为: [two, 4] 5; [two, 4] 6, 传入last, 保留最后一个做代表'

one

two

one

two

one

two

数据映射转换函数 map

For many datasets, you may wish to perform some transformation based on the values in an array, Series, or column in a DataFrame. Consider the following hypothetical(假设) data collected about various kinds of meat:

data = pd.DataFrame({'food': ['bacon', 'pulled pork', 'bacon',
    'Pastrami', 'corned beef', 'Bacon',
    'pastrami', 'honey ham', 'nova lox'],
    'ounces': [4, 3, 12, 6, 7.5, 8, 3, 5, 6]})

data

food

ounces

bacon

4.0

pulled pork

3.0

bacon

12.0

Pastrami

6.0

corned beef

7.5

Bacon

8.0

pastrami

3.0

honey ham

5.0

nova lox

6.0

Suppose you wanted to add a column indicating the type of animal that each food came from. Let's write down a mapping of each distinct meat type to the type of animal:

meat_to_animal = {
'bacon': 'pig',
'pulled pork': 'pig',
'pastrami': 'cow',
'corned beef': 'cow',
'honey ham': 'pig',
'nova lox': 'salmon'
}

The map method on a Series accepts a function or dict-like object containing a maping, but here we have a small ploblem in that some of the meats are capitalized and others are not. Thus, we need to convert each value to lowercase using the str.lower Series method:
(Series的map(), 接收一个mapping函数or字典对象, 在本例中, 我们首先要进行大小写转换)

lowercased = data['food'].str.lower()

lowercased

0          bacon
1    pulled pork
2          bacon
3       pastrami
4    corned beef
5          bacon
6       pastrami
7      honey ham
8       nova lox
Name: food, dtype: object

"map() 值为映射字段的值"

data['animal'] = lowercased.map(meat_to_animal)

data

'map() 值为映射字段的值'

food

ounces

animal

bacon

4.0

pig

pulled pork

3.0

pig

bacon

12.0

pig

Pastrami

6.0

cow

corned beef

7.5

cow

Bacon

8.0

pig

pastrami

3.0

cow

honey ham

5.0

pig

nova lox

6.0

salmon

"We could also have passed a function that does all the work"

data['food'].map(lambda x: meat_to_animal[x.lower()])

'We could also have passed a function that does all the work'






0       pig
1       pig
2       pig
3       cow
4       cow
5       pig
6       cow
7       pig
8    salmon
Name: food, dtype: object

Using map is a convenient way to perform element-wise transformations and other data cleaning-related operations.
使用 map() 是处理相关联字段,的一种快捷方式

# cj test

cj_df = pd.DataFrame({
    'a':[1,3],
    'b':[2,4]
})

cj_df

'df[新增字段] = df[关联的键].map(对应的值-字典)'

cj_df['c'] = cj_df['a'].map({1:'cj', 3:'youge'})

cj_df

'df[新增字段] = df[关联的键].map(对应的值-字典)'

youge

异常值替换 Replacing

Filling in missing data with the fillna method is a special case of more general value replacement. As you've already seen, map can be used to modify a subset of values in an object but replace provides a simpler and more flexible way to do so. Let's consider this Series:
(在处理缺失值的时候, fillna其实只是, 替换的一种特例, 后面我们看到了 map 可对数据集进行映射, 而 replace 则更为通用化和灵活处理)

data = pd.Series([1., -999., 2., -999., -1000., 3.])

data

0       1.0
1    -999.0
2       2.0
3    -999.0
4   -1000.0
5       3.0
dtype: float64

The -999 values might be sentinel values(哨兵值, 标记值) for missing data. To replace these with NA values that pandas understands, we can use replace, producing a new Series(unless you pass inplace=True)

"先用df.replace(old, NA), 将其让 pandas能识别"

data.replace(-999, np.nan, inplace=False)

'先用df.replace(old, NA), 将其让 pandas能识别'






0       1.0
1       NaN
2       2.0
3       NaN
4   -1000.0
5       3.0
dtype: float64

data

0       1.0
1    -999.0
2       2.0
3    -999.0
4   -1000.0
5       3.0
dtype: float64

If you want to replace multiple values as once, you instead pass a list and the the substitute(替代的) value:

data.replace([-999, -1000], np.nan)

0    1.0
1    NaN
2    2.0
3    NaN
4    NaN
5    3.0
dtype: float64

"To use a different replacement for each value, pass a list of substitutes"

data.replace([-999, -1000], [np.nan, 0])

'To use a different replacement for each value, pass a list of substitutes'






0    1.0
1    NaN
2    2.0
3    NaN
4    0.0
5    3.0
dtype: float64

"The argument passed can also be a dict"

data.replace({-999:np.nan, -1000:0})

'The argument passed can also be a dict'






0    1.0
1    NaN
2    2.0
3    NaN
4    0.0
5    3.0
dtype: float64

The data.replace method is distinct from data.str.replace, which performs string substitution element-wise. We look at these string methods on Series later in the chapter.

重命名轴索引

Like values a Series, axis labels can be similarly transformaed by a function or mapping of some form to produce new, differently labeled objects. You can also modify the axes in-palce without creating a new data structure. Here's a simple example.

data = pd.DataFrame(np.arange(12).reshape((3, 4)),
    index=['Ohio', 'Colorado', 'New York'],
    columns=['one', 'two', 'three', 'four'])

data

one

two

three

four

Ohio

Colorado

New York

Like a Series, the axis indexes hava a map method:

"map 通过原来的映射"
transform = lambda x: x[:4].upper()

data.index.map(transform)

'map 通过原来的映射'






Index(['OHIO', 'COLO', 'NEW '], dtype='object')

"You can assign to index, modifying the DF in-place"
"不提供inplace哦"

data.index = data.index.map(transform)
data

'You can assign to index, modifying the DF in-place'






'不提供inplace哦'

one

two

three

four

OHIO

COLO

NEW

If you want to create a transformed version of a dataset without modifying the original, a useful method is rename:

data.rename(index=str.title, columns=str.upper)

ONE

TWO

THREE

FOUR

Ohio

Colo

New

Notably(值得注意的是), rename can be used in conjunction with a dict-like object providing new values for a subset of the axis labels:

data.rename(
    index={'OHIO': 'INDIANA'},
    columns={'three': 'peekaboo'})

one

two

peekaboo

four

INDIANA

COLO

NEW

rename saves you from the chore(零星的) of copying the DataFrame manually(手动地) and assigning to its index and columns attributes. Should you wish to modify a dataset in-place.

data.rename(index={'OHIO':'cj'}, inplace=True)
data

one

two

three

four

COLO

NEW

连续数据离散化和装箱

Continuous data is offen discretized(离散化) or otherwise separated into "bins" for analysis. Suppose you hava data about a group of people in a study, and you want to group them into discrete age buckets:
(对连续变量进行离散化, 分桶)

ages = [20, 22, 25, 27, 21, 23, 37, 31, 61, 45, 41, 32]

Let's divide these into bins of 18 to 25, 26 to 35, 36 to 60, and finally 61 and order. To do so, you have to use cut, a function in pandas:

bins = [18, 25, 35, 60, 100]

cats = pd.cut(ages, bins)

cats

"cut 通常配合 pd.value_counts() 实现分组统计"
pd.value_counts(cats)

[(18, 25], (18, 25], (18, 25], (25, 35], (18, 25], ..., (25, 35], (60, 100], (35, 60], (35, 60], (25, 35]]
Length: 12
Categories (4, interval[int64]): [(18, 25] < (25, 35] < (35, 60] < (60, 100]]






'cut 通常配合 pd.value_counts() 实现分组统计'






(18, 25]     5
(35, 60]     3
(25, 35]     3
(60, 100]    1
dtype: int64

The object pandas returns is a special Categorical object. The output you see describels the bins coputed by pandas.cut. You can treat it like an array of strings indicating the bin name; internally(内部地) it contains a categories array specifying the distinct category names alone with a labeling for the ages data in the codes attribute:
(离散化后的对象, 像一个array, 值是区间)

cats.codes

array([0, 0, 0, 1, 0, 0, 2, 1, 3, 2, 2, 1], dtype=int8)

cats.categories

IntervalIndex([(18, 25], (25, 35], (35, 60], (60, 100]]
              closed='right',
              dtype='interval[int64]')

pd.value_counts(cats)  # 统计一波, 可做直方图哦

(18, 25]     5
(35, 60]     3
(25, 35]     3
(60, 100]    1
dtype: int64

# 这里就不显示图片了
pd.value_counts(cats).plot("bar")

Note that pd.value_counts(cats) are the bin counts for the result of pandas.cut

Consistent with mathematical notation(符号) for intervals, a parenthesis(插入) means that the side is open, while the square bracket means it is closed(inclusive). You can change which side is closed by passing right=False: (区间的开,闭可以灵活设置)

pd.cut(ages, [18,26,36,61,100], right=False)

[[18, 26), [18, 26), [18, 26), [26, 36), [18, 26), ..., [26, 36), [61, 100), [36, 61), [36, 61), [26, 36)]
Length: 12
Categories (4, interval[int64]): [[18, 26) < [26, 36) < [36, 61) < [61, 100)]

You can also pass your own bin names by passing a list or array to the labels option:

group_names = ['Youth', 'YoungAdult', 'MiddleAged', 'Senior']

tmp = pd.cut(ages, bins, labels=group_names)

pd.value_counts(tmp)

Youth         5
MiddleAged    3
YoungAdult    3
Senior        1
dtype: int64

If you pass an integer number of bins to cut instead of explicit bin edges(明晰的边界), it will compute equal-lenth bins base on the nimimum and maximum values in the data. Consider the case of some uniformly distributed data chopped(砍断) into fourths:

data = np.random.rand(20)

tmp = pd.cut(data, 4, precision=2)

tmp

pd.value_counts(tmp)

[(0.55, 0.76], (0.55, 0.76], (0.55, 0.76], (0.34, 0.55], (0.55, 0.76], ..., (0.14, 0.34], (0.76, 0.97], (0.55, 0.76], (0.34, 0.55], (0.14, 0.34]]
Length: 20
Categories (4, interval[float64]): [(0.14, 0.34] < (0.34, 0.55] < (0.55, 0.76] < (0.76, 0.97]]






(0.55, 0.76]    6
(0.34, 0.55]    6
(0.14, 0.34]    5
(0.76, 0.97]    3
dtype: int64

The pricision=2 option limits the dicimal precision to two digits.

A closely related function, qcut, bins the data based on sample quantiles(样本分位数). Depending on the distribution of the data, using cut will not usually result in each bin having the same number of data points. Since qcut uses sample quantiles instead, by definition you will obtain roughly equal-size bins:
(qcut 是按样本分为数来选的, 每个分区的样本点是一样的, 可控制)

data = np.random.randn(1000) # Normally distributed

cats = pd.qcut(data, 4)  # Cut into quartiles

cats

pd.value_counts(cats)

[(-2.855, -0.667], (0.0288, 0.663], (0.0288, 0.663], (0.663, 3.065], (-0.667, 0.0288], ..., (-0.667, 0.0288], (0.0288, 0.663], (0.0288, 0.663], (0.663, 3.065], (0.663, 3.065]]
Length: 1000
Categories (4, interval[float64]): [(-2.855, -0.667] < (-0.667, 0.0288] < (0.0288, 0.663] < (0.663, 3.065]]






(0.663, 3.065]      250
(0.0288, 0.663]     250
(-0.667, 0.0288]    250
(-2.855, -0.667]    250
dtype: int64

Similar to cut you can pass your own quantiles (number between 0 and 1, inclusive):

tmp = pd.qcut(data, [0, 0.1, 0.5, 0.9, 1])

tmp

pd.value_counts(tmp)

[(-2.855, -1.311], (0.0288, 1.256], (0.0288, 1.256], (0.0288, 1.256], (-1.311, 0.0288], ..., (-1.311, 0.0288], (0.0288, 1.256], (0.0288, 1.256], (0.0288, 1.256], (1.256, 3.065]]
Length: 1000
Categories (4, interval[float64]): [(-2.855, -1.311] < (-1.311, 0.0288] < (0.0288, 1.256] < (1.256, 3.065]]






(0.0288, 1.256]     400
(-1.311, 0.0288]    400
(1.256, 3.065]      100
(-2.855, -1.311]    100
dtype: int64

We'll return to cut and qcut later in the chapter during our discussion of aggregation and group oprations, as these discretization functions are especially usefull for quantile and group analysis.

检测和过滤异常值

Filteringor transform outliers is largely a matter of applying array operations. Consider a DataFrame with some normally distributed data:
(异常值检测, 通常就是一个数组操作)

data = pd.DataFrame(np.random.randn(1000, 4))

data.describe()

count

1000.000000

mean

-0.004497

0.059209

-0.041191

0.038118

std

0.993640

1.011040

1.020444

0.999947

min

-3.147704

-2.843186

-4.352625

-3.129246

25%

-0.695843

-0.660866

-0.727212

-0.640414

50%

0.022815

-0.010976

-0.024072

0.059942

75%

0.699713

0.774702

0.674284

0.712486

max

3.005114

3.279071

3.033638

3.515276

Suppose you wanted to find values in one of the columns exceeding 3 in absolute value:

col = data[2]  # 名字为 2 的列

col[np.abs(col) > 3]

26    -3.495486
324   -3.130600
364    3.007297
380    3.033638
566   -4.352625
791    3.010874
833   -3.226842
Name: 2, dtype: float64

To select all rows having a value exceeding 3 or -3 you can use the any method on a boolean DataFrame:

data[(np.abs(data)>3).any(1)]  # 选取所有行, 值在-3,3之间

0.756734

-0.109773

0.737890

-3.025528

-0.766688

0.791026

-3.495486

0.689195

158

-0.403342

0.347707

1.014756

3.279205

324

-0.070037

-0.240627

-3.130600

0.104002

339

-1.450547

1.596675

0.609930

3.317804

364

-1.005375

-0.176153

3.007297

-0.488084

380

-0.520558

-1.530794

3.033638

-0.437202

391

0.991961

-0.441668

-1.225294

3.515276

394

0.201471

3.006423

0.052278

0.329850

404

1.129113

3.279071

-0.251223

-0.479738

439

0.535682

3.241431

1.109137

-0.726348

491

-3.147704

0.563707

0.017993

-1.139543

566

-1.652402

-1.073253

-4.352625

-0.036447

584

3.005114

-1.024521

-0.213738

1.480222

630

-3.023823

-0.623671

-1.414060

-0.996899

674

0.045482

-0.189843

0.817160

3.027287

760

2.101176

3.251508

-1.328292

-0.406980

791

1.190759

0.527168

3.010874

-1.135035

833

0.321461

-0.049159

-3.226842

0.264828

885

-3.031641

1.004770

-0.674609

0.702201

933

0.603761

-0.154076

0.061579

-3.129246

Values can be set based on these criteria. Here is code to cap values outside the interval -3 to 3.

data[np.abs(data) > 3] = np.sign(data) * 3

data.describe()

count

1000.000000

mean

-0.004298

0.058431

-0.039037

0.037133

std

0.993000

1.008671

1.012805

0.995857

min

-3.000000

-2.843186

-3.000000

25%

-0.695843

-0.660866

-0.727212

-0.640414

50%

0.022815

-0.010976

-0.024072

0.059942

75%

0.699713

0.774702

0.674284

0.712486

max

3.000000

The statement np.sign(data) produces 1 and -1 base on whether the values in data are positive or negative:

np.sign(data).head() # np.sing(data), 正数赋值为1, 零为 0, 负数为-1

1.0

-1.0

1.0

-1.0

1.0

-1.0

1.0

-1.0

1.0

-1.0

# cj test

np.sign(100)
np.sign(0)
np.sign(-888)

排列和随机采样

Permuting(randomly reordering 随机排列) a Series or the rows in a DataFrame is easy to do using the numpy.random.permutation function. Calling permutation with the lenght of the axis you want to permute produces an array of integers indicating the new ordering:

df = pd.DataFrame(np.arange(5*4).reshape((5,4)))

"随机排列 0 - 5"
sampler = np.random.permutation(5)

sampler

'随机排列 0 - 5'






array([2, 1, 0, 3, 4])

That array can then be used in iloc-based indexing or the equivalent take function:

df

df.take(sampler)  # 按行的随机 [2,1,3,0,4]

To select a random subset without replacement, you can use the sample method on Series and DataFrame.

"随机选3行, 不带重复"
df.sample(n=3)

'随机选3行, 不带重复'

To generate a sample with replacement (to allow repeat choices), pass replace=True to sample:

choices = pd.Series([5, 7, -1, 6, 4])

"随机, 可重复采样, boosting"
draws = choices.sample(n=10, replace=True)

draws

'随机, 可重复采样, boosting'






1    7
1    7
2   -1
4    4
0    5
3    6
2   -1
4    4
2   -1
4    4
dtype: int64

虚拟变量哑变量one-hot编码

Another type of transformation for statistical modeling or machine learing applications is converting a categorical variable into "dummy" or "indicator" matrix or DataFrame with k columns containing all is 0s.(热独编码呗) pandas has a get_dummies function for doing this, though devising(发明) one youself is not difficult(尽管自己实现也不难). Let's return to an earlier example DataFrame:

df = pd.DataFrame({'key':"b,b,a,c,a,b".split(','),
                 'data1': range(6)})

df

key

data1

"热独编码而已"
pd.get_dummies(df['key'])

'热独编码而已'

In some cases, you may want to add a prefix to the columns in the indicator DataFrame, which can then be merged with the other data.get_dummies has a prefix argument for doing this:

dummies = pd.get_dummies(df['key'], prefix='key')  

"添加新列, [[]], 这种参数, 试一试就ok"
df_with_dummy = df[['data1']].join(dummies)

df_with_dummy

'添加新列, [[]], 这种参数, 试一试就ok'

data1

key_a

key_b

key_c

If a row in DataFrame belongs to multiple categories, things are a bit more complicated Let's look at the MovieLens 1 M dataset, which is investigated(研究) in more detail in Chapter 14.

movies = pd.read_table("../datasets/movielens/movies.dat", sep="::",
                      header=None)

# pandas 啥都能读, 而且还能自己编码成utf8

c:\python\python36\lib\site-packages\ipykernel_launcher.py:2: ParserWarning: Falling back to the 'python' engine because the 'c' engine does not support regex separators (separators > 1 char and different from '\s+' are interpreted as regex); you can avoid this warning by specifying engine='python'.

movies.info()
movies.head()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3883 entries, 0 to 3882
Data columns (total 3 columns):
0    3883 non-null int64
1    3883 non-null object
2    3883 non-null object
dtypes: int64(1), object(2)
memory usage: 91.1+ KB

Toy Story (1995)

Animation|Children's|Comedy

Jumanji (1995)

Adventure|Children's|Fantasy

Grumpier Old Men (1995)

Comedy|Romance

Waiting to Exhale (1995)

Comedy|Drama

Father of the Bride Part II (1995)

Comedy

# 自己重写取列名, 给属性columns, df原地的哦

movies.columns = ['movie_id', "title", 'genres']

movies.head()

movie_id

title

genres

Toy Story (1995)

Animation|Children's|Comedy

Jumanji (1995)

Adventure|Children's|Fantasy

Grumpier Old Men (1995)

Comedy|Romance

Waiting to Exhale (1995)

Comedy|Drama

Father of the Bride Part II (1995)

Comedy

Adding indicator variables(添加指标变量) for each genre requires a little bit of wrangling. First, we exract the list of unique genres in the dataset:

all_genres = []

for x in movies.genres:
    all_genres.extend(x.split('|'))
    
"去重后"    
genres = pd.unique(all_genres)

genres

'去重后'






array(['Animation', "Children's", 'Comedy', 'Adventure', 'Fantasy',
       'Romance', 'Drama', 'Action', 'Crime', 'Thriller', 'Horror',
       'Sci-Fi', 'Documentary', 'War', 'Musical', 'Mystery', 'Film-Noir',
       'Western'], dtype=object)

One way to construct the indicator DataFrame is to start with a DataFrame of all zeros:

# zero_matrix = np.zeros((movies.shap[0], len(genres)))

zero_matrix = np.zeros((len(movies), len(genres)))

Now, iterate through each movie and set entries in each row of dummies to it. To do this, we use the dummies. columns to compute the column indices for each genre:

gen = movies.genres[0]  # 传数字, 行

"Animation|Children's|Comedy"

gen.split('|')

['Animation', "Children's", 'Comedy']

dummies.columns.get_indexer(gen.split("|"))

array([-1, -1, -1], dtype=int64)

Then, we can use .iloc to set values based on thees indices:

for i, gen in enumerate(movies.genres):
    indices = dummies.columns.get_indexer(genres.split('|'))
    dummies.iloc[i, indices] = 1

---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

<ipython-input-62-c595faeffed8> in <module>
      1 for i, gen in enumerate(movies.genres):
----> 2     indices = dummies.columns.get_indexer(genres.tostring.split('|'))
      3     dummies.iloc[i, indices] = 1


AttributeError: 'builtin_function_or_method' object has no attribute 'split'

arr = np.array([1,2,3])

arr.tolist().split(',')

---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

<ipython-input-69-a97f9bb706d8> in <module>
----> 1 arr.tolist().split(',')


AttributeError: 'list' object has no attribute 'split'

For much large data, this method of consturction indicator variables with multiple membership is not especially speedy. It would be better to write a lower-level function that writes directly to a NumPy array, and then wrap the result in a DataFrame.
对于大量数据，这种具有多个成员资格的建立指标变量的方法并不是特别快。编写一个直接写入NumPy数组的低级函数，然后将结果包装在DataFrame中会更好。

np.random.seed(123456)  # cj 还是会变, 有点打脸, 都是正数吧

values = np.random.rand(10)

values

array([0.37301223, 0.44799682, 0.12944068, 0.85987871, 0.82038836,
       0.35205354, 0.2288873 , 0.77678375, 0.59478359, 0.13755356])

bins = [0, 0.2, 0.4, 0.6, 0.8, 1]

pd.get_dummies(pd.cut(values, bins))

(0.0, 0.2]

(0.2, 0.4]

(0.4, 0.6]

(0.6, 0.8]

(0.8, 1.0]

We set the random seed with numpy.random.seed to make the example deterministic. We will look again at pandas.get_dummies later in the book. (cj, 设置的随机种子, 目的其实是为了做测试, 保证在n次的抽样, 结果是一样的.)

耐心和恒心, 总会获得回报的.

本文地址：https://www.henghost.com/jishu/5694/