背景#

polars学习系列文章，第8篇分类数据处理（Categorical data）

该系列文章会分享到github，大家可以去下载jupyter文件，进行参考学习

仓库地址：https://github.com/DataShare-duo/polars_learn

小编运行环境#

1
import sys
2

3
print('python 版本：',sys.version.split('|')[0])
4
#python 版本： 3.11.9
5

6
import polars as pl
7

8
print("polars 版本：",pl.__version__)
9
#polars 版本： 0.20.22

分类数据 Categorical data#

分类数据就是平时在数据库中能进行编码的数据，比如：性别、年龄、国家、城市、职业等等，可以对这些数据进行编码，可以节省存储空间

Polars 支持两种不同的数据类型来处理分类数据：Enum 和 Categorical

当类别预先已知时使用 Enum，需要提前提供所有类别
当不知道类别或类别不固定时，可以使用 Categorical

1
enum_dtype = pl.Enum(["Polar", "Panda", "Brown"])
2
enum_series = pl.Series(
3
    ["Polar", "Panda", "Brown", "Brown", "Polar"],
4
    dtype=enum_dtype)
5

6
cat_series = pl.Series(
7
    ["Polar", "Panda", "Brown", "Brown", "Polar"],
8
    dtype=pl.Categorical
9
)

Categorical 类型#

Categorical 相对比较灵活，不用提前获取所有的类别，当有新类别时，会自动进行编码

当对来自2个不同的 Categorical 类别列直接进行拼接时，以下这种方式会比较慢，polars 是根据字符串出现的先后顺序进行编码，不同的字符串在不同的序列里面编码可能不一样，直接合并的话全局会再进行一次编码，速度会比较慢：

1
cat_series = pl.Series(
2
    ["Polar", "Panda", "Brown", "Brown", "Polar"], dtype=pl.Categorical
3
)
4
cat2_series = pl.Series(
5
    ["Panda", "Brown", "Brown", "Polar", "Polar"], dtype=pl.Categorical
6
)
7

8
#CategoricalRemappingWarning: Local categoricals have different encodings,
9
#expensive re-encoding is done to perform this merge operation.
10
#Consider using a StringCache or an Enum type if the categories are known in advance
11
print(cat_series.append(cat2_series))

可以通过使用 polars 提供的全局字符缓存 StringCache，来提升数据处理效率

1
with pl.StringCache():
2
    cat_series = pl.Series(
3
        ["Polar", "Panda", "Brown", "Brown", "Polar"], dtype=pl.Categorical
4
    )
5
    cat2_series = pl.Series(
6
        ["Panda", "Brown", "Brown", "Polar", "Polar"], dtype=pl.Categorical
7
    )
8
    print(cat_series.append(cat2_series))

Enum#

上面来自2个不同类型列进行拼接的耗时的情况，在Enum中不会存在，因为已经提前获取到了全部的类别

1
dtype = pl.Enum(["Polar", "Panda", "Brown"])
2
cat_series = pl.Series(["Polar", "Panda", "Brown", "Brown", "Polar"], dtype=dtype)
3
cat2_series = pl.Series(["Panda", "Brown", "Brown", "Polar", "Polar"], dtype=dtype)
4

5
print(cat_series.append(cat2_series))
6
#shape: (10,)
7
#Series: '' [enum]
8
[
9
  "Polar"
10
  "Panda"
11
  "Brown"
12
  "Brown"
13
  "Polar"
14
  "Panda"
15
  "Brown"
16
  "Brown"
17
  "Polar"
18
  "Polar"
19
]

如果有编码的字符串类别，当不在提前获取的Enum中时，则会报错：OutOfBounds

1
dtype = pl.Enum(["Polar", "Panda", "Brown"])
2
try:
3
    cat_series = pl.Series(["Polar", "Panda", "Brown", "Black"], dtype=dtype)
4
except Exception as e:
5
    print(e)
6
#conversion from `str` to `enum` failed
7
#in column '' for 1 out of 4 values: ["Black"]
8
#Ensure that all values in the input column are present
9
#in the categories of the enum datatype.

比较#

Categorical vs Categorical
Categorical vs String
Enum vs Enum
Enum vs String(该字符串必须要在提前获取的Enum中)

Categorical vs Categorical#

1
with pl.StringCache():
2
    cat_series = pl.Series(["Brown", "Panda", "Polar"], dtype=pl.Categorical)
3
    cat_series2 = pl.Series(["Polar", "Panda", "Black"], dtype=pl.Categorical)
4
    print(cat_series == cat_series2)
5
#shape: (3,)
6
#Series: '' [bool]
7
[
8
  false
9
  true
10
  false
11
]

Categorical vs String#

1
cat_series = pl.Series(["Brown", "Panda", "Polar"], dtype=pl.Categorical)
2
print(cat_series <= "Cat")
3
#shape: (3,)
4
#Series: '' [bool]
5
[
6
  true
7
  false
8
  false
9
]
10

11
cat_series = pl.Series(["Brown", "Panda", "Polar"], dtype=pl.Categorical)
12
cat_series_utf = pl.Series(["Panda", "Panda", "A Polar"])
13
print(cat_series <= cat_series_utf)
14
#shape: (3,)
15
#Series: '' [bool]
16
[
17
  true
18
  true
19
  false
20
]

Enum vs Enum#

1
dtype = pl.Enum(["Polar", "Panda", "Brown"])
2
cat_series = pl.Series(["Brown", "Panda", "Polar"], dtype=dtype)
3
cat_series2 = pl.Series(["Polar", "Panda", "Brown"], dtype=dtype)
4
print(cat_series == cat_series2)
5
#shape: (3,)
6
#Series: '' [bool]
7
[
8
  false
9
  true
10
  false
11
]

Enum vs String(该字符串必须要在提前获取的Enum中)#

1
try:
2
    cat_series = pl.Series(
3
        ["Low", "Medium", "High"], dtype=pl.Enum(["Low", "Medium", "High"])
4
    )
5
    cat_series <= "Excellent"
6
except Exception as e:
7
    print(e)
8
#conversion from `str` to `enum` failed
9
#in column '' for 1 out of 1 values: ["Excellent"]
10
#Ensure that all values in the input column are present
11
#in the categories of the enum datatype.
12

13
dtype = pl.Enum(["Low", "Medium", "High"])
14
cat_series = pl.Series(["Low", "Medium", "High"], dtype=dtype)
15
print(cat_series <= "Medium")
16
#shape: (3,)
17
#Series: '' [bool]
18
[
19
  true
20
  true
21
  false
22
]
23

24
dtype = pl.Enum(["Low", "Medium", "High"])
25
cat_series = pl.Series(["Low", "Medium", "High"], dtype=dtype)
26
cat_series2 = pl.Series(["High", "High", "Low"])
27
print(cat_series <= cat_series2)
28
#shape: (3,)
29
#Series: '' [bool]
30
[
31
  true
32
  true
33
  false
34
]

历史相关文章#

以上是自己实践中遇到的一些问题，分享出来供大家参考学习，欢迎关注微信公众号：DataShare ，不定期分享干货