如何像使用熊貓數(shù)據(jù)框那樣查看Spark數(shù)據(jù)框中每種數(shù)據(jù)類型的計(jì)數(shù)？例如，假設(shè)df是熊貓數(shù)據(jù)幀：>>> df.info(verbose=True)<class 'pandas.core.frame.DataFrame'>RangeIndex: 5 entries, 0 to 4Data columns (total 3 columns):int_col 5 non-null int64text_col 5 non-null objectfloat_col 5 non-null float64**dtypes: float64(1), int64(1), object(1)**memory usage: 200.0+ bytes我們可以很清楚地看到每種數(shù)據(jù)類型的計(jì)數(shù)。如何使用Spark數(shù)據(jù)框執(zhí)行類似操作？也就是說(shuō)，如何看到有多少列是浮動(dòng)的，有多少列是int的，有多少列是對(duì)象的？
3 回答
茅侃侃
TA貢獻(xiàn)1842條經(jīng)驗(yàn) 獲得超22個(gè)贊
下面的代碼應(yīng)該可以為您帶來(lái)理想的結(jié)果
# create data frame
df = sqlContext.createDataFrame(
[(1,'Y','Y',0,0,0,2,'Y','N','Y','Y'),
(2,'N','Y',2,1,2,3,'N','Y','Y','N'),
(3,'Y','N',3,1,0,0,'N','N','N','N'),
(4,'N','Y',5,0,1,0,'N','N','N','Y'),
(5,'Y','N',2,2,0,1,'Y','N','N','Y'),
(6,'Y','Y',0,0,3,6,'Y','N','Y','N'),
(7,'N','N',1,1,3,4,'N','Y','N','Y'),
(8,'Y','Y',1,1,2,0,'Y','Y','N','N')
],
('id', 'compatible', 'product', 'ios', 'pc', 'other', 'devices', 'customer', 'subscriber', 'circle', 'smb')
)
# Find data types of data frame
datatypes_List = df.dtypes
# Querying datatypes_List gives you column and its data type as a tuple
datatypes_List
[('id', 'bigint'), ('compatible', 'string'), ('product', 'string'), ('ios', 'bigint'), ('pc', 'bigint'), ('other', 'bigint'), ('devices', 'bigint'), ('customer', 'string'), ('subscriber', 'string'), ('circle', 'string'), ('smb', 'string')]
# create empty dictonary to store output values
dict_count = {}
# Loop statement to count number of times the data type is present in the data frame
for x, y in datatypes_List:
dict_count[y] = dict_count.get(y, 0) + 1
# query dict_count to find the number of times a data type is present in data frame
dict_count
添加回答
舉報(bào)
0/150
提交
取消