我正在嘗試像下面這樣劃分?jǐn)?shù)據(jù)框：from io import StringIOimport pandas as pddata = """A,B,C87jg,28,3012h372,28,3011kj87,27,30112yh8,54,3010802h,53,30105d8b,52,3010"""df = pd.read_csv(StringIO(data), sep=',')for key, group in df.groupby(['C','B']): group.to_csv(f'df_{key}.csv', index=False)這會(huì)將按數(shù)據(jù)幀分組的結(jié)果導(dǎo)出到本地機(jī)器。有沒有辦法執(zhí)行此操作并將這些多個(gè)拆分 csv 上傳到 s3（類似于 boto3 的 put_object）
2 回答
人到中年有點(diǎn)甜
TA貢獻(xiàn)1895條經(jīng)驗(yàn) 獲得超7個(gè)贊
您也可以使用必須安裝的s3fs 。可以使用 完成安裝pip，例如：
pip install s3fs
根據(jù)您的代碼驗(yàn)證的示例：
import os
from io import StringIO
import pandas as pd
import s3fs
# I did not use my default aws profile
# so had to provide key and secret. If you use
# the default aws profile, providing `key`
# and `secret` should not be required
fs = s3fs.S3FileSystem(
? ? ? ? anon=False,
? ? ? ? key='<access_key>',
? ? ? ? secret='<secret_key>')
data = """?
A,B,C
87jg,28,3012
h372,28,3011
kj87,27,3011
2yh8,54,3010
802h,53,3010
5d8b,52,3010
"""
df = pd.read_csv(StringIO(data), sep=',')
for key, group in df.groupby(['C','B']):
? ? group.to_csv(fs.open(f's3://<bucket-name>/df_{key[0]}-M{key[1]}.csv', 'w'), index=False)
代碼正確上傳文件：
一只萌萌小番薯
TA貢獻(xiàn)1795條經(jīng)驗(yàn) 獲得超7個(gè)贊
from io import StringIO
import pandas as pd
import boto3
data = """
A,B,C
87jg,28,3012
h372,28,3011
kj87,27,3011
2yh8,54,3010
802h,53,3010
5d8b,52,3010
"""
df = pd.read_csv(StringIO(data), sep=',')
client = boto3.client('s3')
for key, group in df.groupby(['C', 'B']):
group.to_csv(f'df_{key}.csv', index=False)
client.upload_file(f'df_{key}.csv', 'my-another-test-bucket-2',
f'df_{key[0]}-M{key[1]}.csv')
S3 桶
添加回答
舉報(bào)
0/150
提交
取消