我正在使用命令sbatch和下一個配置排隊任務：#SBATCH --job-name=dask-test#SBATCH --ntasks=1#SBATCH --cpus-per-task=10#SBATCH --mem=80G#SBATCH --time=00:30:00#SBATCH --tmp=10G#SBATCH --partition=normal#SBATCH --qos=normalpython ./dask-test.pypython腳本大致如下：import pandas as pdimport dask.dataframe as ddimport numpy as npfrom dask.distributed import Client, LocalClusterprint("Generating LocalCluster...")cluster = LocalCluster()print("Generating Client...")client = Client(cluster, processes=False)print("Scaling client...")client.scale(8)data = dd.read_csv( BASE_DATA_SOURCE + '/Data-BIGDATFILES-*.csv', delimiter=';',)def get_min_dt(): min_dt = data.datetime.min().compute() print("Min is {}".format())print("Getting min dt...")get_min_dt()第一個問題是文本“Generating LocalCluster...”打印了 6 次，這讓我懷疑腳本是否同時運行了多次。其次，在幾分鐘不打印任何內(nèi)容后，我收到以下消息：/anaconda3/lib/python3.7/site-packages/distributed/node.py:155: UserWarning: Port 8787 is already in use.Perhaps you already have a cluster running?Hosting the HTTP server on port 37396 instead http_address["port"], self.http_server.port
1 回答
慕斯王
TA貢獻1864條經(jīng)驗 獲得超2個贊
經(jīng)過一些研究，我可以得到一個解決方案。不太確定原因，但非常確定它有效。
LocalCluster、Client 及其之后的所有代碼（將分發(fā)執(zhí)行的代碼）的實例化不得在 Python 腳本的模塊級別。相反，此代碼必須位于方法中或 __main__ 塊內(nèi)，如下所示：
import pandas as pd
import dask.dataframe as dd
import numpy as np
from dask.distributed import Client, LocalCluster
if __name__ == "__main__":
print("Generating LocalCluster...")
cluster = LocalCluster()
print("Generating Client...")
client = Client(cluster, processes=False)
print("Scaling client...")
client.scale(8)
data = dd.read_csv(
BASE_DATA_SOURCE + '/Data-BIGDATFILES-*.csv',
delimiter=';',
)
def get_min_dt():
min_dt = data.datetime.min().compute()
print("Min is {}".format())
print("Getting min dt...")
get_min_dt()
這個簡單的改變帶來了不同。在該問題線程中找到了解決方案：https://github.com/dask/distributed/issues/2520#issuecomment-470817810
添加回答
舉報
0/150
提交
取消