我正在使用命令sbatch和下一個(gè)配置排隊(duì)任務(wù)：#SBATCH --job-name=dask-test#SBATCH --ntasks=1#SBATCH --cpus-per-task=10#SBATCH --mem=80G#SBATCH --time=00:30:00#SBATCH --tmp=10G#SBATCH --partition=normal#SBATCH --qos=normalpython ./dask-test.pypython腳本大致如下：import pandas as pdimport dask.dataframe as ddimport numpy as npfrom dask.distributed import Client, LocalClusterprint("Generating LocalCluster...")cluster = LocalCluster()print("Generating Client...")client = Client(cluster, processes=False)print("Scaling client...")client.scale(8)data = dd.read_csv( BASE_DATA_SOURCE + '/Data-BIGDATFILES-*.csv', delimiter=';',)def get_min_dt(): min_dt = data.datetime.min().compute() print("Min is {}".format())print("Getting min dt...")get_min_dt()第一個(gè)問題是文本“Generating LocalCluster...”打印了 6 次，這讓我懷疑腳本是否同時(shí)運(yùn)行了多次。其次，在幾分鐘不打印任何內(nèi)容后，我收到以下消息：/anaconda3/lib/python3.7/site-packages/distributed/node.py:155: UserWarning: Port 8787 is already in use.Perhaps you already have a cluster running?Hosting the HTTP server on port 37396 instead http_address["port"], self.http_server.port
在 HPC 集群中創(chuàng)建 Dask LocalCluster 實(shí)例時(shí) SLURM 任務(wù)失敗
繁星點(diǎn)點(diǎn)滴滴
2023-02-15 16:11:48
