我在 OS X 中安裝了 Hadoop，并且運行良好。我的經(jīng)驗是最近的，并且正在努力學習有關使用 Hadoop 進行應用程序開發(fā)的更多信息。昨天，當我需要在 Hadoop 中查找目錄和/或文件列表時，我只需鍵入$ hadoop fs -ls 并且，它會顯示集群中的所有內容。今天，它顯示了文件系統(tǒng)中的所有本地內容。我必須提供 hdfs 的確切地址才能獲得內容列表，$ hadoop fs -ls hdfs://localhost:8020/user/myName我的core-site.xml文件和以前一樣，<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file.--><!-- Put site-specific property overrides in this file. --><!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>hadoop.tmp.dir</name> <value>/usr/local/Cellar/hadoop/hdfs/tmp</value> <description>A base for other temporary directories.</description> </property> <property> <name>fs.default.name</name> <value>hdfs://localhost:8020</value> </property></configuration>在啟動 hadoop 守護進程之前，我停止了集群并再次使用以下命令重新格式化分布式文件系統(tǒng)，以便我們可以在執(zhí)行 map-reduce 作業(yè)時將數(shù)據(jù)源放入 hdfs 文件系統(tǒng)$ hdfs namenode -format我收到管理員報告通知 FileSystem file:/// 不是 HDFS 文件系統(tǒng)，$ hadoop dfsadmin -reportWARNING: Use of this script to execute dfsadmin is deprecated.WARNING: Attempting to execute replacement "hdfs dfsadmin" instead.2018-10-18 18:01:27,316 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicablereport: FileSystem file:/// is not an HDFS file systemUsage: hdfs dfsadmin [-report] [-live] [-dead] [-decommissioning] [-enteringmaintenance] [-inmaintenance]
2 回答
慕桂英3389331
TA貢獻2036條經(jīng)驗 獲得超8個贊
您需要確保已添加了一個環(huán)境變量HADOOP_CONF_DIR，該變量被調用以設置到包含來自 Hadoop 的 XML 文件的目錄中。
您可以.bashrc在主文件夾中執(zhí)行此操作
否則，您將獲得默認文件系統(tǒng) ，file://它仍然有效并且仍然可以正常運行 MapReduce 作業(yè)
FWIW，這是我的核心站點
$ cat /usr/local/Cellar/hadoop/3.1.1/libexec/etc/hadoop/core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>file:///tmp/hadoop/hdfs/tmp</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
和 hdfs 站點
$ cat /usr/local/Cellar/hadoop/3.1.1/libexec/etc/hadoop/hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///tmp/hadoop/hdfs/names</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>file:///tmp/hadoop/hdfs/checkpoint</value>
</property>
<property>
<name>fs.checkpoint.edits.dir</name>
<value>file:///tmp/hadoop/hdfs/checkpoint-edits</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///tmp/hadoop/hdfs/data</value>
</property>
</configuration>
梵蒂岡之花
TA貢獻1900條經(jīng)驗 獲得超5個贊
在 core-site.xml 文件中進行如下編輯。
<value>hdfs://localhost.localdomain:8020/</value>
我相信缺少斜線 (8020/) 會造成這個問題。嘗試一下。
添加回答
舉報
0/150
提交
取消