🗂️ HDFS Distributed File System

🗂️ HDFS 分布式文件系统

Overview of HDFS

HDFS 概述

“Hadoop Distributed File System (HDFS) is a Distributed File System, built on the idea of one write, many reads, high fault tolerance, and high throughput.”

“Hadoop 分布式文件系统 (HDFS) 是一个分布式文件系统，基于一次写入、多次读取、高容错性和高吞吐量的理念构建。”

Key Features of HDFS

HDFS 的主要特性

• One Write, Many Reads: Allows multiple reads after a single write operation, enhancing efficiency.
• 一次写入，多次读取：允许在单次写入操作后进行多次读取，从而提高效率。
• High Fault Tolerance: Designed to handle hardware failures gracefully.
• 高容错性：旨在优雅地处理硬件故障。
• High Throughput: Optimized for large data sets and batch processing.
• 高吞吐量：针对大数据集和批处理进行了优化。

Components

组件

• DataNode: Responsible for reading and writing requests from file system clients.
• DataNode (数据节点)：负责处理来自文件系统客户端的读写请求。

🛠️ Hadoop Shell Commands

🛠️ Hadoop Shell 命令

HDFS is accessed through a set of shell commands to perform various file operations, such as moving, deleting, and changing file permissions. Below are the essential commands categorized by their functionality.

HDFS 通过一组 shell 命令进行访问，以执行各种文件操作，例如移动、删除和更改文件权限。以下是按功能分类的基本命令。

Basic Commands: View and Create Files and Directories

基本命令：查看和创建文件及目录

Count and Size Commands

计数和大小命令

Copy and Move Commands

复制和移动命令

Remove or Delete Commands

删除命令

Other Commands

其他命令

🛠️ Hadoop Administration Commands

🛠️ Hadoop 管理命令

🗂️ Hadoop Commands and HDFS Architecture

🗂️ Hadoop 命令和 HDFS 架构

🛠️ Hadoop Namenode Command

🛠️ Hadoop Namenode 命令

The Namenode Command is utilized to manage the Hadoop Namenode. The general syntax is:

Namenode 命令 用于管理 Hadoop Namenode。常规语法是：

1

hadoop namenode [-format] | [-upgrade] | [-rollback] | [-finalize] | [-importCheckpoint]

Options and Descriptions

选项和说明

🚀 Hadoop Job Command

🚀 Hadoop 作业命令

The Hadoop Job Command is essential for interacting with MapReduce jobs. The syntax is:

Hadoop 作业命令 对于与 MapReduce 作业交互至关重要。语法是：

1

hadoop job [GENERIC_OPTIONS] [-submit] | [-status] | [-counter] | [-kill] | [-events <#-of-events>] | [-history [all]] | [-list [all]] | [-kill-task] | [-fail-task]

Options and Descriptions

选项和说明

📁 HDFS Write Architecture

📁 HDFS 写入架构

HDFS follows the Write Once – Read Many principle. Data cannot be edited once stored, but new data can be appended.

HDFS 遵循 一次写入 – 多次读取 的原则。数据一旦存储就无法编辑，但可以追加新数据。

HDFS Write Request Steps

HDFS 写入请求步骤

1. Client Interaction: The client interacts with the NameNode to gain permission and obtain the IPs of DataNodes for writing.
2. 客户端交互：客户端与 NameNode (名称节点) 交互以获取权限并获取用于写入的 DataNodes (数据节点) 的 IP 地址。
3. DataNode Interaction: The client writes data directly to the DataNodes, which creates replicas based on the defined replication factor (e.g., 3).
4. DataNode 交互：客户端直接将数据写入 DataNodes，DataNodes 根据定义的复制因子（例如 3）创建副本。

Replication Factor

复制因子

If the replication factor is 3, then at least 3 copies of each data block are created across different DataNodes.

如果复制因子为 3，则每个数据块至少在不同的 DataNodes 上创建 3 个副本。

📥 HDFS Read Architecture

📥 HDFS 读取架构

The HDFS read operation is straightforward. Here’s how it works:

HDFS 读取操作非常简单。其工作原理如下：

HDFS Read Request Steps

HDFS 读取请求步骤

1. The client requests block metadata from the NameNode.
2. 客户端向 NameNode (名称节点) 请求块元数据。
3. The NameNode returns a list of DataNodes where each block is stored.
4. NameNode 返回存储每个块的 DataNodes (数据节点) 列表。
5. The client connects to the DataNodes and reads data in parallel (e.g., Block A from DataNode 1 and Block B from DataNode 3).
6. 客户端连接到 DataNodes 并并行读取数据（例如，从 DataNode 1 读取块 A，从 DataNode 3 读取块 B）。
7. The client combines the blocks to reconstruct the original file.
8. 客户端组合这些块以重建原始文件。

Acknowledgment Flow

确认流程

After creating the required replicas during a write operation, an acknowledgment is sent to the client.

在写入操作期间创建所需副本后，会向客户端发送确认。

📝 Client File Writing Operation

📝 客户端文件写入操作

The file operation classes in Hadoop are found in the org.apache.hadoop.fs package. These APIs support operations including opening, reading, writing, and deleting files.

Hadoop 中的文件操作类位于 org.apache.hadoop.fs 包中。这些 API 支持包括打开、读取、写入和删除文件在内的操作。

Writing to HDFS

写入 HDFS

1. The client sends a create request to the DistributedFileSystem.
2. 客户端向 DistributedFileSystem (分布式文件系统) 发送创建请求。
3. The DistributedFileSystem makes an RPC call to the namenode to create a new file.
4. DistributedFileSystem 向 namenode 发出 RPC 调用以创建新文件。
5. The client receives an FSDataOutputStream to start writing data.
6. 客户端接收一个 FSDataOutputStream 以开始写入数据。

File Write Process

文件写入过程

• Data is streamed to the first DataNode, which forwards it to the next in the pipeline.
• 数据被流式传输到第一个 DataNode，然后由该 DataNode 将其转发到管道中的下一个 DataNode。
• An internal queue, known as the ack queue, maintains packets waiting for acknowledgment.
• 一个内部队列，称为 ack 队列 (确认队列)，维护等待确认的数据包。

Example Code for Writing to HDFS

写入 HDFS 的示例代码

1
2
3
4
5
6
7
8
9
10
11
12

InputStream inputStream = new BufferedInputStream(new FileInputStream("/Data/dependencies.txt"));
OutputStream outputStream = hdfs.create(new Path(uri), new Progressable() {
    public void progress() {
        System.out.println("*");
    }
});
try {
    IOUtils.copyBytes(inputStream, outputStream, 4096, false);
} finally {
    IOUtils.closeStream(inputStream);
    IOUtils.closeStream(outputStream);
}

📖 Client File Read Operation

📖 客户端文件读取操作

Reading from HDFS

从 HDFS 读取

1. The client opens the file using the open() function of FileSystem.
2. 客户端使用 FileSystem (文件系统) 的 open() 函数打开文件。
3. The DistributedFileSystem retrieves data block information from metadata nodes.
4. DistributedFileSystem (分布式文件系统) 从元数据节点检索数据块信息。
5. The client reads data from the nearest DataNodes.
6. 客户端从最近的 DataNodes (数据节点) 读取数据。

Example Code for Reading from HDFS

从 HDFS 读取的示例代码

1
2
3

InputStream inputStream = hdfs.open(path);
IOUtils.copyBytes(inputStream, System.out, 4096, false);
IOUtils.closeStream(inputStream);

Handling Errors

处理错误

If communication fails with a DataNode, the client attempts to connect to another DataNode containing the required data block.

如果与 DataNode (数据节点) 的通信失败，客户端会尝试连接到包含所需数据块的另一个 DataNode。

🗂️ File Operations

🗂️ 文件操作

Renaming Files

重命名文件

To rename a file in HDFS, you need to pass:

要在 HDFS 中重命名文件，您需要传递：

• src: The original file name.
• src：原始文件名。
• dst: The new file name.
• dst：新文件名。

1

fs.rename(currentName, reName);

Deleting Files

删除文件

To delete a file, the command is:

要删除文件，命令是：

1

public abstract boolean delete(Path f, boolean recursive) throws IOException

• f: The path of the file to be deleted.
• f：要删除的文件的路径。
• recursive: If true, deletes the directory and its contents.
• recursive：如果为 true，则删除目录及其内容。

🗂️ HDFS Operations

🗂️ HDFS 操作

File Deletion

文件删除

To delete a file in HDFS, you can use the following code:

要在 HDFS 中删除文件，可以使用以下代码：

1
2
3

Path delef = new Path("/home/test_2.txt");
boolean delResult = fs.delete(delef, true);
System.out.println(delResult == true ? "Delete success" : "Delete failure");

• Return Values:

• 返回值：

  • True: Indicates successful deletion.
  • True (真)：表示删除成功。
  • False: Indicates failed deletion.
  • False (假)：表示删除失败。

Directory Creation

目录创建

To create a directory at a specified location, use:

要在指定位置创建目录，请使用：

1

public boolean mkdirs(Path f) throws IOException

• FileSystem Instance:

• 文件系统实例：

1

FileSystem fs2 = FileSystem.get(URI.create(address), conf);

• Creating the Directory:

• 创建目录：

1

boolean isCreated = fs2.mkdirs(new Path(dir));

• Return Values:

• 返回值：

  • True: Indicates successful creation.
  • True (真)：表示创建成功。
  • False: Indicates failed creation.
  • False (假)：表示创建失败。

Example Output

示例输出

1
2
3
4
5

if (isCreated) {
    System.out.println("Directory created, check your HDFS");
} else {
    System.out.println("Directory can’t create check Hadoop status in Linux");
}

1
2
3
4
5

if (isCreated) {
    System.out.println("目录已创建，请检查您的 HDFS");
} else {
    System.out.println("目录无法创建，请检查 Linux 中的 Hadoop 状态");
}

Listing Folder Contents

列出文件夹内容

To display all directories and files in the current directory, use:

要显示当前目录中的所有目录和文件，请使用：

1

public abstract FileStatus[] listStatus(Path f) throws FileNotFoundException, IOException

• FileStatus:

• FileStatus (文件状态)：

  • Represents the file status including path information of the given directory.

• 表示文件状态，包括给定目录的路径信息。

Example Code

示例代码

1
2
3
4
5
6
7

uri = "hdfs://niit:9000/niit";
FileStatus[] filestatus = fs.listStatus(new Path(uri));
Path[] paths = FileUtil.stat2Paths(filestatus);
System.out.println("*******Files in NIIT directory are******");
for (Path path : paths) {
    System.out.println(path);
}

1
2
3
4
5
6
7

uri = "hdfs://niit:9000/niit";
FileStatus[] filestatus = fs.listStatus(new Path(uri));
Path[] paths = FileUtil.stat2Paths(filestatus);
System.out.println("*******NIIT 目录中的文件如下******");
for (Path path : paths) {
    System.out.println(path);
}