Top 30 Most Asked Hadoop Admin Interview Question

Are you looking for Hadoop Admin Interview Question? Then you are in the right place. Here I will discuss the Top 30 Most Asked Hadoop Admin Interview Question with Answer. Give your few minutes here and mug these Hadoop Admin Interview Question with their answers. I have collected these questions from various sources and sum up here just for you.

Hello, & Welcome!

In this blog, I am gonna tell you-

1. Top 30 Most Asked Hadoop Admin Interview Question with Answer

So without wasting your time, let’s get started,

Top 30 Most Asked Hadoop Admin Interview Question and Answer-

Question 1- Which operating system(s) supports for production Hadoop deployment?

Answer- The main supported operating system is Linux. However, with some additional software Hadoop can be deployed on Windows.

Question 2- What is the role of the namenode?

Answer- The namenode is the “brain” of the Hadoop cluster and responsible for managing the distribution blocks on the system based on the replication policy. The namenode also supplies the specific addresses for the data based on the client requests.

Question 3- What happens on the namenode when a client tries to read a data file?

Answer- The namenode will look up the information about a file in the edit file and then retrieve the remaining information from the filesystem memory snapshot. Since the name node needs to support a large number of the clients, the primary name node will only send information back for the data location. The data node itself is responsible for the retrieval.

Question 4- What are the hardware requirements for a Hadoop cluster (primary and secondary name nodes and data nodes)?

Answer- There are no requirements for data nodes. However, the name nodes require a specified amount of RAM to store the filesystem image in memory. Based on the design of the primary name node and secondary name node, the entire filesystem information will be stored in memory. Therefore, both name nodes need to have enough memory to contain the entire filesystem image.

Question 5- On What mode(s) Hadoop code can run in?

Answer- You can deploy Hadoop in stand-alone mode, pseudo-distributed mode or fully-distributed mode. The design of Hadoop is to deploy on a multi-node cluster. However, we can deploy on a single machine and as a single process for testing purposes.

Question 6- How would a Hadoop administrator deploy various components of Hadoop in production?

Answer- Deploy name node and job tracker on the master node, and deploy data nodes and task trackers on multiple slave nodes. There is a need for only one name node and job tracker on the system. The number of data nodes depends on the available hardware.

Question 7- What is the best practice to deploy the secondary name node?

Answer- Deploy a secondary name node on a separate standalone machine. The secondary name node needs to be deployed on a separate machine. It will not interfere with primary name node operations in this way. The secondary name node must have the same memory requirements as the main name node. 

Question 8- Is there a standard procedure to deploy Hadoop?

Answer- No, there are some differences between various distributions. However, they all require that Hadoop jars be installed on the machine. There are some common requirements for all Hadoop distributions but the specific procedures will be different for different vendors. Since they all have some degree of proprietary software.

Question 9- What is the role of the secondary name node?

Answer- Secondary name node performs CPU intensive operation of combining edit logs and current filesystem snapshots. The secondary name node was separated out as a process due to having CPU intensive operations and additional requirements for metadata back-up.

Question 10- What are the side effects of not running a secondary name node?

Answer- The cluster performance will degrade over time since the edit log will grow bigger and bigger. If the secondary name node is not running at all, the edit log will grow significantly and it will slow the system down. Also, the system will go into a safe mode for an extended time. Since the name node needs to combine the edit log and the current filesystem checkpoint image.

Question 11- What happens if a data node loses a network connection for a few minutes?

Answer- The namenode will detect that a data node is not responsive and will start replication of the data from the remaining replicas.  The name node maintains replication factor. The name node monitors the status of all data nodes and keeps track of which blocks are located on that node. The moment the data node is not available it will trigger replication of the data from the existing replicas. However, if the data node comes back up, over replicated data will be deleted.

Note: the data might be deleted from the original data node.

Question 12- What happens if one of the data nodes has a much slower CPU?

Answer- The task execution will be as fast as the slowest worker. However, if speculative execution is enabled, the slowest worker will not have such a big impact Hadoop was specifically designed to work with commodity hardware. The speculative execution helps to offset the slow workers. The multiple instances of the same task will be created. And the job tracker will take the first result into consideration and the second instance of the task will be killed.

Question 13- What is speculative execution?

Answer- If we enable speculative execution, the job tracker will issue multiple instances of the same task on multiple nodes. And will take the result of the task that finished first. The other instances of the task will be killed.

We can use speculative execution to offset the impact of the slow workers in the cluster. The job tracker creates multiple instances of the same task and takes the result of the first successful task. The rest of the tasks will be discarded.

Question 14- How many racks do you need to create a Hadoop cluster in order to make sure that the cluster operates reliably?

Answer- In order to ensure reliable operation, it is mandatory to have at least 2 racks with rack placement configured. Hadoop has a built-in rack awareness mechanism. That allows data distribution between different racks based on the configuration.

Question 15- Are there any special requirements for the name node?

Answer- Yes, the namenode holds information about all files in the system and needs to be extra reliable. The name node is a single point of failure. It needs to be extra reliable and replicate the metadata in multiple places. Note that the community is working on solving the single point of failure issue with the name node.

Question 16- If you have a file 128M size and the replication factor is set to 3, how many blocks can you find on the cluster that will correspond to that file (assuming the default apache and Cloudera configuration)?

Answer- Based on the configuration settings the file will be divided into multiple blocks according to the default block size of 64M. 128M / 64M = 2 . Each block replicates according to replication factor settings (default 3). 2 * 3 = 6.

Question 17- What is distributed copy (distcp)?

Answer- Distcp is a Hadoop utility for launching MapReduce jobs to copy data. The primary usage is for copying a large amount of data. One of the major challenges in the Hadoop environment is copying data across multiple clusters. Distcp will allow multiple data nodes to be leveraged for parallel copying of the data.

Question 18- What is the replication factor?

Answer- The replication factor controls how many times each individual block can be replicated. Data is replicated in the Hadoop cluster based on the replication factor. The high replication factor guarantees data availability in the event of failure.

Question 19- What daemons run on Master nodes?

Answer- NameNode, Secondary NameNode, and JobTracker.

Hadoop consist of five separate daemons and each of these daemons run in its own JVM. NameNode, Secondary NameNode, and JobTracker run on Master nodes. DataNode and TaskTracker run on each Slave nodes.

Question 20- What is rack awareness?

Answer- Rack awareness is the way in which the name node decides how to place blocks based on the rack definitions. Hadoop will try to minimize the network traffic between data nodes within the same rack. And will only contact remote racks if it has to. The name node is able to control this due to rack awareness

Question 21- What is the role of the job tracker in a Hadoop cluster? 

Answer- The job tracker is responsible for scheduling tasks on slave nodes, collecting results, retrying failed tasks. The job tracker is the main component of the map-reduce execution. It controls the division of the job into smaller tasks, submits tasks to individual task trackers, tracks the progress of the jobs and reports results back to calling code.

Question 22- How does the Hadoop cluster tolerate data node failures?

Answer- Since the design of Hadoop is to run on commodity hardware. So you can expect data node failure. Namenode keeps track of all available data nodes and actively maintains the replication factor on all data.

The name node actively tracks the status of all data nodes. And acts immediately if the data nodes become non-responsive. The namenode is the central “brain” of the HDFS and starts replication of the data the moment when it detects disconnect.

Question 23- What is the procedure for namenode recovery?

Answer- We can recover a name node in two ways. One is starting a new name node from backup metadata. Second is promoting the secondary name node to the primary name node. 

The name node recovery procedure is very important to ensure the reliability of the data. You can accomplish by starting a new name node using backup data. Or by promoting the secondary name node to the primary.

Question 24- Web-UI shows that half of the data nodes are in decommissioning mode. What does that mean? Is it safe to remove those nodes from the network?

Answer- This means that the name node is trying to retrieve data from those data nodes by moving replicas to remaining data nodes. There is a possibility that we lost data if the administrator removes those data nodes before decommissioning finished. 

Due to the replication strategy, it is possible to lose some data due to data nodes removal en masse prior to completing the decommissioning process. Decommissioning refers to the name node trying to retrieve data from data nodes by moving replicas to remaining data nodes.

Question 25- What does the Hadoop administrator have to do after adding new data nodes to the Hadoop cluster?

Answer- Since the new nodes will not have any data on them. The administrator needs to start the balancer to redistribute data evenly between all nodes.

The Hadoop cluster will detect new data nodes automatically. However, in order to optimize the cluster performance, it is important to start to rebalance to redistribute the data between data nodes evenly.

Question 26- If the Hadoop administrator needs to make a change, which configuration file does he need to change?

Answer- Each node in the Hadoop cluster has its own configuration files and the changes need to be made in every file. One of the reasons for this is that configuration can be different for every node.

Question 27- Map Reduce jobs are failing on a cluster that was just restarted. They worked before restart. What could be wrong?

Answer- The cluster is in a safe mode. The administrator needs to wait for the name node to exit the safe mode before restarting the jobs again 

This is a very common mistake by Hadoop administrators when there is no secondary name node on the cluster and the cluster has not been restarted in a long time. The name node will go into safe mode and combine the edit log and current file system timestamp

Question 28- Map Reduce jobs take too long. What you can do to improve the performance of the cluster?

Answer- One of the most common reasons for performance problems on the Hadoop cluster is the uneven distribution of the tasks. The number of tasks has to match the number of available slots on the cluster

Hadoop is not a hardware aware system. It is the responsibility of the developers and the administrators to make sure that the resource supply and demand match.

Question 29- How often do you need to reformat the name node?

Answer- Never. The name node needs to formatted only once in the beginning. Reformatting of the namenode will lead to loss of the data on the entire

The namenode is the only system that needs to be formatted only once. It will create the directory structure for file system metadata and create a namespace ID for the entire file system.

Question 30- After increasing the replication level, I still see that data is under replicated. What could be wrong?

Answer- Data replication takes time due to large quantities of data. The Hadoop administrator should allow sufficient time for data replication. Depending on the data size the data replication will take some time. Hadoop cluster still needs to copy data around and if the data size is big enough it is not uncommon that replication will take from a few minutes to a few hours.

I hope you have read and understood all Hadoop Admin Interview Question. All the best for your career.

Enjoy Learning

All the Best!

Related Search

Best Online Courses for Data Science to become A Skilled Data Scientist

15 Best Books on Data Science Everyone Should Read in 2024
Data Science vs Data Analyst: Ultimate Guide to Clear Doubts
How to make Data Science Resume to Get Hired?
What is Big Data Analytics? Things no one tells you
Data Science: Top 8 Most Demanding Skills to Get You Hired

Explore More about Data Science, Visit Here

Thank YOU!

Though of the Day…

‘ It’s what you learn after you know it all that counts.’

– John Wooden