Big data has a big reputation

Just like Taylor Swift’s popularity, the use of big data has grown in recent years. As companies have expanded, they’ve found new ways of collecting and keeping data. And as they stored more data, the traditional database management systems became less efficient. This is where the term big data comes to play. Big data refers to the large volume of data that businesses are having to deal with day-to-day. This new era has brought forth new challenges and new technologies to address them. In this post, we’ll focus on Hadoop systems. 

Hadoop systems 

Hadoop is an open-sourced framework used to store and manage big data. Its cluster design allows for distributed storage and processing across a number of computers using programming nodes. Its structure is based on normal, commodity hardware which allows for increasing expansion as the needs to the business change. Hadoop uses horizontal scaling-based solutions to be able to process the large volumes of data it stores. The Hadoop ecosystem is comprised of four main components, and each allows different capabilities. 

Hadoop Distributed File System (HDFS) – Hadoops’s primary component, manages the filing system that allows for the storage of data across the computer cluster. The data is stored in blocks, with individual Hadoop nodes operating on the data included in these blocks. There are two types of nodes: Name nodes, which holds all the metadata and Data nodes, which handle the data requests including creation, deletion, and replication. HDFS allow for a global view and doesn’t require users to define schemas beforehand. 

MapReduce – a distributed programming model that assists with data processing. The model starts with key-value pairs that are mapped to each other, producing more key-value pairs. These outputs are grouped according to their key and the reduce function is applied. The final outputs of this process are individual subsets combined into smaller datasets that are easier to understand and analyze. It’s important to point out that MapReduce is not a programming language, but a model/paradigm. 

Yet Another Resource Negotiator (YARN) – the coordination and scheduling component of Hadoop. YARN distributes MapReduce, manages the resources to schedule user’s applications and keeps log of all jobs. 

Hadoop Common – this component includes the libraries and utilities across Hadoop modules. It provides the services and processes behind the operating and filing systems. Its library stores documentation and procedures and includes a contribution section with the Hadoop community. 

Hadoop is just one of many systems and technologies that have surfaced in response to the big data phenomenon. Spark is another framework that’s popularly used by companies. It is important for businesses to establish their needs before investing in systems like this. Sometimes, the data could be easily managed using data warehouses and data lakes instead of a system like Hadoop or Spark. 


Sources

Craig Stedman, M. L. (2021). TechTarget. Retrieved from Big Data Management: https://www.techtarget.com/searchdatamanagement/definition/big-data-management

Google Cloud. (n.d.). Retrieved from What is Hapache Hadoop?: https://cloud.google.com/learn/what-is-hadoop

Rouse, M. (2014). Techopedia. Retrieved from Hadoop Common: https://www.techopedia.com/definition/30427/hadoop-common




Comments

Post a Comment

Popular posts from this blog

The Importance of the Basics: Data Management

Image
Every day we interact with data, whether it’s through our everyday decision-making or at a business operational level. However, data is unstructured and our goal as business and data analysts is to turn it into information and knowledge. Sometimes, analysts forget this goal and focus too much on building complex databases and data warehouses without paying attention to the quality and applicability of the data. Data management is the foundation block to everything else that has to do with analyzing data. If our first steps are thoughtless, then our whole structure could crumble. In data management, we gather, process, and examine our data based on the end goal established.  1. Information extraction - Our first step is information extraction, where we use different tools such as website, documents, and other sources to retrieve the specific information we’re trying to analyze. The data we extract needs to help answer the goal that’s set at the beginning. There are different ways w...
Read more

The 4s of Database Management Systems (DBMS)

Image
Databases are a collection of logically related data. The data and the relationships included built into databases to fulfill a need of the person or organization in charge of the database. To understand database design, we’ll look at its properties, steps and finally core operations.  The 4 properties of DBMS 1. Atomicity - an operation in a database system should be executed completely or not at all. This means that an operation cannot be interrupted or done partially because it would be completely aborted, and the user would be forced to restart. In other words, it maintains the atomicity of the data. 2. Consistency - all the data in a database system is updated before and after an operation is performed. Every fact in our database needs to be checked to make sure everything is consistent and running smoothly. 3. Isolation - each operation runs individually in a database system, preventing interruptions from each other. The data in one database should not affect the other dat...
Read more