In this post we discuss Apache Arrow’s native support for Hadoop, Hive and Spark. Firstly, some of the core Apache Arrow concepts are Operators, Sensors, and Hooks.
Github link for some reference implementations on Apache Airflow
Airflow native support for Hadoop HDFS:
| Operators | None Available |
| Hooks | hdfs hook, hive hook |
| Sensors | web_hdfs_sensor |
Caution:
hdfs hook mentioned above is developed by Snakebite and they are currently not maintaining this hook codebase consistely. Also, in the source code of this hook it is hardcoded to use only “Python 2” and does not work with “Python 3”.
For the above reasons, it is highly recommended not to use hdfs hook in your Apache Airflow DAG codebase.
Having said above caution, Hadoop and Apache Airflow combo based ETL developers are literally does not have any Airflow “native” support (i.e. no operators or no hooks) to integrate with Hadoop HDFS. This is true as we write this article and this assertment is made after reviewing the code and trying to run couple of Airflow and Hadoop HDFS based examples.
Airflow native support for Apache Hive / Apache Sqoop:
Reason for combining both Hive and Sqoop in the above heading is, through Sqoop we can manage Hive metastore
| Operators | hive operator, sqoop operator |
| Hooks | hive hook |
| Sensors | None Available |
Recommendations:
Below are some of the recommendations just in case you need to integrate with Hadoop HDFS or Hive through Apache Airflow but do not use any of the above operators or hooks or sensors
Recommendation for Apache HDFS integration:
Use HDFS’s WebHDFS REST API which can be interacted using Linux “curl” command.
Example: Below “curl” command interacts with a Cloudera Quickstart VM running on the IP 192.168.1.24.
Below “curl” command creates (see MKDIRS at the end of the below URL) a HDFS folder with path and name as “/user/cloudera/ent_home/bus_unit_01/dl_home/
pre_process_zone” for the user “cloudera”
curl -X PUT "http://192.168.1.24:50070/webhdfs/v1/user/cloudera/ent_home/bus_unit_01/dl_home/pre_process_zone?user.name=cloudera&op=MKDIRS"
Through this WebHDFS API approach, your Airflow Workflow (say BashOperator) can be free from delays in software upgrades of above mentioned HDFS operators and/or hooks because above code uses Hadoop HDFS REST API which is always available in a typical Hadoop cluster.
Recommendation for Apache Hive integration:
Just like Hadoop HDFS, even Hive provides a REST API in the form of WebHCat. Here is the WebHCat documentation link https://cwiki.apache.org/confluence/display/Hive/WebHCat+Reference
So in your Apache Airflow DAG, develop a BashOperator or PythonOperator and invoke the Hive Rest API to read/write/list/delete Hive data.
Github Link
Here are some of our Apache Airflow reference implementations available in Github
https://github.com/javakishore-veleti/ApacheAirflow
- Becoming an AI Consulting Architect: Ability To Inform When To Use Claude Sonnet and Opus
- Becoming an AI Consulting Architect: Value Calculation Techniques for an AI-enabled Disability Claims Chatbot
- Becoming an AI Consulting Architect: What Is “Value Proposition” & “Business Impact” Differences
- Scaling Intelligent eCommerce: Deploying ADK Agents to Google Cloud Run
- My Hands-On GitHub Copilot CLI Patterns I Use to Build Shopify Apps on Azure
- AWS AI Architecture for Enterprise Content Compliance Alerting and Reporting
- Before You Design an AI Platform: Understand Horizontal and Vertical AI Solutions Across Business Functions and Processes
- From Requirements to Running Code: Can Spec-Driven Development Close the AI Software Delivery Gap?
- Picking a Model to Fine-Tune: Instruct, 4-bit, and safetensors