Ai大模型相关案例,AI
为了帮助企业用户寻找更为有效、加快Hadoop数据查询的方法，Apache软件基金会近日发起了一项名为“Drill”的开源项目。Apache Drill 实现了 Google’s Dremel.
据Hadoop厂商MapR Technologies公司产品经理Tomer Shiran介绍，“Drill”已经作为Apache孵化器项目来运作，将面向全球软件工程师持续推广。
该项目将会创建出开源版本的谷歌Dremel Hadoop工具（谷歌使用该工具来为Hadoop数据分析工具的互联网应用提速）。而“Drill”将有助于Hadoop用户实现更快查询海量数据集的目的。
“Drill”项目其实也是从谷歌的Dremel项目中获得灵感：该项目帮助谷歌实现海量数据集的分析处理，包括分析抓取Web文档、跟踪安装在Android Market上的应用程序数据、分析垃圾邮件、分析谷歌分布式构建系统上的测试结果等等。
通过开发“Drill”Apache开源项目，组织机构将有望建立Drill所属的API接口和灵活强大的体系架构，从而帮助支持广泛的数据源、数据格式和查询语言。
Apache Drill是为大数据集的互动分析而生，是Google的Dremel的开源版本。它的目标是可以高效地对大数据集进行分析，可以运行在1000台以上的服务器，在几秒内处理PB级的数据和万亿条的数据记录，目前Drill还在Apache进行孵化。近日，Apache Drill 0.4.0发布，该版本是一个开发者预览版本。该版本是一个具有大量改进和新特征引入：
A new way to work with data and metadata including the first query engine to champion advanced Apache Parquet format files to support self-describing data, completely avoiding a central metadata repository.
A completely new columnar execution engine that leverages both runtime code compilation and advanced memory management for query execution.
Advanced cost-based query optimization that works with or without stats providing complex distributed query planning.
Focus on full SQL capability with support for correlated subqueries, complex subexpressions and scalar subqueries.
The first query engine to support JSON everything, enabling instant analysis of semi-structured and partially schemed data without setup or extra effort.
Full complex data semantics combined with complete SQL data types allow you to use JavaScript notation to access and interact with complex fields and data structures. This includes support for exact Decimal, Date, Time and Interval types.
In-query dynamic schema discovery allows you to redefine blob fields as complex objects, using advanced CONVERT_FROM and CONVERT_TO semantics.
Support for more than 150 data formats and thousands of existing function libraries through strong integration with Hive Serdes and UDFs.
Additional support for high performance native Drill storage plugins and UDFs.
A friendly web interface with query and profiling tools including an advanced query plan visualizer and execution flow visualizations.
A complete set of interfaces and APIs including support for JDBC, C++, Java, ODBC*, REST and CLI
Advanced dynamic analysis capabilities on top of HBase including dynamic schema discovery, high speed parallel scanning and operator pushdown.
Support for in-memory and beyond memory datasets with an multi-staged innovative sort algorithm that produces faster time to first record sorting than traditional query engines.
Ability to meet query SLAs and avoid resource starvation with multiple query resource queues.
Support for wide rows with thousands of columns within a single query.
An advanced modular design with extensibility points at storage, query, planning and operator execution to work for a large set of standalone or embedded setups.
Full scaling: run embedded on Linux, Mac or PC for development purposes or scale up to a full cluster on any platform.
Support for use of Zookeeper and HBase for Drill configuration and profiling management.
The only open source distributed query engine architected to work with all types of big data, not just Hadoop data sources.
相关文章
