Apache CarbonData 1.0.0完成了80+ 特性，100+ bugfixs，亮点特性如下：
1.New load data solution
The old CarbonData load solution depends on Kettle engine, but Kettle engine is not designed for handling big data domain and the code maintainability is complex in this flow. So in the 1.0 version, a new data loading solution without kettle dependency is added and makes more modular and improved performance.
2.Support Spark2.1 integration in carbon
Spark 2.1 has added many features and improved the performance. CarbonData also gets the advantage of it after upgrading.
3.Data update/delete SQL support
Now user can delete and update the carbon table using standard sql syntax. This feature currently is supported in Spark 1.5/1.6 integration, it will be support in Spark 2.1 integration soon.
4.Support adaptive data compression for int/bigint/decimal to increase compression ratio
This feature can adapt the data to the smaller data type that fits the value, and it also supports delta compression technique to reduce the store size.
5.Support to define Date/Timestamp format for different columns
Now user can provide Date/Timestamp format for each column while loading the data. Provide option in the create table DDL itself to define the format for each Timestamp column, also provide defaults so that users can create table with Timestamp columns without having to always define the Date/Timestamp format.
6.Implement LRU cache for B-Tree
Btree in CarbonData keeps the information of blocks and blocklets of carbon tables inside memory. If number of tables increases or data increases there is a possibility of going out of memory. LRU cache of Btree now keep only recently or frequently used block/blocklet information in memory and evicts the unused or less used block/blocklet information.
7.CarbonData V2 format to improve first time query performance
This V2 format is more organized and maintains less metadata(reads metadata on demand) so that first time queries are faster. And also it has less IO cost compare to V1. Several testcases show that first time query response time reduced around 50%.
8.Vectorized reader support
It reads the data in batches, column by column. This feature reduces GC time and improve performance during data scan.
9.Fast join using bucket table
This feature enable bucket table support for CarbonData. It can improve the join query performace by avoiding shuffling if both tables are bucketed on same column with same number of buckets.It is supported in Spark 2.1 version.
10.Leveraging off-heap memory to reduce GC
By leveraging off-heap memory, it improves both loading and reading performance. In data loading, it improves data sorting performance and in reading, also it reduces GC overhead as it stores data in off-heap
11.Support single-pass loading
Currently data loading happens in 2 jobs (generate dictionary first, then do the actual data loading), this feature enables single job to finish the data loading with dictionary generation on the fly. It can improve the performance for the scenario that data loading with less incremental updates on dictionary, which usually is this case after initial data load.
12.Support pre-generated dictionary for data loading
User can use the generated dictionary, this feature also supports with customized dictionary by users to improve data load efficiency.
- Apache JIRA：https://issues.apache.org/jira/browse/CARBONDATA/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel