Impala-lzo provides code to allow Impala to read the LZO compressed tables. LZO is GPL licensed, which is why this support is not included directly. The Impala-lzo code interacts with internal Impala code at a level that is error prone and intricate. Given the low adoption of LZO and the other compression options available, Impala removes Impala-lzo support along with the low level interface it used.
Remove support for Sentry
Starting from 4.0, Impala will only support Ranger in authorization.
Set minimum CPU requirement to AVX for x86_64 (IMPALA-9690)
Before 4.0, the minimum CPU requirement is SSSE3. Now we bump it to AVX. For machines that only support AVX but not AVX2, please use --enable_legacy_avx_support flag to launch Impala. We may bump the minimum CPU requirement to AVX2 in near future.
Dropped support for dateless timestamps (IMPALA-9531)
Add support for string concatenation operator using ||
(IMPALA-452) Previously, "||" means "OR" for logical OR expressions. Now if the type of the left operand is STRING, "||" means "concat" for string concatenations.
Default to not allow ordinals in HAVING clause (IMPALA-7844)
New Features
Multi-threading (MT_DOP) support in all operators (IMPALA-3902)
The MT_DOP query option sets the degree of parallelism used for certain operations that can benefit from multithreaded execution.
Previously Impala only supported setting MT_DOP in queries that have only scans and aggregates. Now we can set MT_DOP for all kinds of queries.
Denser (aggregated) runtime profile, i.e. profile-v2 (experimental, IMPALA-9378)
Impala can produce a denser runtime profile which reduces a lot of cpu and mem consumption, especially for large clusters or queries with MT_DOP > 1.
Enabled by setting --gen_experimental_profile=true.
Support all TPC-DS 99 queries without manual rewrites
Support ROLLUP, CUBE and GROUPING SETS (IMPALA-7204)
Support uncorrelated subqueries in SelectList (IMPALA-8954)
Queries that fail due to cluster membership changes can be transparently retried by the coordinator. Enabled by setting retry_failed_queries query option to true.
Skip locked tables from topic updates (IMPALA-6671)
Previously, long running operations on a locked table (refresh, recover partitions, compute stats) may block the topic update thread, which causes unrelated queries that are waiting on metadata updates to unnecessarily block. We now add a mechanism for topic-update thread to skip a table which is locked for more than a configurable interval.
In the legacy catalog mode, previously table updates are propagated in a new snapshot of the table metadata, now they are populated in partition level deltas. This helps to avoid OOM errors of hitting JVM array limit (2GB) in sending catalog updates. Note that DDL/DML responses still contain the whole snapshot of table metadata. This will be tracked in IMPALA-9937.
In the LocalCatalog mode, previously when a table is updated, all the partitions in the coordinator’s local cache will be invalidated. Now only updated partitions will be invalidated.
Slim down metastore Partition objects in LocalCatalog cache (IMPALA-7501)
This helps to avoid OOM errors of hitting JVM array limit (2GB) in LocalCatalog mode. Also improve cache efficiency.
