[removed]
Sorry, this post was removed by Reddit’s filters.
Kent Yao
u/kentyao
Feed options
Hot
New
Top
View
Card
Compact
Apache Spark 3.5.2 has been released.
Spark 3.5.2 is the second maintenance release containing security and correctness fixes. This release is based on the branch-3.5 maintenance branch of Spark. We strongly recommend all 3.5 users to upgrade to this stable release.
We would like to acknowledge all community members for contributing to this release. This release would not have been possible without you.
-
Installing with pypi : pip install pyspark
-
Installing with docker: docker pull spark:latest
Please visit for more information.
Hi,
We are not adding caching semantics into Spark SQL in this PR. Actually, Spark already has a CacheManager as its implementation of caching and supports [CACHE/UNCACHE statements](https://spark.apache.org/docs/latest/sql-ref-syntax-aux-cache-cache-table.html).
What we do here is to add HINTS for giving users a way to suggest how Spark SQL uses specific approaches to generate its execution plan, w/ or w/o cached data.
BR
A place for members of to chat with each other
I've submitted to introduce cache hints to Spark SQL. This feature works as a supplement for CACHE/UNCACHE commands, pure SQL users can operate the cache in their queries directly without extra definitions. Also, cache skipping is supported by hints.
This PR intends to add some query hints for cache behaviors, users can perform actions the use/skip/cache/uncache, etc on the cached results by the following hints.
-
RESULT_CACHE: spark will use cache by default, this hint keeps this behavior but will cache the child plan if uncached yet.
-
NO_RESULT_CACHE: with this hint, we can skip the default behavior to bypass the cached data if it is cached. This is useful when the underlying data is changed, especially in a multitenant environment, a user may not want to use cached data by others.
-
RESULT_UNCACHE: with this hint, we can remove the cached data.
For example,
For the above hints, RESULT_CACHE and NO_RESULT_CACHE are borrowed from , while RESULT_UNCACHE is added accordingly via the naming rule.
Similar ideas can also be found in
If people feel this feature is useful and support it being merged to the spark main codebase, we prepare to work on other cache behaviors, such as enabling/disabling metadata cache, datasource paths cache, refreshing different caching data, etc.
WDYT? Comments are welcome here or directly to the Pull requests on Github.
Bests,
Kent Yao
References
The 2022 ACM SIGMOD Systems Award goes to “Apache Spark”, an innovative, widely-used, open-source, unified data processing system encompassing relational, streaming, and machine-learning workloads.
The award recognizes the contributions of Michael Armbrust, Tathagata Das, Ankur Dave, Wenchen Fan, Michael J. Franklin, Huaxin Gao Maxim Gekk, Ali Ghodsi, Joseph Gonzalez, Liang-Chi Hsieh, Dongjoon Hyun, Hyukjin Kwon, Xiao Li, Cheng Lian, Yanbo Liang, Xiangrui Meng, Sean Owen, Josh Rosen, Kousuke Saruta, Scott Shenker, Ion Stoica, Takuya Ueshin, Shivaram Venkataraman, Gengliang Wang, Yuming Wang, Patrick Wendell, Reynold Xin, Takeshi Yamamuro, Kent Yao, , Ruifeng Zheng, and Shixiong Zhu.
