Making Nested Columns as First Citizen in Apache Spark SQL

Apple Siri is the world's largest virtual assistant service powering every iPhone, iPad, Mac, Apple TV, Apple Watch, and HomePod. We use large amounts of data to provide our users the best possible personalized experience. Our raw event data is cleaned and pre-joined into an unified data for our data consumers to use. To keep the rich hierarchical structure of the data, our data schemas are very deep nested structures. In this talk, we will discuss how Spark handles nested structures in Spark 2.4, and we'll show the fundamental design issues in reading nested fields which is not being well considered when Spark SQL was designed. This results in Spark SQL reading unnecessary data in many operations. Given that Siri's data is super nested and humongous, this soon becomes a bottleneck in our pipelines.

Then we will talk about the various approaches we have taken to tackle this problem. By making nested columns as first citizen in Spark SQL, we can achieve dramatic performance gain. In some of our production queries, the speed-up can be 20x in wall clock time and 8x less data being read. All of our work will be open source, and some has already been merged into upstream.

参考译文：

Apple Siri是世界上最大的虚拟助理服务，为每部 iPhone，iPad，Mac，Apple TV，Apple Watch 和 HomePod 提供服务支持。我们使用大量数据来为用户提供最佳的个性化体验。所有的原始事件数据被清理并预先加入到统一数据中，供我们的数据使用者使用。为了保持数据的丰富层次结构，我们的数据模式采用了非常深的嵌套结构。

在本次演讲中，我将讨论 Spark 如何处理 Spark 2.4 中的嵌套结构，还会展示读取嵌套字段时的基本设计问题，这些问题在设计 Spark SQL 时并未得到充分考虑。这就导致了 Spark SQL 在许多操作中读取不必要的数据。鉴于 Siri 超级嵌套的数据非常庞大，它很快就成了瓶颈所在。

之后，我会介绍为解决这个问题所采取的各种方法。将嵌套列作为 Spark SQL 中的第一个公民，在性能上获得显着的提升。在我们的一些生产查询中，加速20倍，读取的数据减少8倍。我们所有的工作都将开源，有些已经合并到了核心区域。