Big Data Platform at Pinterest - Now and in the Future

Big data platform at Pinterest is a public cloud based platform at massive scale (100+PB data, hundreds of billions of new events per day, ~PB new data ingested per day) focusing on empowering all engineers.

In this talk, Yongsheng will deep dive into the current technology landscape of big data platform at Pinterest across data ingestion (real-time events, logging, database snapshots, database incremental dump), batch/streaming data processing platforms (Hadoop, Spark, Flink, Kafka Streams), query platforms (Hive, Presto, Spark SQL), and their homegrown workflow engine at Pinterest; he will also offer insights into how these technologies will evolve in their ecosystem in the future. Besides key technologies powering big data platform at Pinterest, Yongsheng will also cover what enables Pinterest engineering to encourage ownership and accountability, improve platform efficiency, and elevate platform team from being overwhelmed by operation and support to focus on platform advancement.

You will leave this talk with great insights into how big data ecosystem works at Pinterest and being inspired to re-envision your own big data platform.

参考译文：

Pinterest 的大数据平台是一个大规模的公共云平台（100+PB数据，每天数千亿新事件，每天接收约PB级新数据），它的重点在于赋予所有工程师工作处理能力。

在这次演讲中，永胜将深入探讨Pinterest数据平台技术栈，围绕数据摄入（实时事件，日志记录，数据库快照，数据库增量转储），批量/流式数据处理平台（Hadoop, Spark, Flink, Kafka Streams），查询平台（Hive，Presto，Spark SQL）以及 Pinterest 本土工作流引擎；他还将提供有关未来这些技术如何在其生态系统中演变的个人见解。除了在 Pinterest 支持大数据平台的关键技术之外，永胜还将分享涵盖 Pinterest 工程鼓励所有权和问责制，提高平台效率，避免平台团队受到运营压力，并致力于平台进步。

参会者会从永胜的演讲中获益，深入了解大数据生态系统在 Pinterest 的工作原理，并启发参会者如何重新审视目前自己大数据平台。

Scaling Pinterest

In this talk, Yongsheng will cover how they scale the online infrastructure at Pinterest in the past 8 years to serve 200-300M MAUs. Over the years, online infrastructure at Pinterest has evolved from a Python Django application on a single MySQL instance in 2010 to a modern microservice-based architecture. Yongsheng will cover the following key technologies that allows Pinterest to horizontally scale to serve hundreds of millions of users with great user experiences:

• Microservice framework, powered by Twitter Finagle with highly resilient service discovery and real-time configuration management systems built on top of Apache Zookeeper.

• Pinlater, an asynchronous job processing framework, open sourced by Pinterest, which embraces idempotency/commutativity, and allows them to execute non-critical part of request processing logic asynchronous reliably to deliver delightful online user experiences.

• Distributed caching, storage and serving systems.

• Pinterest adopted caching proxy, Mcrouter, open sourced by Facebook, to enable AZ failure resilient, highly consistent, and highly performant distributed caching system.

• Pinterest uses Apache HBase and sharded MySQL to deliver a homegrown distributed storage system with integrated caching layer, code named Zen, to enable rapid product innovations in self-serviced manner through graph data model. This system was inspired by TAO and Dragon from Facebook.

• Scorpion Serving, a high throughput, low latency ML serving system at Pinterest powered by C++, Folly, and RocksDB, which scores tens of millions of (user, pin) pairs per second with P99 latency less than 20ms.

• Muse: a homegrown search engine, implemented in C++, which Pinterest used to replace Apache Solr and Apache Lucene to deliver high throughput, low long-tail latency search at scale, with capability of real-time indexing, as services to the entire Pinterest engineering.

As their business grows, Pinterest needed to address the business continuity risk of single geo region outage. Yongsheng will also share the major changes which they had to make to enable Pinterest to serve active-active across multiple geo locations with strong consistency between caching and persistent storage tiers.

Yongsheng will wrap up his talk with their future plan, and key learnings acquired throughout the years as they work on scaling online infrastructure at Pinterest to enable hundreds of millions of people to discover and do what they love.

参考译文：

在本演讲中，永胜老师会分享在过去的8年他们服务 200-300M MAUs 过程中，是如何扩展 Pinterest 在线基础设施架构的。经过多年迭代，Pinterest 的在线基础设施从2010年的一个基于单个 MySQL 实例的 Python Django 应用，不断演进成最新的基于微服务的架构体系。永胜会分享使 Pinterest 能够横向扩展，为数以亿计的用户提供卓越用户体验的关键技术点：

1. 微服务框架，由 Twitter Finagle 提供支持，具有高度弹性的服务发现和基于 Apache Zookeeper 构建的实时配置管理系统。
2. Pinlater 是一个异步任务处理框架，由 Pinterest 开源，包含幂等/交换性，并允许它们执行非关键部分的请求处理逻辑，以提供高质量的在线用户体验。
3. 分布式缓存，存储和服务系统：
   • Pinterest 采用了​​由 Facebook 开源的缓存代理 Mcrouter，以实现 AZ 失效弹性，高度一致和高性能的分布式缓存系统。
   • Pinterest 使用 Apache HBase 和 MySQL 将分散式存储系统与集成缓存层（代号为 Zen）分离开来，通过图形数据模型以自助服务的方式实现快速产品创新。这个系统受 Facebook 的 TAO 和 Dragon 的启发。
   • Scorpion Serving 是一款由 C++，Folly 和 RocksDB 提供支持的高吞吐量，低延迟 ML 服务系统，每秒可以获得数千万用户配对，P99延迟小于20ms。
4. Muse: 这是Pinterest内部研发的搜索引擎，采用 C++实现，Pinterest 用于替代 Apache Solr 和 Apache Lucene，以提供高吞吐量，低延迟大规模搜索，具有实时索引功能，为整个Pinterest 工程提供服务。

随着业务的增长，Pinterest 需要解决单个地理区域系统故障的业务连续性风险。永胜也将分享他们必须做出的主要变化，以确保 Pinterest 能够在多个地理位置之间积极主动地进行服务，并且在缓存和持久存储层之间具有强大的一致性。

永胜会以未来计划开始他的演讲，以及多年来致力于在Pinterest上扩展在线基础设施所获得的关键知识，帮助了数以亿计的人能够发现并做他们热爱的事。