Marketing and sales intelligence with Elixir at PepsiCo
Welcome to our series of case studies about companies using Elixir in production. See all cases we have published so far.
PepsiCo is one of the world’s leading food and beverage companies serving more than 200 countries and territories around the world. Today Elixir is used at varying capacities inside PepsiCo by six different teams. This article explores how the Search Marketing and Sales Intelligence Platform teams adopted and use Elixir to build internal tools.
Although we will explore only two teams in this article, PepsiCo is hiring Elixir engineers across multiple teams. Let’s get started.
The first steps
The project that would become the first Elixir project and open the door for future Elixir applications inside PepsiCo was started by Jason Fertel back in 2016.
Initially, the application provided workflow automation for managing search marketing operations on multiple web platforms. The product was a success and ended up integrated into PepsiCo in 2018.
Now, the Elixir application plays a central role in a data pipeline that empowers PepsiCo’s marketing and sales teams with tools to query, analyze, and integrate with several search marketing partners.
The pipeline starts with the Data Engineering team, which collects and stores data into Snowflake Data Cloud. The Elixir application reads data from Snowflake’s platform, pre-process, and stores it in two databases: PostgreSQL or Apache Druid, according to the data characteristics. Finally, a Phoenix application serves this data to internal teams and communicates directly with third-party APIs.
Why Elixir?
Elixir helps PepsiCo eCommerce focus and get things done fast. “Elixir allows our team to develop quickly with confidence,” says David Antaramian, a Software Engineering Manager at PepsiCo. “In turn, that lets us deliver value to the business quickly, and it’s the reason we’ve stuck with the language. Whether it’s streaming changes to the front-end or orchestrating concurrent data operations across multiple storage systems, Elixir offers a robust developer experience that translates to a great consumer experience.”
Different Elixir features came together to help the PepsiCo team build compelling development and user experiences. Thanks to its functional and extensible aspects, PepsiCo used Elixir to create a domain-specific language that translates business queries into data structures sent to different stores. This gave them a stable foundation where they can continually add new queries and integrations, even as they grow in complexity.
Furthermore, the reports generated by PepsiCo’s marketing and sales teams often have to query different tables or even separate storages, all while juggling long-running connections to different third-party APIs. Elixir’s programming model, inherited from the Erlang Virtual Machine, makes it trivial to run all of these operations concurrently, leading to fast and rich user interactions while the development team remains focused on delivering features.
Libraries and frameworks
David Antaramian is quick to praise the Erlang runtime and its standard library. He says: “Since we are working with large amounts of data, it is also essential to avoid hitting the database whenever possible. Thankfully, Erlang ships with an in-memory table storage called ETS, which we use to store hundreds of thousands of rows”.
The Erlang standard library was also handy when communicating to some data stores. In particular, the Snowflake platform requires ODBC connections. The PepsiCo team built a library called Snowflex, designed on top of Erlang’s built-in ODBC drivers.
The Elixir ecosystem nicely complements the Erlang one. The front-end, written in React, talks to the server via the Absinthe GraphQL toolkit running on top of the Phoenix web framework. The Ecto database library manages the communication to PostgreSQL. They also use the esaml and Samly libraries to provide authentication within PepsiCo’s organization - another example of leveraging the tools within both Erlang and Elixir communities.
Finally, the team also recognizes the Erlang Ecosystem Foundation’s efforts, which PepsiCo are sponsors of, particularly the Observability Working Group. David remarks: “The adoption of Telemetry by the ecosystem has been a massive help in bringing monitoring visibility and metrics to our system. Now when we see spikes in one place, we can easily correlate them with other system parts”.
Hiring
Today there are approximately 40+ Elixir engineers within PepsiCo distributed across six teams. Eight of those engineers are part of the Search Marketing and Sales Intelligence Platform teams.
While the team recognizes that there aren’t as many Elixir engineers compared to communities like JavaScript, they were effective in hiring qualified Elixir candidates. Chase Gilliam, a Software Engineering Manager at PepsiCo, explains: “We have met many engineers that, like us, found Elixir due to being burned out by previous experiences. So when it came to hiring, many Elixir candidates had a mindset similar to ours, which ultimately sped up the process.”
This initial group of Elixir engineers paved the way for the language’s growth inside PepsiCo. David adds: “At first, we looked for engineers with Elixir experience to help us build a team that could guide other developers. Then we extended the pool to anyone who has a functional programming background and now to developers with either Ruby or Erlang experience. However, if someone is the right candidate, we onboard them even if they have no Elixir experience and train them”. He continues: “We also make extensive use of the learning resources available in the community, such as conferences, books, online courses, and others.”
As the team grew, they adopted best practices and saw the quality of the codebase improve at the same time. Chase concludes: “At the beginning, there were some large modules in our application. Luckily, refactoring in a functional programming language is straightforward, thanks to immutability and limited side-effects. Adopting tools like Credo, ExDoc, and the code formatter was also essential to standardize how we use Elixir internally.” For those interested in learning more about the different use cases for Elixir inside PepsiCo and help continue its growth, they are hiring.
