Using the cloud to scale Etsy

Strategic Principles

Mike Fisher (CTO) and Keyur Govande (Chief Architect) created the initial cloud migration strategy with these principles:

Minimum viable product - A typical anti-pattern Etsy wanted to avoid was rebuilding too much and prolonging the migration. Instead, they used the lean concept of an MVP to validate as quickly and cheaply as possible that Etsy’s systems would work in the cloud, and removed the dependency on the data center.

Local decision making - Each team can make its own decisions for what it owns, with oversight from a program team. Etsy’s platform was split into a number of capabilities, such as compute, observability and ML infra, along with domain-oriented application stacks such as search, bid engine, and notifications. Each team did proof of concepts to develop a migration plan. The main marketplace application is a famously large monolith, so it required creating a cross-team initiative to focus on it.

No changes to the developer experience - Etsy views a high-quality developer experience as core to productivity and employee happiness. It was important that the cloud-based systems continued to provide capabilities that developers relied upon, such as fast feedback and sophisticated observability.

There also was a deadline associated with existing contracts for the data center that they were very keen to hit.

Using a partner

To accelerate their cloud migration, Etsy wanted to bring on outside expertise to help in the adoption of new tooling and technology, such as Terraform, Kubernetes, and Prometheus. Unlike a lot of Thoughtworks’ typical clients, Etsy didn’t have a burning platform driving their fundamental need for the engagement. They are a digital native company and had been using a thoroughly modern approach to software development. Even without a single problem to focus on though, Etsy knew there was room for improvement. So the engagement approach was to embed across the platform organization. Thoughtworks infrastructure engineers and technical product managers joined search infrastructure, continuous deployment services, compute, observability and machine learning infrastructure teams.

An incremental federated approach

The initial “lift & shift” to the cloud for the marketplace monolith was the most difficult. The team wanted to keep the monolith intact with minimal changes. However, it used a LAMP stack and so would be difficult to re-platform. They did a number of dry runs testing performance and capacity. Though the first cut-over was unsuccessful, they were able to quickly roll back. In typical Etsy style, the failure was celebrated and used as a learning opportunity. It was eventually completed in 9 months, less time than the full year originally planned. After the initial migration, the monolith was then tweaked and tuned to situate better in the cloud, adding features ​​like autoscaling and auto-fixing bad nodes.

Meanwhile, other stacks were also being migrated. While each team created its own journey, the teams were not completely on their own. Etsy used a cross-team architecture advisory group to share broader context, and to help pattern match across the company. For example, the search stack moved onto GKE as part of the cloud, which took longer than the lift and shift operation for the monolith. Another example is the data lake migration. Etsy had an on-prem Vertica cluster, which they moved to Big Query, changing everything about it in the process.

Not surprising to Etsy, after the cloud migration the optimization for the cloud didn’t stop. Each team continued to look for opportunities to utilize the cloud to its full extent. With the help of the architecture advisory group, they looked at things such as: how to reduce the amount of custom code by moving to industry-standard tools, how to improve cost efficiency and how to improve feedback loops.

Figure 1: Federated cloud migration

As an example, let’s look at the journey of two teams, observability and ML infra:

The challenges of observing everything

Etsy is famous for measuring everything, “If it moves, we track it.” Operational metrics - traces, metrics and logs - are used by the full company to create value. Product managers and data analysts leverage the data for planning and proving the predicted value of an idea. Product teams use it to support the uptime and performance of their individual areas of responsibility.

With Etsy’s commitment to hyper-observability, the amount of data being analyzed isn’t small. Observability is self-service; each team gets to decide what it wants to measure. They use 80M metric series, covering the site and supporting infrastructure. This will create 20 TB of logs a day.

When Etsy originally developed this strategy there weren’t a lot of tools and services on the market that could handle their demanding requirements. In many cases, they ended up having to build their own tools. An example is StatsD, a stats aggregation tool, now open-sourced and used throughout the industry. Over time the DevOps movement had exploded, and the industry had caught up. A lot of innovative observability tools such as Prometheus appeared. With the cloud migration, Etsy could assess the market and leverage third-party tools to reduce operational cost.

The observability stack was the last to move over due to its complex nature. It required a rebuild, rather than a lift and shift. They had relied on large servers, whereas to efficiently use the cloud it should use many smaller servers and easily scale horizontally. They moved large parts of the stack onto managed services and third party SaaS products. An example of this was introducing Lightstep, which they could use to outsource the tracing processing. It was still necessary to do some amount of processing in-house to handle the unique scenarios that Etsy relies on.

Migration to the cloud-enabled a better ML platform

A big source of innovation at Etsy is the way they utilize their Machine learning.

Etsy leverages machine learning (ML) to create personalized experiences for our millions of buyers around the world with state-of-the-art search, ads, and recommendations. The ML Platform team at Etsy supports our machine learning experiments by developing and maintaining the technical infrastructure that Etsy’s ML practitioners rely on to prototype, train, and deploy ML models at scale.

-- Kyle Gallatin and Rob Miles

The move to the cloud enabled Etsy to build a new ML platform based on managed services that both reduces operational costs and improves the time from idea generation to production deployment.

Because their resources were in the cloud, they could now rely on cloud capabilities. They used Dataflow for ETL and Vertex AI for training their models. As they saw success with these tools, they made sure to design the platform so that it was extensible to other tools. To make it widely accessible they adopted industry-standard tools such as TensorFlow and Kubernetes. Etsy’s productivity in developing and testing ML leapfrogged their prior performance. As Rob and Kyle put it, “We’re estimating a ~50% reduction in the time it takes to go from idea to live ML experiment.”

This performance growth wasn’t without its challenges however. As the scale of data grew, so too did the importance of high-performing code. With low-performing code, the customer experience could be impacted, and so the team had to produce a system which was highly optimized. “Seemingly small inefficiencies such as non-vectorized code can result in a massive performance degradation, and in some cases we’ve seen that optimizing a single tensor flow transform function can reduce the model runtime from 200ms to 4ms.” In numeric terms, that’s an improvement of two orders of magnitude, but in business terms, this is a change in performance easily perceived by the customer.

What were the challenges of the cloud?

Etsy had to operate its own infrastructure, and a lot of the platform teams' skills were in systems operation. Moving the cloud allowed teams to use a higher abstraction, managed by infrastructure as code. They changed their infrastructure hiring to look for software engineering skills. It caused friction with the existing team; some people were very excited but others were apprehensive about the new approach.

While the cloud certainly reduced the number of things they had to manage and allowed for simpler planning, it didn’t fully get them away from capacity planning. The cloud services still run on servers with CPUs and Disks, and in some situations, there is right-sizing for future load that has to be done. Going forward, as on-demand cloud services improve, Etsy is hopeful they can reduce this capacity planning.

The stress test of the pandemic

Etsy had always been data center based, which had kept them constrained in some ways. Because they'd been so heavily invested in their data center presence, they hadn't been taking advantage of new offerings cloud vendors had developed. For example, their data center setup lacked robust APIs to manage provisioning and capacity.

When Mike Fisher came onboard, Etsy then began their cloud migration journey. This set them up for success for the future, since the migration was basically finished at the start of the pandemic. There were a few ways this manifested: they had no capacity crunch, although traffic exploded 2-3X overnight, as events had increased from 1 billion to 6 billion.

And there were specific examples of ways the cloud gave them agility during the pandemic. For example, the cloud enabled efforts to close the "semantic gap", ensuring searches for "masks" surfaced cloth masks not face masks of the cosmetic or costume variety. This was possible because Google Cloud enabled Etsy to implement more sophisticated machine learning and the agility to retrain algorithms in real time. Another example was their database management changed from the datacenter to the cloud. Specifically, around backups, Etsy's DR posture improved in the cloud, since they leveraged block storage snapshotting as a way of restoring databases. This enabled them to do fast restores, have confidence and be able to test them quickly, unlike the older method, where a restore would take several hours and not be perfectly scalable.

Etsy performs extensive load and performance testing. They use chaos engineering techniques, having a ‘scale day’ that stresses the systems at max capacity. After the pandemic the increased load was no longer a spike, it was now the daily average. The load testing architecture and techniques needed to be just as scalable as any other system in order to handle the growth.

Continually Improving the platform

One of Etsy’s next focus areas is to create “paved roads” for engineers. A set of suggested approaches and machinery to reduce friction when launching and developing services. During the initial four years of the cloud migration, they decided to take a very federated strategy. They took the “let 1000 flowers bloom” approach as described by Peter Seibel in his article on engineering effectiveness at Twitter. The systems had never existed in the cloud before. They did not know what the payoffs would be, and wanted to maximize the chances of discovering value in the cloud.

As a result, some product teams are reinventing the wheel because Etsy doesn’t have existing implementation patterns and services. Now that they have more experience operating in the cloud, platform teams know where the gaps are and can see where tooling is needed.

To determine if the investments are paying off. Etsy is tracking various measures. For example, they monitor trends in SLI/SLOs related to reliability, debuggability and availability of the systems. One other key metric is Time to Productive – the time it takes for a new engineer to be set up with their environments and make the first change. What exactly that means changes by domain; for example it might be the first website push or the first data pipeline working in the big data platform. Something that used to take 2 hours now takes 20 minutes.

They combine these quantitative metrics with regularly measuring engineering satisfaction, using a form of an NPS survey to assess how engineers enjoy working in their respective engineering environments, and give an opportunity to point out problems and suggest improvements. Another interesting stat is that the infrastructure has expanded to use 10x the number of nodes but only requires 2x the number of people to manage them.

Measuring Cost and Carbon Consumption

Etsy continues to embrace measuring everything. Moving to the cloud made it easier for teams to identify and track their operational costs than it had been in the datacenters. Etsy built tools on top of Google Cloud to provide dashboards which give insight into spending, in order to help teams understand which features were causing costs to rise. The dashboards included rich contextual information to help them make optimization decisions, measured against their understanding of what ideal efficiency should be.

A very important company pillar is sustainability. Etsy reports their energy consumption in their quarterly SEC filings, and have made commitments to reduce it. They had been measuring energy consumption in the data center, but trying to do this in the cloud was initially more difficult. A team at Etsy researched and created Cloud Jewels, an energy estimation tool, which they open-sourced.

We’ve been unable to measure our progress against one of our key impact goals for 2025 -- to reduce our energy intensity by 25%. Cloud providers generally do not disclose to customers how much energy their services consume. To make up for this lack of data, we created a set of conversion factors called Cloud Jewels to help us roughly convert our cloud usage information (like Google Cloud usage data) into approximate energy used. We’re proud that our work and methodology have been leveraged by Google and AWS to build into their own models and tools.

-- Emily Sommer (Etsy sustainability architect)

These metrics have recently been added to their product dashboard, allowing product managers and engineers to find opportunities to reduce energy consumption and spot whether a new feature has had any effect. Thoughtworks, who has a similar sustainability mission, also created an open-source tool called the Cloud Carbon Footprint, which was inspired by initial research into Cloud Jewels, and further developed by an internal Thoughtworks team.