The PlayerZero Debugging Process

Throughout our time researching and building PlayerZero, we’ve always been struck by two things: First, how crucial it is to effectively construct narratives around customer support and triage scenarios that accurately connect customers to code in a way that’s immediately actionable. Second, how difficult it is to actually do this in practice.

The importance of a story: why we built the debug button

The importance of constructing a cohesive narrative in debugging cannot be overstated. In war rooms and customer triage settings, the pressure is intense. Teams are tasked with piecing together fragmented information from various sources—logs (logs, and more logs... IYKYK), errors, user behavior data, and, most crucially, customer complaints. These pieces of information are like puzzle pieces from different puzzles; they just don't intuitively fit together at first glance. The difficulty lies not just in identifying the issues but in connecting them to form a narrative that accurately represents the user's experience and pinpoints the root cause of their frustration. Since business and technical leaders speak different languages, it’s crucial that their understanding meets in the middle - with the customer.

Traditionally, this process has relied heavily on the manual efforts of engineers specifically skilled in connecting disconnected datapoints who comb through technical and customer data, attempting to correlate disparate events. The challenge here is twofold. First, there's the sheer volume of data to sift through, which can be overwhelming even for the most experienced professionals. Second, and perhaps more significantly, is the human factor. Cognitive biases and varying interpretations of data can lead to oversights or misdirection, prolonging the debugging process and delaying resolutions.

In war rooms, where time is of the essence, constructing a narrative that everyone can agree on is often as hard as finding the bug itself. Different stakeholders may have different views on what the problem is, based on their interpretation of the data. This can lead to circular discussions, where the focus often shifts from solving the problem to agreeing on what the problem actually is.

The Debug Button mitigates these challenges by providing a more intuitive way of linking customer complaints directly to the underlying code issues. By automating the correlation of user feedback with specific changes in the codebase, PlayerZero enables developers to bypass the initial confusion and jump straight to investigating the root cause. This not only speeds up the debugging process but also ensures that the narrative constructed is grounded in data, reducing the room for error and misinterpretation.

Technical breakdown: how we built the debug button

The main challenge inherent in traditional observability is that an issue/code correlation could be staring you in the face, but the burden of making that connection can be extremely high. Things don’t seem related but actually are, and unforeseen variables compound to the point that constructing an accurate narrative requires a person who not only knows the entire history of the codebase and the customer - skillsets which are rarely shared.

The first step towards automating this process was identifying each relevant variable (think code changes, customer complaints, logs, and analytics events). Once we gained a deep theoretical understanding of how each variable contributed to the overarching narrative in a theoretical sense, we were then able to map the relationships between each in a hierarchy that logically aligned with a typical engineering quality workflow. These relationships are easily represented via a descending graph with different nodes embodying each variable and its logical relationship to the others:

On a technical level, what we’ve created here is a singular index that understands the relationships between each variable and has a firm hierarchy with which to be able to distinguish high-confidence relationships at scale. By mapping each interaction between the nodes on the tree (ie, this backend action happens after a specific button is pressed, etc.), we now have a comprehensive system for drilling down to a single line of code from any entry point. Here’s what this flow looks like for a practical example in which an end user got stuck uploading a video while using a client’s video management solution:

Conclusion

By leveraging this singular index that captures the essence of the software's operation and its interaction with users, we can not only simplify the debugging process but also enhance it. Developers can quickly identify the source of a problem by following the connections within the graph, significantly reducing the time and effort required for troubleshooting. This approach not only makes the debugging process more efficient but also more intuitive, allowing for a deeper understanding of the underlying issues affecting software performance and user satisfaction.

You can get a hands-on look at how our new debug workflow works here with our new interactive product demo.