Modern Data Stack

"Modern Data Stack™ is dead" begs the question: was it ever alive?

This blog post articulates a truth universally recognized but conveniently overlooked by data companies.

  1. The hype around MDS obscured the fact that data stacks were simply moved to the cloud. IMO, this happened to every software vertical and it was the writing on the wall — and so it happened to analytics. Beyond this, there was nothing "modern" about the modern data stack.
  2. The incentives aligned for a fragmented data stack with multiple interoperable tools — until they didn't. As a side-effect of cloud optimization and layoffs, willingness build a data stack with tools from 8-12 (!!!) vendors has reduced a lot. Instead, companies are much more likely today to expect to buy 2-4 products as the core of their analytics infrastructure.
  3. Analytics products are always purchased from an analytics budget, MDS or not. And AI has (or will very quickly) make it possible to have a no-BS data stack that is a single or few tools that help drive results, not infra-for-infra's sake.

In my opinion, every software purchase decision has to pass the same checklist:
  1. Is it actually useful and does it justify it's cost?
  2. What are the hidden costs (people, servers, etc)
  3. Is it reliable? Is there a cost associated with making it reliable?

Most data stacks fail this basic litmus test on all accounts.

Businesses care about moving the bottom line, not whether your dashboard is powered by 10 cloud-native tools.

More so now than ever — It is clear that data is a growth center, not a cost sink. Analytics is critical, but expending time, energy and focus in maintaining large teams of data engineers and independent services is a massive undertaking with a lot of latent costs.

Mind monkey, will horse

At first, meditation uncovered how little I could control the things that my mind prioritized for me and then went on to expose the brittle nature of focus itself. 

When seen through the prism of the Buddhist  "mind monkey, will horse" metaphor, the value of meditation is a lot more tangible. Loosely, this metaphor compares the mind to a fickle, fidgety, easily-distracted yet pure-hearted monkey. The monkey is at the reins of a horse, who himself is powerful but also impulsive and prone to poor judgment. 

Taking a step back — these are metaphors for the mind and the will respectively. In this model, trying to focus is a lot like trying to ride the horse (or "steer the will") except you cannot directly grab the horse by the reins. Instead, you have to relay all instructions to the horse via the monkey. 

The monkey has great cognitive flexibility (the ability to switch between thinking about two different things) which is a superpower that gets in the way of the horse's knack for picking a direction and sticking to it without much thought. While meditating, I'd often try and brute force my way to getting my mind to settle. Either by trying hard to think about "nothing" or by partaking in breathing rituals that worked, but only partially. 

On understanding that we are tied to reality by delicate and taut strings of consciousness, I learned not to fight the nature of consciousness. By trying to change the shape of a string under tension, we only put it in an unnatural and unsustainable position which is bound to fail. 

Instead, going back to the mind-monkey-will-horse metaphor, as opposed to trying to "cure" the monkey's superpowers or scare him into obedience—make friends with the monkey. The mind monkey is scared, optimistic, raw, and sometimes naive, but the will (the horse) only listens to him. Without restraint, the monkey causes mayhem (procrastination, lack of productivity, etc) and the horse causes havoc (haste, poorly thought-out decisions, etc). The two (the mind/will or the monkey/horse) have their shortcomings but must work together if one is to get anywhere. 

Meditation was never the same when I sought to get closer to the "mind monkey", first through playful deception and then through a deep understanding of how to incentivize him to be productive, empathize with him, and ultimately trust him to make (roughly) responsible decisions on how to commandeer the spirit.