Beyond the Dashboard: Collaborative Analytics in Slack

Ben Lorica

Nick Schrock on Orchestration, Context Engineering, and AI-Accelerated Coding.

Subscribe: Apple • Spotify • Overcast • Pocket Casts • AntennaPod • Podcast Addict • Amazon •  RSS.

Nick Schrock, CTO of Dagster, discusses the critical role of data orchestration in the AI era, framing “context pipelines” as the new data pipelines that form the foundation of any AI strategy. He introduces Compass, a new Slack-native tool for collaborative, exploratory data analysis designed to replace the 80% of ad-hoc BI dashboards. The conversation also explores the transformative impact of AI coding assistants on software engineering and why companies should be embracing, not fearing, junior developers in this new paradigm.

Subscribe to the Gradient Flow Newsletter

Interview highlights – key sections from the video version:

Jump to transcript

Related content:

Support our work by subscribing to our newsletter📩

Transcript

Below is a heavily edited excerpt, in Question & Answer format.

Orchestration and AI Infrastructure

What types of orchestration are relevant for AI applications, and which one should teams focus on?

There are two distinct types of orchestration in the AI era that teams need to understand. The first is highly dynamic, probabilistic orchestration for agentic workflows in the critical path of user interactions – this involves AI agents orchestrating tool calls, deciding which tools to call in what sequence. This is fundamentally different from traditional orchestration, almost different enough to need a new term like “probabilistic agentic orchestration.” It’s essentially an LLM managing a series of prompts, which can be brittle and highly dependent on specific models.

The second type, which is absolutely essential for AI success, is orchestration for building data pipelines and data platforms. This is foundational to any AI strategy because all value in AI, especially in the enterprise, depends on having structured data where you understand the semantics. The data platform is the AI platform – they’re truly one surface. Even for seemingly “model-first” AI applications, the value still sits in well-modeled, well-governed data. For mainstream AI use cases—training, retrieval, evaluation, monitoring—you need reliable pipelines for data ingest, transformation, validation, and scheduling. Companies like Poolside use Dagster to orchestrate their entire platform from data ingest all the way through to foundation model training.

What is context engineering and why does it matter for AI applications?

Context engineering became a recognized concept in June 2024, and it’s fundamentally about the management of context fed into agents. Context includes features, summaries, embeddings, business definitions, guardrails, evaluation sets – essentially everything you feed models and agents to shape their behavior. Context pipelines are becoming the new data pipelines.

The context you feed to agents is often expensive to compute, requires precise scheduling control, needs quality tests, and requires LLM-as-evaluator assessments. When you analyze these requirements, they’re primarily data pipeline issues: lineage tracking, retries, SLAs, validation, versioning, and promotion through environments. This means orchestration becomes even more critical as the beating heart of both data and AI pipelines. Teams want to schedule recomputation of context, version it, test it (including using LLMs as judges for quality evaluation), and promote it through different environments just like they would with traditional data assets.

Context Stores vs Semantic Layers

What is a context store and how does it differ from a semantic layer?

A context store is broader than a traditional semantic layer. While semantic layers traditionally define metrics and measures that feed into deterministic SQL generation, context stores contain any text or structured knowledge that shapes how an analytics or AI agent behaves. This includes schema notes, business definitions, disambiguations (like noting that “Core” is a product codename, not a database column), prompt snippets, evaluated examples, and routing hints.

Context stores are designed to be more generic and informal, allowing knowledge to accrete over time. In Dagster’s implementation, the context store is backed by a Git repository, treating context as a new form of code – it’s bytes on disk that determines system behavior. This makes changes reviewable, diff-able, testable, and deployable like code. Technically, it’s a tree of files in a Git repository with a search index built over it, starting with the simplest thing that works before adding more advanced capabilities like vector search.

The critical difference is making it extremely lightweight for the right people to add context in a governed fashion. Business stakeholders can submit context corrections directly from their workflow (like Slack), which then creates a pull request to the governed context store that the data team must approve. This keeps governance with data/platform teams while tapping into the knowledge of people who actually understand the data’s quirks.

How do you solve the cold start problem with context stores?

The approach focuses on making the feedback loop between business stakeholders and the context store extremely lightweight and highly visible. When the data team onboards, they connect data warehouse connections, which triggers initial context collection by automatically querying all schemas, tables, and metadata. Systems can also seed with query logs, documentation, and organizational data (like who owns what).

The data team then seeds the context store with the most obvious problems – like internal project codenames that would confuse an LLM. The key is creating a fast, visible loop: ingest → use → observe error → propose fix → review → re-ask the question and see it improved immediately. This visible improvement builds trust and adoption.

The system is very transparent about how it works and what custom context is being injected. Users can see it get better over time through their contributions, and psychologically, the data team takes ownership and becomes proud of it. They transform from analysts manually writing SQL to context engineers governing the process of SQL generation.

Compass: A New Approach to Data Analysis

What problem does Compass solve that traditional BI tools don’t address?

Compass is not a BI tool – it’s an exploratory data analysis platform designed to handle the reality that about 80% of dashboards built in BI tools are effectively expensive, one-off exploratory analyses. A business stakeholder asks a question, an analyst builds a dashboard, the stakeholder looks at it once or twice, and it’s never used again.

The genesis came from observing that business stakeholders don’t see data platform teams as a source of value, but rather as cost centers, even though they rely entirely on the data products these teams produce. Traditional BI tools require significant analyst time to create dashboards, and this creates a major value disconnect.

Compass allows business stakeholders to ask questions directly and iterate quickly, while data teams govern the process through context engineering rather than manually writing SQL for every request. It leaves the 10-20% of mission-critical, highly deterministic dashboards that drive compensation or core business processes to traditional BI tools. Once an analysis proves valuable through iteration in Compass, it can be scheduled as a recurring report in a Slack channel – for instance, a daily report on product engagement that identifies new sign-ups, top usage, and drop-offs.

Why build this as a Slack-native application rather than a web app?

Nobody wants to go to another web app or learn an entire new ontology. Slack is where cross-functional teams already work, and “multiplayer agentic chat with integrated workflows” is as foundational an interaction primitive for the enterprise as single-player chat is for consumer applications.

The channel abstraction is incredibly valuable – different channels can have access to different data warehouse tables and even point to different context stores, because the same words have different meanings in different business contexts. For example, #sales-compass and #marketing-compass channels can have different permissions and contexts. While Slack may be terrible for communicating human-authored information internally, it’s excellent for driving ChatOps and workflows. The ability to bring people into a conversation, mention bots, and collaborate is incredibly powerful for data exploration.

Implementation and Trust

How do you maintain user trust and avoid over-promising with LLM-based analysis?

Transparency is critical at multiple levels. First, the system explains its reasoning in real-time as it works: “I am searching data documentation for X,” “I am searching the context for X,” “I am building the SQL query.” Users can follow along, see intermediate steps, and review the full history. This teaches users how the system works and builds appropriate trust.

Second, the system is explicit about uncertainty. Nearly every thread with data visualization includes a disclaimer that “AI might be inaccurate.” The UI design reinforces this – the bot talks in all lowercase and has a slightly cheeky, informal personality, communicating that it’s a collaborative assistant, not an infallible oracle. When something doesn’t look right, the system will preemptively suggest correcting its own context.

Third, the feedback loop is incredibly fast. When users provide corrections, they can immediately re-run queries and see the system incorporate the new information, reinforcing the value of their contribution. This combination of explicit warnings, explanation of process, demonstrated fallibility, and visible improvement builds appropriate trust without killing usage.

What happens when the data warehouse can’t answer a question?

When the LLM determines that required data isn’t in the warehouse, it automatically creates a well-scoped data request in your ticketing system of choice. This captures who asked for it, why they care, attempted SQL, business rationale, and all the context about what was attempted. This becomes perfect input for a code generation step to produce the needed data pipeline.

In an ideal integration with an orchestrator like Dagster, you can scaffold a new pull request targeting your data platform to create the production asset. Once developed and pushed to production, the new table or asset is registered back into the context store, and the original Slack thread can be pinged to retry the original question with the new data available. This creates clearer demand signals for what to model next based on actual question volume.

How does this change the data team’s day-to-day work?

Analysts shift from hand-writing ad hoc SQL to being “context engineers.” Instead of manually authoring SQL for every request, they govern definitions, review context pull requests, and convert repeatedly asked questions into durable assets and pipelines. They manage the context store like a code repository, reviewing and approving changes submitted by business users.

The data team becomes proud of governing this system rather than feeling burdened by manual SQL requests. They focus on seeding critical business knowledge, maintaining quality, and building pipelines for frequently requested data. This transformation is visible in metrics: reduction in manual SQL requests, increase in self-service data exploration by business stakeholders, and improvements in data request quality with more specific, better-contextualized asks.

AI-Assisted Software Engineering

How are AI coding tools like Claude Code changing software engineering workflows?

The transformation is dramatic and represents the most significant change in software engineering workflows in many engineers’ careers. Experienced engineers can now generate 70-90% of their code with AI, shifting from IDE-centric to agent-centric development. The workflow changes from writing individual functions to interacting directly from terminal to code review tool without touching an IDE.

The fastest way to onboard to a new system is now checking out the repository and asking the AI to explain how things work. Engineers can ask it to explore Git history to understand why code exists, what was happening during each change, and traverse the codebase with natural language queries. The biggest frontier isn’t writing functions anymore; it’s using agents effectively inside large, multi-engineer systems – setting boundaries, tests, and contexts so agents act predictably within complex codebases.

What are the practical considerations around cost and efficiency of AI coding tools?

These tools are expensive – it’s possible to spend thousands of dollars in weeks of heavy usage. Currently, the economics of many AI-assisted coding businesses are upside down, with tools that can brute-force tickets quickly but at high cost. Claude Code has smartly trained users to understand and accept consumption-based pricing with transparency.

At the application layer, there’s enormous room for improvement. Teams need to focus on token-efficient contexts, requiring fewer model turns, using smaller models for more precise outcomes, and implementing better retrieval and evaluation harnesses. The progression of underlying models is outstripping the application layer’s ability to effectively use them, suggesting years of optimization work ahead even if models stopped improving today. Engineering platforms need to include consumption transparency and guardrails like budgets, per-task ceilings, and cheaper-model fallbacks.

What’s the impact on junior engineers and career progression?

Companies not hiring junior engineers because they think AI can replace them are acting both shortsightedly and foolishly. Younger engineers are learning machines who will be far more fluent with AI-native tools. These tools dramatically accelerate learning – juniors can onboard faster with agents by querying the repository, traversing history, and asking why code exists.

However, they still need mentorship to develop systems intuition – debugging complex issues, understanding observability, and managing cost awareness. The open question is whether AI-accelerated learning translates into understanding primitives well enough to develop intuition about system behavior and fix really hard bugs. Companies that slow down intern and new grad programs will regret it, creating an arbitrage opportunity for smaller companies willing to invest in developing AI-native engineering talent. The cohort most fluent with AI-native workflows shouldn’t be excluded from the industry just as these tools are transforming it.