2025 Agenda

Data streaming is a really difficult problem. Despite 10+ years of attempting to simplify it, teams building real-time data pipelines can spend up to 80% of their time optimizing it or fixing downstream output by handling bad data at the lake. All we want is a service that will be reliable, handle all kinds of data, connect with all kinds of systems, be easy to manage, and scale up and down as our systems change.

Oh, it should also have super low latency and result in good data. Is it too much to ask?

In this presentation, you’ll learn the basics of data streaming and architecture patterns such as DLQ, used to tackle these challenges. We will then explore how to implement these patterns using Apache Flink and discuss the challenges that real-time AI applications bring to our infra. Difficult problems are difficult, and we offer no silver bullets. Still, we will share pragmatic solutions that have helped many organizations build fast, scalable, and manageable data streaming pipelines.

Apache Flink is a powerful stream processing framework that enables complex real-time data processing. One of the most common use cases in streaming ETL is enriching events with reference data, such as Slowly Changing Dimensions (SCDs). However, real-world streaming systems are anything but perfect: events arrives late, reference data updates unpredictably, and standard join patterns can fall short.
In this session, we go beyond the basics to explore advanced enrichment techniques for handling late-arriving events and evolving reference data. Attendees will learn how to ensure consistency and meet latency requirements, even when dealing with unreliable data sources. We’ll also dive into lesser-known but powerful features of Flink’s API that can help designing resilient, high-performance real-time data pipelines.

Imagine your startup is being acquired by a big tech company. The technology you designed and built now needs to scale to meet the demands of that company. That’s what happened to us 3 years ago, when LinkedIn acquired Oribi. We had to rebuild our system to match LinkedIn’s scale: over 3 million requests per second and more than 45 trillion Kafka messages per day! More than that – we had to make sure we supported a whole new level of compliance and legal requirements, and fast.
In this talk, I’ll share how we redesigned our system from scratch to meet LinkedIn’s requirements & scale, what guided us, the changes we made, and how we successfully completed the project ahead of a very tight timeline.
Whether you’re working at a small startup or a large company, scale can take many forms. Don’t let it catch you by surprise – develop the right mindset and methodologies to prepare in advance!

Model Context Protocol (MCP) is emerging as a lightweight standard for connecting large language models to actual systems—not just chat interfaces. In this fast-paced session, you’ll learn what MCP is, why it matters, and how it can help automate the repetitive glue work that clogs modern data workflows. We’ll walk through a real-world example for automating daily tasks using MCP to make life easier.

At Coralogix, we process massive volumes of observability data (logs, metrics, traces) with a hard requirement: the data must be available for query within five minutes of arrival. To achieve this, we built a near real-time ingestion pipeline into object storage — but one that doesn’t stop at speed.
Our system makes data queryable immediately upon arrival using a custom file format. Meanwhile, background compaction processes kick in — restructuring that data into optimized files for efficient querying.
This talk explores how we make data instantly queryable while still optimizing it later for maximum throughput — all while (trying to) meet both latency and performance goals under strict SLA pressure.

AI memory is the collection of interactions of an AI agent with humans and other AI agents. As the world of AI rapidly develops, making LLMs smarter and more capable, the idea of unified memory for AI is still undefined. In this talk Roy will try to convince the audience that 1) Artificial General Memory is “a thing”; 2) There is likely a missing ‘data structure’ in our world; and 3) traditional systems like Snowflake and Databricks should be concerned; and 4) There’s opportunity for new players.

This is going to be a fast paced, 15 minutes talk that covers the problem, the solution that exists today and the opportunity that lies ahead.

This talk will explore the challenges of sharing insights in a way that resonates with stakeholders, and reveal a surprising solution to this common problem. By attending this session, you’ll learn how to overcome the pitfalls of traditional analysis and discover a proven approach to driving impact and engagement through concise and effective communication. Join us to find out what this game-changing solution is and how it can transform your approach to insights sharing.

What if your DWH could deliver petabyte-scale capabilities directly to applications in real-time, without the cost, complexity, or limits of conventional solutions? We’ve reimagined the DWH by adopting a fully stateless architecture that leverages Kubernetes, Spot Instances, S3, ClickHouse, and Parquet.
Building a stateless DWH is more than just a technical challenge- it’s a strategic shift in our mindset. It requires stepping away from familiar methods, critically evaluating the long-term impact of architectural decisions, and embracing experimentation over assumptions. To innovate, you must stop and think deeply about your infrastructure.
In this talk, we’ll share how deliberate decision-making and experimentation enabled us to build a scalable, cloud-native DWH. By the end, you’ll gain insights on how to move beyond conventional solutions, focus on outcomes, embrace a state of mind of bottleneck-free architecture, and what to watch out for when introducing a novel data architecture.

What do you do when your most valuable dataset wasn’t designed, labeled, or even intended to exist?
In this talk, we’ll share how The Haibulans, a volunteer wildlife rescue network, unintentionally built a high-impact dataset from WhatsApp messages, spreadsheets, and field notes. Originally used for coordination, this informal data revealed spatial and temporal patterns in injuries, urban hazards, and human-wildlife interactions.

We’ll show how manual tagging and analysis led to changes in dispatch strategy, public outreach, and collaboration with city agencies — and explore the ethical challenges of working with emotionally charged, unstructured data.

Gain practical insights into extracting value from messy, real-world data, and learn how meaning can emerge even without a formal model. This talk is especially relevant for anyone navigating the space between fieldwork and data work — where meaning emerges before models.

Building AI products often involves a handoff: data scientists develop models, and then developers figure out how to productionize them. This division can lead to friction, delays, and suboptimal results, especially when working with complex LLMs. As AI becomes more integrated into production environments, the gap between data science and software development must be addressed.

In this talk, I’ll share how to move beyond the traditional handoff model by creating shared ownership across roles. We’ll discuss practical strategies for building heterogeneous teams, the challenges of merging distinct mindsets, and the tangible benefits of this integration.
This isn’t just about team structure—it’s about how people work together, test together, and ship together.

Drawing from real examples, I’ll cover principles, tools, and pitfalls that helped us build LLM-powered products more effectively—without the handoff headaches.

LLMs can do a lot, but can they label and evaluate data like a human? Sometimes. This talk shares hard-earned lessons from the front lines of data operations (those who manage human judgment on data operations and ensure its quality), where one practitioner set out to recreate textual human-labeled data with the generous help of GenAI, and discovered what works, what doesn’t, and why it’s not as simple as it seems.

Apache Airflow is a powerful data pipeline orchestrator—but what if it could do more? What if a single Airflow instance on Kubernetes could serve an entire org, empowering teams beyond data engineering?

In this talk, I’ll share how we evolved Airflow into a self-service, org-wide workflow orchestration platform. By scaling its distribution, we replaced two legacy systems and enabled 20+ R&D teams to own their workflows. We also built an internal Airflow community that fostered collaboration and achieved milestones, some shared with the broader Airflow community. A key enabler was “Wrappers”—an abstraction layer bridging user code with Airflow’s core. It let us scale while enforcing best practices for 100+ users.

You’ll learn how to run multi-tenant Airflow, integrate Okta, ensure ownership, and streamline local development with lightweight testing. If you’re ready to push Airflow beyond its boundaries, this talk is for you.

We’ve all used data export mechanisms in one way or another – whether it’s by clicking an “Export to CSV” button, calling a designated API, or using another common solution to get the data we need. But what if we had to build such a mechanism ourselves?
While integrating cost data reports from popular major systems into our platform, we realized the strategy to approach this differs widely: AWS, for example, relies on snapshots stored in Parquet files, Oracle Cloud relies on incremental CSV updates, and others like GCP and Azure implement their own unique approach.
In this talk, we’ll uncover the hidden complexities of building efficient data export tools. We’ll explore trade-offs between snapshot-based and incremental exports, key choices for file formats, and scalability. With real-world lessons and practical approaches, you’ll learn how smart design simplifies workflows, enhances user experience, and future-proofs systems against growing data demands.

See how Unity iAds transformed its data lake into a self-serve Iceberg powerhouse: any engineer now spins up an autoscaling EMR-on-EKS streaming pipeline in minutes, hammering through millions of events per second, with Iceberg maintenance on autopilot and table-level FinOps insight baked in.
In 30 minutes we’ll unpack the key design moves, killer tools, and cultural shifts that made this dream into reality.

Our data is spread across many data sources (Iceberg, Vertica, BigQuery, MySql, Druid just to name a few) across thousands of tables and countless columns, even if someone knows where the data sits, they don’t necessarily know how to query it, let alone validate their result.
At this point you’re probably thinking, just GPT and RAG it right? Well, it just doesn’t work.
In this talk we’ll describe the different angles we tried and how all “best practices”, and we took it to the next level to not only answer questions, but completely own deep data investigations (why was there a drop in revenue yesterday? In what user segment variant B is losing?).

At Wix, we’ve adopted a 10-step methodology for building high-quality DWH – starting from gathering business questions, through designing logical models, and ending with ensuring real adoption across teams with managed certified tables.
But building a great DWH is still a time-consuming process – especially when translating product and business intent into scalable, governed, and tested tables.
In this talk, I’ll share how we used gen AI to support and enhance this journey. Starting with just a product manager’s presentation, our AI-powered modeling approach aims to extract meaningful metadata, identifies core business entities and questions, builds ERDs and logical models, writes quality tests, and even helps with visualized documentation and onboarding.
We’ll explore what worked well, what needed refinement, and which parts of the process still rely heavily on human expertise.