Danny Zuckerman’s Post

Many of our customers and prospects face the challenge of having their data scattered across various systems. The real issue lies in the shareable data that remains locked within silos. By integrating and sharing this data through Snowflake, whether directly or using Iceberg tables, we can unlock exciting new AI and BI possibilities that depend on interconnected data. These advancements not only provide instant access to the necessary data but also ensure unified governance, nearly limitless scalability, a variety of programming language options, and a vast array of partner solutions that seamlessly integrate with Snowflake. https://okt.to/h3d2R9

🚀 Unlocking New Data Potential with Microsoft Fabric and Yugabyte DB In today's data-driven landscape, managing and analyzing data efficiently is crucial for business success. I came across a blog by Yugabyte that outlines a powerful integration between Microsoft Fabric and YugabyteDB, enhancing data pipeline capabilities. 💧 Seamless Data Flow: Learn how to establish a seamless data pipeline where transactional data from YugabyteDB can be smoothly ingested into Microsoft Fabric's Lakehouse. This integration ensures data integrity and consistency, making real-time analytics more accessible. 🏪 Centralized Storage: With this setup, you can centralize your data storage, leveraging Fabric's robust data processing capabilities alongside YugabyteDB's transaction management. This not only simplifies data management but also boosts performance. 📈 Enhanced Analytics: The blog explains how to use the Fabric Semantic Data Model for deeper business insights, turning complex data into actionable intelligence. It's all about making data work for you! 🪜 Step-by-Step Guide: The article provides practical steps on setting up your Fabric account in Microsoft Azure, connecting YugabyteDB via PostgreSQL, and using Fabric's Data Pipeline for incremental data loading. It's perfect for anyone looking to streamline their data operations. For those keen on optimizing their data strategy, this is a must-read! Check out the full blog for a detailed walkthrough and start transforming your data pipeline here - https://lnkd.in/gibeFJKs #DataAnalytics #MicrosoftFabric #YugabyteDB #DataPipeline #BigData #DataIntegration

Dueling open-source storage formats are upending the data business, as companies increasingly seek more freedom and build data analytics services that work across formats. In this article, folks from Databricks, Snowflake, Starburst, Theory Ventures, and Microsoft weigh in on how that could move the competition beyond format wars and on to features like security and governance.

A few thoughts: 0) The article buries the lead: "Snowflake data items will light up experiences in any of the Fabric engines." to bullet #5. Folks, storing data in tables and querying it is well beyond being played out. I really want to be snarky... (Alas... Stop focusing so much attention on marginal differences in query speeds or some new SQL function. Invest in solid data management patterns, quality via constraints, observability, a decent catalog, and cost management. If you nail those, you'll be fine.) The next era of data platforms will be aimed at "bringing apps to data" or "your data to apps" (I don't know the right marketing slogan here). Bringing these worlds closer (and yes, apps can be AI-powered workflows too) represents the post-BI/reporting era I think we're all entering, quickly. 1) Interop with Snowflake & Fabric (and Databricks and BigQuery and OSS SQL engines ...). Good. Love them all. Data are everywhere. Make it easy to read/write from any engine. Stop moving data around if you don't have to. Apache XTable is cool, as long as it doesn't create a YAML (yet another management layer, ha!) of headaches.

Would any customers say no to technologies that help them do more with less? No my experience so far. In fact, customers love leveraging the innovations and heavy-lifting from technology vendors. They could then put their focus on ways to deliver more value to their own business. From the following blog, learn how Databricks SQL delivers 4x performance boost over 2 years. Make sure you read till the end, where you will see the amazing innovations (with links) in the last 3 months with Databricks SQL. https://lnkd.in/g_UQseBZ

Weekend Reading: „Microsoft’s vision of an open data lake ecosystem: Open lakes, not walled gardens“ Microsoft #Fabric is an #analytics solution for businesses. It handles everything from moving data to doing data science, real-time analytics, and business intelligence. It includes a bunch of services like #data lake, data engineering, and data integration, all in a single package. In my opinion, the basic idea behind Fabric is particularly interesting for new innovations and offers a lot of potential. I can only recommend this blogpost. You will also find a linked whitepaper in the comments. https://lnkd.in/eZFH2s8q

🔄 Uninterrupted Data Syncs with Airbyte Checkpointing: The Secret to Reliable Data and AI Engineering 🔄 Data and AI engineers, ever faced the frustration of sync failures due to transient issues like network outages or memory shortages? We’ve got you covered! Introducing Airbyte Checkpointing—a game-changing feature designed to ensure your data syncs are robust and reliable, no matter what interruptions occur. 🌐⚙️ Why Checkpointing Matters: 🔹 Seamless Resumption: If a sync fails, Airbyte can pick up right where it left off, minimizing data replay and maximizing efficiency. 🔹 Rapid Checkpoints: Our system guarantees checkpoints every 30 minutes or less, so you’re never left dealing with massive data losses or prolonged downtime. 🔹 Versatile Support: From incremental syncs with various sources to destinations like Snowflake, BigQuery, and Redshift, our checkpointing covers it all! How It Works: Checkpointing relies on STATE messages. When a source sends records and state messages, and the destination confirms receipt, a checkpoint is created. This means that if a sync crashes, Airbyte can skip over records that have already been processed and continue from the last successful checkpoint, saving you time and resources. Here‘s an article that dives deeper into it: https://lnkd.in/empXqebc As we continue to enhance our platform, we're focused on adding checkpointing to more destinations and speeding up syncs. Stay tuned for updates as we make Airbyte even more resilient and efficient! 🎉 Exciting News: All this and more is part of the launch of Airbyte 1.0 on 09/24! Don't miss out—sign up for the event at airbyte.com/v1 and discover how we’re revolutionizing data syncs. #DataEngineering #DataSync #Airbyte1.0 #DataPipeline #ELT #DataManagement

Address the challenges of scalability, agility, and governance with data mesh. Unlock the full potential of your data with #Dataplex in #BigQuery. ✨ Improve data quality, speed up insights, and empower your teams. #Virtusa #googlecloudpartners

Let’s Talk About How Fabric is Centralizing Data Engineering If you’ve been working as a data engineer for a while, you’ll immediately notice the ease once you switch to Fabric. Instead of juggling multiple tools for ETL/ELT (yes, ELT!), you can handle everything within a single, centralized platform. Exciting, right? But, as with every silver lining, there’s a cloud. Recently, while working on a client project, I created a table to store job records. Later, I realized I had declared the wrong data type for one of the columns. Normally, this is a simple fix, you just run an ALTER TABLE...ALTER COLUMN command, and like magic, the problem is resolved. But guess what? Every time I ran the command, I kept getting an error. Surprising, right? Frustrated, I did a quick Google search and discovered that Fabric doesn’t yet support the ALTER COLUMN operation. Turns out, Fabric still uses an older version of the Delta Lake read/write protocol, which doesn’t support Delta column mapping. The workaround? Create an exact duplicate of the table, delete the old one, and use the sp_rename function to swap the table names. That’s a million light-years of work, especially when dealing with large tables! Big shoutout to Andy Cutler, who wrote an excellent blog providing deeper insights into this issue. So, Microsoft, can we get ALTER TABLE...ALTER COLUMN added to Fabric’s feature roadmap? Thanks in advance! P.S. Despite this hiccup, I still believe Microsoft Fabric is one of the best data engineering tools on the market right now. Until next time, love and light, people! 🚀✨ #DataEngineering #MicrosoftFabric #AzureData #ETL #ELT #BigData #DataPlatform #DeepSeek

I began my journey with #SingleStore in 2018 while managing the #SAS/ACCESS team. Back then, the company was known as #MemSQL. This article takes you through the incredible transformation from #MemSQL to #SingleStore, highlighting the innovations that are redefining the data management landscape