Matthew Tyler

Building resilient Kafka applications? Don't let message failures disrupt your data pipelines!     This blog post explores Spring Kafka's retry mechanism to ensure data integrity and application resilience. Learn about exponential backoff, fixed delays, and dead letter topics (DLT).     Ensure your Kafka apps can handle failures gracefully. Explore our blog for more insights https://lnkd.in/gTCd8D74   #EdstemTechnologies #ApacheKafka #SpringKafka #DataPipelines #FaultTolerance #DevOps

13

Introducing the QuixStreams CSV source connector! You can now integrate Kafka and CSV handling systems in seconds with Quix Streams. This high-performance pure Python Kafka Connect alternative handles multi-threaded pre-processing and checkpointing so you can transform your data with a Streaming DataFrame before producing it to Kafka. Obviously, you’ve got custom requirements - who hasn’t? The best thing about the Quix Streams CSV source connector is that you can easily customize it. Optimize write performance, change the serialization format, or implement custom pre-processing logic - it’s all simple when working in pure Python. Check out the repo for more details 👉https://lnkd.in/eyhpuhVQ

9

📊 Hear from one of our Open-Source Engineers on unlocking valuable insights and cheap metrics from datasets! 📊 Dive into the world of data efficiency with our latest article on observing Spark aggregates. Discover powerful techniques to optimise your data processes and drive smarter decision-making. https://lnkd.in/eJKbdgnS #OpenSource #DataScience #Spark #Analytics

17

Build a Real-Time Data Pipeline with Apache Kafka: Step-by-Step Guide 🚀 Are you eager to master real-time data processing? Our latest blog post provides a comprehensive, step-by-step guide on building a real-time data pipeline using Apache Kafka. What You'll Learn: ✅ Setting up a Kafka cluster ✅ Simulating sensor data ✅ Processing data in real time ✅ Storing processed data ✅ Implementing logging and error handling This assignment not only enhances your understanding of Kafka but also equips you with practical skills applicable in today's data-driven industries. Need assistance? Our experts at Codersarts are ready to help you excel in your Kafka projects. #ApacheKafka #DataEngineering #RealTimeData #DataPipeline #Codersarts Check out my blog post https://wix.to/aHFczgJ #newblogpost

1

When used together, Project Jupyter and Redis assist in caching, deduplication, fast data ingest, feature stores, session management, and vector search. Jupyter Notebooks have long been an essential tool for data scientists, but they’re becoming an increasingly important tool for devs building GenAI apps. Learn more about Redis and Jupyter’s integration can be useful for Vector Search: https://bit.ly/4fxUf7B

29

Let's transform raw page view data from Apache Kafka into aggregated insights directly loaded into Databricks Delta tables. 🧱 We explore real-time data aggregation, demonstrating how to downsample page view data across a 10-second time window using just a few SQL queries. By processing data before it reaches Databricks, you can dramatically reduce storage and compute costs while minimizing latency. It's all here 👇 https://lnkd.in/g7QwbSUD

3

Always nice to see approachable and easy-to-follow guides for data engineering beginners. 👍 After introducing how to build your first data platform using MySQL, Apache Doris, and Apache Flink, Mohamed Amine Turki moves a little upstream and explains on Change Data Capture (CDC) with step-by-step instructions. https://lnkd.in/g9igQNjY (P.S. Apache Doris provides a Flink-Doris-Connector with built-in CDC support: https://lnkd.in/gmPESd3V) #dataengineering #beginner #Flink #CDC

21

1 Comment

LSEG didn't let their technical debt get them down 💪 , instead they: ✅ Reduced time to market for new products and services ✅ Faster engineer onboarding ✅ Production pipelines delivered within a week ✅ Able to use all features within Snowflake The full story: https://lnkd.in/ehKP2ipd

12

PDP has published version 1.3 of the data standards, which provide the basis for data interoperability across the pensions dashboards ecosystem. This includes updates to the find and view JSON schema. Industry should be working towards version 1.3. View the data standards: https://ow.ly/lWRo50UrEyK #PensionsDashboards #Pensions #Standards

1

✋ Stop! In the name of data Before you break your stack, take this quick course to learn your way around the #streamingdata ecosystem and get hands-on practice with today’s most popular tools (that are fully compatible with #Kafka). https://rpnda.co/3OwAeTL

6

Real-world data is complex, and traditional large language models fall short when using structured data like prices and product categories. With Superlinked and Redis, you can integrate structured and unstructured data for high-quality information retrieval. This means faster, more accurate recommendations that truly resonate with your customers' needs. Read more here. https://lnkd.in/erngxz-n

31

1 Comment

Databricks Announced the GA of Serverless Compute for Notebooks, Workflows and Delta Live Tables ! This would immensely help Customers with: ➡Rapid workload startup ➡Automatic infrastructure scaling ➡Seamless version upgrades of Databricks runtime ➡Cost Savings, billed only when compute is assigned to your workloads and not for the time to acquire and set up compute instances. Fast ! Simple ! Reliable ! Databricks Digivate Labs https://lnkd.in/gHNF6SGB #Databricks #Serverless #Datalakehouse #DataEngineering #DataScience #Datalake

10

Learn How to Run Hugging Face Models Programmatically Using Ollama  and #Testcontainers 🤖💻 Dive into the future of AI/ML with easy model deployment 👋👋 Say goodbye to complex setups and hello to seamless integration 👉👉 Docker, Inc blog https://lnkd.in/g_F89xCk by Ignacio López Luna

8

In todays #XataLaunchWeek demo, Andrew shows us to how add a NOT NULL constraint to an existing column with a multi-version schema migration, accessing both the old and new schema simultaneously until the migration is complete ✨ Watch now: https://lnkd.in/ekcuabS2

3