Sijie Guo

Worth reading: How the Apache Arrow Format Accelerates Query Result Transfer by Ian Cook One of the hidden problems of intersystem communication is transcription of data formats. Ian does a wonderful job of explaining the "why" of Apache Arrow as well as the "how" of its advantages. Read on...

8

AI memory is fundamental for conversational AI and AI agents. It is critical for building personalized experiences and improving reasoning. Check out Letta’s course to understand how to use AI memory to enhance the customization and quality of your AI app.

15

➰ APIContext's latest insights on cloud-based API performance have been featured on Nordic APIs! In the 2024 report, we delve into crucial aspects affecting API efficiency: 🔐 Authentication is key. Ensuring robust authentication mechanisms in APIs is crucial, yet offloading this function can introduce complex challenges. 📖 DNS configurations matter. How you handle DNS settings can significantly impact API performance, particularly when tied to authentication processes. 🌐 The role of third parties. Critical third-party services play a significant role in 🔗 API transactions, underscoring the need for careful integration and monitoring. ⛅ Not all clouds are created equal. Differences in cloud services can drastically affect API performance, an essential consideration as APIs account for over 80% of all internet traffic. #APIs #CloudComputing #TechInsights #APIContext

15

1 Comment

Cedana enables live migration of stateful workloads including databases and long-running batch analytics. This is important as it enables higher reliability and lower costs (20%-80%). When your spot instance gets revoked, you can feel safe knowing that your workload is resumed on another instance without losing work. Here's a how-to guide to get started running Redis on spot. https://lnkd.in/eNt_hRGX If you're headed to #kubecon let's find time to chat.

58

We're launching the inaugural edition of Beyond Benchmarks, a deep dive into the metrics and trends observed across the early-stage enterprise cloud market. This is the first look at some of the business data around GenAI in SaaS. Some initial observations: 1) 60% of SaaS companies have already integrated GenAI into their offerings (60% of these as part of their current product and 40% as a new product), and an additional 20% are planning to do so this year. Almost half aren't monetizing these features. Those that are are primarily experimenting with both usage-based and flat fee pricing models. 2) Most companies are utilizing OpenAI as their primary LLM, but many are experimenting with multiple models - and we are noticing an early trend towards intelligently routing GenAI inference requests to different models based on cost, performance, and security. 3) Though we are in the initial innings of GenAI, we are seeing some positive signals - companies that implemented GenAI had 7% higher NDR than those that did not. Still early days, but we'll be doing this benchmarking report regularly to track and share how things progress in this new era. Link to report in comments.

104

6 Comments

I like this idea of "technical taste". It gets into where software engineering is as much creative art as it is science. Four great takeaways (IMHO) for software devs in this era in particular: 1. Aspiring engineers should cultivate a sense of curiosity, experiment with different tools and technologies, and embrace a mindset of continuous learning. 2. AI has the potential to streamline processes and enhance productivity for engineers, but it may also lead to disruptions in traditional software development workflows. 3. Developing technical taste and judgment is essential for making informed decisions about which technologies and approaches to pursue. 4. Collaboration and open-mindedness are key to leveraging the full potential of AI and staying ahead of technological advancements. I also really liked these two quotes from Sam Schillace: 1. "The right time to do something is when you have that feeling in the pit of your stomach that's like, 'oh, this is a great idea and it's going to suck to build because nothing's ready yet.'" 2. "Technical taste is like, 'how well have you consolidated that set of experiences and heuristics into judgment that you can apply accurately when you see new things?'" 3. "It may be the case that very small teams can do very large projects, or like we were talking about before, it may be the case we're just going to get really ambitious about what we try to do with the same size of teams, which is kind of where I would put my money." Great job on continuing to put out great episodes, Brett Berson and team First Round Capital! #ai #engineering #softwaredevelopment

5

Worth reading: Generative AI is not going to build your engineering team for you. So, so good - peppered with gem quotes and observations- chuckle. https://lnkd.in/gZBKtU4X Thanks to Eric Brooke for sharing.

16

Batch processing for LLM inference activity: "One sign of that is the rise of batch processing. Today, popular consumer AI products like ChatGPT and Perplexity provide users with near instantaneous responses, otherwise known as real-time inference. However, businesses don't always need immediate responses and are often willing to wait hours, days, or even weeks for responses, as long as they don't have to shell out as much money. Founders of cloud and inference providers have told me there's growing pressure from business customers for this kind of flexibility, otherwise known as batch processing." -- Nathaniel Whittemore on "The AI Daily Brief" (with much deserved shout-out to his inspo from Stephanie Palazzolo at The Information) I fall into this bucket as we have a lot of backfilling to do as we're building out the underlying data in our service, and some of that could just as well be done in the wee hours of the night. The new stuff flowing in I want processed ASAFP, but these backfills—I'd mos def take more cost-efficient delayed batch processing. There's also use cases like needing (near) real-time inference for the first inference pass for timeliness / contemporaneousness value, but then a second pass slow-time inference for wider classification exercises over an ever-growing data corpus. I could see us mixing Sonnet & Haiku models in both cases. Just a heads up to Anthropic Mike Krieger as you contemplate this, along with Amazon Web Services (AWS) Jeff Barr Mark Roy Swami Sivasubramanian on the Bedrock side of things...and I don't mean the "batch" mode that doesn't seem to have any real value other than helping AWS manage its own rate limit challenges. (DeepCast is using Bedrock.)

11

2 Comments

LLMs are trained on internet-scale data. But to make AI work for work, models need to be grounded in your enterprise data -- a much harder problem than meets the eye! How do you connect to all your data sources and ingest them frequently enough while managing compute/storage costs? How do you make sure that you only show users results from documents they have access to? How do you rank results without internet-scale click and link data? How do you process the content of documents that are often hidden in PDFs and presentations? These are tricky problems, and it's why enterprise search didn't work for nearly a decade, until Glean came along. Turns out it's the exact same set of challenges needed to give LLMs the right context to work their magic on your enterprise content. Glean spent 5 years building the ingestion, search, and ranking infra needed to solve this problem well. This has given them the right to run the "Work AI" race and delight users at work. CEO Arvind Jain shares more on the latest episode of Training Data. Listen here, or wherever you get your podcasts: YouTube: https://seq.vc/ytd Apple: https://seq.vc/pit Spotify: https://seq.vc/jnz 

203

10 Comments

Outcome-Based Pricing, AgentOS, and User Simulators: 3 takeaways from Sequoia Capital's Pat Grady & Ravi Gupta's chat with Sierra's co-founder Clay Bavor 🧠 1. Outcome-Based Pricing for High-Value AI Tasks: Charge based on the value delivered, not just usage 💰 2. AgentsOS: A powerful concept for coordinating and managing multiple AI agents 🤖 3. User Simulators: Testing AI agents with simulated users to accelerate user alignment and build a moat 🤯 User simulators could be a game-changer, providing a massive dataset of simulated customer behavior to train and refine AI agents. This could lead to a powerful "simulated customer data flywheel," offering a unique competitive advantage. What are your thoughts on these AI agent trends? 🤔 #AI #Agents #Sequoia #Sierra #Innovation #TechTrends Link to the original interview: https://lnkd.in/eK6MG6wx

6

The shift from traditional ML to LLMs has pushed infrastructure requirements to new limits. At neptune.ai, we’re tackling these challenges by focusing on: → Scalable infrastructure, built to handle high-speed data ingestion and processing. → Real-time data display, adapting seamlessly to complex and large volumes. → Consistent UX quality, even under intense backend demands. — (link to the full interview in the comments) #generativeai #genai #llm

3

1 Comment

If you are working with Docker... I highly recommend looking at the Slim toolkit. https://lnkd.in/gX_Gb9Kf Why? Slim allows developers to inspect, optimize, and debug their containers using its xray, lint, build, debug, run, images, merge, registry, vulnerability (and other) commands. It simplifies and improves your developer experience building, customizing and using containers. It makes your containers better, smaller and more secure while providing advanced visibility and improved usability working with the original and minified containers. Don't change anything in your container image; minify it by up to 30x to make it secure, too! Optimizing images isn't the only thing it can do, though. It can also help you understand and author better container images. Keep doing what you are doing. No need to change anything. Use the base image you want. Use the package manager you want. Don't worry about hand-optimizing your Dockerfile. You shouldn't have to throw away your tools and your workflow to have small container images. Node.js application images: from ubuntu:14.04 - 432MB => 14MB (minified by 30.85X) from debian:jessie - 406MB => 25.1MB (minified by 16.21X) from node:alpine - 66.7MB => 34.7MB (minified by 1.92X) from node:distroless - 72.7MB => 39.7MB (minified by 1.83X) Go application images: from golang:latest - 700MB => 1.56MB (minified by 448.76X) from ubuntu:14.04 - 531MB => 1.87MB (minified by 284.10X) from golang:alpine - 258MB => 1.56MB (minified by 165.61X) from centos:7 - 615MB => 1.87MB (minified by 329.14X) Rust application images: from rust:1.31 - 2GB => 14MB (minified by 147.16X) Java application images: from ubuntu:14.04 - 743.6 MB => 100.3 MB

4

Anthropic continues to deliver new models that customers enjoy using in a variety of ways. Today, Anthropic released an upgraded Claude 3.5 Sonnet model with awesome new capabilities, and it’s available now in Amazon Bedrock. The updated Claude 3.5 Sonnet shows wide-ranging improvements on industry benchmarks, with strong gains in agentic coding and tool use tasks. Anthropic also introduced a new capability in public beta: computer use. In Amazon Bedrock, developers can direct Claude to use computers the way people do—by looking at a screen, moving a cursor, clicking buttons, and typing text. Claude 3.5 Sonnet is the first frontier AI model to reliably use computers in this way—albeit experimentally—and Anthropic expects the capability to improve over time. Such a cool feature that I know our customers will find useful. https://lnkd.in/gwBQf8zH

2,508

46 Comments

Nearly every great product company is a great infrastructure software company. Infrastructure enabled Google to deliver the most relevant search results and the fastest browser, Facebook to provide the most personalized social feed, Netflix to stream with the best viewer experience (minimal lag/interruption), Uber to determine the best real-time ride matches, etc. Infrastructure was key to helping these companies (and many others) win their markets. It’ll be the same in AI. The companies that leverage AI infrastructure in the smartest ways - figuring out the right breakdown and sequence of agentic tasks, utilizing data curation and RAG to get best-in-class model performance, managing multi-model and multi-modal complexity - will gain a strategic / competitive advantage. AI founders: invest in your infrastructure early.

125

9 Comments

Exciting news for teams leveraging monorepos! Coherence now supports glob patterns for selective build configuration, bringing unprecedented efficiency to your development pipeline. 🔧 Technical Highlights: • Granular control over build and deploy processes • Intelligent file change detection • Seamless integration with existing CI/CD workflows 💡 Key Advantages: 1. Dramatic reduction in build and deploy times 2. Optimized resource allocation 3. Precision targeting of updates for changed code only 🏗️ Real-world Impact: • Accelerated development cycles • Enhanced productivity for large-scale projects • Significant cost savings on computational resources This feature empowers you to fine-tune your build process, ensuring only necessary services are built and deployed based on actual code changes. It's a game-changer for maintaining velocity in complex, multi-service architectures. Ready to supercharge your monorepo? Dive into our comprehensive documentation to implement selective builds and experience the future of efficient development.

6

Hard-coded model routing takes away intelligence from your system instead of adding to it! Interesting read below on how Ema Unlimited's Fusion model is a potent solution for the heterogeneous multi-model future of AI in the enterprise.

14

1 Comment

💥 Nice preInvent announcement from DynamoDB -- the ability to pre-warm the max throughput on your table. Previously, you'd have to briefly set your table to provisioned mode with high throughput (see Yan Cui's article on this), but now you can do it yourself with an API call. Be careful on oversizing though! You'll see the impact of this if using DDB Streams -- lots of small / empty batches. And if you're not consuming from Lambda, that will be more expensive. https://lnkd.in/dHV-q3zc

102

3 Comments

🪦 RPA is dead – right? Yesterday, Andreessen Horowitz posted an investment thesis about the end of Robotic Process Automation (RPA), and the subsequent rise of intelligent automation. Kimberly Tan made great points, and I think the thesis is pretty spot on. RPA as it exists today has fallen drastically short of what it promised to achieve. There’s a couple key problems: 1. Long and expensive (!!!) implementation process. If you want to implement RPA within your organization today, you have to go through a multimillion dollar implementation phase that can take months. Unless you’re a massive company, there’s no way you can make this work. 2. Brittle, hardcoded flows. RPAs today are heavily dependent on hardcoded rules. These automations break as soon as the underlying websites change, which kind of defeats the whole purpose of having an automation in the first place When you combine the two, you quickly reach a cost structure that makes a positive ROI impossible to achieve. As a result, the majority of repetitive, back-office workflows are still un-automated. LLM driven RPA will close this gap and deliver what the first wave of RPA companies promised 15 years ago. I don’t think RPA is ‘dead’, per se. But, I do think that new, vertical specific automation tools are going to force incumbents to decide between cannibalizing their business models or slowly fading into irrelevance. I can’t think of a more exciting space to be building in. —----- If anyone has anything to add or thoughts on these I'd love to hear them below!

19

7 Comments