From the course: Cloud-Based AI Solution Design Patterns
Join today to access over 24,900 courses taught by industry experts.
Continuous data validation
From the course: Cloud-Based AI Solution Design Patterns
Continuous data validation
- We often need a lot of data to train AI models. Even if we take steps to prepare and filter this data in advance, there is always the possibility that it will still contain pockets of flawed, biased, or corrupt data that could negatively influence how the model is trained and developed. Similarly, when we are ingesting new data into a production AI system, we can't always be sure that that data is also error-free or whether there is drift that is causing it to no longer be in alignment with the type of data the model was originally trained on. The continuous data validation pattern addresses these concerns by establishing a comprehensive system of regular automated data checks into our data architecture. There are several ways this pattern can be applied. There is real-time data validation, scheduled data validation, and event-triggered validation. Let's have a look at each approach. With real-time data validation, the data can be checked when it is first ingested into the AI system…
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.