Systematically Improving RAG Applications
Год выпуска: 5/2025
Производитель: Maven
Сайт производителя:
https://maven.com/applied-llms/rag-playbook
Автор: Jason Liu
Продолжительность: 30h 13m 25s
Тип раздаваемого материала: Видеоурок
Язык: Английский
Субтитры: Отсутствуют
Описание:
Stop building RAG systems that impress in demos but disappoint in production
Transform your retrieval from "good enough" to "mission-critical" in weeks, not months
Most RAG implementations get stuck in prototype purgatory. They work well for simple cases but fail on complex queries—leading to frustrated users, lost trust, and wasted engineering time. The difference between a prototype and a production-ready system isn't just better technology, it's a fundamentally different mindset.
The RAG Implementation Reality
What you're experiencing right now:
❌ Your RAG demo impressed stakeholders, but real users encounter hallucinations when they need accuracy most
❌ Engineers spend countless hours tweaking prompts with minimal improvement
❌ Colleagues report finding information manually that your system failed to retrieve
❌ You keep making changes but have no way to measure if they're actually helping
❌ Every improvement feels like guesswork instead of systematic progress
❌ You're unsure which 10% of possible enhancements will deliver 90% of the value
What your RAG system could be:
With the RAG Flywheel methodology, you'll build a system that:
✅ Retrieves the right information even for complex, ambiguous queries
✅ Continuously improves with each user interaction
✅ Provides clear metrics to demonstrate ROI to stakeholders
✅ Allows your team to make data-driven decisions about improvements
✅ Adapts to different content types with specialized capabilities
✅ Creates value that compounds over time instead of degrading
What Makes This Course Different
Unlike courses that focus solely on technical implementation, this program gives you the
systematic, data-driven approach used by companies to transform prototypes into production systems that deliver real business value:
✅
The Improvement Flywheel: Build synthetic evaluation data that identifies exactly what's failing in your system—even before you have users
✅
Fine-tuning Framework: Create custom embedding models with minimal data (as few as 6,000 examples)
✅
Feedback Acceleration: Design interfaces that collect 5x more high-quality feedback without annoying users
✅
Segmentation System: Analyze user queries to identify which segments need specialized retrievers for 20–40% accuracy gains
✅
Multimodal Architecture: Implement specialized indices for different content types (documents, images, tables)
✅
Query Routing: Create a unified system that intelligently selects the right retriever for each query
The Complete RAG Implementation Framework
Week 1: Evaluation Systems
Build synthetic datasets that pinpoint RAG failures instead of relying on subjective assessments
BEFORE: "We need to make the AI better, but we don't know where to start."
AFTER: "We know exactly which query types are failing and by how much."
Week 2: Fine-tune Embeddings
Customize models for 20–40% accuracy gains with minimal examples
BEFORE: "Generic embeddings don't understand our domain terminology."
AFTER: "Our embedding models understand exactly what 'similar' means in our business context."
Week 3: Feedback Systems
Design interfaces that collect 5x more feedback without annoying users
BEFORE: "Users get frustrated waiting for responses and rarely tell us what's wrong."
AFTER: "Every interaction provides signals that strengthen our system."
Week 4: Query Segmentation
Identify high-impact improvements and prioritize engineering resources
BEFORE: "We don't know which features would deliver the most value."
AFTER: "We have a clear roadmap based on actual usage patterns and economic impact."
Week 5: Specialized Search
Build specialized indices for different content types that improve retrieval
BEFORE: "Our system struggles with anything beyond basic text documents."
AFTER: "We can retrieve information from tables, images, and complex documents with high precision."
Week 6: Query Routing
Implement intelligent routing that selects optimal retrievers automatically
BEFORE: "Different content requires different interfaces, creating a fragmented experience."
AFTER: "Users have a seamless experience while the system intelligently routes to specialized components."
Real-world Impact From Implementation
✅ 85% blueprint image recall: Construction company using visual LLM captioning
✅ 90% research report retrieval: Through better text preprocessing techniques
✅ $50M revenue increase: Retail company enhancing product search with embedding fine-tuning
✅ +14% accuracy boost: Fine-tuning cross-encoders with minimal examples
✅ +20% response accuracy: Using re-ranking techniques
✅ -30% irrelevant documents: Through improved query segmentation
Join 400+ engineers who've transformed their RAG systems with this methodology
Your Instructor
Jason Liu has built AI systems across diverse domains—from computer vision at the University of Waterloo to content policy at Facebook to recommendation systems at Stitch Fix that boosted revenue by $50 million. His background in managing large-scale data curation, designing multimodal retrieval models, and processing hundreds of millions of recommendations weekly has directly informed his consulting work with companies implementing RAG systems.
This Course Is For You If You Are
- A product leader, engineer, or data scientist looking to move beyond ad-hoc RAG prototypes into scalable, production-grade AI solutions.
- A professional who understands LLM basics but wants a repeatable, data-driven methodology to improve retrieval relevance, latency, and user
- Eager to create feedback loops that continuously refine and enhance the quality of RAG applications as models, data, and user needs evolve.
By the end of this course, participants will be able to
Adopt a Systematic, Data-First Methodology
Implement the Data and Evals Flywheel approach to continuously develop and improve RAG applications—breaking free from guesswork and relying on measurable, iterative enhancements.
Run Fast, Unit-Test-Like Evaluations
Quickly assess your retrieval systems using precision and recall metrics, identify bottlenecks, and confidently validate changes without sinking into endless trial-and-error cycles.
Leverage Synthetic Data for Rapid Iteration
Generate and utilize synthetic data sets to speed up experimentation, enabling you to test new approaches, embeddings, and architectures before committing full resources.
Master Fine-Tuning & Hard Negative Mining
Apply fine-tuning strategies for embedding models to boost search relevance and explore hard negative examples to further sharpen retrieval performance.
Classify Queries & Identify Bottlenecks
Use query classification and segmentation techniques to pinpoint exactly where your RAG system falls short—whether it’s due to limited inventory or insufficient capabilities.
Design Specialized Indices for Multiple Modalities
Create tailored indices for documents, images, tables, SQL databases, and more. Learn when and how to fuse or layer these indices to handle complex retrieval tasks elegantly.
Enhance Retrieval with Summarization & Chunking
Implement synthetic text chunk generation and strategic summarization methods to improve retrieval results, ensuring end-users get clear, concise, and contextually rich answers.
Implement Query Routing & Index Fusion
Develop systems that intelligently route queries to the right indices, tools, or pipelines. Blend and fuse indices effectively to handle nuanced, multi-step queries.
Optimize Both Global & Local Performance
Evaluate the performance of your routing logic and each individual index separately. Gain the nuance to fine-tune global system performance and local retrieval accuracy in tandem.
Integrate Feedback Loops for Continuous Improvement
Design explicit and implicit feedback mechanisms—capturing user signals, automating re-labeling, and applying improvements in real-time to keep your RAG systems on an upward trajectory.
Формат видео: MP4
Видео: avc, 1920x1080 и 1828x1080, 5:3, 18.000 к/с, 423 кб/с
Аудио: aac lc, 44.1 кгц, 128 кб/с, 2 аудио
MediaInfo
General
Complete name : D:\2\Maven - Systematically Improving RAG Applications (5.2025)\04. Week 1 - 2 _ Benchmarking Retrieval - Descript.mp4
Format : MPEG-4
Format profile : Base Media
Codec ID : isom (isom/iso2/avc1/mp41)
File size : 53.3 MiB
Duration : 13 min 23 s
Overall bit rate : 556 kb/s
Frame rate : 18.000 FPS
Writing application : Lavf60.16.100
Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L4
Format settings : CABAC / 4 Ref Frames
Format settings, CABAC : Yes
Format settings, Reference frames : 4 frames
Codec ID : avc1
Codec ID/Info : Advanced Video Coding
Duration : 13 min 23 s
Bit rate : 423 kb/s
Width : 1 828 pixels
Height : 1 080 pixels
Display aspect ratio : 5:3
Frame rate mode : Constant
Frame rate : 18.000 FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.012
Stream size : 40.5 MiB (76%)
Writing library : x264 core 164
Encoding settings : cabac=1 / ref=3 / deblock=1:0:0 / analyse=0x3:0x113 / me=hex / subme=7 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=34 / lookahead_threads=5 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=3 / b_pyramid=2 / b_adapt=1 / b_bias=0 / direct=1 / weightb=1 / open_gop=0 / weightp=2 / keyint=250 / keyint_min=18 / scenecut=40 / intra_refresh=0 / rc_lookahead=40 / rc=crf / mbtree=1 / crf=23.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / ip_ratio=1.40 / aq=1:1.00
Color range : Limited
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709
Codec configuration box : avcC
Audio
ID : 2
Format : AAC LC
Format/Info : Advanced Audio Codec Low Complexity
Codec ID : mp4a-40-2
Duration : 13 min 23 s
Source duration : 13 min 23 s
Bit rate mode : Constant
Bit rate : 128 kb/s
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 44.1 kHz
Frame rate : 43.066 FPS (1024 SPF)
Compression mode : Lossy
Stream size : 12.3 MiB (23%)
Source stream size : 12.3 MiB (23%)
Default : Yes
Alternate group : 1
