Lucene Near Realtime Search - PowerPoint PPT Presentation

lucene near realtime search l.
Skip this Video
Loading SlideShow in 5 Seconds..
Lucene Near Realtime Search PowerPoint Presentation
Lucene Near Realtime Search

play fullscreen
1 / 17
Download Presentation
Download Presentation

Lucene Near Realtime Search

Presentation Transcript

  1. Lucene Near Realtime Search Jason Rutherglen & Jake Mannix LinkedIn 6/3/2009 SOLR/Lucene User’s Group San Francisco

  2. What is NRT? • Search on documents nearly as fast as they are indexed • Delete documents in a way that is immediate and IO efficient • Good for things like Twitter and other apps that require realtime searching (Social 2.0)

  3. Today? • Users expect to search their data immediately after updating it (Web/Social 2.0 apps) • Search engines are designed to perform efficient batch indexing (not realtime) • Batch indexing is slow and updates take a while to be searchable

  4. NRT in Lucene • Uses core Lucene code to make existing batch indexing nearly realtime • Required retrofitting of some of the core implementation • Details are hidden • Hopefully really easy for developers to use

  5. Lucene NRT Patches • LUCENE-1314 – IndexReader.clone • LUCENE-1516 – IndexWriter.getReader • LUCENE-1313 – RAMDir in IndexWriter • LUCENE-1483 – Fast FieldCache loading • LUCENE-1231 – Column stride fields • LUCENE-1526 – Incremental copy-on-write

  6. LUCENE-1314 • IndexReader.clone is like reopen • However it performs a copy-on-write of norms and deletes • Used by LUCENE-1516 to keep deletes in RAM (rather than flush them to disk)

  7. LUCENE-1516 • Adds ability to obtain an IndexReader from IndexWriter • Efficient in ram deletes • Call IndexWriter.getReader instead of IndexReader.reopen • All updating, deletes, roepening, and flushing details hidden from user • Will be in Lucene 2.9

  8. Sample IW.getReader Code IndexWriter writer; Document doc = new Document(); writer.addDocument(doc); IndexReader reader = writer.getReader(); Document sameDoc= reader.document(0); assert doc.equals(sameDoc);

  9. LUCENE-1313 • Near Realtime Search • Makes IW.getReader faster • New segments are flushed to IndexWriter internal RAMDirectory • Could increase overall indexing performance because there’s no pause while the ram buffer is being written to disk • Will be in Lucene 2.9?

  10. LUCENE-1483 • Searches on fieldcaches at the segment level • Means faster field cache loading and more efficient memory usage • Good for realtime because field cache loading is less of a bottleneck, less ram usage • Will be in Lucene 2.9

  11. LUCENE-1526 • Optimize copy-on-write • When we’re doing IndexReader.clone, we may be creating a huge new array for a small number of deletes or norms updates • So we need to do incremental copy-on-write of things like deletes, norms, and field caches (?) • Lucene 3.0?

  12. LUCENE-1231 • Column stride fields will make field cache loading faster because data will be loaded sequentially from disk • Today there are potentially two hard drive seeks per field cache value (, • Lucene 3.0?

  13. Future of Lucene NRT • LUCENE-1292 – Realtime parallel untokenized field index (for tags) • Pulsing - Store smaller postings directly in the term dictionary (to avoid seeks) for faster field cache loading • Replication • More benchmarks

  14. LinkedIn Open Source Projects • Bobo – Facet library that counts using custom field caches • Zoie – Realtime search on top of Lucene • Voldemort – Distributed key-value storage

  15. BoboBrowse: facet features • MultiSelect • Runtime-defined facets (query-based, etc) • Fast (custom field-cache based) • Custom facet types: • Hierarchical (/a/b/c) • Range • Multivalued

  16. Zoie: realtime features • No modifications to core lucene • Multiple read/write: RAMDir + FSDir • IndexReader on (small) RAMDir opened per request: instantly realtime • IndexReaderDecorator for custom Reader • Transparent Indexing: implement StreamDataProvider then inject

  17. Next Steps • Help work on the patches? • LinkedIn is hiring • Contact: or