Download as KEY, PPTX
![... tags ... record-type: content id: world/picture/2010/may/14/formula-one-monaco tag-ids: [ world/series/eyewitness, sport/formulaone, world/monaco ...] tag-external-names: [ Eyewitness, Formula One, Monaco, ...]](https://image.slidesharecdn.com/apacheluceneeurocon2010-100521093240-phpapp02/75/The-Guardian-Open-Platform-Content-API-Implementation-32-2048.jpg)
![... tags ... record-type: content id: world/picture/2010/may/14/formula-one-monaco tag-ids: [ world/series/eyewitness, sport/formulaone, world/monaco ...] tag-external-names: [ Eyewitness, Formula One, Monaco, ...] record-type: tag id: world/series/eyewitness section-name: World news web-title: Eyewitness type: series internal-name: Eyewitness (centespread photo series)](https://image.slidesharecdn.com/apacheluceneeurocon2010-100521093240-phpapp02/75/The-Guardian-Open-Platform-Content-API-Implementation-33-2048.jpg)
![... tags ... record-type: content id: world/picture/2010/may/14/formula-one-monaco tag-ids: [ world/series/eyewitness, sport/formulaone, world/monaco ...] tag-external-names: [ Eyewitness, Formula One, Monaco, ...] record-type: tag id: world/series/eyewitness section-name: World news web-title: Eyewitness Included in search type: series internal-name: Eyewitness (centespread stored=false photo series)](https://image.slidesharecdn.com/apacheluceneeurocon2010-100521093240-phpapp02/75/The-Guardian-Open-Platform-Content-API-Implementation-34-2048.jpg)
![... factboxes ... record-type: content id: world/picture/2010/may/14/formula-one-monaco factbox-data: [ 197544~|~~|~photography-tip~|~ ] fact-data: [ 197544~|~pro-tip~|~The photographer has framed the cars between the boats and spectators and played with the scales of the components of the scene ] fact-value: [ The photographer has framed the cars between the boats and spectators and played with the scales of the components of the scene ]](https://image.slidesharecdn.com/apacheluceneeurocon2010-100521093240-phpapp02/75/The-Guardian-Open-Platform-Content-API-Implementation-36-2048.jpg)
![... media record-type: content id: world/picture/2010/may/14/formula-one-monaco media-asset-ids: [ PICTURE|362634152|IMAGE|362629791, ...]](https://image.slidesharecdn.com/apacheluceneeurocon2010-100521093240-phpapp02/75/The-Guardian-Open-Platform-Content-API-Implementation-38-2048.jpg)
![... media record-type: content id: world/picture/2010/may/14/formula-one-monaco media-asset-ids: [ PICTURE|362634152|IMAGE|362629791, ...] record-type: media id: PICTURE|362634152|IMAGE|362629791 credit: Mark Thompson/Getty Images width: 1024 height: 768 path: /sys-images/Guardian/About/General/2010/5/14/1273823813621/66-lap- Monaco-grand-prix-002.jpg](https://image.slidesharecdn.com/apacheluceneeurocon2010-100521093240-phpapp02/75/The-Guardian-Open-Platform-Content-API-Implementation-39-2048.jpg)
Uploaded byThe Guardian Open Platform
The Guardian Open Platform Content API: Implementation
The document discusses the implementation of Solr for The Guardian's Open Platform Content API, highlighting its scalability and ease of use compared to other database solutions. It details the architecture, including the microapp framework and data storage, and emphasizes the efficiency of Solr in building indexes quickly from a large content database. The author concludes that Solr has effectively enabled free access to their content API and streamlined operations.
The Guardian Open Platform Content API: Implementation
- 1. Solr in theWild: The Guardian’s Open Platform Content API Graham Tackley guardian.co.uk 1
- 2.
- 3.
- 4.
- 5.
- 6. • Content API • MicroApp Framework • Politics API • Data Store http://www.guardian.co.uk/open-platform
- 7. • Content API • MicroApp Framework • Politics API • Data Store http://www.guardian.co.uk/open-platform
- 8. • Content API • pis.com MicroApp Framework ard iana Politics API ten •ttp://con t.gu h Data Store • http://www.guardian.co.uk/open-platform
- 9.
- 11.
- 12.
- 13.
- 14. Implementation • Traffic patternsmuch less predictable than a web site • Need to easily scale on demand... • ... and never take down guardian.co.uk due to API traffic
- 15. Core Webservers App server Memcached (20Gb) rdbms CMS
- 16. Core Webservers App server Memcached (20Gb) rdbms Content API CMS
- 17. Core Webservers App server Memcached (20Gb) rdbms Content API CMS
- 18. Why Solr? • Databasecould not cope... • ... and far too expensive to scale • Solr ... • ... was easy for developers to understand • ... has a great replication model • ... is simple to install
- 19. Core Webservers App server Memcached (20Gb) CMS
- 20. Core Webservers App server Memcached (20Gb) Solr Master Indexer CMS
- 21. Core Api Web servers Solr & Api App server Solr & Api Memcached (20Gb) Replication Solr & Api Solr Solr Master Solr & Api Indexer Solr & Api CMS Cloud, EC2
- 24.
- 25. Solr Schema • 350+tables in database schema
- 26. Content fields arejust fields...
- 28.
- 29.
- 30. Tags Factbox Media
- 31. Keywor Article d Contributor Video Series Tags Content Audio Publication Gallery Tone Cartoon
- 32. ... tags ... record-type:content id: world/picture/2010/may/14/formula-one-monaco tag-ids: [ world/series/eyewitness, sport/formulaone, world/monaco ...] tag-external-names: [ Eyewitness, Formula One, Monaco, ...]
- 33. ... tags ... record-type: content id: world/picture/2010/may/14/formula-one-monaco tag-ids: [ world/series/eyewitness, sport/formulaone, world/monaco ...] tag-external-names: [ Eyewitness, Formula One, Monaco, ...] record-type: tag id: world/series/eyewitness section-name: World news web-title: Eyewitness type: series internal-name: Eyewitness (centespread photo series)
- 34. ... tags ... record-type: content id: world/picture/2010/may/14/formula-one-monaco tag-ids: [ world/series/eyewitness, sport/formulaone, world/monaco ...] tag-external-names: [ Eyewitness, Formula One, Monaco, ...] record-type: tag id: world/series/eyewitness section-name: World news web-title: Eyewitness Included in search type: series internal-name: Eyewitness (centespread stored=false photo series)
- 35.
- 36. ... factboxes ... record-type:content id: world/picture/2010/may/14/formula-one-monaco factbox-data: [ 197544~|~~|~photography-tip~|~ ] fact-data: [ 197544~|~pro-tip~|~The photographer has framed the cars between the boats and spectators and played with the scales of the components of the scene ] fact-value: [ The photographer has framed the cars between the boats and spectators and played with the scales of the components of the scene ]
- 37.
- 38. ... media record-type: content id:world/picture/2010/may/14/formula-one-monaco media-asset-ids: [ PICTURE|362634152|IMAGE|362629791, ...]
- 39. ... media record-type: content id:world/picture/2010/may/14/formula-one-monaco media-asset-ids: [ PICTURE|362634152|IMAGE|362629791, ...] record-type: media id: PICTURE|362634152|IMAGE|362629791 credit: Mark Thompson/Getty Images width: 1024 height: 768 path: /sys-images/Guardian/About/General/2010/5/14/1273823813621/66-lap- Monaco-grand-prix-002.jpg
- 40. The Code • Writtenin Scala • Uses SolrJ • Plan to open source in the new few months
- 41.
- 42.
- 44. Creating the Index •Existing search index takes 20 hours to build • Solr index takes 1 hour • Here’s how...
- 45. 1.1 million+ itemsof content in the database
- 46. 1.1 million+ itemsof content in the database Split into Batches SELECT id FROM ( SELECT id, ROWNUM rownumber FROM content_live ORDER BY id ) WHERE MOD(rownumber, 10000) = 0
- 47. 1.1 million+ itemsof content in the database Split into Batches SELECT id FROM ( SELECT id, ROWNUM rownumber FROM content_live ORDER BY id ) WHERE MOD(rownumber, 10000) = 0
- 48. 1.1 million+ itemsof content in the database Actor 1 Actor 2 Each actor: Actor 3 1. reads data from database 2. builds solr input document Actor 4 3. submits to solr
- 49. 1.1 million+ itemsof content in the database Actor 1 Actor 2 Each actor: Actor 3 1. reads data from database 2. builds solr input document Actor 4 3. submits to solr
- 50. Summary • Solr madefree access to our content API possible • Replication rocks for scaling • Solr just works for us (thank you!) • NoSQL really isn’t that scary
- 51. • http://guardian.co.uk/open-platform • http://content.guardianapis.com graham.tackley@guardian.co.uk · @tackers 37
Editor's Notes
- #3 As Stephen said: Very basic links to interesting content
- #4 Note the registration paywall
- #5 Broadcast, stories, basic community Rebuild started in 2005
- #6 “Web 2.0”, community, (full fat) RSS, discoverability, tagging. Where do we go from here? Other newspaper sites - looking to restrict access to content via paywalls etc - we’re looking to open up
- #7 We’ve spent the last 12 months experimenting around open distribution and open partnerships - 4 initiatives make up the open platform (right now) (As stephen said)
- #8 This talk focuses on the content API - provides a way for others to re-present our content in their applications
- #9 http://content.guardianapis.com
- #10 http://content.guardianapis.com/search.json?q=prague%20beer&order-by=relevance (most users want most recent content, so default ordering is newest) This is just a dismax search
- #11 Can also retrieve extra metadata, including tags http://content.guardianapis.com/search.json?q=prague%20beer&order-by=relevance&show-fields=all&show-tags=all
- #12 If you have an API key can get full content. (You need to apply for this and agree to some T&Cs - mostly to ensure that we can take down content for legal reasons.) This example key is only valid for this conference, will be disabled afterwards :) http://content.guardianapis.com/search.json?q=prague%20beer&order-by=relevance&show-fields=all&show-tags=all&api-key=eurocon2010
- #13 Refinements give the ability to narrow down your result set (ofc these are just solr facets) http://content.guardianapis.com/search.json?q=prague%20beer&order-by=relevance&show-refinements=all
- #15 Our current architecture - perhaps we could feed the content api off the database?
- #16 Our current architecture - perhaps we could feed the content api off the database?
- #17 time to developer understanding: about 2 hours
- #18 currently rebuild every night, incrementals during the day [next] expose solr master to EC2, create hosts in EC2 that replicate using solr replication - works fantastically. 6GB index size. Load-balancer config. We use solr.war from 1.4 dist totally unchanged - run api webapp in same jetty container
- #19 currently rebuild every night, incrementals during the day [next] expose solr master to EC2, create hosts in EC2 that replicate using solr replication - works fantastically. 6GB index size. Load-balancer config. We use solr.war from 1.4 dist totally unchanged - run api webapp in same jetty container
- #20 Lots of talk nowadays on “no sql” solutions
- #21 No. Designed a new logo that better reflects where we currently are
- #23 disclaimer: the next slides describe how *we* did it; not necessarily best practice! We took the opportunity to simplify our domain model....
- #24 Content fields are just fields But also need to map tags, media, and factboxes
- #28 Here’s how we model tags & content
- #32 Fact boxes associate arbitary information with content We need to search them, but 1-to-1 relationship with content So no separate record
- #33 Fact boxes associate arbitary information with content We need to search them, but 1-to-1 relationship with content So no separate record
- #34 show-media allows access to the non-text assets of an item of content
- #37 Code mostly just takes input params, converts to solr query, and transforms result to json or xml I’m not here to talk about scala, but here’s a quick couple of snippets
- #38 RichSolrDocument makes SolrDocument more “scala” ish
- #39 Scala can make writing understandable code much easier
- #40 Supporting auto scaling in EC2 - our base images all have empty index (EC2 load balance is configured to check this url & add server to list on 200 response)
- #41 Thanks to Grant Ingersoll from Lucid Imagination for guiding us down this route (were planning to do something much more complicated), Also thanks to Francis Rhys-Jones to actually implementing this This is game changing - suddenly we’re prepared to change the index -- and NoSQL solutions seem a whole lot less scary: we migrate our entire database every night!
- #53 Effectively the batch divisions become a work queue fed to a set of actors (Actually, we found that 8 worked best with our hardware) Each actor reads the data from the database; creates a solrinputdocument; submits
- #54 Effectively the batch divisions become a work queue fed to a set of actors (Actually, we found that 8 worked best with our hardware) Each actor reads the data from the database; creates a solrinputdocument; submits
- #55 Effectively the batch divisions become a work queue fed to a set of actors (Actually, we found that 8 worked best with our hardware) Each actor reads the data from the database; creates a solrinputdocument; submits
- #56 Effectively the batch divisions become a work queue fed to a set of actors (Actually, we found that 8 worked best with our hardware) Each actor reads the data from the database; creates a solrinputdocument; submits
- #57 Effectively the batch divisions become a work queue fed to a set of actors (Actually, we found that 8 worked best with our hardware) Each actor reads the data from the database; creates a solrinputdocument; submits
- #58 Effectively the batch divisions become a work queue fed to a set of actors (Actually, we found that 8 worked best with our hardware) Each actor reads the data from the database; creates a solrinputdocument; submits
- #59 Effectively the batch divisions become a work queue fed to a set of actors (Actually, we found that 8 worked best with our hardware) Each actor reads the data from the database; creates a solrinputdocument; submits
- #60 Effectively the batch divisions become a work queue fed to a set of actors (Actually, we found that 8 worked best with our hardware) Each actor reads the data from the database; creates a solrinputdocument; submits
- #61 Effectively the batch divisions become a work queue fed to a set of actors (Actually, we found that 8 worked best with our hardware) Each actor reads the data from the database; creates a solrinputdocument; submits
- #62 Effectively the batch divisions become a work queue fed to a set of actors (Actually, we found that 8 worked best with our hardware) Each actor reads the data from the database; creates a solrinputdocument; submits
- #63 Effectively the batch divisions become a work queue fed to a set of actors (Actually, we found that 8 worked best with our hardware) Each actor reads the data from the database; creates a solrinputdocument; submits
- #64 Effectively the batch divisions become a work queue fed to a set of actors (Actually, we found that 8 worked best with our hardware) Each actor reads the data from the database; creates a solrinputdocument; submits
- #65 Effectively the batch divisions become a work queue fed to a set of actors (Actually, we found that 8 worked best with our hardware) Each actor reads the data from the database; creates a solrinputdocument; submits
- #66 Effectively the batch divisions become a work queue fed to a set of actors (Actually, we found that 8 worked best with our hardware) Each actor reads the data from the database; creates a solrinputdocument; submits
- #67 Effectively the batch divisions become a work queue fed to a set of actors (Actually, we found that 8 worked best with our hardware) Each actor reads the data from the database; creates a solrinputdocument; submits
- #68 Effectively the batch divisions become a work queue fed to a set of actors (Actually, we found that 8 worked best with our hardware) Each actor reads the data from the database; creates a solrinputdocument; submits
- #69 Effectively the batch divisions become a work queue fed to a set of actors (Actually, we found that 8 worked best with our hardware) Each actor reads the data from the database; creates a solrinputdocument; submits
- #70 Effectively the batch divisions become a work queue fed to a set of actors (Actually, we found that 8 worked best with our hardware) Each actor reads the data from the database; creates a solrinputdocument; submits
- #71 Effectively the batch divisions become a work queue fed to a set of actors (Actually, we found that 8 worked best with our hardware) Each actor reads the data from the database; creates a solrinputdocument; submits
- #72 Effectively the batch divisions become a work queue fed to a set of actors (Actually, we found that 8 worked best with our hardware) Each actor reads the data from the database; creates a solrinputdocument; submits
- #73 All we wanted was a search engine... but actually we got an easy to work with, fast, scalable NoSQL solution!
- #74 All we wanted was a search engine... but actually we got an easy to work with, fast, scalable NoSQL solution!