Processing Linked Data with Catmandu
5 likes2,601 views
This document discusses using the Catmandu tool to process linked data. It describes how Catmandu can be used to convert between different data formats like MARC, JSON, RDF, and load/export to systems. The document outlines a typical workflow of: 1) Exporting from a catalog to MARC, 2) Converting MARC to JSON, 3) Converting JSON to RDF, and 4) Enriching the RDF with links to external datasets. It provides examples of Catmandu commands for conversion, fixing fields, and importing/exporting to different systems.
![Processing Linked Data with Catmandu | http://librecat.org
CATMANDU
catmandu import JSON to MongoDB --opt ... --opt ...
catmandu import MARC to ElasticSearch
catmandu import DC to FedoraCommons
catmandu import CSV to DBI
catmandu export MongoDB to JSON
catmandu export Solr to YAML
catmandu export DBI to CSV
catmandu export FedoraCommons to Template --template test.tt
test.tt: (TemplateToolKit)
[%- FOREACH f IN record %]
[% _id %] [% f.shift %][% f.shift %][% f.shift %][% f.join(“:”) %]
[%- END %]
IMPORT / EXPORT](https://image.slidesharecdn.com/presentation-140313031740-phpapp01/85/Processing-Linked-Data-with-Catmandu-15-320.jpg)
More Related Content
What's hot (6)
Viewers also liked (20)
Similar to Processing Linked Data with Catmandu (20)
More from Patrick Hochstenbach (20)
Recently uploaded (20)
Processing Linked Data with Catmandu
- 1. Processing Linked Data with Catmandu Patrick Hochstenbach | UGent http://librecat.org
- 2. Processing Linked Data with Catmandu | http://librecat.org LUND GHENT BIELEFELD
- 3. Processing Linked Data with Catmandu | http://librecat.org RATIONALE
- 4. Processing Linked Data with Catmandu | http://librecat.org KAHN-WILENSKI WEB HANDLE SERVICE PROVIDER REPOSITORY REPOSITORY I search a paper about...
- 5. Processing Linked Data with Catmandu | http://librecat.org Hypothesis 1: one network with a common schema Hypothesis 2: object-oriented design Hypothesis 3: the resource is the message
- 6. Processing Linked Data with Catmandu | http://librecat.org Hypothesis 1: one network with a common schema GOOGLE EUROPEANA OPENAIRE CRIS Videos Images Books Data sets
- 7. Processing Linked Data with Catmandu | http://librecat.org Hypothesis 2: object-oriented design Drive Race Park Economy Compact Minivan Convertible Wheel Half car Bicycle Zeppelin
- 8. Processing Linked Data with Catmandu | http://librecat.org Hypothesis 3: the resource researcher is the message DNS GOOGLE REPOSITORY CLOUD Dr. Müller
- 9. Processing Linked Data with Catmandu | http://librecat.org LIBRECAT/CATMANDU
- 10. Processing Linked Data with Catmandu | http://librecat.org CATMANDU PubMed MARC MODS EXCEL DSPACE Fedora SRU OAI-PMH DBI ISI Twitter DBI Atom EXCEL RDF JSON XML Solr ElasticSearch MongoDB Fedora Aleph Fix
- 11. Processing Linked Data with Catmandu | http://librecat.org FUNCTIONAL DESIGN JSON } each slice take group select map reduce add_field join_field lookup remove_field marc_mab count
- 12. Processing Linked Data with Catmandu | http://librecat.org LIBRECAT Institutional Repositories Search Engines Image Databases Archival Systems Data cleaning workbench Citation Style Processor
- 13. Processing Linked Data with Catmandu | http://librecat.org CATMANDU catmandu convert MARC to JSON < records.mrc catmandu convert OAI --url http://server/OAI to JSON catmandu convert SRU --url http://server/SRU --query dna to JSON catmandu convert DBI --query ‘SELECT * FROM table’ to JSON catmandu convert MARC to JSON < records.mrc catmandu convert OAI --url http://server/OAI to XML catmandu convert SRU --url http://server/SRU --query dna to YAML catmandu convert ArXiv --query ‘all:electron’ to CSV CONVERT
- 14. Processing Linked Data with Catmandu | http://librecat.org CATMANDU catmandu convert X to Y --fix ‘marc_map(“245”,”title”)’ catmandu convert X to Y --fix ‘prepend(“title”,”abcd-”)’ catmandu convert X toY --fix fixes.txt fixes.txt: remove_field(“_id”); marc_map(“001”, “merge.id”); prepend(“merge.id”, “author:”); add_field(“merge.source”,”author”); copy_field(“merge.id”,”_id”); FIX set_field add_field move_field copy_field remove_field upcase downcase capitalize trim substring prepend append lookup lookup_in_store count cmd split_field join_field retain_field replace_all collapse expand clone if_all_match if_any_match if_exists
- 15. Processing Linked Data with Catmandu | http://librecat.org CATMANDU catmandu import JSON to MongoDB --opt ... --opt ... catmandu import MARC to ElasticSearch catmandu import DC to FedoraCommons catmandu import CSV to DBI catmandu export MongoDB to JSON catmandu export Solr to YAML catmandu export DBI to CSV catmandu export FedoraCommons to Template --template test.tt test.tt: (TemplateToolKit) [%- FOREACH f IN record %] [% _id %] [% f.shift %][% f.shift %][% f.shift %][% f.join(“:”) %] [%- END %] IMPORT / EXPORT
- 16. Processing Linked Data with Catmandu | http://librecat.org CATMANDU https://metacpan.org/pod/Catmandu https://github.com/LibreCat/Catmandu http://librecat.org/tutorial/ http://librecat.org/catmandu/2013/06/21/catmandu-cheat-sheet.html
- 17. Processing Linked Data with Catmandu | http://librecat.org LIBRECAT http://biblio.ugent.be
- 18. Processing Linked Data with Catmandu | http://librecat.org LIBRECAT http://pub.uni-bielefeld.de/en
- 19. Processing Linked Data with Catmandu | http://librecat.org LIBRECAT http://adore.ugent.be
- 20. Processing Linked Data with Catmandu | http://librecat.org LIBRECAT http://libnew.ugent.be
- 21. Processing Linked Data with Catmandu | http://librecat.org Architecture FEDORA MEDIAMOSA VLE BIBLIO SCANNING RECEIVE ALEPH ABS CLOUD DEDUP/MERGE/AUGMENT BLACKLIGHT
- 22. Processing Linked Data with Catmandu | http://librecat.org LINKED DATA
- 23. Processing Linked Data with Catmandu | http://librecat.org PRODUCTION CATALOG MARC 245 $$a ... $$b 260 $$a ... 700 $$a ... JSON/YAML LINKED DATA
- 24. Processing Linked Data with Catmandu | http://librecat.org STAGE 1: CATALOG to MARC CATALOG MARC 245 $$a ... $$b 260 $$a ... 700 $$a ... $ catmandu export ALEPH to MARC
- 25. Processing Linked Data with Catmandu | http://librecat.org STAGE 2: MARC to JSON MARC 245 $$a ... $$b 260 $$a ... 700 $$a ... JSON/YAML
- 26. Processing Linked Data with Catmandu | http://librecat.org MARC 245 $$a ... $$b 260 $$a ... 700 $$a ... JSON/YAML STAGE 2: MARC to JSON Tolstoj, Lev Nikolaevič, Author War and peace / Title 1952. Publication Year Napoleonic Wars, Subject
- 27. Processing Linked Data with Catmandu | http://librecat.org STAGE 2: MARC to JSON MARC 245 $$a ... $$b 260 $$a ... 700 $$a ... JSON/YAML Tolstoj, Lev Nikolaevič, War and peace / 1952. Napoleonic Wars, Author Title Year Subject Tolstoj, Lev Nikolaevič War and peace 1952 Napoleonic Wars Author Title Year Subject FIX
- 28. Processing Linked Data with Catmandu | http://librecat.org STAGE 3a: JSON to RDF JSON/YAML LINKED DATA Tolstoj, Lev Nikolaevič War and peace 1952 Napoleonic Wars Author Title Year Subject ? Tolstoj, Lev Nikolaevič War and peace 1952 Napoleonic Wars dc:creator dc:title dc:date dc:subject FIX
- 29. Processing Linked Data with Catmandu | http://librecat.org JSON/YAML LINKED DATA STAGE 3a: JSON to RDF <http://example.org/000000008> <http://purl.org/dc/elements/1.1/creator> “Tolstoj, Lev Nikolaevič”; <http://purl.org/dc/elements/1.1/title> “War and peace” ; <http://purl.org/dc/elements/1.1/date> “1952” ; <http://purl.org/dc/elements/1.1/subject> “Napoleonic Wars” ; a <http://www.europeana.eu/schemas/edm/Book> . FIX RDF/Turtle http://demo.librecat.org/
- 30. Processing Linked Data with Catmandu | http://librecat.org STAGE 3b: RDF to Linked Data JSON/YAML LINKED DATA <http://example.org/000000008> <http://purl.org/dc/elements/1.1/creator> “Tolstoj, Lev Nikolaevič”; <http://purl.org/dc/elements/1.1/title> “War and peace” ; <http://purl.org/dc/elements/1.1/date> “1952” ; <http://purl.org/dc/elements/1.1/subject> “Napoleonic Wars” ; a <http://www.europeana.eu/schemas/edm/Book> . <http://example.org/000000008> <http://purl.org/dc/elements/1.1/creator> <http://viaf.org/viaf/96987389>; <http://purl.org/dc/elements/1.1/title> “War and peace” ; <http://purl.org/dc/elements/1.1/date> “1952” ; <http://purl.org/dc/elements/1.1/subject> <http://dbpedia.org/page/Napoleonic_ Wars> ; a <http://www.europeana.eu/schemas/edm/Book> . FIX
- 31. Processing Linked Data with Catmandu | http://librecat.org THANK YOU Nicolas Steenlant Nicolas Franck Snorri Briem Jörgen Eriksson Maria Hedberg Dave Sheroman Friedrich Summann Najko Jahn Vitali Peil Petra Kohorst Christian Pietsch Mathias Lösch Johan RolschewskiJakob Voß UGENT LUND BIELEFELD GBV STAATSBIBLIOTHEK ZU BERLIN Wouter Willaert INUITS