Skip to content

Cookbook

Jump to bottom Edit New page
Vitali Peil edited this page Aug 22, 2022 · 31 revisions

Install Catmandu OAI processing on your computer

Make sure you have cpanm (hint: $ cpan App::cpanminus) installed.

$ cpanm Catmandu::OAI

Read Dublin Core records from an OAI repository from the command line

  1. Goto: http://www.opendoar.org/
  2. Find a repository of choice
  3. Read the base URL of the repository from the 'OAI-PMH'
  4. Execute in a terminal the catmandu import command with the URL found in the OAI-PPMH field

E.g.

$ catmandu convert OAI --url https://biblio.ugent.be/oai

Read Dublin Core records from an OAI repository in your Perl code

use Catmandu; Catmandu->importer('OAI',url => 'https://biblio.ugent.be/oai')->each(sub { my $record = shift; print "$record\n"; });

Convert Dublin Core records from an OAI repository into YAML from the command line

$ catmandu convert OAI --url https://biblio.ugent.be/oai to YAML

Convert Dublin Core records from an OAI repository into YAML in your Perl code

use Catmandu -all; my $importer = importer('OAI',url => 'https://biblio.ugent.be/oai'); my $exporter = exporter('YAML'); $exporter->add_many($importer); $exporter->commit;

Extract all identifiers from an OAI repository from the command line

$ catmandu convert OAI --url https://biblio.ugent.be/oai --fix 'retain("_id")'

or if you like an CSV file

$ catmandu convert OAI --url https://biblio.ugent.be/oai to CSV --fix 'retain("_id")'

Extract all identifiers from an OAI repository into CSV in your Perl code

use Catmandu; my $importer = Catmandu->importer('OAI',url => 'https://biblio.ugent.be/oai'); my $fixer = Catmandu->fixer('retain("_id")'); my $exporter = Catmandu->exporter('CSV'); $exporter->add_many( $fixer->fix($importer) ); $exporter->commit;

Show the speed of importing records from the command line

Hint: use the -v option

$ catmandu convert -v OAI --url https://biblio.ugent.be/oai to CSV --fix 'retain("_id")' > /dev/null

Here we send the output to the /dev/null to show the verbose messages.

Show the speed of importing records from your Perl program

use Catmandu; my $importer = Catmandu->importer('OAI',url => 'https://biblio.ugent.be/oai'); my $fixer = Catmandu->fixer('retain("_id")'); my $exporter = Catmandu->exporter('CSV'); $exporter->add_many( $fixer->fix($importer->benchmark) ); $exporter->commit;

See some debug messages

Make sure you have Log::Log4perl installed (hint: $ cpan Log::Any::Adapter::Log4perl).

In your main program do:

use Catmandu; use Log::Any::Adapter; use Log::Log4perl; Log::Any::Adapter->set('Log4perl'); Log::Log4perl::init('./log4perl.conf'); # The lines above should be enough to activate logging for Catmandu. # Include the lines below to activate logging for your main program. my $logger = Log::Log4perl->get_logger('myprog'); $logger->info("Starting main program"); ...your code...

with log4perl.conf like:

# Send a copy of all logging messages to STDERR log4perl.rootLogger=DEBUG,STDERR # Logging specific for your main program log4perl.category.myprog=INFO,STDERR # Logging specific for on part of Catmandu log4perl.category.Catmandu::Fix=DEBUG,STDERR # Where to send the STDERR output log4perl.appender.STDERR=Log::Log4perl::Appender::Screen log4perl.appender.STDERR.stderr=1 log4perl.appender.STDERR.utf8=1 log4perl.appender.STDERR.layout=PatternLayout log4perl.appender.STDERR.layout.ConversionPattern=%d [%P] - %p %l time=%r : %m%n

You will see now Catmandu log messages (e.g. for Fixes).

If you want to add logging functionality in your own Perl modules you have two options;

  1. Your package is a Catmandu::Importer or Catmandu::Exporter. In this case you are lucky because you have a logger as part of your instance:

    $self->log->debug('blablabla'); # where $self is an Importer,Fix or Exporter instance

  2. You need to create the logger yourself.

    package Foo::Bar;

    use Moo;

    with 'Catmandu::Logger';

    sub bar { my $self = shift; $self->log->debug('tadaah'); }

If you want to see the logging messages only of your package, then use a this type of line in your log4perl.conf:

log4perl.category.Foo::Bar=DEBUG,STDOUT

or if you want to see all the log messages for Foo packages:

log4perl.category.Foo=DEBUG,STDOUT

How to create a new Catmandu::Store

A Catmandu::Store is used to store items. Stores can have one or more compartments where to store the items. Each such compartment is a Catmandu::Bag. You can compare a Store with a database and a Bag with a table in a database. Like tables, Bags have names. When no name is provided for a Bag, then 'data' is used.

To implement a Catmandu store you need to create at least two packages:

  1. A 'Catmandu::Store', defining the general parameters, possible connection parameters and actions for the whole store.
  2. A 'Catmandu::Bag', which is used to list, add,fetch and delete items from a Bag.

As example, this is a skeleton for a 'Foo' Catmandu::Store which requires at least one 'foo' connection parameter:

package Catmandu::Store::Foo; use Moo; use Catmandu::Store::Foo::Bag; with 'Catmandu::Store'; has 'foo' => (is => 'ro' , required => 1); 1;

For this Catmandu::Store::Foo we can define a module 'Catmandu::Store::Foo::Bag' to implement the Bag functions. Notice how in the generator the bag can access the Catmandu::Store instance:

package Catmandu::Store::Foo::Bag; use Moo; with 'Catmandu::Bag'; sub generator { my $self = shift; sub { # This subroutine is used to loop over all items # in a store and should return a item HASH for # every call return { name => $self->name, foo => $self->store->foo }; }; } sub get { my ($self,$id) = @_; # return a item HASH given an $id return {}; } sub add { my ($self,$data) = @_; # add/update an item HASH to the bag and return the item with an _id field set return $data; } sub delete { my ($self,$id) = @_; # delete an item from the bag given an $id 1; } sub delete_all { my ($self) = @_; # delete all items $self->each(sub { $self->delete($_[0]->{_id}); }); } 1;

With this skeleton Store you have enough code to run basic tests. Save these package in a lib directory:

lib/Catmandu/Store/Foo.pm lib/Catmandu/Store/Foo/Bag.pm

and a catmandu command to test your implementation:

$ catmandu -I lib export Foo --foo bar

{"foo":"bar","name":"data"} {"foo":"bar","name":"data"} {"foo":"bar","name":"data"} . . .

Or create a test.pl script to access your new Store via Perl:

#!/usr/bin/env perl use lib qw(./lib); use Catmandu; my $store = Catmandu->store('Foo', foo => 'bar'); $store->add({ test => 123});

Clone this wiki locally