Edurep: Metadata verwerking/en

Edurep harvest diverse repositories via het OAI-PMH protocol. Vervolgens kunnen eindgebruikers via de zoekindex zoeken in het materiaal dat Edurep op deze wijze heeft verzameld. Om de kwaliteit en uniformiteit te garanderen worden er allerlei validaties, bewerkingen en translaties uitgevoerd op de metadata. In het schema hieronder staat een versimpelde weergave van de processen die plaatsvinden binnen Edurep.

records komen binnen via een specifieke interface
naast een initiële validatie (xml/toegang) vindt er inhoudelijke validatie plaats
er worden verschillende opvraagbare representaties gemaakt
de Schema.org variant wordt gebruikt in de aggregator en zoekindex

OAI-PMH Harvester

The harvester retrieves the records from the provider repository and thus acts as a gateway to Edurep. Records that do not validate are rejected and the status can be viewed on the harvester status page.

Harvester Status page

The meaning of each column in the status overview is explained below:

Repository: The identifier of the repository (or a link to the repository data, only available to the Edurep administrator).
Last successful harvest: Timestamp of the last time the harvester made an "error free" harvest.
Total records: Total number of records retrieved by the harvester. (This number is not necessarily the same as the number in Edurep. For example, our deadlink checker may clean up records with dead links.)
Harvested/Uploaded/Deleted: The ratio between the number of new, changed, and deleted records from the last harvest visit.
#Validation Errors: The number of validation errors. The link points to a list of all errors at the bottom of the status page.
#Errors: The number of errors. The link points to a list of all errors at the bottom of the status page.
RSS: Provides access to an RSS feed for a specific connected repository.

Validation errors

When a record is successfully harvested, it can still be rejected for inclusion in the search engine based on a validation error. Currently, validation is only performed against the LOM XML schema (both IEEE and IMS bindings can be offered).

Unlike an error, harvesting continues after a validation error, to a certain extent. In principle, no further harvesting takes place after 100 validation errors. Then an error occurs for the harvester.

Each validation error can be viewed individually from the error overview. The first line of such an error message contains the generated summary of the error. In a number of cases this message will be sufficient to detect the error in question. The xml shown here is in the IEEE LOM binding, but is identical to the entered record.

Sometimes this line says "Line 105: Unable to transform record". The error itself can then be found in the xml file and is marked with explicit Edurep error xml elements:

105<edurep:error xmlns:edurep="http://meresco.org/namespace/users/kennisnet/edurep">
106 <lom:keyword xmlns:lom="http://www.imsglobal.org/xsd/imsmd_v1p2">
107  <lom:langstring xml:lang="nl"/>
108 </lom:keyword>
109</edurep:error>

If a validation error is repaired by the provider and is offered with a new updated timestamp according to OAI, the record will be harvested regularly again and the validation error will disappear.

Deadlink Checker

The Deadlink Checker checks whether a record contains a valid and working URL in the URL field. A record can be given one of the following statuses:

OK: The result of the URL is a 2.x.x or 3.x.x HTTP status code
NTL: The record does not contain a URL (No Technical Location)
FAILED: The URL is not valid or leads to a timeout or a 4.x.x or 5.x.x HTTP status code

Records with status FAILED are then not shown in Edurep search results.

About once a week, all records with the status "OK" are checked. The records with the status "FAILED" are checked every day.

It may still happen that some learning materials are not classified as a dead link because the URL ends up on a so-called landing page that reports as an OK instead of FAILED. These situations are difficult to recognize.

Deadlink Checker Status Page

There is a status page where you can request an overview of the dead links per repository.

Production: https://wszoeken.edurep.kennisnet.nl/status
Staging: https://staging.edurep.kennisnet.nl/status

A short description for each column in the status overview:

Repository: The repository identifier as it is known in Edurep.
Vindbare records: The number of harvested records minus the records with dead links.
Deadlink records: The number of records with dead links.
Totaal: The number of harvested records.

When you click on one of the repositories, a special page opens where you can subscribe to the RSS feeds of the harvester and deadlink checker, and where you can download an overview of all dead link record identifiers.

Edits and Validation

There are various editing and validation processes in Edurep to increase the quality of records.

missing values

In order to improve the quality of metadata, Edurep fills in a certain number of metadata fields if they have not been filled in by the provider. This concerns:

costs: Edurep enters costs=yes if no costs have been entered.
publisher: Edurep fills in the repository_id as publisher when the provider does not provide a publisher.

vocabulary values

Various vocabulary fields are validated in Edurep.

legacy:
The legacy solution only works with XSLTs . Here, old values are partly mapped to new values, incorrect values are removed or adjusted. This fix will be phased out after the 2021-11 release.

new:
The new solution makes it easier for us to apply more improvement options, but it is also easier to find out more explicitly what has not been validated.

classificaties

All NL LOM classification fields are converted to schema.org in a certain way. The goal is to use only curriculum-defined terms in the schema.org for subjects, educational levels and competences. Generally speaking, we apply a series of rules in the following order:

the OBK identifier (http://purl.edustandaard.nl/begrippenkader/*) for purpose discipline, educational level or comptency will be placed in respectively in schema:educationalAlignment, schema:educationalLevel and schema:teaches.
for specific old VDEX vocabulary values, a translation is made from old value to OBK identifier
special rules are made for other classification purposes, for example mapping for access rights.
anything that does not match is stored as a keyword, while retaining vocabulary information.

A complete overview of the conversion can be found on the 2021 migration page.

OBK

Filling in a label for a classification identifier is not mandatory, but it is convenient for search portals to display meaningful names in search results without setting up their own lookup service. Edurep therefore always fills in the labels for each valid purl.edustandard.nl/begrippenkader classification identifier of learning level, subject or goal.

Any existing label will be overwritten.
If a taxon does not contain an ID, but only an entry, the taxon is removed from the record because the validity of the entry cannot be determined.

Example input:

<taxonpath>
  <source>
    <langstring xml:lang="x-none">http://purl.edustandaard.nl/begrippenkader</langstring>
  </source>
  <taxon>
    <!-- OBK-id for Secondary Education -->
    <id>2a1401e9-c223-493b-9b86-78f6993b1a8d</id>
  </taxon>
  <taxon>
    <id>512e4729-03a4-43a2-95ba-758071d1b725</id>
    <entry>
        <langstring xml:lang=”nl”>PO</langstring>
    </entry>
  </taxon>
</taxonpath>

Result:

<taxonpath>
  <source>
    <langstring xml:lang="x-none">http://purl.edustandaard.nl/begrippenkader</langstring>
  </source>
  <taxon>
    <id>2a1401e9-c223-493b-9b86-78f6993b1a8d</id>
    <!-- The entry is automatically completed -->
    <entry>
        <langstring xml:lang=”nl”>Voortgezet Onderwijs</langstring>
    </entry>
  </taxon>
  <taxon>
    <id>512e4729-03a4-43a2-95ba-758071d1b725</id>
    <!-- The entry has been automatically overwritten -->
    <entry>
        <langstring xml:lang=”nl”>Primair Onderwijs</langstring>
    </entry>
  </taxon>
</taxonpath>

vCard

A VCARD in a centity is scanned by Edurep to enable searches by "author" or "publisher". Of all the possible variables that a vCard can contain, N, FN or ORG are used as possible values for the names of authors or publishers. The value from the first of those three variables that is entered will be used. vCard must be compatible with version 3.0. In that case, FN, N and VERSION are mandatory fields and therefore cannot be omitted.

Connections

During processing, connections are established between records so that, for example, they can be searched based on the average rating for all related records.

Review

A link between a review and a learning material record is made based on the hreview:info field in SMO and 1st of the object identifiers in a learning material record. After the 2021-11 release, a match will not only be made on the 1st record identifier, but a match can be made on all record identifiers. This also means that a review can be linked to a record that did not have a direct match with the original review. For example, a review that points to id:1 via id:2 can also be linked to record B:

record A
- id:1
- id:2
record B
- id:2
- id:3

Edurep:Metadata verwerking/en