Skip to end of metadata
Go to start of metadata

Current status of Stability Service

At the begining of this sprint (04/09/2012) the implementation of the Stability Service has not been finished. The stability service is using the cheklist evaluation in order to evaluate independent ROs that in the stability context simulate snapshots.

The Stability Service has an API (link here) that can be tested and also a webpage with a very simple interface (link here). But the results do not show a real stability trace because of the formentioned reasons. It is faking snapshots of an RO.

Next steps to improve it

To keep improving the stability service we need to make it work together with a "real" roevo trace (it can be handcrafted). With this information we will be able to get different snapshots of a RO, evaluate them and then create the stability measure.

Apart from the snapshots we will need to have information of the modifications in between them. In that way we will be able to provide additional information and explanations of the reasons that may have taken part in the variation of the stability.

In that way we need to create a scenario that will become a roevo trace.

Scenrario for the roevo trace

?Draft

We need to have at least three or four snapshots of the same RO. The snapshots should have differences in between them in order to get different results after a checklist evaluation.

We can use an existing RO. For instance, the one that was created and used before in the showcase of the checklist evaluation (purl1 and purl2).
The idea is that those snapshot are going to be evaluated by the checklists for a specific purpose. I think that a good choice would be "Repeatability", which tests the existence of inputs and outputs in an RO (and it's reflected in the results).

This information will let us compare, for instance if Q(S1)>Q(S2). If the quality decreases the user will be interested about which factors may have influenced on the resolts, which are the modifications. That is the reason why we would like to have a trace of the modifications (ROEVO info about changes). In that way, we will be able to provide information to the user that could explain what has happened with the RO.

Having this information allows us to create different queries that could show, for instance, if there's a clumsy user involved in the project or if there's an action that uses to "break" the RO.

Example:
The first snapshot doesn't have outputs.
Mr. Clumsy doesn't know very well what happens with the RO. He modifies the workflow and deletes the inputs. Second snapshot.
Mr. Curator sees that the RO needs inputs and outputs so he adds them. And he also fixes the workflow. Third snapshot.

Proposed ROs for snapshots

Three different versions of the wf74 about protein discovery that will become into snapshots and will be evaluated against the "repeatability" purpose of the checklist evaluation service.

ROEVO trace

There have been created (by hand) the required roevo annotations for the scenario. In the first version of the trace we have an rdf in each one of the Snapshots and also in the Live RO.

In that way we can see which are the snapshots starting from the Live RO (http://purl.org/wf4ever/roevo#hasSnapshot). Once we get the snapshots we can evaluate them (using the checklist evaluation) and look for the changes that have been done on them (http://purl.org/wf4ever/roevo#hasChange).

ROEVO annotations:

Live RO: http://purl.org/net/wf4ever/ro/wf74/.ro/wf74-live-evoAnnotation001.rdf

Explanation of the annotations

The roevo trace has been based the proposed scenario, so it is very similar. All the ROs have had their manifest modified to aggregate the roevo annotations.

First Snapshot:

  • Changes: None 
  • Evaluation: Lack of outputs (for the repeatability purpose)
  • Creator: Mr. Clumsy

Second Snapshot:

  • Changes: Two files (inputs) have been deleted: Query.text and maxHits_parameter.text.
    The wfdesc has been modified.
  • Evaluation: Lack of outputs and inputs (for the repeatability purpose)
  • Creator: Mr. Clumsy

Third Snapshot:

  • Changes: The two inputs have been added again.  
    392 outputs have been added.
    The wfdesc has been modified.
  • Evaluation: Everything OK (for the repeatability purpose)
  • Creator: Mr. Curator

Live RO: Reflects the current state of the RO. All the snapshots were done and represent this RO at a specific point of its evolution. The roevo annotations of the Live RO have links to the snapshots.

Querying the trace

The ROs  with the roevo annotations in RODL

Snapshot1: http://sandbox.wf4ever-project.org/rosrs5/ROs/wf74-repeat-fail/

Snapshot2: http://sandbox.wf4ever-project.org/rosrs5/ROs/wf74-repeat-fail2/

Snapshot3: http://sandbox.wf4ever-project.org/rosrs5/ROs/wf74-repeat-ok/

Live RO: http://sandbox.wf4ever-project.org/rosrs5/ROs/wf74/

Useful queries

Once we have the roevo traces available in RODL we are able to do some queries in order to extract the information needed (roevo provides the annotations and now the way to use this information is up to us).

From the point of view of Integrity and Authenticity and the Stability service we are interested in using this information to detect errors and provide explanations to the user.

The best way to see what we are trying to do is following a simple story.

Story

For the stability measure we want to evaluate different states of an RO over time. The checklist service provides an evaluation of a single RO for a specific purpose, but we need to decide which RO's are we going to evaluate. So, first of all we want to retrieve all the snapshots that have been done for a RO.

Query:

Specific example (snapshots of wf74):

Now that we have the list of snapshots we can evaluate them using the checklist service. The results provided by the executions of the checklist service are used to calculate the stability (this value and analysis is provided by the stability service).

Now that we have the results, the user may be interested on why the quality of his RO has increased or decreased. Let's say that the user is interested on a specific snapshot with low quality (decreasing in the stability of his RO). When has it happened?

Query:

Specific example (snapshot wf74-repeat-fail2):

Which resources have changed form a snapshot to another?

Query:

Specific example (modifications in the wf74-repeat-fail2 snapshot):

Who is the responsible of that?

Query:

Specific example (the creator of the wf74-repeat-fail2 snapshot):

Which is the previous snapshot? (Might be useful to compare and fix the problems in our Live RO).

Query:

Specific example (previous snapshot of wf74-repeat-fail2:

All this questions can be answered now, and those explanations may be useful for the user

  • No labels