1.1. Triggers

In order to write new or change existing handlers, let's look how handlers are triggered:

1.1.1. System Triggers

Following triggers are predefined in MetaFS:

◹

Non-Item Triggers

• init: initialization of MetaFS volume at the moment of mounting
• fsck: mfsck or metabusy fsck called
• fsinfo: mfsinfo or metabusy fsinfo called
• format: metabusy format called

Item Related Triggers

• open / close: opening an item for reading, closing it again (no change of content)
• create & change: create or change an item (altering content)
• update: end of change or update
• delete / undelete / purge: delete, undelete, or purge entirely
• meta: metadata was altered
• mkdir: creating new directory
• import / export from or to a volume (e.g. via marc archiving)

% rm AA.txt
% mrm AA.txt

% mrm -u AA.txt

% mrm -p

1.2. Writing Handlers

Writing custom handlers is one of the features of MetaFS. There is a template handlers/template which helps you to start to code your own handler:

#!/usr/bin/perl

# -- Template
#    written by YOUR NAME
#
# Description:
#    DESCRIBE WHAT IT DOES
#
# History:
# YYYY/MM/DD: 0.0.1: first version, minimal functionality

my $NAME = 'template';       # -- CHANGE IT

if($0=~/\/$NAME$/) {         # -- when testing stand-alone
   while($#ARGV>=0) {
      my($k,$v) = (shift(@ARGV), shift(@ARGV));
      $arg->{$k} = $v;
   }
}

main($arg);

# Global variables:
#    $conf: contains structured configuration variables (from conf/metafs.conf and all other conf/*), e.g. $conf->{myhandler}->{...} <= conf/myhandler.conf (JSON)
#      $db: current MongoDB connection of the default volume
#      $fs: current filesystem information
        
sub main {
   my($arg) = @_;
   if($conf->{verbose}) {
      print "$NAME started:\n";  
      foreach (sort keys %$arg) {
         print "\t$_: $arg->{$_}\n";
      }
   }
   my $uid = $arg->{uid};
   if($arg->{type}eq'init') {
   } elsif($arg->{type}eq'fsck') {
      return 0;         # -- return amount of errors
   } elsif($arg->{type}eq'fsinfo') {
      my $i; 
      # $i->{info} = "SOMETHING INTERESTING"; 
      return $i;
   } elsif($arg->{type}eq'create') {
   } elsif($arg->{type}eq'change') {
   } elsif($arg->{type}eq'update') {
   } elsif($arg->{type}eq'delete') {
   } elsif($arg->{type}eq'undelete') {
   } elsif($arg->{type}eq'purge') {       # -- clean up all files you created in this handler
   } elsif($arg->{type}eq'open') {
   } elsif($arg->{type}eq'close') {
   } elsif($arg->{type}eq'mkdir') {
   } elsif($arg->{type}eq'meta') {
   }
}

Following global variables are available:

• $conf contains all configurations, e.g. also your own $conf->{myhandler}-> etc. if you define it either in conf/metafs.conf or conf/myhandler.conf.
• $db contains the current MongoDB / TokuMX db connection, volume specific[1]
• $fs contains the filesystem specific information (mfsinfo)

Best study the existing handlers as stored in /var/lib/metafs/handlers/, given MetaFS is installed at the default path.

1.3. Connect Handlers

...
   "image": {
      "triggers": {
         # -- items/file and often mime related triggers
         "update": { "mime": [ "image/*", "application/pdf" ], "priority": 6, "nice": 10 },
         "delete": { "mime": [ "image/*", "application/pdf" ], "priority": 6, "nice": 10 }
      }
   },
   "video": {
      "triggers": {
         "update": { "mime": [ "application/ogg", "video/*" ], "priority": 6, "nice": 10 },
         "delete": { "mime": [ "application/ogg", "video/*" ], "priority": 6, "nice": 10 }
      }
   },
   "audio": {
      "triggers": {
         "update": { "mime": "audio/*", "priority": 6 }
      }
   },
...

Hint: Whenever you edit metafs.conf, it will be considered at the next trigger event.

1.4. Handler Execution

◹

• sync: synchronous, be aware: a handler might block the fs operation
• async: asynchronous, a separate process (fork) is started, be aware: it might create a lot of processes and cause overhead
• queue: this is default, the execution will be queued and processed asynchronous but just in one process

...
   "hash": {
      "triggers": {
         "update": { "exec": "sync" },
         "fsck": { }
      }
   },
   "something": {
      "triggers": {
         "update: { "exec": "async" }
      }
   }, 
   "image": {
      "triggers": {
         "update": { 
            "mime": [ "image/*", "application/pdf" ], 
            "priority": 2, 
            "exec": "queue" 
         },
         "delete": { 
            "mime": [ "image/*", "application/pdf" ], 
            "priority": 2,
            "exec": "queue" 
         }
      }
   },
...

By default as mentioned, all handlers are queue executed (there is no requirement for "exec": "queue"), this is the most light approach.

1.5. Dedicated Handler Configuration

In conf/metafs.conf you can define your own trigger (e.g. "myhandler": { }), yet handler specific settings you can place to conf/myhandler.conf which are mapped to $conf->{myhandler}->{etc}.

Example conf/myhandler.conf:

{
   "type": "mytype"
   "list": [ 1, 2, 3 ],
   "object": {
      "sub1": "me"
   },
   "types": {
      "myhandler": {
         "name": "string",
         "mytime": "date",
         "duration": "time",
         "distance": "number"
      }
   },
   "units": {
      "myhandler": {
         "distance": "m"
      }
   }
}

and so $conf->{myhandler}->{type} is set, @{$conf->{myhandler}->{list}} and $conf->{myhandler}->{object}->{sub1}, and can be referenced in your handler code.

You optionally can define types and units which are specially treated as they are merged to $conf->{types} and $conf->{units}:

• types defines the type of data (string,number,date,time, etc), and are important in conjunction with mfind and mmeta
• units defines the units of data, e.g. 'm' for meter, or 'deg' for degrees

• myhandler.name (type: string as defined in conf/myhandler.conf)
• myhandler.mytime (type: date)
• myhandler.duration (type: time)

% mmeta --myhandler.mytime=2015/06 --myhandler.duration=109 BB.txt
myhandler.mytime = 2015/06/15 12:00:00.000 (5months 4days 13hrs 1m 54secs ahead)
myhandler.duration = 1min 39secs

% mls -l BB.txt
   ...
   myhandler: {
      mytime: 2015/06/15 12:00:00.000 (5months 4days 13hrs 1min 38secs ahead)
      duration: 1min 39secs
      distance: 1,500 m
   }
   ...

1.6. Trigger: fsinfo

1.7. Trigger: fsck

If a task takes long time, you may use a function of dispProgress($title,$count,$max), where as $count: 0..$max, and $max - please make sure $count does not go over $max, but reaches $max.

my $max = 1024+48;               # -- calculate the max

for(my $i=0; $i<=$max; $i++) {
   # doing something here
   dispProgress("\tperforming something",$i,$max);
}
print "\n";

which gives then the user some feedback, see handlers/hash and the section of fsck in there.

1.8. Trigger: format

2.1. MetaFS REST

MetaFS REST provides a layer of functionality to the MetaFS core, it does not access the FUSE mounted disk, but the volume direct (Item.pm, Trigger.pm, NoSQL.pm etc).

2.2. Firefox OS App & Chrome App

(coming soon)

3.1. Query

{
   "name": "test"
}

{
   "$and": [
      { 
         "name": { 
            "$regex": "test",
            "$options": "i"
         }
      },
      {
         "image.width": {
            "$gt": 1000
         }
      }
   ]
}

3.2. Aggregation

[
   {
      '$match': {                         // stage 1: perform query (match)
         'parent': '1234'
      }
   },
   { 
      '$project': {                       // stage 2: project (not really required in this example)
         'tags': 1                        //          only carry tags further
      },
   },
   {
      '$unwind': '$tags'                  // stage 3: explode or unwind array tags
   },
   {
      '$group': {                         // stage 4: group tags, and count occurance
         '_id': '$tags',
         'count': { 
            '$sum': 1 
         }
      }
   }
   {
      '$sort': {                          // stage 5: nicely sort in descending order
         'count': -1
      }
   }
]

Results

[
   { 
      '_id': 'sun',
      'count': 100
   },
   { 
      '_id': 'moon',
      'count': 63
   },
   { 
      '_id': 'mars',
      'count': 14
   },
]
   

4.1. Operations

Every query must be an object ({ .. }).

use MetaFS::Query;

my $db = { db => "metafs_alpha", col => "items" };
#      or
my @db = (
   { name => "something", type => "A" },
   { name => "something else", type => "B" },
);
my $r = MetaFS::Query::query($db || \@db,{
   name => 'something'
});
# -- $r is reference to an array of results

<script src="MetaFS/Query.js"></script>

<script>
var db = { db: "metafs_alpha", col: "items" };
// or
var db = [ 
   { name: "something", type: "A" }, 
   { name: "something else", type: "B" } 
];
var r = MetaFS.Query.query(db,{
   name: "something"
});
// -- r is an array of results
</script>

5.1. Stage Operations

Function	Aggregate.pm	Aggregate.js	Operand
$project	✔	✔	object (e.g. { '$project': { ... } })
$match	✔	✔	Query object (e.g. { '$match': { ... } })
$redact
$limit	✔	✔	literal (e.g. { '$limit': 100 })
$skip	✔	✔	literal (e.g. { '$skip': 10 })
$unwind	✔	✔	literal (e.g. { '$unwind': 'tags' })
$group	✔	✔	object (e.g. { '$group': { ... } })
$sample	✔	✔	object (e.g. { '$sample': { size: 10 } })
$sort	✔	✔	object (e.g. { '$sort': { name: 1 } })
$geoNear
$lookup
$out
$indexStats

use MetaFS::Query;

my $db = { db => "metafs_alpha", col => "items" };
#      or
my @db = (
   { name => "something", type => "A" },
   { name => "something else", type => "B" },
);
my $r = MetaFS::Aggregation::aggregation($db || \@db,[
   { '$project' => { name => 1, type => 1 } }
]);
# -- $r is reference to an array of results

<script src="MetaFS/Aggregation.js"></script>

<script>
var db = { db: "metafs_alpha", col: "items" };
// or
var db = [ 
   { name: "something", type: "A" }, 
   { name: "something else", type: "B" } 
];
var r = MetaFS.Aggregation.aggregation(db,[
   { '$project': { name: 1, type: 1 } }
]);
// -- r is an array of results
</script>

{
   '$project': {
      a: 1,             # -- regard key "a"
      b: { 
         '$cmp': [
            '$b', 10
         ]
      },
      c: {
         '$lt': [
            '$z.date', 100
         ]
      }
   }
} 

{
   '$match': {
      'a': {
         '$lt': 100,
      },
      'b': {
         '$gt': 10,
      }
   }
}

{
   '$group': {
      '_id': '$name',            // value is an expression therefore '$' ahead is required of keys
      'some': '$time',           // dito
      'some2': {
         '$lt': [                // returns true(1) or false(0)
            '$time', 200
         ]
      },
      ...
   }
}

5.2. Expression

{ '$and': [ { '$gt': [ '$a', 50 ] }, { '$lt': [ '$a', 100 ] } ] }
{ '$or': [ { '$gt': [ '$a', 100 }, { '$lt': [ '$a', 50 } ] }
{ '$not': [ { '$lt': [ '$a', 100 ] } ] }

{ result: { '$cmp': [ '$name', "test" ] } }
{ result: { '$eq': [ '$name', "test" ] } }
{ result: { '$lt': [ '$line', 50 ] } }