modelverse/doc/internal.rst

Internal workings
=================

For more detailed information on the Modelverse specification, which this project is implementing, we refer to the `Modelverse Specification <http://msdl.cs.mcgill.ca/people/yentl/files/Modelverse.pdf>`_.

Information on the implementation can be found below.

Modelverse State
----------------

The Modelverse State is basically just an implementation of a graph library.
As we have a particular kind of graph, this implementation is mostly done by hand at the moment.
The notable exception to this is the RDF backend, proving that other implementations can also be used.

The basic implementation just stores everything as dictionaries.
All operations are then defined by doing operations on these dictionaries.
The most interesting operations here are dictionary operations, which need to traverse these dictionaries in complex ways.
To overcome performance problems for these operations, all results are cached (and validated afterwards).

RDF backend
^^^^^^^^^^^

The RDF backend requires the *rdflib* module in Python.
The Modelverse graph is then stored in RDF representation and all operations on it are done using SPARQL queries.
Due to this level of indirection, performance is extremely slow.
To increase performance, we would likely have to make more *composite* operations, or even group different requests together internally.

Status codes
^^^^^^^^^^^^

The MvS returns, apart from its actual return value, a status code for the request.
This value is not used by the MvK at all, since sometimes a request is expected to give an error (*e.g.*, checking whether an element is present).
When debugging the MvS, however, these status codes can come in handy.

Modelverse Kernel
-----------------

Precompiled functions
^^^^^^^^^^^^^^^^^^^^^

Modelverse Interface
--------------------

Semantics visitor
^^^^^^^^^^^^^^^^^

Constructors visitor
^^^^^^^^^^^^^^^^^^^^

Primitives visitor
^^^^^^^^^^^^^^^^^^

Bootstrap visitor
^^^^^^^^^^^^^^^^^

Model visitor
^^^^^^^^^^^^^

Bootstrapping
-------------

To bootstrap, you just have to run the file *bootstrap.py*.
Here, we explain what this file actually does...

The bootstrap script primarily creates the initial graph manually.
This manual creation is done by writing data to a file, which is later read by the MvS.
The initial graph consists of several parts:

* The Modelverse Root node;
* A global definition of all primitives;
* The stack frame of the initial user *user_manager*, which manages all other users;
* The code for the *user_manager* user;
* The code for all new users, as assigned by the *user_manager*;
* Bindings in the compilation manager for bootstrap files.

These are all fairly simple in themselves.
For some parts, such as the code, the HUTN compiler is invoked on a temporary piece of code.
All bootstrap files are also compiled and made available to the compilation manager with their MD5 hash.

How to add a primitive
----------------------

Probably the most important reason why you would want to know about the Modelverse internals, is if you want to add a Modelverse primitive.
Primitives are functions implemented in the MvK core, and thus become hardcoded.

.. warning::
    While these functions are hardcoded, their implementation needs to follow strict rules, such that their semantics is identical in all possible programming languages.
    For example, a primitive *getTime()* cannot simply be implemented as *time.time()* in Python, as this gives different results on Linux and Windows.

To add a primitive, follow these steps:

1. Add the code of the primitive to the MvK implementation by adding it in the file *primitives.py*.

    a. Name the function identically to how the primitive will be invoked later on.
    b. Take a set of parameters. Parameters for primitives are always positional and start from *a* onwards.
    c. The final parameter should be a catch-all element, as it is possible that a high number of additional information is passed. In Python, this is done with a parameter prepended with \*\*.
    d. When asking the MvS for information, use yields, just like the implementation of transformation rules in the MvK.
    e. Instead of return, use a *raise* with the exception *PrimitiveFinished*. This exception takes one argument: the returnvalue.

2. Add the signature to *bootstrap.py* in the topmost dictionary *primitives*. The key is the name of the function, and the value is a list starting with the return type, followed by all parameter types (as string).
3. Add the declaration to *includes/primitives.alh* to make them available everywhere.
4. Add the definition to *primitives.alc* and assign the Modelverse reference *?primitives/function_name*.
5. Recreate the bootstrap file by running the script *generate_bootstrap.py*.
6. Restart the Modelverse to reload the bootstrap file.
7. From now on, all files including *primitives.alh* have access to the defined function.

Adding a precompiled function
-----------------------------

Adding a precompiled function is way easier: only step 1 of the addition of a primitive should be done, but in the file *compiled.py* instead of *primitives.py*.
All other steps are done automatically since there is an action language implementation present.
The MvK will then automatically pick the precompiled function or the explicitly modelled operation, depending on preferences.
A restart of the MvK is needed for Python to pick up the new functions.