Some internal documentation on how to add primitives and precompiled

functions, as well as the use of bootstrapping.

doc/_build/html/_sources/internal.txt

@@ -1,2 +1,113 @@
 Internal workings
 =================
+
+For more detailed information on the Modelverse specification, which this project is implementing, we refer to the `Modelverse Specification <http://msdl.cs.mcgill.ca/people/yentl/files/Modelverse.pdf>`_.
+
+Information on the implementation can be found below.
+
+Modelverse State
+----------------
+
+The Modelverse State is basically just an implementation of a graph library.
+As we have a particular kind of graph, this implementation is mostly done by hand at the moment.
+The notable exception to this is the RDF backend, proving that other implementations can also be used.
+
+The basic implementation just stores everything as dictionaries.
+All operations are then defined by doing operations on these dictionaries.
+The most interesting operations here are dictionary operations, which need to traverse these dictionaries in complex ways.
+To overcome performance problems for these operations, all results are cached (and validated afterwards).
+
+RDF backend
+^^^^^^^^^^^
+
+The RDF backend requires the *rdflib* module in Python.
+The Modelverse graph is then stored in RDF representation and all operations on it are done using SPARQL queries.
+Due to this level of indirection, performance is extremely slow.
+To increase performance, we would likely have to make more *composite* operations, or even group different requests together internally.
+
+Status codes
+^^^^^^^^^^^^
+
+The MvS returns, apart from its actual return value, a status code for the request.
+This value is not used by the MvK at all, since sometimes a request is expected to give an error (*e.g.*, checking whether an element is present).
+When debugging the MvS, however, these status codes can come in handy.
+
+Modelverse Kernel
+-----------------
+
+Precompiled functions
+^^^^^^^^^^^^^^^^^^^^^
+
+Modelverse Interface
+--------------------
+
+Semantics visitor
+^^^^^^^^^^^^^^^^^
+
+Constructors visitor
+^^^^^^^^^^^^^^^^^^^^
+
+Primitives visitor
+^^^^^^^^^^^^^^^^^^
+
+Bootstrap visitor
+^^^^^^^^^^^^^^^^^
+
+Model visitor
+^^^^^^^^^^^^^
+
+Bootstrapping
+-------------
+
+To bootstrap, you just have to run the file *bootstrap.py*.
+Here, we explain what this file actually does...
+
+The bootstrap script primarily creates the initial graph manually.
+This manual creation is done by writing data to a file, which is later read by the MvS.
+The initial graph consists of several parts:
+
+* The Modelverse Root node;
+* A global definition of all primitives;
+* The stack frame of the initial user *user_manager*, which manages all other users;
+* The code for the *user_manager* user;
+* The code for all new users, as assigned by the *user_manager*;
+* Bindings in the compilation manager for bootstrap files.
+
+These are all fairly simple in themselves.
+For some parts, such as the code, the HUTN compiler is invoked on a temporary piece of code.
+All bootstrap files are also compiled and made available to the compilation manager with their MD5 hash.
+
+How to add a primitive
+----------------------
+
+Probably the most important reason why you would want to know about the Modelverse internals, is if you want to add a Modelverse primitive.
+Primitives are functions implemented in the MvK core, and thus become hardcoded.
+
+.. warning::
+    While these functions are hardcoded, their implementation needs to follow strict rules, such that their semantics is identical in all possible programming languages.
+    For example, a primitive *getTime()* cannot simply be implemented as *time.time()* in Python, as this gives different results on Linux and Windows.
+
+To add a primitive, follow these steps:
+
+1. Add the code of the primitive to the MvK implementation by adding it in the file *primitives.py*.
+
+    a. Name the function identically to how the primitive will be invoked later on.
+    b. Take a set of parameters. Parameters for primitives are always positional and start from *a* onwards.
+    c. The final parameter should be a catch-all element, as it is possible that a high number of additional information is passed. In Python, this is done with a parameter prepended with \*\*.
+    d. When asking the MvS for information, use yields, just like the implementation of transformation rules in the MvK.
+    e. Instead of return, use a *raise* with the exception *PrimitiveFinished*. This exception takes one argument: the returnvalue.
+
+2. Add the signature to *bootstrap.py* in the topmost dictionary *primitives*. The key is the name of the function, and the value is a list starting with the return type, followed by all parameter types (as string).
+3. Add the declaration to *includes/primitives.alh* to make them available everywhere.
+4. Add the definition to *primitives.alc* and assign the Modelverse reference *?primitives/function_name*.
+5. Recreate the bootstrap file by running the script *generate_bootstrap.py*.
+6. Restart the Modelverse to reload the bootstrap file.
+7. From now on, all files including *primitives.alh* have access to the defined function.
+
+Adding a precompiled function
+-----------------------------
+
+Adding a precompiled function is way easier: only step 1 of the addition of a primitive should be done, but in the file *compiled.py* instead of *primitives.py*.
+All other steps are done automatically since there is an action language implementation present.
+The MvK will then automatically pick the precompiled function or the explicitly modelled operation, depending on preferences.
+A restart of the MvK is needed for Python to pick up the new functions.

doc/_build/html/index.html

@@ -102,7 +102,15 @@
 </ul>
 </li>
 <li class="toctree-l1"><a class="reference internal" href="problems.html">Common problems and solutions</a></li>
-<li class="toctree-l1"><a class="reference internal" href="internal.html">Internal workings</a></li>
+<li class="toctree-l1"><a class="reference internal" href="internal.html">Internal workings</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="internal.html#modelverse-state">Modelverse State</a></li>
+<li class="toctree-l2"><a class="reference internal" href="internal.html#modelverse-kernel">Modelverse Kernel</a></li>
+<li class="toctree-l2"><a class="reference internal" href="internal.html#modelverse-interface">Modelverse Interface</a></li>
+<li class="toctree-l2"><a class="reference internal" href="internal.html#bootstrapping">Bootstrapping</a></li>
+<li class="toctree-l2"><a class="reference internal" href="internal.html#how-to-add-a-primitive">How to add a primitive</a></li>
+<li class="toctree-l2"><a class="reference internal" href="internal.html#adding-a-precompiled-function">Adding a precompiled function</a></li>
+</ul>
+</li>
 </ul>
 </div>
 </div>

doc/_build/html/internal.html

@@ -47,6 +47,117 @@
             
   <div class="section" id="internal-workings">
 <h1>Internal workings<a class="headerlink" href="#internal-workings" title="Permalink to this headline">¶</a></h1>
+<p>For more detailed information on the Modelverse specification, which this project is implementing, we refer to the <a class="reference external" href="http://msdl.cs.mcgill.ca/people/yentl/files/Modelverse.pdf">Modelverse Specification</a>.</p>
+<p>Information on the implementation can be found below.</p>
+<div class="section" id="modelverse-state">
+<h2>Modelverse State<a class="headerlink" href="#modelverse-state" title="Permalink to this headline">¶</a></h2>
+<p>The Modelverse State is basically just an implementation of a graph library.
+As we have a particular kind of graph, this implementation is mostly done by hand at the moment.
+The notable exception to this is the RDF backend, proving that other implementations can also be used.</p>
+<p>The basic implementation just stores everything as dictionaries.
+All operations are then defined by doing operations on these dictionaries.
+The most interesting operations here are dictionary operations, which need to traverse these dictionaries in complex ways.
+To overcome performance problems for these operations, all results are cached (and validated afterwards).</p>
+<div class="section" id="rdf-backend">
+<h3>RDF backend<a class="headerlink" href="#rdf-backend" title="Permalink to this headline">¶</a></h3>
+<p>The RDF backend requires the <em>rdflib</em> module in Python.
+The Modelverse graph is then stored in RDF representation and all operations on it are done using SPARQL queries.
+Due to this level of indirection, performance is extremely slow.
+To increase performance, we would likely have to make more <em>composite</em> operations, or even group different requests together internally.</p>
+</div>
+<div class="section" id="status-codes">
+<h3>Status codes<a class="headerlink" href="#status-codes" title="Permalink to this headline">¶</a></h3>
+<p>The MvS returns, apart from its actual return value, a status code for the request.
+This value is not used by the MvK at all, since sometimes a request is expected to give an error (<em>e.g.</em>, checking whether an element is present).
+When debugging the MvS, however, these status codes can come in handy.</p>
+</div>
+</div>
+<div class="section" id="modelverse-kernel">
+<h2>Modelverse Kernel<a class="headerlink" href="#modelverse-kernel" title="Permalink to this headline">¶</a></h2>
+<div class="section" id="precompiled-functions">
+<h3>Precompiled functions<a class="headerlink" href="#precompiled-functions" title="Permalink to this headline">¶</a></h3>
+</div>
+</div>
+<div class="section" id="modelverse-interface">
+<h2>Modelverse Interface<a class="headerlink" href="#modelverse-interface" title="Permalink to this headline">¶</a></h2>
+<div class="section" id="semantics-visitor">
+<h3>Semantics visitor<a class="headerlink" href="#semantics-visitor" title="Permalink to this headline">¶</a></h3>
+</div>
+<div class="section" id="constructors-visitor">
+<h3>Constructors visitor<a class="headerlink" href="#constructors-visitor" title="Permalink to this headline">¶</a></h3>
+</div>
+<div class="section" id="primitives-visitor">
+<h3>Primitives visitor<a class="headerlink" href="#primitives-visitor" title="Permalink to this headline">¶</a></h3>
+</div>
+<div class="section" id="bootstrap-visitor">
+<h3>Bootstrap visitor<a class="headerlink" href="#bootstrap-visitor" title="Permalink to this headline">¶</a></h3>
+</div>
+<div class="section" id="model-visitor">
+<h3>Model visitor<a class="headerlink" href="#model-visitor" title="Permalink to this headline">¶</a></h3>
+</div>
+</div>
+<div class="section" id="bootstrapping">
+<h2>Bootstrapping<a class="headerlink" href="#bootstrapping" title="Permalink to this headline">¶</a></h2>
+<p>To bootstrap, you just have to run the file <em>bootstrap.py</em>.
+Here, we explain what this file actually does...</p>
+<p>The bootstrap script primarily creates the initial graph manually.
+This manual creation is done by writing data to a file, which is later read by the MvS.
+The initial graph consists of several parts:</p>
+<ul class="simple">
+<li>The Modelverse Root node;</li>
+<li>A global definition of all primitives;</li>
+<li>The stack frame of the initial user <em>user_manager</em>, which manages all other users;</li>
+<li>The code for the <em>user_manager</em> user;</li>
+<li>The code for all new users, as assigned by the <em>user_manager</em>;</li>
+<li>Bindings in the compilation manager for bootstrap files.</li>
+</ul>
+<p>These are all fairly simple in themselves.
+For some parts, such as the code, the HUTN compiler is invoked on a temporary piece of code.
+All bootstrap files are also compiled and made available to the compilation manager with their MD5 hash.</p>
+</div>
+<div class="section" id="how-to-add-a-primitive">
+<h2>How to add a primitive<a class="headerlink" href="#how-to-add-a-primitive" title="Permalink to this headline">¶</a></h2>
+<p>Probably the most important reason why you would want to know about the Modelverse internals, is if you want to add a Modelverse primitive.
+Primitives are functions implemented in the MvK core, and thus become hardcoded.</p>
+<div class="admonition warning">
+<p class="first admonition-title">Warning</p>
+<p class="last">While these functions are hardcoded, their implementation needs to follow strict rules, such that their semantics is identical in all possible programming languages.
+For example, a primitive <em>getTime()</em> cannot simply be implemented as <em>time.time()</em> in Python, as this gives different results on Linux and Windows.</p>
+</div>
+<p>To add a primitive, follow these steps:</p>
+<ol class="arabic">
+<li><p class="first">Add the code of the primitive to the MvK implementation by adding it in the file <em>primitives.py</em>.</p>
+<blockquote>
+<div><ol class="loweralpha simple">
+<li>Name the function identically to how the primitive will be invoked later on.</li>
+<li>Take a set of parameters. Parameters for primitives are always positional and start from <em>a</em> onwards.</li>
+<li>The final parameter should be a catch-all element, as it is possible that a high number of additional information is passed. In Python, this is done with a parameter prepended with **.</li>
+<li>When asking the MvS for information, use yields, just like the implementation of transformation rules in the MvK.</li>
+<li>Instead of return, use a <em>raise</em> with the exception <em>PrimitiveFinished</em>. This exception takes one argument: the returnvalue.</li>
+</ol>
+</div></blockquote>
+</li>
+<li><p class="first">Add the signature to <em>bootstrap.py</em> in the topmost dictionary <em>primitives</em>. The key is the name of the function, and the value is a list starting with the return type, followed by all parameter types (as string).</p>
+</li>
+<li><p class="first">Add the declaration to <em>includes/primitives.alh</em> to make them available everywhere.</p>
+</li>
+<li><p class="first">Add the definition to <em>primitives.alc</em> and assign the Modelverse reference <em>?primitives/function_name</em>.</p>
+</li>
+<li><p class="first">Recreate the bootstrap file by running the script <em>generate_bootstrap.py</em>.</p>
+</li>
+<li><p class="first">Restart the Modelverse to reload the bootstrap file.</p>
+</li>
+<li><p class="first">From now on, all files including <em>primitives.alh</em> have access to the defined function.</p>
+</li>
+</ol>
+</div>
+<div class="section" id="adding-a-precompiled-function">
+<h2>Adding a precompiled function<a class="headerlink" href="#adding-a-precompiled-function" title="Permalink to this headline">¶</a></h2>
+<p>Adding a precompiled function is way easier: only step 1 of the addition of a primitive should be done, but in the file <em>compiled.py</em> instead of <em>primitives.py</em>.
+All other steps are done automatically since there is an action language implementation present.
+The MvK will then automatically pick the precompiled function or the explicitly modelled operation, depending on preferences.
+A restart of the MvK is needed for Python to pick up the new functions.</p>
+</div>
 </div>
 
 
@@ -55,6 +166,33 @@
       </div>
       <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
         <div class="sphinxsidebarwrapper">
+  <h3><a href="index.html">Table Of Contents</a></h3>
+  <ul>
+<li><a class="reference internal" href="#">Internal workings</a><ul>
+<li><a class="reference internal" href="#modelverse-state">Modelverse State</a><ul>
+<li><a class="reference internal" href="#rdf-backend">RDF backend</a></li>
+<li><a class="reference internal" href="#status-codes">Status codes</a></li>
+</ul>
+</li>
+<li><a class="reference internal" href="#modelverse-kernel">Modelverse Kernel</a><ul>
+<li><a class="reference internal" href="#precompiled-functions">Precompiled functions</a></li>
+</ul>
+</li>
+<li><a class="reference internal" href="#modelverse-interface">Modelverse Interface</a><ul>
+<li><a class="reference internal" href="#semantics-visitor">Semantics visitor</a></li>
+<li><a class="reference internal" href="#constructors-visitor">Constructors visitor</a></li>
+<li><a class="reference internal" href="#primitives-visitor">Primitives visitor</a></li>
+<li><a class="reference internal" href="#bootstrap-visitor">Bootstrap visitor</a></li>
+<li><a class="reference internal" href="#model-visitor">Model visitor</a></li>
+</ul>
+</li>
+<li><a class="reference internal" href="#bootstrapping">Bootstrapping</a></li>
+<li><a class="reference internal" href="#how-to-add-a-primitive">How to add a primitive</a></li>
+<li><a class="reference internal" href="#adding-a-precompiled-function">Adding a precompiled function</a></li>
+</ul>
+</li>
+</ul>
+
   <h4>Previous topic</h4>
   <p class="topless"><a href="problems.html"
                         title="previous chapter">Common problems and solutions</a></p>

doc/internal.rst

@@ -1,2 +1,113 @@
 Internal workings
 =================
+
+For more detailed information on the Modelverse specification, which this project is implementing, we refer to the `Modelverse Specification <http://msdl.cs.mcgill.ca/people/yentl/files/Modelverse.pdf>`_.
+
+Information on the implementation can be found below.
+
+Modelverse State
+----------------
+
+The Modelverse State is basically just an implementation of a graph library.
+As we have a particular kind of graph, this implementation is mostly done by hand at the moment.
+The notable exception to this is the RDF backend, proving that other implementations can also be used.
+
+The basic implementation just stores everything as dictionaries.
+All operations are then defined by doing operations on these dictionaries.
+The most interesting operations here are dictionary operations, which need to traverse these dictionaries in complex ways.
+To overcome performance problems for these operations, all results are cached (and validated afterwards).
+
+RDF backend
+^^^^^^^^^^^
+
+The RDF backend requires the *rdflib* module in Python.
+The Modelverse graph is then stored in RDF representation and all operations on it are done using SPARQL queries.
+Due to this level of indirection, performance is extremely slow.
+To increase performance, we would likely have to make more *composite* operations, or even group different requests together internally.
+
+Status codes
+^^^^^^^^^^^^
+
+The MvS returns, apart from its actual return value, a status code for the request.
+This value is not used by the MvK at all, since sometimes a request is expected to give an error (*e.g.*, checking whether an element is present).
+When debugging the MvS, however, these status codes can come in handy.
+
+Modelverse Kernel
+-----------------
+
+Precompiled functions
+^^^^^^^^^^^^^^^^^^^^^
+
+Modelverse Interface
+--------------------
+
+Semantics visitor
+^^^^^^^^^^^^^^^^^
+
+Constructors visitor
+^^^^^^^^^^^^^^^^^^^^
+
+Primitives visitor
+^^^^^^^^^^^^^^^^^^
+
+Bootstrap visitor
+^^^^^^^^^^^^^^^^^
+
+Model visitor
+^^^^^^^^^^^^^
+
+Bootstrapping
+-------------
+
+To bootstrap, you just have to run the file *bootstrap.py*.
+Here, we explain what this file actually does...
+
+The bootstrap script primarily creates the initial graph manually.
+This manual creation is done by writing data to a file, which is later read by the MvS.
+The initial graph consists of several parts:
+
+* The Modelverse Root node;
+* A global definition of all primitives;
+* The stack frame of the initial user *user_manager*, which manages all other users;
+* The code for the *user_manager* user;
+* The code for all new users, as assigned by the *user_manager*;
+* Bindings in the compilation manager for bootstrap files.
+
+These are all fairly simple in themselves.
+For some parts, such as the code, the HUTN compiler is invoked on a temporary piece of code.
+All bootstrap files are also compiled and made available to the compilation manager with their MD5 hash.
+
+How to add a primitive
+----------------------
+
+Probably the most important reason why you would want to know about the Modelverse internals, is if you want to add a Modelverse primitive.
+Primitives are functions implemented in the MvK core, and thus become hardcoded.
+
+.. warning::
+    While these functions are hardcoded, their implementation needs to follow strict rules, such that their semantics is identical in all possible programming languages.
+    For example, a primitive *getTime()* cannot simply be implemented as *time.time()* in Python, as this gives different results on Linux and Windows.
+
+To add a primitive, follow these steps:
+
+1. Add the code of the primitive to the MvK implementation by adding it in the file *primitives.py*.
+
+    a. Name the function identically to how the primitive will be invoked later on.
+    b. Take a set of parameters. Parameters for primitives are always positional and start from *a* onwards.
+    c. The final parameter should be a catch-all element, as it is possible that a high number of additional information is passed. In Python, this is done with a parameter prepended with \*\*.
+    d. When asking the MvS for information, use yields, just like the implementation of transformation rules in the MvK.
+    e. Instead of return, use a *raise* with the exception *PrimitiveFinished*. This exception takes one argument: the returnvalue.
+
+2. Add the signature to *bootstrap.py* in the topmost dictionary *primitives*. The key is the name of the function, and the value is a list starting with the return type, followed by all parameter types (as string).
+3. Add the declaration to *includes/primitives.alh* to make them available everywhere.
+4. Add the definition to *primitives.alc* and assign the Modelverse reference *?primitives/function_name*.
+5. Recreate the bootstrap file by running the script *generate_bootstrap.py*.
+6. Restart the Modelverse to reload the bootstrap file.
+7. From now on, all files including *primitives.alh* have access to the defined function.
+
+Adding a precompiled function
+-----------------------------
+
+Adding a precompiled function is way easier: only step 1 of the addition of a primitive should be done, but in the file *compiled.py* instead of *primitives.py*.
+All other steps are done automatically since there is an action language implementation present.
+The MvK will then automatically pick the precompiled function or the explicitly modelled operation, depending on preferences.
+A restart of the MvK is needed for Python to pick up the new functions.