External Services
=================

Up to now, everything was contained in the Modelverse.
Nonetheless, this is not always possible, or desired: sometimes we want to run an activity using an external tool, or a service.
There are two main reasons why we would like this: performance and feasability.

For performance, an external service is likely to be faster than a Modelverse-based implementation.
Not only because the action language execution of the Modelverse is comparatively slow, but also because of the optimized implementation of the external tool.
It is clear that a specialized tool will be much faster than a naive implementation, even if the naive implementation were to be done in an efficient language.
For example, PythonPDEVS uses many non-trivial algorithms to speed up DEVS simulation, which are not easily mimicked in a new implementation.
While not impossible, of course, it seems wasteful to spend a significant amount of time to reimplement a tool.

For feasability, an external service can offer operations that the Modelverse would otherwise not be able to execute.
This links back to performance, but also to the algorithms required for the operation.
For example, reimplementing a differential equation solver is wasteful, knowing that specialized tools have spent many man-years to implement and optimize it.
We can never dream to come even close to these implementations.

Both go hand-in-hand: a naive implementation is often possible, but there is always the tradeoff between spending more time in developing the algorithm, and more time in using the algorithm.
The ideal solution would be to use an existing tool, thereby relying on the expertise of other developers, and merely making it available in the Modelverse.

When making this available in the Modelverse, we strife to minimize the difference between an internal and external operation.
This can be done using services: instead of modelling the activity, we model the communication with a tool that does the actual activity.
While there remains a big difference, as we cannot alter the semantics of the external tool, we come fairly close, abstracting away from the tool that is actually used for the operation.

In this section, we will illustrate this operation through the use of a *Fibonacci* service, which takes a single integer, and returns the Fibonacci number corresponding to it.

Modelverse-side
---------------

On the Modelverse, users can use the *comm* operations to communicate with external tools.
This is highly similar to communication for input and output to the designated user.
Code on the Modelverse can set up a connection to a registered service as follows::

    Integer function fibonacci(n : Integer):
        String port

        // Set up a dedicated communication port to the service
        port = comm_connect("fibonacci")

        // Send some data: the integer we got ourselves
        comm_send(port, n)

        // The external service computes the result

        // The comm_get function is blocking
        Integer result
        result = comm_get(port)!

        // Close the communication port from our side
        comm_close(port)
        return result!

In this example, we send an integer parameter to the external service, and expect it to reply with another integer, containing the final result.
When we got our result, we should always call *comm_close* to close the port at the Modelverse side.

Multiple calls to *comm_send* and *comm_get* are possible, depending on the communication protocol devised between the Modelverse and the external service.
For example, if we want to wait for the service to accept the result (e.g., the service can give an error on non-integer input)::

    Integer function fibonacci(n : Integer):
        String port
        port = comm_connect("fibonacci")

        comm_send(port, n)

        String response
        response = comm_get(port)
        if (response == "OK"):
            Integer result
            result = comm_get(port)
            comm_close(port)
            return result!
        elif (response == "Integer value expected!"):
            // error out, as it is not an integer!
            log("Non-integer input to fibonacci function!")
            comm_close(port)
            return -1!

Sometimes, for example when coupled to Statecharts, we want to poll whether or not there is input from the service.
This can be done using the *comm_poll* function, whose signature is identical to *comm_get*, though it returns a boolean and returns immediately.
As expected, it does not consume the value itself, which can subsequently be read out using *comm_get*, which will now return immediately if *comm_poll* was True.

Service-side
------------

The external service itself should be slightly augmented with a minimal wrapper.
This wrapper is responsible for the communication with the Modelverse, by importing the Modelverse wrapper, deserializing the input from the Modelverse, and serializing the output from the tool.
For our fibonacci example, where the fibonacci code is written in Python as well, we can code this as follows::

    >>> def fibonacci_service(port):
    ...     def fibonacci(n):
    ...         if n <= 2:
    ...             return 1
    ...         else:
    ...             return fibonacci(n - 1) + fibonacci(n - 2)
    ...     mv_input = service_get(port)
    ...     result = fibonacci(mv_input)
    ...     service_set(port, result)
    >>> service_register("fibonacci", fibonacci_service)
    >>> try:
    ...     while raw_input() != "STOP":
    ...         pass
    ... finally:
    ...     service_stop()

In this simple piece of code, we define the actual fibonacci code in the function *fibonacci*, which is totally unrelated to the Modelverse and is just pure Python code.
To wrap it, we define *fibonacci_service*, which is a function taking the communication port.
This function is responsible for communication with the Modelverse, through the operations *service_get* and *service_set*.
Both functions have the expected signature.
The service function is invoked on a thread, and therefore runs concurrently with other invocations of this same service.
Upon termination of the function, the thread ceases to exist.
All threads are daemon threads, such that, if the service wants to exit, it can immediately exit without first finishing up the processing of tasks in the Modelverse.
It is up to the Modelverse tasks to be resilient to such service failure or termination anyway, possibly through the use of a statechart that contains timeouts.

When the service is registered, we have to specify the name of the service to be used in the Modelverse on a *comm_connect* call.
The second parameter is the function to invoke when a connection is made, and must take one parameter: the communication port to be used.
After the service is registered, the main thread simply continues.
This makes it possible for the administrators of the service to maintain full control over their service.
If the main thread terminates, all currently running services are terminated.
Note that, after registration, this program is not able to do any other operations, until the *service_stop* operation is sent.

When doing type checking first, we can do multiple *service_set* operations::

    >>> def fibonacci_service(port):
    ...     def fibonacci(n):
    ...         if n <= 2:
    ...             return 1
    ...         else:
    ...             return fibonacci(n - 1) + fibonacci(n - 2)
    ...     mv_input = service_get(port)
    ...     if isinstance(mv_input, int):
    ...         service_set(port, "OK")
    ...         result = fibonacci(mv_input)
    ...         service_set(port, result)
    ...     else:
    ...         service_set(port, "Integer value expected!")
    >>> service_register("fibonacci", fibonacci_service)
    >>> try:
    ...     while raw_input() != "STOP":
    ...         pass
    ... finally:
    ...     service_stop()

The invoked function is in this case other Python code, though this does not need to be that way.
It would also be possible to invoke another executable, such as a *fibonacci* program, as long as the program can be used in an automated way.
Therefore, the external services are not limited to Python in any way, and we merely use Python as a convencience, since we already have a wrapper for it.
Using an external program happens through the *subprocess* Python module::

    >>> def fibonacci_service(port):
    ...     def fibonacci(n):
    ...         import subprocess
    ...         result = subprocess.check_output(["./fibonacci", str(n)])
    ...         return int(result)
    ...     mv_input = service_get(port)
    ...     result = fibonacci(mv_input)
    ...     service_set(port, result)
    >>> service_register("fibonacci", fibonacci_service)
    >>> try:
    ...     while raw_input() != "STOP":
    ...         pass
    ... finally:
    ...     service_stop()

HUTN compiler
-------------

While it might not be obvious, some of the internal Modelverse operations are implemented using external services already.
This is the case for the *compile_code* and *compile_model* functions.
These functions rely on a HUTN compiler running as a service: the string is sent there, and the service sends back the list of operations to execute on the Modelverse (i.e., the compiled data).
This was done to prevent a complete implementation of a parser in the Modelverse.
While this would certainly be possible, at the moment it is not of high priority, and it is easy to shift it externally.
Also, by using external services, it becomes possible to rely on trusted parsers, such as ANTLR or Ply, instead of building one from scratch as well.

Users of the *compile_code* and *compile_model* functions are never exposed to the use of an external service: it looks exactly like any other function call.
Nonetheless, there is a small difference: when the external service encounters a problem, or is simply not running, the functions will not be able to operate, even though the input is correct.
In any case, the compile_code and compile_model operations return a string value if they encounter a problem.
This problem could be anything, such as a syntax error or a semantical analysis error.
Otherwise, the functions return the action language function and model, respectively.

Before you can execute these functions, the HUTN compilation service must be running, which can be done by executing::

    >>> python scripts/HUTN_service.py

The service will connect to the Modelverse specified in its configuration (i.e., the *init* call).
It stays connected until the *STOP* input is given on *stdin*.

Others
------

Many other services have been defined and are automatically ran by the Modelverse.
This includes a JSON service (serialization and deserialization), a file server (persistent storage to files), a DEVS simulation server (both interactive and batch execution), and a PetriNet analysis service.

New services can easily be defined by putting a Python file in the *services* folder of the Modelverse.
All its subfolders are traversed and for each the *main.py* file is executed upon starting the Modelverse.
Upon starting the Modelverse, a log is printed to stdout of which services have started up.
When the Modelverse terminates, all services also quit.