This page contains all notes from meetings with prof. Hans Vangheluwe. Additionally, a list of todo items and mindmaps are included.
Mindmap 2023/07/05
This list contains the following step(s) in my research, possibly annotated with additional questions.
  • AGV Digital Twin
    • Robot Recognition
      • Homography: Automatic detection (add fifth point; red color might not be the best choice?)
    • Study up on SysML v2
    • Create architecture in SysMLv2
    • Long Term (maybe out-of-scope): Create docker containers for DTs

  • CBD Simulator
    • Library and Docs:
      • Add tests
      • Online hosting of docs
    • Plotting
      • Fully add Bokeh backend (Close event is not working)
        ⇒   It appears this is impossible in Bokeh. The session remains active upon closure. Maybe this is due to the periodic call?
    • Simulator
      • Allow for hybrid simulation
      • Add MPI (?)
      • Add parallelization + distribution
      • Add Memoisation
      • Add Checkpointing
      • Add FMI
    • Structure
      • Allow for priorities
      • Cleanup: Split simulation from config (?)
      • refactor simulator to use internal (graph) structure from ModelverseState (MvS) --> Version will become 2.*
      • Add XML Tracer
      • Add VCD Tracer
    • See also: whishlist.txt

  • Logging: Create custom trace plotter w.r.t. Bill Song's DEVS Visual Modeling and Simulation Environment
  • CBD Simulator
    • Write paper for WSC
    • Library and Docs:
      • Data stream blocks?
        ⇒   CSV Streaming already supported
      • Docs for module
    • Plotting
      • Fully add Bokeh backend
        • Close event is not working
      • Add Seaborn
      • Add GGPlot
        ⇒   While a good way for plotting data, the mismatch between GGPlot and other frameworks is too large.
      • Add Label kind to also plot variable text?
    • Simulator
      • Adaptive/Variable step size
        • Keep Fixed as an option.
        • Euler 2-step
        • Runge-Kutta-Fehlberg (RKF45) [link to a paper]
          ⇒   Issues with RKF45: It is defined using y' = f(t, y), but it is unclear what this means.
      • Add Multi-Rate
      • Allow for external interrupt events during simulation
        ⇒   Not useful! The ConstantBlock.setValue() function makes this ineffectual; i.e. add a ConstantBlock to an input and change the value during runtime! See also the "dashboard" example.
    • Structure
      • Extracted scheduler
      • Add more signals (will allow statistics collection outside of the system)
      • Add CBD Tracers (or possibilities thereof)
No large progress updates. Discussed in this meeting:
  1. line-following robot using wheels instead of caterpillar tracks
  2. WSC Methodology Track paper (deadline 30 April): CBD framework (teaching, experimenting with alternative semantics, performance, accuracy)
    • MM + conformance checking, including hierarchy (Andrei)
    • drawio framework, visual libraries (Joeri)
    • (screen and model sharing (Joeri))
    • hierarchical, flattening
    • LaTeX generation (denotational semantics)
    • inline integration (for CT-CBD) vs. export CT-CBD + external solver (*)
    • operational semantics (Alg, DT, CT), fixed step-size, multi-rate, _|_ adaptive step-size (Randy)
    • optimizations: communication interval vs. integration interval
    • DEVS framework (for multiple Xreal-time platforms) (Randy) [realtime module]
    • application
      • used both for simulation and deployment in DT of line-following robot (Randy) OR
      • circle test AND/OR
      • bouncing ball AND/OR
      • train example (DEVS)
      • look at this report (problem with zero-crossing detector outside numerical solver, discuss quantization)
    • future work:
      • (neutral) textual language (should follow from MM infrastructure)
      • FMU (C) code generation (one, or more?) (*)
      • link with OpenModelica (a-causal models)
  • AGV Digital Twin
    • Write paper for AMC (SCS)
      • Iterate over paper (might become a technical report; add full FTG+PM description)
      • Make a presentation (in Draw.IO) on the full process (FTG+PM)
      • Remake architecture model using AADL/SysML/UML
        ⇒   Based on the UML 2.0 Component Diagram, an architecture design metamodel has been created.
      • Add deployment model to FTG+PM
    • Calibration/Verification
      • Use physical system as "accurate" and when the offset is too large, alter the parameter; i.e. build a tracking simulator
        ⇒   Possible to do this with "jumping back to the line"
        ⇒   The computation of the valid r and d values for the correct trajectory is flawed
        ⇒   Temporary solution: Use machine learning to identify the correct values:
        • Create a Q-learning algorithm to find the best sequence of parameters for the path
      • Solve "future measurements" in data collection for tracking simulator
  • AGV Digital Twin
    • Calibration/Verification
      • Check valitity of used formulas
        ⇒   The formulas used can be applied to the AGV (by transforming formulas from this paper.
      • Find parameters
        ⇒   Both r (wheel radius) and d (axis length) are parameters, yet by keeping d constant, only a single parameter needs to be tracked.
      • Find corresponding parameters in virtual world.
        ⇒   They should match theoretically, but don't in practice.
    • Make a presentation (in Draw.IO) on the full process (FTG+PM)
      ⇒   Do some more iterations! The FTG+PM has some issues.
  • AGV Digital Twin
    • Write first version of paper for AMC (SCS)
      • Add "gear" icon (⚙) to FTG+PM for automated tasks
        ⇒   When building the MetaModel for the FTG+PM (future), manual tasks can use the "hand" icon (🖑)
      • Change Motor icon in AGV model to "gear" as well
      • Add Architecture overview
      • Fix figure positions
      • Add PID tuning plot
      • Add resulting simulation plot (path trace)
        ⇒   Presumably due to fluctuations of battery level, inconsistent encoders and failing to meet required deadlines, the experiment used is difficult to precisely reproduce.
      • Refer to other papers
      • Describe Physical World Analyzer

  • CBD Simulator
    • Library and Docs:
      • Changeable constant block
      • Advanced example: change model behaviour based on user input during simulation (tkinter dashboard with CBDs?)
        ⇒   Can be used for AGV control and communication; i.e. "change the robot's speed"
  • AGV Digital Twin
    • Solve mistakes in FTG+PM graph
    • Communication
      • Communicate data from the robot to the simulation
        • Communicate motor speeds to verify validity of the Odometry component
        • Communicate sensor data to actually build a Digital Twin (actually a Digital Shadow?)
      • Implement communication protocol (OPC-UA?)
        ⇒   Too much overhead for current implementation.
This list contains the following step(s) in my research, possibly annotated with additional questions.
  • AGV Digital Twin
    • Make presentation of work
    • Create FTG+PM graph of AGV DT
    • Solve FOV error: the robot is on the line, but the camera does not see it like that
    • Combine data of simulation and view
      • Solve inconsistencies (calibration/verification)
        • Identify inconsistencies
          ⇒   It appears the motors obtained not enough power to do what they thought they did... Solved by dividing phi by 0.5406
          ⇒   Something else is wrong, but I can't seem to pinpoint what exactly
          ⇒   Note from a later deep-dive in this topic: I was using the wrong robot dimensions!
        • Make the simulation work again (with same parameters as physical AGV)
          ⇒   Only works for (model_DELTA_T=0.01, simulation_DELTA_T=0.2, v=0.01, Kp=-0.03, Ki=Kd=0, T=35, line=55, background=5), where the integrator and derivator have been implemented as BaseBlocks to prevent massive computation delays on the hardware.
      • Offline: do both simulations, interlace results and plot them
      • Online: do both simulations and plot results at same time
    • Communication
      • Identify how to communicate with the EV3
        ⇒   Thanks to this comment (local copy), the EV3 now works with the Edimax EW-7811Un V2
      • Research communication protocols
      • Add coordination between simulation and tkinter dashboard (i.e. draw the simulated position)
  • AGV Digital Twin
    • Create video of green rectangle to show "correctness" (can be found here)
    • Make brand new workstation for AGV testing
    • CBD Sim + AGV
      • Add visualization for heading of simulation
        ⇒   This brought out a large problem with the current implementation...
        ⇒   Right now, this problem has been solved by changing the PID values to (Kp = 0.06, Ki=0.0, Kd=0.06; DELTA_T=0.01)
      • Test the efficiency of the CBD Simulator on the AGV robot and compare to computer-obtained efficiency
        ⇒   The efficiency is terrible! In general, the "test" simulation (complete AGV model) takes 200% longer than is required, WITHOUT even running in Real-Time...
        ⇒   Hopefully the "hardware" model (without odometry and simulated env, but using actual sensors) takes less time and can be used in actual simulations.
      • Test if this algorithm actually works on the hardware itself
        ⇒   Efficiency is too bad to actually test this... See also previous point.
      • Test the efficiency of the "simple" SinGen and LCG models
      • Test the actual sensors and actuators instead of builtin simulation
        ⇒   Acceptable delays; use DELTA_T = 0.2
        ⇒   The PID controller is hence redefined by (Kp=0.02, Ki=0.0, Kd=0.01; DELTA_T=0.2)
      • Not Important: Test out multiple shapes to verify viability of PID controller.
      • Long Term (maybe out-of-scope): Add some Machine Learning PID controller (Evolution algorithm? Q Learning? Neural Net?)
    • Identify path for (virtual) simulation model

  • CBD Simulator
    • Read from CSV block (time column identifies output moments); add hold order (i.e. interpolation method)
    • Write to CSV block
  • AGV Digital Twin
    • Create TkInter Dashboard
      • Basic Dashboard to allow variation of clipping panes
      • Add tabbed pane for different views; i.e. a "depth" view
    • Recreate AGV model in CBD simulator and tune the controller using a somewhat random shape.
      ⇒   Also show heading on plot?
      ⇒   Add error cumulation to model, instead of afterwards
    • Robot Recognition
      • Optimize robot recognition
      • Add heading to the Kalman filter
      • Create an additional CBD library for OpenCV Image Manipulation
        • Create all components that the image manipulation can be broken up in
        • Create a Draw.IO library for these components
        • Recreate the main loop for image recognition
        • Test the system (efficiency? correctness?)
          ⇒   Apparently, the pipeline does not appreciate the multithreaded approach of a realtime CBD simulation
          ⇒   Maybe create a new "formalism" that just generates a simple loop; heavily based on Blender's Node Editor? (Coordinated project with IDLab?)
          ⇒   Maybe use ImagePlay? (Latest release: 2017)
          ⇒   FUTURE WORK: find a "data flow" formalism to do this (and more)
      • Homography
        • Find corners of "path area"
          ⇒   There can be a lot of "flickering" due to impreciseness of color detection.
          ⇒   How is it rotated? What is the topleft corner?⇒   Add a 5th point as "reference" for position
        • Image to coordinate system
        • Solve Keystone Effect
        • Find coordinate-mapping from original/transformed image view to "path area" view
        • Find coordinate-mapping from "path area" view to image view
        • Include homography in general video feed

Meeting 2021-01-25
  • We discussed the video of the AGV and the image/depth recognition that is happening.
  • (x, y, omega) should all be sent through the Kalman filter.
Maintained by Randy Paredis.