E1 Freeze diagnostics improvements

Ideas / Feature requests
1

Hi,
I am developing a fairly sophisticated Electra One custom MIDI integration with VCVRack, which I hope to share on this forum very soon (sneak peak at the VCV Rack module on Github) . Looks really promising. The E1 shines in this setup, with its screen and tactile encoders.

I wanted share a little of my experience of working on the development and testing of the E1 preset and the associated VCV Rack C++ module (I am a software developer in my day job). Overall, the online patch editor has been very useful, mostly I work in the Lua editor and the debugger / logger. Adding print statements has helped me got me out of most script debugging situations.

A few small UX frustrations aside, it has enabled me to get the job done. But the biggest problem is when Electra One freezes and becomes unresponsive. It seems the firmware is pretty sensitive to situations it is put in that it cannot handle, and hangs. Most of the issues that have caused freezes fall squarely at my door 100% - mistakes in the lua script, or wrong midi data. But each time it happens (a) I have to recover the unit - by resetting, turning off and on again, or occasionally reloading the firmware then (b) start figuring out what I did to make it freeze.

The problem is that E1 gives no feedback or diagnostic info at all what state it was in when it froze. So each time, I have to start adding print statements and systematically narrow down to the cause of the problem … which might take many freeze / reload cycles to get to the culprit. I have and am still getting freezes when I am just testing the preset. No errors in the log (but after a freeze, the online editor generally loses connection with the E1 anyway so it wouldn’t be able to send log info even if there was an error log to be shown). I suspect these might be to do with sending too much MIDI data to the E1 from my VCV rack module. But, I have no way of knowing that, other than throttling the data rate sent to E1 , then testing and seeing if it freezes again.

I want to create a solid controller experience that could be used for live playing etc. I expect to need to engineer towards that goal and spend the effort needed to get there. But, without having any feedback at all about what is going on inside the E1 firmware when it gets into a freeze (or almost as usefully, just before it gets into a freeze), its more a random guessing game than engineering. And if there was some info available, how to use that info to stop it happening!

Some ideas:

  1. I think it would be very useful to at least have a documented “pseudocode” workflow (without giving away any IP of course!) of how E1 handles in / out data streams, rate controls, as a sort of checklist we could work through. How does E1 handle incoming midi data stream volume? Does it have internal throttle limits that systems sending it midi should avoid hitting ?
  2. Some kind of switchable overlay metering on the E1 display that shows how close E1 is getting to its runtime “capacity” (memory, data rate ,. cpu ?) That would help understanding what is happening in realtime during testing prior to a freeze.
  3. Maybe write a crash dump or something onto the internal storage, that could be transferred and sent to E1 support for help.

I wonder if my experience mirrors any other E1 preset developer?

Despite these problems, I am really motivated to get this to work. 20 years ago, I created a patch for NI Reaktor that implemented 2 way control feedback with my Evolution UC33, so have been looking for a solution for hardware control of software modulars for a long time!

1 Like
2

Absolutely mirrors my experience developing complex presets.

Usually it comes down to trashing the stack by referencing out of bounds on an array or passing something unexpected to an E1 method.

Debugging these things has sucked up considerable time. I have a USB-enabled hw debugger version of the mk I hardware (at least I think it’s the mk I), but getting the environment set up and getting all the bits and pieces compiled correctly is such a challenge that I rarely have the time to use it.

I always hope it’s something simple and stupid, so after 2 hrs of prints and reboots, I regret not doing it

2 Likes
3

Yes, this is a difficult issue. I encounter this problem occasionally. It’s not very frequent, but I believe it is due to my in-depth knowledge of the underlying system. In scenarios like the one @oldgearguy described, what I experience is that Lua doesn’t trigger an error but fails silently. This typically occurs when passing complex data. I’m not blaming Lua for this; I think the issue lies somewhere between the E1 firmware and the Lua interpreter.

There are two main reasons that can cause the system to freeze:

  1. A high-priority task enters an infinite loop.
  2. An out-of-bounds memory access.

E1 can detect both of these situations. For the first case, it is relatively straightforward to implement corrective actions. The second case is more challenging, but still manageable.

I plan to add a watchdog to monitor for the first issue. This watchdog would allow the user to stop the Lua interpreter and dump a stack trace when needed. After reading your messages, I think it might be possible to switch the Lua interpreter to a “debugging” mode. This mode would retain information about the last successfully executed Lua chunk and make this information available. Such a feature would help in detecting and debugging both infinite loops and out-of-bounds memory accesses.

E1 runs a slightly modified version of the latest Lua release of Lua: version history. I will investigate whether a solution like the one I described already exists. If not, I will explore what I can do to implement it myself.

Having written all this I think that are ways to improve this.

1 Like