Performance and dropped Lua calls issue - possibly MIDI bitrate related

Developers
1

I’m observing some slow UI updates and dropped messages under high incoming MIDI message load on the E1. I’m trying to understand where these are coming from and what can be done to mitigate them. I’m developing a control plugin for REAPER that sends custom sysex messages to the E1 for up to 36 VST parameters.
The control plugin is operating on roughly 30 update calls per second, checks for state changes (parameter name, parameter value, parameter value text, …) and sends these messages to the E1. These messages are evaluated in a Lua script on the E1.

Especially under high load situations like the initial state messages or when automating all 36 parameters, the E1 UI can start to lag behind by several seconds and not carry out some Lua calls like control:setName().

My first idea was that the MIDI message rate could be too high (the original MIDI spec allows a baudrate of 31250, which is roughly the equivalent of 3125 MIDI bytes), eating up messages somewhere between my control plugin and the E1 OS. So I’ve implemented a rate limiter into my control plugin ensuring that with it’s update rate of 30hz it’ll never send out a MIDI message with a total over 104 bytes. Unfortunately, that didn’t fix the issue of some Lua calls being ignored under high load.

Another problem is that while my control plugin sends out the initial batch of messages rate-limited in about 666ms, the E1 takes more than 2 seconds to fully process everything and be done with all visual updates. Here’s a log from my control plugin (the print call gets run 30 times per second):

MIDI out block with size of 85 bytes done in 1 microseconds. 35 parameter left to be updated.
MIDI out block with size of 90 bytes done in 3 microseconds. 33 parameter left to be updated.
MIDI out block with size of 90 bytes done in 2 microseconds. 31 parameter left to be updated.
MIDI out block with size of 92 bytes done in 3 microseconds. 29 parameter left to be updated.
MIDI out block with size of 100 bytes done in 1 microseconds. 27 parameter left to be updated.
MIDI out block with size of 94 bytes done in 2 microseconds. 25 parameter left to be updated.
MIDI out block with size of 92 bytes done in 2 microseconds. 23 parameter left to be updated.
MIDI out block with size of 101 bytes done in 2 microseconds. 21 parameter left to be updated.
MIDI out block with size of 98 bytes done in 1 microseconds. 21 parameter left to be updated.
MIDI out block with size of 99 bytes done in 3 microseconds. 19 parameter left to be updated.
MIDI out block with size of 92 bytes done in 1 microseconds. 17 parameter left to be updated.
MIDI out block with size of 93 bytes done in 2 microseconds. 16 parameter left to be updated.
MIDI out block with size of 100 bytes done in 2 microseconds. 14 parameter left to be updated.
MIDI out block with size of 100 bytes done in 1 microseconds. 12 parameter left to be updated.
MIDI out block with size of 96 bytes done in 2 microseconds. 10 parameter left to be updated.
MIDI out block with size of 94 bytes done in 1 microseconds. 8 parameter left to be updated.
MIDI out block with size of 100 bytes done in 2 microseconds. 6 parameter left to be updated.
MIDI out block with size of 101 bytes done in 2 microseconds. 4 parameter left to be updated.
MIDI out block with size of 72 bytes done in 3 microseconds. 0 parameter left to be updated.

So rather quick. However, on the E1:

00:23:00.961 lua: Received plugin param 1 range set with min 0 max 1 and default 0
00:23:00.963 lua: Received plugin param 1 value change of 0
00:23:00.965 lua: Received plugin param 1 with value text Off
00:23:00.965 lua: Received plugin param 1 with parameter name Bypass
00:23:00.966 lua: Received plugin param 2 range set with min 0 max 16383 and default 0
00:23:00.968 lua: Received plugin param 2 value change of 8192
00:23:00.969 lua: Received plugin param 2 with value text 0.00 dB
00:23:00.982 lua: Received plugin param 2 with parameter name Ip Trim
00:23:01.044 lua: Received plugin param 3 range set with min 0 max 1 and default 0
00:23:01.046 lua: Received plugin param 3 value change of 16383
00:23:01.046 lua: Received plugin param 3 with value text On
00:23:01.047 lua: Received plugin param 3 with parameter name PreEQ
00:23:01.050 lua: Received plugin param 4 range set with min 0 max 1 and default 0
00:23:01.051 lua: Received plugin param 4 value change of 0
00:23:01.052 lua: Received plugin param 4 with value text Off
00:23:01.053 lua: Received plugin param 4 with parameter name LPF
00:23:01.124 lua: Received plugin param 5 range set with min 0 max 16383 and default 0
00:23:01.126 lua: Received plugin param 5 value change of 16383
00:23:01.128 lua: Received plugin param 5 with value text 24000 Hz
00:23:01.128 lua: Received plugin param 5 with parameter name LPFFreq
00:23:01.129 lua: Received plugin param 6 range set with min 0 max 4 and default 0
00:23:01.130 lua: Received plugin param 6 value change of 4096
00:23:01.132 lua: Received plugin param 6 with value text 6 6/12/18
00:23:01.133 lua: Received plugin param 6 with parameter name LPFSlope
00:23:01.134 lua: Received plugin param 7 range set with min 0 max 1 and default 0
00:23:01.136 lua: Received plugin param 7 value change of 0
00:23:01.137 lua: Received plugin param 7 with value text Off
00:23:01.182 lua: Received plugin param 7 with parameter name HPF
00:23:01.184 lua: Received plugin param 8 range set with min 0 max 16383 and default 0
00:23:01.184 lua: Received plugin param 8 value change of 0
00:23:01.186 lua: Received plugin param 8 with value text 20.0 Hz
00:23:01.187 lua: Received plugin param 8 with parameter name HPFFreq
00:23:01.206 lua: Received plugin param 9 range set with min 0 max 4 and default 0
00:23:01.207 lua: Received plugin param 9 value change of 4096
00:23:01.208 lua: Received plugin param 9 with value text 6 6/12/18
00:23:01.210 lua: Received plugin param 9 with parameter name HPFSlope
00:23:01.211 lua: Received plugin param 10 range set with min 0 max 1 and default 0
00:23:01.212 lua: Received plugin param 10 value change of 0
00:23:01.213 lua: Received plugin param 10 with value text Off
00:23:01.214 lua: Received plugin param 10 with parameter name Noise
00:23:01.216 lua: Received plugin param 11 range set with min 0 max 16383 and default 0
00:23:01.218 lua: Received plugin param 11 value change of 0
00:23:01.218 lua: Received plugin param 11 with value text -200.00 d
00:23:01.220 lua: Received plugin param 11 with parameter name NoiseLvl
00:23:01.221 lua: Received plugin param 12 range set with min 0 max 2 and default 0
00:23:01.222 lua: Received plugin param 12 value change of 0
00:23:01.223 lua: Received plugin param 12 with value text Off
00:23:01.250 lua: Received plugin param 12 with parameter name NoisePre
00:23:01.252 lua: Received plugin param 13 range set with min 0 max 16383 and default 0
00:23:01.253 lua: Received plugin param 13 value change of 16383
00:23:01.272 lua: Received plugin param 13 with value text 100 %
00:23:01.274 lua: Received plugin param 13 with parameter name Wet/Dry
00:23:01.275 lua: Received plugin param 14 range set with min 0 max 16383 and default 0
00:23:01.276 lua: Received plugin param 14 value change of 16383
00:23:01.278 lua: Received plugin param 14 with value text 24000 Hz
00:23:01.279 lua: Received plugin param 14 with parameter name DryFreq
00:23:01.281 lua: Received plugin param 15 range set with min 0 max 3 and default 0
00:23:01.282 lua: Received plugin param 15 value change of 0
00:23:01.283 lua: Received plugin param 15 with value text 0 Off/12d
00:23:01.285 lua: Received plugin param 15 with parameter name DrySlope
00:23:01.286 lua: Received plugin param 16 range set with min 0 max 3 and default 0
00:23:01.288 lua: Received plugin param 16 value change of 0
00:23:01.304 lua: Received plugin param 16 with value text Off
00:23:01.306 lua: Received plugin param 16 with parameter name LP SplitEQ
00:23:01.307 lua: Received plugin param 17 range set with min 0 max 16383 and default 0
00:23:01.309 lua: Received plugin param 17 value change of 8192
00:23:01.309 lua: Received plugin param 17 with value text 0.00 dB
00:23:01.310 lua: Received plugin param 17 with parameter name Split Mix
00:23:01.312 lua: Received plugin param 18 range set with min 0 max 16383 and default 0
00:23:01.313 lua: Received plugin param 18 value change of 5461
00:23:01.315 lua: Received plugin param 18 with value text -0.00 dB
00:23:01.316 lua: Received plugin param 18 with parameter name Gain
00:23:01.317 lua: Received plugin param 19 range set with min 0 max 16383 and default 0
00:23:01.318 lua: Received plugin param 19 value change of 8192
00:23:01.319 lua: Received plugin param 19 with value text 0.00 dB
00:23:01.321 lua: Received plugin param 19 with parameter name Output Trim
00:23:01.323 lua: Received plugin param 20 range set with min 0 max 1 and default 0
00:23:01.324 lua: Received plugin param 20 value change of 0
00:23:01.324 lua: Received plugin param 20 with value text Off
00:23:01.326 lua: Received plugin param 20 with parameter name SC Ext
00:23:01.327 lua: Received plugin param 21 range set with min 0 max 1 and default 0
00:23:01.329 lua: Received plugin param 21 value change of 0
00:23:01.330 lua: Received plugin param 21 with value text Off
00:23:01.331 lua: Received plugin param 21 with parameter name SCEQ
00:23:01.332 lua: Received plugin param 22 range set with min 0 max 1 and default 0
00:23:01.333 lua: Received plugin param 22 value change of 0
00:23:01.334 lua: Received plugin param 22 with value text Off
00:23:01.336 lua: Received plugin param 22 with parameter name SCHPF
00:23:01.337 lua: Received plugin param 23 range set with min 0 max 16383 and default 0
00:23:01.338 lua: Received plugin param 23 value change of 0
00:23:01.339 lua: Received plugin param 23 with value text 20.0 Hz
00:23:01.341 lua: Received plugin param 23 with parameter name SCHPFFreq
00:23:01.343 lua: Received plugin param 24 range set with min 0 max 4 and default 0
00:23:01.344 lua: Received plugin param 24 value change of 4096
00:23:01.345 lua: Received plugin param 24 with value text 6 6/12/18
00:23:01.346 lua: Received plugin param 24 with parameter name SCHPFSlope
00:23:01.348 lua: Received plugin param 25 range set with min 0 max 2 and default 0
00:23:01.349 lua: Received plugin param 25 value change of 0
00:23:01.351 lua: Received plugin param 25 with value text HPF
00:23:01.352 lua: Received plugin param 25 with parameter name SCHPFType
00:23:01.353 lua: Received plugin param 26 range set with min 0 max 16383 and default 0
00:23:01.354 lua: Received plugin param 26 value change of 8192
00:23:01.355 lua: Received plugin param 26 with value text 0.00 dB
00:23:01.357 lua: Received plugin param 26 with parameter name SCHPFGain
00:23:01.358 lua: Received plugin param 27 range set with min 0 max 16383 and default 0
00:23:01.359 lua: Received plugin param 27 value change of 8192
00:23:01.360 lua: Received plugin param 27 with value text 0.00 dB
00:23:02.128 lua: Received plugin param 27 with parameter name SCEQGain
00:23:02.132 lua: Received plugin param 28 range set with min 0 max 16383 and default 0
00:23:02.137 lua: Received plugin param 28 value change of 9039
00:23:02.139 lua: Received plugin param 28 with value text 1000.0 Hz
00:23:02.141 lua: Received plugin param 28 with parameter name SCEQFreq
00:23:02.142 lua: Received plugin param 29 range set with min 0 max 16383 and default 0
00:23:02.143 lua: Received plugin param 29 value change of 6430
00:23:02.144 lua: Received plugin param 29 with value text 0.80 
00:23:02.146 lua: Received plugin param 29 with parameter name SCEQQ
00:23:02.147 lua: Received plugin param 30 range set with min 0 max 1 and default 0
00:23:02.150 lua: Received plugin param 30 value change of 0
00:23:02.151 lua: Received plugin param 30 with value text Off
00:23:02.154 lua: Received plugin param 30 with parameter name SCLPF
00:23:02.156 lua: Received plugin param 31 range set with min 0 max 16383 and default 0
00:23:02.157 lua: Received plugin param 31 value change of 16383
00:23:02.158 lua: Received plugin param 31 with value text 24000 Hz
00:23:02.159 lua: Received plugin param 31 with parameter name SCLPFFreq
00:23:02.162 lua: Received plugin param 32 range set with min 0 max 4 and default 0
00:23:02.163 lua: Received plugin param 32 value change of 4096
00:23:02.164 lua: Received plugin param 32 with value text 6 6/12/18
00:23:02.165 lua: Received plugin param 32 with parameter name SCLPFSlope
00:23:02.166 lua: Received plugin param 33 range set with min 0 max 2 and default 0
00:23:02.168 lua: Received plugin param 33 value change of 0
00:23:02.169 lua: Received plugin param 33 with value text LPF
00:23:02.170 lua: Received plugin param 33 with parameter name SCLPFType
00:23:02.171 lua: Received plugin param 34 range set with min 0 max 16383 and default 0
00:23:02.173 lua: Received plugin param 34 value change of 8192
00:23:02.175 lua: Received plugin param 34 with value text 0.00 dB
00:23:02.176 lua: Received plugin param 34 with parameter name SCLPFGain
00:23:02.177 lua: Received plugin param 35 range set with min 0 max 16383 and default 0
00:23:02.178 lua: Received plugin param 35 value change of 0
00:23:02.180 lua: Received plugin param 35 with value text -200.00 d
00:23:02.181 lua: Received plugin param 35 with parameter name SCNoise
00:23:02.182 lua: Received plugin param 36 range set with min 0 max 2 and default 0
00:23:02.183 lua: Received plugin param 36 value change of 0
00:23:02.186 lua: Received plugin param 36 with value text Off
00:23:02.188 lua: Received plugin param 36 with parameter name SCNoisePre

So it took from xx:xx:00.961 to xx:xx:02.188. Again, visually, it seems to take roughly a second more than what the log times state.

Also, all messages have been received by the E1 successfully as the log demonstrates, yet multiple controls didn’t update to the values received by the E1.

The messages in the quoted log affect the following properties of all 36 controls on a single page:

  • parameter range (min, max, default, 14 bit)
  • parameter name (14 chars)
  • parameter value (14 bit)
  • parameter value text (9 chars)

Should I limit the rate of MIDI bytes sent to the E1 via USB to a specific value or is it unlikely to be the cause of this because it may support higher bitrates via USB?

I’m clearly hitting a bottleneck somewhere, but I’m unsure where it could be. What could I do to further analyze this?

It also seems that under these high load situations, the E1 will not carry out certain Lua calls, which seems to be a bug?

EDIT: I’ve just realized that the seemingly dropped Lua calls become visible once I press a different control set button than the one currently active. This will cause the outdated control properties to be applied, even minutes later. So the dropped Lua calls strongly look like a bug.

2

Changing parameter range, min, max have always been problematic as far as seeing the change in real time. Once you make the updates, I’ve had to refresh the page to see it and even then turning the controls resulted in wonky response.

3

For my remote script it helped tremendously to call window.stop() before sending a lot of updates and window.resume() afterwards.

3 Likes
4

100% agree with @jhh - window.stop(), window.resume() bracketing lots of screen updates has helped tremendously.

2 Likes
5

Thanks a lot, the window.stop() and window.resume() functions help a lot with the not carried out Lua calls. Setting a bunch of parameter names via control:setName(“”) now works reliably! It might also show the final result a tad quicker.

But it has added a new problem: some parameter values with custom value texts now use the wrong alignment, causing values on the very ends of the ranges to be pushed out and become invisible. It always seems to affect the same parameters. Adding a print() line printing some variables and text before running the parameterMap.set() call weirdly prevents this. With the print() call removed, I can see the affected controls quickly with the intended text and after that the text is gone:

Screenshot_20230731-014055~2
Screenshot_20230731-014055~21080×529 197 KB

IMG_20230731_014124
IMG_20230731_0141241080×518 26.9 KB

6

yes, the individual lua calls for updating controls result in scheduling a repaint of the control on the screen. Electra firmware merges several subsequent updates of the same control to one repaint. If it, however, receives updates of many controls and possibly many of their attributes, things can get slow. The window.stop() temporarily suspends the repaint scheduler. When suspended, all control updates modify the data model only. When the window.resume() is called at later stage, a single repaint of whole page is performed.

I will

I will look at it.

1 Like