Electra one mini core - is that really that slow?

Bug reports
1

I was trying to optimalize my preset for roland jv2080, to make it faster, and wanted to add buffor, because i was thinking, i send too much sysex to it, and make big latency problem. But i was wrong.

I analyzed on debugger timing of notes what i send to roland, and realize it have really big latency when changing parameters. I’m using external controller as a USB host, and sending all commands thru electra one mini. I don’t use MIDI out in my controller directly to roland, to send notes, everything is send thru electra one.

I cleaned all the lua code from unnecesary loops, ui refreshing and other things, that can make any latency, everything is totally clean, and can’t make anything faster.
Notes still have enormous latency. When i play fast arpeggio, notes are playing however they want when i send sysex to change any of parameters.
That’s not problem with roland buffering, when i was using ctrlr, there was no problems like that, but then i was sending notes directly to my roland.
I was trying to add to Json “rate” function into device, but it’s not working at all i think. Whatever time i set, it seems like it’s not working.

I didn’t try with lua buffer functions, but i’m almost sure, that’s not problem in enqueue, but efficiency of the core. If that’s true, then there’s no function what will help with that, and adding more rules to output, will make it only worse. I think, the only way to solve that, is send notes directly to roland, using midi merge, but i don’t have any right now, i wanted to ask about that here, before i buy one, i wanted to make my setup simple as possible, but afraid, there’s no other options.
Anybody had problems like that before? Maybe i’m wrong, and creating lua buffor will give priority for notes or CC above the sysex, without any latency?
I’m totally disgusted.

I can record how it “works” right now, but not today, im busy for a while. Thanks for anwsers

2

Are you forwarding the midi messages from the external controller (USB host) to the JV using the E1 router or is the forwarding done in the preset using Lua?

let me explain why am asking. The E1 router runs as the priority task on the controller. It means, a message that arrives will be forwarded immediately. That may result in slowing down of other activities, such as drawing graphics, etc.

An idle E1 controller has latency of around 1msec. That is rather good. This latency stays unaffected when you send sysex messages (JSON or Lua) in parallel to the forwarded stream of the data. There is another thing, however, a contention. As both forwarded messages and the sysex messages generated on the controller share the same MIDI IO out port, the controller has two blend them in. An event such as a Midi note cannot interrupt a sysex. ie. if sending a sysex is in progress, the note has to wait until it is finished. JV2080 parameter change is 12 bytes (roughly 4msec), so if the message collide, the overall time of forwarding the note message can grow to 5msec. if the sysex messages were 200bytes, it grow up to 60msec. This is how MIDI IO works.

If you forwarded or processed your notes using Lua, the situation is a bit different, the Lua runs as a lower priority task. On an idle e1 controller the latency will be about the same, but on a controller which is under heavy lua load, eg. intense Lua graphics, the latency will be affected.

As I am working on the upcoming firmware release, there have been things that I improved in this area, but the fundamental design stays described above.

3

Awww, i was in big mistake again.

I was asking for that few weeks ago here, when had this same problem, but then this problem was much more complex. I solve that things, mostly by changing all of the architecture of presets, but like somone suggest, i make every of route rules exactly in Lua code. I didn’t wanted to make routing rules in controller settings, coz when i experimenting with presets, really often had to make hard resets to bring back electra to life, but i see, You answer while ago in other topic to avoid it with other way, so i need to change way of doing this.

More here in old topic:

Anyway, You’re right, i use routing with:

  1. midi.onInput(port, message)
  2. midi.onNoteOn(...)
  3. midi.onNoteOff(...)
  4. midi.onPitchBend(...)
  5. midi.onProgramChange(...)
  6. midi.onChannelPressure(...)

And seems i need to delete all this functions, and make routing rules in controller setup.

Thanks a lot!

1 Like
4

One thing I would check before spending money on a merge box: whether the worst timing hits line up with page or patch changes rather than with single knob moves. A single JV parameter write is short, but a full refresh of a page of controls is dozens of them back to back, and if that happens while an arpeggio is running the notes have to wait for the whole batch. Spreading those refreshes out from a timer, a few messages per tick instead of all at once, gave me back gaps for the note stream to slip through and it cost nothing in feel. Also worth repeating your debugger test with the USB host controller as the only source, since a busy incoming stream can look like output latency when you only watch the destination end.

1 Like
5

Yup, but, when You touch one encoder and set it’s value, from, for example, 0 to 20, then You send 20 commands of sysex in really short time, You send much of microvalues in between, and best way, is to buffer it, You don’t need to send all values at the time, because, these microvalues are to small, and not hearable after all. The last important value is that one, when You stop to change parameter (the last one value). When You pick 2, 3 or 4 tones to edit with one parameter, You send 2x, 3x, or 4x more sysex commands on the output to edit them all, and there’s much more propably, You will fill the buffer out.
Anyway, i didn’t trying with any buffer right now, i will do it in nearby time, i just was thinking, it’s not problem with that, and have right. Martin said exactly why i have problem with that, i didn’t understand it before, but that’s the truest truth, the Lua code will always be below the routing rules.
I hope it will really solve my problem, and will try with it later. Buffer should make clean sending of sysex, and will optimalize transmission, for making commands lighter for my roland, but it’s not important as much, as sending equal notes. 1ms of delay it’s like nothing, when midi channel will transfer notes and sysex at this same time, 5ms delay it’s totally understable. Midi merge should have equal delays to it, just physics. I had delays up to 3500ms or more, what’s make it totally unusable, so separate lua code from routing seems like good way to solve that.

Have good day

6

he-he-he… It’s not obvious like i was thinking. I delete everything from my preset what concerned notes, and any communication between Lua Script and my external controler. There’s nothing, everything is clean. I was testing it on totally clean default presets too.
I set on default preset routing rule → USB host port 1 to MIDI I/O. Notes are send without problem, when i change parameters on the screen, there’s no problems, but it’s obvious, there’s no complicated commands to send. Everything is equal.

I set this same on my script, where i deleted all service for my external controller. There’s only routing rule exactly like on clean preset, made in controller → configuration → Router.

image
image907×163 7.82 KB

Nothing changed, when i change any parameter on the screen, notes have big latency, so separating it from Lua Code, nothing changed.

Script, and preset in itself working very fast, and have no problems with it right now, i’ve almost done working on it. But it’s totally unusable right now.

7

That’s totally noooot wooorking :confused:

And here’s from console logs.

Firstly i made one tone edit, it’s working quite good. When i send sysex for all 4 tones You can see debugger not follow right in time and everything is messed up…

8

spam, spam, more spam. I’m fighting with buffor right now. Electra can’t cope with to much sysex on output. I can’t find any solve for that. I want to make dynamic buffor, what will be depending on quantity of edited tones, and few more things. The basic question is: Is there’ ANY type of timer to use? Ai telling me once there’s getMillis(), Millis(), os.clock(), getTime(), os.time(), and when nothing work’s telling me there’s no timer at all. And after few more dialogues, can tell me, there’s another timer, trying it once again, and then not working again. I’m dumbfounded. It’s not first time when i’m trying to do functions with timers, but never did it after all, so i need clear answer. There’s any timer to use, or not? It will make some things reaaaaally easy to do.

9

Yes. But AFAIK only the E1 lua API timer functions. Not the general lua ones. See the lua API doc. I find these timer functions sufficient for my needs. Good luck.

1 Like
10

Your description of the 0 to 20 sweep is exactly the shape of the problem, and the nice thing is you do not need a real buffer to fix most of it - you just need to stop sending the in-between values in the first place. Two filters that stack well:

First, a coalesce window. Keep a “pending value” per parameter plus a “last sent at” stamp. When the encoder moves, overwrite the pending value and only actually emit if at least 30 to 50ms has passed since the last emit for that parameter; otherwise leave it pending. Then when the gesture goes quiet, emit the pending value one final time unconditionally. That single trailing send is the important half - without it you land on a stale value. A 0 to 20 twist that took a second becomes maybe five or six sysex messages instead of twenty, and the destination value is still exactly right.

Second, a change threshold for parameters where small steps are inaudible anyway. If the new value differs from the last sent one by less than N steps, hold it as pending rather than emitting. For a coarse thing like an arpeggio setting on the JV, N of 1 already kills the duplicate-value spam, and you can go higher on anything with a wide range where nobody hears a two-step difference. Combined with the trailing send you never lose the final target.

If you would rather keep it very simple: send only on release. Track the value while turning, display it locally so you still get visual feedback, and emit once when the encoder stops being touched. For patch-level edits like tone selection or arpeggio parameters that is usually all you want, and it takes the output load down to one message per gesture. Where release-only hurts is anything you want to hear sweep in real time, so I would use release-only for the structural parameters and the coalesce window for the ones you actually perform with.

One more thing worth checking on your side: make sure the comparison is against the last value you actually sent, not the last value the encoder reported. Otherwise a stationary-but-jittery encoder keeps re-sending the same number and you refill the buffer for no gain.

2 Likes
11

Yeeees, that’s right! That’s the way to do it, more and less. My head is after dozen last hours of optimalizing it, and few more things, and it boils right now, but made it in 3 of 4 of my presets, i will try to explain it quickly.
Firstly, thank You KiwiGrass i found it as a timer.onTick functions, i didn’t see it before, and work perfectly for me.

So mostly standard buffer with 30ms queue, with overwriting values make it perfect smooth, and give time for notes for equal sending of everything.
Mostly, but not in tones section.
I needed to add additional Buffer’s what make:
Change message resolution picking more tones to edit, when editing 1 tone, value step is 1. Editing 2 tones, step is 2, editing 3 tones, step is 3, and 4 tones, step is 4. Different is almost not hearable, when any of parameter independently stopped to change it’s own value, after 100ms, checking it’s value in dirty table, if values not this same, then sending actual value to roland to equal it. There’s no any big jumps, bcz delta is max 4 for 4 edited tones, so it’s almost nothing. When value is 0 or 127, it’s sending values without question of delta.

The other thing is, 9ms delay between all tones send. When i pick 1 and 2 tones to edit, there’s a 9ms delay between 1 value steps. So if You change parameter from 20 to to 22 (one step for 2 tones), these 2 sysex have 9ms of delay. When You pick 3 tones, delta is 3, so 20, to 23 (one step), 1 and 2 tones have 9ms delay, and third command have 18ms delay from this first. Analogously, for edited 4 tones. Even if You change value from 20 to 24, 28, and 30 for the 4 tones, then You send only 28 value (last with properly delta), but when You change editing this parameter for 100ms, then dirty table has check, and send this last value ALWAYS. I was checking it with changing 4 parameters in one time for all the tones turned on, and it works. It was hard work, but im content right now.

I think, there’s no way to make any preset stable without bufforing output enqueue.

AHH almost forgot. I needed to add white list for parameters with small value changes, bank selection, wave selection. There’s can’t be any of delta for any tones editing, value step needed to be always 1. Small, but important thing.

Have good day!

1 Like
12

+1 on building an output queue that sends midi on a timer.

It’s also a useful tool to limit the rate of sending midi to other devices. Each midi device has it’s own limits and fail in odd ways when it gets too many messages.

13

Device object already has the rate parameter and the feature was available in the past. The implementation was too naive though. Therefore it was removed from the code at some point.

lessons learned:

  1. a simple queue with slower rate of outbound messages does not work well as huge queue of messages can build up quickly. It becomes very annoying when rate is slow and In case of larger sysex messages.

  2. skipping value changes of the same parameter between the rate windows on MIDI level works for simple message types such as CC but become harder to handle multi-messages such as NRPN/RPN. it needs extra logic. Also, this is harder to apply to sysex messages.

  3. Viable solution is to queue changes and make decisions on the parameterMap inside the processing core of the firmware. ie where the actually value changes are happening. It also needs some extra care due to snapshots, etc.

I will leave this for broader discussion and other developers when the firmware source is out…

1 Like