Patch Parsing tool - testing and help needed

ok, this was quite some work, but it is getting somewhere. I released firmware 2.1.6 earlier today (check the downloads section on the website) and updated the app.electra.one. The release brings a beta version of the visual editor for the Patch requests and the Patch parsing rules. I am releasing it as a beta because I need some feedback and help from all of you. It looks like that the hardest part on this project is to design a good UI that users would understand. Another reason why is marked as beta is that the visual editor does not support more than one response per request as well as it does not support older format of the parsing rules (bPos, pPos, etc).

For making the first step I suggest to work on an extra copy of your preset or start a new preset from scratch. Just in case that something went wrong…

I used the new editor recently to create a preset for Sequential Mopho and Waldorf Wave XT. It took me only a few hours in the evening to make a complete preset with all controls and full bi-directional support. On top of that, Mopho does not have the patch dump SysEx documented. So, the editor can be seen also as kind of a tool for doing reverse engineering of patch dump SysEx messages. With Wave XT I did not even bother with looking at the information about the parameters, I simply used the MIDI learn to get the controls right and the patch editor to map it to patch dump SysEx messages. I did not write a single line of JSON while I worked on these presets.

What I am after now? It would be great if some of you were willing to give it a test ride. It would be great to hear some feedback on the UI and possibly some proposals on how to improve it. Some parts, especially the Parameter detail, are still a bit too technical. I do feel that if we do this right, it will become real easy to add patch dump parsing for any device and almost by any user.

As the documentation is missing, here are a few steps about how I used to tool so far:

Create a new preset and add a control. In this case it is Mopho synth and I will work with the Cutoff parameter:

Upload the preset with the Cutoff control to Electra One controller and open the Patch editor. You can find it in the Device section of the preset editor sidebar - ‘Edit device’ button.

In order to get started, you need to know if the synth supports a Patch dump request SysEx call. Review the device User guide or MIDI implementation guide. Look for patch dump request, ideally a edit buffer dump. eg. for Mopho you can find following request in the docs:

The next step is to register the patch Request in the Patch editor. Simply add bytes to the request and use the form on the right side to put the correct values in. You can enter the byte values in hex, cec, and binary notation. Do not enter the F0h and F7h bytes. These are markers of the SysEx message itself. The following request corresponds to the request shown above in the Mopho’s docs:

Click the Send to electra button to make sure that controller gets info about the request. If you now press the Patch request button on the controller, the MIDI Out indicator should flash shortly, meaning the request message is sent out.

Now it is time to start working on the parsing rules to get the data out of the synth’s response. To do so, switch to Responses tab. The Responses tab allows you to see incoming patch dump SysEx messages, assign bits and bytes to preset controls:

Click the Enable MIDILearn button at the bottom of the window and then press the Patch Request button on Electra One controller. If the device is configured correctly, the request will be send and the device will respond with the SysEx patch dump message. The right pane of the window allows you to review full content of the patch dump message in hex, dec, ascii, and binary form:

the first important step is to mark the fixed part of the message - the header. The header is used to identify incoming SysEx patch dump messages. Thus it informs Electra for what messages it should start with the parsing.

The header can be marked by clicking the byte position number. Click on the byte position number to tell electra where the patch dump header ends. This is not very user friendly yet…

Your patch dump data will have some bits marked with a green colour. This indicates that the bit has changed since the previous message has been received. For now, just click Clear marked button to make sure that no bits are marked as changed and you start with a clean list:

Now the magic comes, twist the Cutoff control on the controller (the that was set up earlier) to its maximum. Then, hit the Patch request button on the controller again. If you scroll down, you will noticed that some bits are marked now:

Observe the bits and then twist the Cutoff knob again to some other value. Looking at the marked bits, value 127 might be a good one to test. When the value is set, click the Request patch button on the controller again. The bits used by the parameter should emerge. Good values to discover all bits are 0, 1, 3, 7, 15, 31, 63, 127, 254, min, max. I hope you catch my drift:

It’s time to assign the marked bits to the Cutoff control. Click on the Cutoff / value on the left side. It will become selected (blue) and the detail info will be shown in the mid section of the window.

Now, click on all the marked bits. Start with the lower ones and go from the right to the left. When all bits of byte 23 are marked click the bit of the byte 16. By doing so, you will create the parsing rules. If you click the bits from the right to the left and you click the bit next to the selected bit, you will create a continuous range of bits, producing one rule. If you click bit in another byte, a new rule will be added automatically:

In this particular case, there has been two parsing rules added. (This is actually MS packed format). You can see the rules in the middle section of the parameter detail. The last thing you need to do, is to move the bit “P0 R0 B16 6…6” to the bit position 7 of the parameter to form an 8-bit number. I will add some visual for this. For now, I took this simple approach:

If you wondered what the letters mean, then:
P → patch request identifier
R → response indentifier
B → byte position
n…n → range of bits
The selection list is the position of the parsed data (n…n) in the parameter value

Now, click the Disable MIDILearn button. Twist the Cutoff knob to some value, eg. 100. When done, click the Send to electra to make sure the preset is transferred to the controller. Transferring the preset will also reset the value of Cutoff control to 0.

When the preset is uploaded, hit the Patch request button controller. If everything is done right, cutoff will be set to 100. Meaning you just retrieved the value from the connected device!

You can repeat this process for other parameters.

if you are in doubts, or if you feel something is not right, or if you just feel like editing the JSON source, just click on the JSON tab. It always shows and allows editing of the raw patch parsing JSON:

Happy testing!

edit1: If the device does not support patch request calls (eg. MKS-50), leave the request empty. It is not required. Just enable the MIDI learn and initiate the patch dump on the synth/device.

edit2: It is quite handy to test things with the sendmidi tool.

PS: a month ago @Resyn told me that he thinks he will never be able to make a preset as he does not know JSON and stuff. This is my attempt to make it possible for him! :slight_smile:

6 Likes

Thanks Martin, great news! I will give it a try with the Shruthi tomorrow and post some UX feedback :slight_smile:

Looks quite useful.

Funny - I just came upstairs after spending a couple hours creating a basic quick-n-dirty editor for the Microwave XT. I just wanted the 16 Mod Matrix entries and the Modifiers entries; the rest is pretty easy from the front panel.

I was using your previous app to edit the sysex messages for each of the controls plus LUA to copy the sysex buffer and do the initial parsing. I’ll have to give it a try with this version tomorrow.

1 Like

Uograded the electra One, but the browers app is showing only the simple json patch editor to me.

Is there a second version of the brower app I need to access?

EDIT : it’s okay now !

Hi @NewIgnis, sometimes forced reload of the page is needed. CMD+R or equivalent on Windows.

Findings so far after first attempts.

  • It would be good to be able not just to ‘clear marked’ but also to ‘clear response buffer’, so you can easier see when a new message is received. An additonal timestamp would help also.
  • Is there a rule to respect when you want to assign an existing control to a parsed value from a dump request? Can the control itself be CC, NRPN upfront, or should this be avoided?
  • I had a control of type ‘CC’, I mapped it to bit 6…0 from byte 78 . Then afterwards I changed it from type ‘CC’ to type virtual. When returning to the patch editor it look like the first screenshot below.
    What do the green indicators on byte 78 (right screen part) mean? They don’t disappear using ‘clear marked’. The parameter attributes in the middle don’t show any relationship anymore with that byte 78. Is it lost? Apparently not, because if I convert the control back to cc7, the assignment is shown again (second screenshot).It’s just no handy the relationship gets hidden when the control type changes. The relationship should remain visible somehow.

  • Can we use the same way to assign parameter changes to controls ? Or should these currently remain assigned via the SysEx editor of each independent control?

  • In the patch editor, there is room to add requests. What is its purpose ?

I’ll now start over again, but first create a cutoff assigned to a sysex parameter change, and only then assign it to a parsed dump byte.

1 Like

Second Test results.

I now first created a SysEx control for the Waldorf Filter Cutoff using string ‘3E’,‘13’,‘00’,20’,‘00’,‘00’,4E’,value. That sysex works.

However, when I now go to patch editor in order to also assign the parsed byte 78 from the patch dump, to the same control the editor wants me to add it as a second byte to the Parameter Change string, which I believe is not desired.

First screenshot shows the parameter assignment, made in the control itself, to the Parameter Change (4th byte is 20h). At the right is the response from a patch dump request.

Second screenshot shows what happens if I assign the 78th byte of the Patch Dump as well…

The result works partially:

  • the patch dump changes the value correctly on the Electra
  • the parameter change from Electra to Blofeld works as well

However, the parameter change on the Blofeld is not reflected in the Electra, allthough the standard Midi Learn tells me the SysEx message is seen by the Electra.
When I assigned that received SysEx Message (via MIDI learn, as the Blofeld was transmitting Sysex only messages while changing the cutoff on the Blofeld) to my existing control it changed its Parameter Number into 70, so I guess it has overwritten the existing control, but without throwing out the sysEx JSON I added to the previous control 69.
Indeed: this control 70 has no relationship in the Patch editor anymore:

On setting the control back to Parameter Number 69, the relationships were shown again

But allthough the Electra receives the correct value from the patch dump, and transmits the correct value back from the parameter change, I couldn’t receive a Sysex response anymore in the patch editor. The right part doesn’t change anymore:

Two more attention points:

  • I discovered the Blofeld not to respond to a subsequent request for a buffer patch dump, it you don’t change the preset in between. Be aware when testing with Blofeld.
  • When Midilearn is enabled in the patch editor, the already mapped control on the Electra does not change according to the patch dump . When midilearn is disabled it does it again. That is a bit confusing.

My third test will be: first map the control via the new parsing tool, and afterwards give it the appropiate Sysex parameter json for parameter changes.

1 Like

The third test proved to be difficult, and I needed to break it off

I see the Electra One receives information through MIDI 1 IN, but it gives no response whatsoever on the Patch Editor right screen part. It would be great if the right part has a mode in which it explains what it is receiving even if it is not expected. Without that, it’s hard to find what is going wrong in the traffic.

So for now I have the right part sometimes showing response on incoming messages , sometimes not, but I don’t know how to enforce it. Is it only updating the right screen part when Midilearn is enabled? I can’t tell … Is it only updating the right screen part if the header matches the request? I can’t tell as well.
Therefor having a mode where it always tells you something about the messages it receives would be desirable.

Have a nice weekend!

1 Like

@NewIgnis - all points taken. I just switched my Blofeld on and I am taking over :slight_smile:

1 Like

I’ll be doing some tests with a different synth meanwhile

this is what I have done:

  1. used midi learn to create the Cutoff control. It results indeed in:

  2. Created an Edit buffer request:

  3. Set up a response rule to get the Cutoff using the MIDI learn:

Seems to work fine. Still, I agree that your (@NewIgnis) points are perfectly valid and I will reflect them. But I seem to be have no difficulty in using the MIDI learn to map the bytes to the parameters. But of course, I have all the knowledge about how it is implemented in my mind…

1 Like

Hej @martin, I just got my first correct Shruthi parameter via sysex - that’s like second christmas for me after fishing in the sysex dumps and be puzzled for a long time - even with the help of others from this forum :partying_face: Thanks for the great work!

As a electra absent for some time now, I had to search for the “patch edit” link… I think it can be more up front named as “SysEx Patch Editor” next to Edit Lua Script to give easy access.

In the editor area a link to this tutorial text would be splendid - opening a new window. I used the split view of my browser with your post next to the editor which worked great. So all other UX hurdles have been maneuvered by following your guide… I think direct links between the documentation parts and the editor parts would help newbies here. Also your earlier posts on how sysex works would be worth a link here.

Or more sophisticated: a integrated help / tutorial like ableton does it:

I will give you more feedback while building the shruthi preset.

1 Like

One layout problem:
The last sysex lines are cut in the browser window and cannot be scrolled to… (zooming out helps)

Repeating test 3 and replicating your steps. Much more success this time.

1 Used midi learn to create the Cutoff control. It results in the same json, except it still had the head and tail. I took those away.

2 Set up the pach request sequence without an issue
3 This time getting the sound patch dump in was immediate and without error.
4. However, after finishing the assignments. The E1 is still not responding to the Paramter Changing sent out by the Blofeld. The App sees them, I verified through Midi learn, but the value on the control doesn’t change accordingly. The parameter change should be bidirectional in this version?

Test 4 : I will now use Lobo’s preset and try to convert into what I learned today

As all the parameters in my preset are done by midi learn I hope that they are in the right synthax yet. Good luck!! And if someday soon the preset will be upgraded with a working patch request function would be amazing!!! :slight_smile:

1 Like

I’m working on it :wink: . You did a great job already with setting up all the overlays. More news tomorrow evening

1 Like

This is still something that needs doing. The If the parameter change is a sysex message, Electra will not currently track its value when it is changed on the synth. The main reason why I have not done it yet is performance reasons. A naive implementation would be too slow. But there are definitely ways to implement that right…

1 Like

Having bidirectional sysex for param changes is rather a nice-to-have, not a must. I haven’ t encountered any reason yet for having it other than midi learn at this point in time, and that you did implement!

1 Like

Next suggestion: in a preset with a lot of controls you need to know the placement of the control (page and position on the page) in the patch editor to work quickly.

For instance int he example below I need to switch between various screens to know for sure what control I’m looking at:

image

1 Like