# Sysex dumps, updating controls - what am I missing?

**URL:** <https://forum.electra.one/t/sysex-dumps-updating-controls-what-am-i-missing/1025>\
**Category:** Developers\
**Created:** [October 23, 2021, 1:16pm UTC](https://forum.electra.one/t/sysex-dumps-updating-controls-what-am-i-missing/1025 "2021-10-23T13:16:30Z")\
**Posts on this page:** 17\
**Page:** 3

<div class="post-metadata">

**Author:** ![Ramyt](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.electra.one/ramyt/32/293_2.png) [@Ramyt](https://forum.electra.one/u/Ramyt)\
**Post date:** [November 2, 2021, 8:02am UTC](https://forum.electra.one/t/sysex-dumps-updating-controls-what-am-i-missing/1025/42 "2021-11-02T08:02:19Z")

</div>

Thanks for the shout out @martin !!

I’m still quite enthusiastic about it, actually, despite not having tinkered with my E1 (or any of my gear) in the past X months. I’m slowly starting to move past some life “complications” and fear of being a bad coder so that I can start fiddling around with the Lua extensions. Seeing that our initial conversations may have influenced you to incorporated Lua, lights a tiny flame under my ass! I will try to dedicate at least an hour a week moving forward. So much yet to learn on this incredible piece of gear you guys have created, Thank you for all your work, brother!

---

<div class="post-metadata">

**Author:** ![kris](https://avatars.discourse-cdn.com/v4/letter/k/82dd89/32.png) [@kris](https://forum.electra.one/u/kris)\
**Post date:** [December 4, 2021, 10:18pm UTC](https://forum.electra.one/t/sysex-dumps-updating-controls-what-am-i-missing/1025/43 "2021-12-04T22:18:24Z")

</div>

Hi everybody, I am a bit late to this party/thread, but I think I read through most of it and can feel the pain. As the author of the open source sysex librarian KnobKraft Orm I had to tackle exactly the same problems, less that for most synths I do (currently) not extract the individual parameter data.

Years of development of MIDI software can’t be summarized in such a small box, but let me state: I appreciate the Lua possibilities with the Electra, because I believe the JSON version is too limited. Many MIDI librarians try to do a data driven approach, but in the end the code-driven approach is more future proof. In the Orm, I don’t use Lua but Python, but the principle is the same.

I might be missing something, but I wanted to look at the Lua @oldgearguy wrote for the Alesis Micron, but when downloading the preset from the App I always only get the epr file, not the Lua file. Surely I don’t have to pipe it through the E1 to get it?

What I have done in the past is to use BCR2000, but as this is a pain to program and the software editors for it are stone age, I decided to write generators that generate the BCR2000’s peculiar little programming language. Works brilliantly.

Now I want to move my editing to the E1, but have hit multiple roadblocks, the sysex dump parsing being the most important one - I simply want to keep editing from a patch, and when I switch the patch I want the UI to jump and show all current data points. So it seems the Lua onResponse handler is what I need, and it should populate the parameter map.

I appreciate I can read the json epr file (or the instrument file, but those seem to be bit neglected right now as Martin himself wrote above) and get a list of all the parameters a synth offers - to be useful for the librarian I need to be able to decode a dump and show the individual parameters’ values. It would be even better if I could reuse the parsing logic of the preset that populates the parameter map. I can run Lua from within Python, which runs from within a C++/JUCE app 😉

I would also be willing to try to rewrite my generator for the BCR2000 editor templates for the E1.

Is there an example preset that shows how to do the 100% Lua parsing way?

---

<div class="post-metadata">

**Author:** ![oldgearguy](https://avatars.discourse-cdn.com/v4/letter/o/f4b2a3/32.png) [@oldgearguy](https://forum.electra.one/u/oldgearguy)\
**Post date:** [December 4, 2021, 11:45pm UTC](https://forum.electra.one/t/sysex-dumps-updating-controls-what-am-i-missing/1025/44 "2021-12-04T23:45:45Z")

</div>

attached is the current LUA (more or less) for the Micron.  
Also attached is the more-or-less correct sysex specs so you can see what I was basing it on.

So far, I’ve been doing it 100% with LUA. I’m going to start a thread on a new preset using LUA and JSON to create two (hopefully) equivalent presets for the same piece of gear (tc 2290).

Given the craziness in the Micron data, I don’t think it is doable just using JSON.

[Alesis Micron.zip](https://forum.electra.one/uploads/short-url/izXjM8hcbuZIvmJTbs6LzNLPjBO.zip) (583.5 KB)

---

<div class="post-metadata">

**Author:** ![akira](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.electra.one/akira/32/799_2.png) [@akira](https://forum.electra.one/u/akira)\
**Post date:** [December 6, 2021, 7:56am UTC](https://forum.electra.one/t/sysex-dumps-updating-controls-what-am-i-missing/1025/45 "2021-12-06T07:56:05Z")

</div>

Here is a preset and LUA script for the Microwave – UI is just virtual controls that reflect the `parameterMap` (but don’t send data) and everything is controlled from LUA. I’m close to having a full editor for the Matrix 6 too, with mod matrix update capability, all in 200 lines of LUA.

[Waldorf Microwave.zip](https://forum.electra.one/uploads/short-url/ur1o1K2rM6uIvSdM4B1T7EOxsN3.zip) (13.6 KB)

EDIT: I’m attaching the Matrix 6 preset too – it’s been a while since I used or worked on it so I’m not sure what state it is exactly in but it’s definitely WIP.

[Oberheim Matrix 6.zip](https://forum.electra.one/uploads/short-url/4S8c2irrhQrPlWBc3OQaYX4zGV3.zip) (7.4 KB)

---

<div class="post-metadata">

**Author:** ![Resyn](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.electra.one/resyn/32/727_2.png) [@Resyn](https://forum.electra.one/u/Resyn)\
**Post date:** [December 11, 2021, 11:03pm UTC](https://forum.electra.one/t/sysex-dumps-updating-controls-what-am-i-missing/1025/47 "2021-12-11T23:03:10Z")

</div>

Pretty fantastic Presets!

---

<div class="post-metadata">

**Author:** ![kris](https://avatars.discourse-cdn.com/v4/letter/k/82dd89/32.png) [@kris](https://forum.electra.one/u/kris)\
**Post date:** [December 13, 2021, 10:44pm UTC](https://forum.electra.one/t/sysex-dumps-updating-controls-what-am-i-missing/1025/48 "2021-12-13T22:44:24Z")

</div>

Impressive! That is a lot more code than I would have initially expected. But pretty straightforward and even if I haven’t done a Lua script for the Electra before I think I get what it is doing. Thank you!

---

<div class="post-metadata">

**Author:** ![kris](https://avatars.discourse-cdn.com/v4/letter/k/82dd89/32.png) [@kris](https://forum.electra.one/u/kris)\
**Post date:** [December 13, 2021, 10:49pm UTC](https://forum.electra.one/t/sysex-dumps-updating-controls-what-am-i-missing/1025/49 "2021-12-13T22:49:57Z")

</div>

Interesting! How does the PT\_VIRTUAL parameter type work?

---

<div class="post-metadata">

**Author:** ![martin](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.electra.one/martin/32/145_2.png) [@martin](https://forum.electra.one/u/martin)\
**Post date:** [December 14, 2021, 8:17am UTC](https://forum.electra.one/t/sysex-dumps-updating-controls-what-am-i-missing/1025/50 "2021-12-14T08:17:03Z")

</div>

PT\_VIRTUAL parameter does no track any specific MIDI message type. Instead, it just keeps a value. And this value that can be accessed/set with Lua functions.

---

<div class="post-metadata">

**Author:** ![mariankalus](https://avatars.discourse-cdn.com/v4/letter/m/838e76/32.png) [@mariankalus](https://forum.electra.one/u/mariankalus)\
**Post date:** [January 2, 2022, 1:11am UTC](https://forum.electra.one/t/sysex-dumps-updating-controls-what-am-i-missing/1025/51 "2022-01-02T01:11:23Z")

</div>

Recently, I spend some time parsing the undocumented Sysex patch dump of a nord drum 3p and thought I’d chime in here to share a real-world example.

As mentioned in this thread before, there are various ways how different manufacturers are storing data in their Sysex dump. The following steps are meant to give an idea how to approach deciphering an unknown patch dump. The concrete parsing might be different depending on the device.

Most values that we are collecting from a Sysex patch dump are numbers. So before diving into the details of the parsing process, I believe it is a good idea to recap how numbers are composed in a binary bit stream.  
When a number is transmitted in more than one byte, you can use Electra’s _parameterBitPosition_ to put the bits back together. When parsing unknown data, it is very helpful to understand how this calculation works.

Let’s start with a simple example. Assumed we have an 8-bit parameter that is transmitted in the first four bits of two bytes `%0000yyyy` `%0000yyyy`. Given, we are receiving a Hex value `03` in the first byte and `0A` in the second byte, which is 3 in decimal and `111` in binary, or 10 and `1010` respectively. The result of putting the binary back together is really just putting them back together. So `111` and `1010` will be `1111010`. But we can also calculate the decimal value from the details we have so far.  
To put a parameter part to its corresponding bit position, we just need to multiply it by **2 power ‘length of first parameter part’** and **add the multiplied result to the first value**. In our example, the first parameter part has a length of 4. 2 power 4 equals 16.  
So we can calculate the result like this: 3 \* 16 + 10 = 58

Sometimes we also see other types of numbers, like signed, two’s complement or even fractions.  
To put a (simple) float back together, we first follow the same steps mentioned above. Then we **divide the result by 2 power ‘position of decimal point’**.  
The nord drum is storing its _Delay Rate_ parameter as float in 16-bits, distributed across 3 bytes. The lowest value as complete binary looks like this `1010101010101001`. That is 43689 as decimal. The decimal point position is 15 (from the right). 2 power 15 equals 32768.  
Which means we can calculate the float like this : 43689 / 32768 = 1.333 (The correct binary notation would be `1.010101010101001`)

Besides a power of 2 table as reference, personally I also find it helpful to have a list at hand to quickly look up the highest number that can be stored in a certain bit length:  
2-bit: 3  
3-bit: 7  
4-bit: 15  
5-bit: 31  
6-bit: 63  
7-bit: 127

Now, let’s dive into the actual parsing workflow.  
For the most part, I’m using [snoize’s MIDI monitor](https://www.snoize.com/midimonitor/) and the text editor [BBEdit](https://www.barebones.com/products/bbedit/download.html), but there are several other tools which get the job done as well.

**Note** : the following steps only work when the patch dump sends out the current parameter position. Otherwise, it might be necessary to save the patch on the device between sending the patch dumps.

1. Make sure MIDI monitor is running and listening to incoming MIDI messages

2. Turn a first parameter on the MIDI device to its minimal position

3. Trigger a patch dump

4. Turn the same parameter to its maximum position

5. Trigger a second patch dump

6. Create two empty documents in BBedit

7. In MIDI monitor, double click the first patch response message and copy its content into the first created BBEdit document  

8. Copy the second patch dump message into the second BBEdit document

9. In the first BBEdit document press [cmd] + [F] to open the Find window

10. Put a space character into the _Find_ field and a line return **\n** into the _Replace_ field. Click _Replace All_. → This will convert all space characters between the Hex values into line returns, meaning that the line number is now equal to the bytes offset  

11. Repeat the same process in BBEdit’s second document

12. In BBEdit’s sidebar, select both documents and right-click to open the context menu. Select _Compare_ from the context menu.  

13. In the resulting window, you should see the Hex values (bytes) highlighted that are different in the patch dump messages.  

14. If the data type is unknown, I would continue with converting these highlighted Hex values into binary and decimal and check if that already makes sense. To do so, I’m using the following [online calculator](https://www.rapidtables.com/convert/number/binary-to-decimal.html)

15. When looking at the binary, I’m mostly interested in detecting the boundaries of the parameter. So I’m looking for the highest and lowest bit that are changing. If the parameter is distributed across more than one byte, I would also try putting the binary parts together in the aforementioned calculator and see if the result corresponds with the number of parameter values.

Example:  
Many parameters on the nord drum range from 0 to 50. And most of them are distributed across two bytes.  
Here is the binary of a parameter at its minimal and maximum position:

00: `0000 0010110`  
50: `1100 1010110`

From the bit length list above, we know that a parameter ranging from 0 to 50 needs at least 6 bits.  
In the example, we can see that the first five bits of the second byte (reading from right to right) did not change. The length of the remaining bits is 6. If we remove the unchanged bits and put only these remaining 6 bits of the maximum parameter position again into the online calculator, we will be happy to see that `110010` is the binary representation of 50.

The only missing piece of the puzzle is the byte number required in Electra’s JSON file. All we need to calculate that is the **line number** of the changed bytes in the BBEdit document and the **length of the Sysex response header** , defined in the JSON file.  
The Sysex response header for the nord drum 3p looks like this:

```auto
{
    "responses": [
        {
            "header": [
                "33",
                "7F",
                "1A",
                "08",
                "05",
                "06",
                "00"
             ],

```

So it has a length of 7 bytes. To complete our calculation, we need to add 2 to the length of the response header. 1 for the unlisted start byte F0 and another 1 because Electra’s byte index is 0 based, but our line numbers in BBEdit start at 1.  
Now, we can calculate the byte number by subtracting 9 (the header length + 2 for the start byte and 0 index) from the line number in BBEdit.

With that information, I would move to Electra’s JSON and add it like this:

```auto
{
    "type": "sysex",
    "parameterNumber": 1,
    "parameterBitPosition": 2,
    "byte": 41,
    "byteBitPosition": 0,
    "bitWidth": 4
},
{
    "type": "sysex",
    "parameterNumber": 1,
    "parameterBitPosition": 0,
    "byte": 42,
    "byteBitPosition": 5,
    "bitWidth": 2
}

```

Next, I would load the updated preset into Electra and send the same patch dump messages with the first parameter being at its minimum and maximum position and check if the control is responding correctly.

This is just scratching at the surface, but maybe it helps a bit when starting to parse a Sysex patch dump.

Happy New Year!

---

<div class="post-metadata">

**Author:** ![martin](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.electra.one/martin/32/145_2.png) [@martin](https://forum.electra.one/u/martin)\
**Post date:** [January 3, 2022, 3:19pm UTC](https://forum.electra.one/t/sysex-dumps-updating-controls-what-am-i-missing/1025/52 "2022-01-03T15:19:49Z")

</div>

Hi @mariankalus,

Thanks for taking time to write all this down. Not only that it explains the general concept but it covers stuff specific to Electra. @Flyweight did quite some work on documenting this in the past too. Both texts provide good foundation for working with electra’s JSON.

I have had some ideas on how to make this more user friendly, but all the suggestions discussed and described here on the forum helped to move it one level up. I used most of the holiday time to build a proof of concept of a visual editor based on those ideas. I must say that I am very happy with that and I’ve already used the tool to reverse engineer DSI Mopho’s patch dumps. With this new tool (that will be part of the app.electra.one), it was easy and actually quite a fun activity.

 ![image](https://europe1.discourse-cdn.com/flex017/uploads/electraone/original/1X/1ce77817628e87d89497c212148b290171279e5e.jpeg)

_eg. the blue bits are assigned to the Resonance parameter, and it is actually two nibbles as one nibble is placed at the 0 position of the parameter while the other goes to the 4th position._

The idea is based on having all available preset parameters and patch dump bytes on one screen and being able to link the bits to the parameters by clicking. The patch dump bytes listing is integrated with Electra’s MIDI learn. Therefore twisting the knobs on the synth (or even on Electra) marks the changing bits and give a clear visual indication where and how the parameter value is stored in the patch dump data. Of course this is much inspired by all those howtos and posts from users on the forum. I just wrapped that up to a visual tool.

 ![image](https://europe1.discourse-cdn.com/flex017/uploads/electraone/original/1X/b83c827b88b34062c73363a529952252f9c47184.jpeg)

_the green colour marks bits that have changed between individual MIDI learn calls._

There is still stuff to do but I would love to release this some time this week. The JSON text editor works nicely along with this, so it is safe to use even though it is not fully completed.

What I like about that is that it is not only tool to build Electra’s patch dump parsing rules. It actually makes reverse engineering of undocumented sysex implementations possible and easy.

This is taking quite some of my time, so I am not fast in responding to other topics on the forum. I will take care of that once this is out.

---

<div class="post-metadata">

**Author:** ![mariankalus](https://avatars.discourse-cdn.com/v4/letter/m/838e76/32.png) [@mariankalus](https://forum.electra.one/u/mariankalus)\
**Post date:** [January 3, 2022, 4:52pm UTC](https://forum.electra.one/t/sysex-dumps-updating-controls-what-am-i-missing/1025/53 "2022-01-03T16:52:27Z")

</div>

Hi @martin, this looks truly amazing!

Just when one thinks Electra can hardly get any better, you’re making it even more incredible! 🤩

---

<div class="post-metadata">

**Author:** ![Flyweight](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.electra.one/flyweight/32/342_2.png) [@Flyweight](https://forum.electra.one/u/Flyweight)\
**Post date:** [January 5, 2022, 12:30pm UTC](https://forum.electra.one/t/sysex-dumps-updating-controls-what-am-i-missing/1025/54 "2022-01-05T12:30:56Z")

</div>

wow @martin that looks just awesome.

but can you as change a parameter on the synth and then catch the changing bits?

this is useful for synth with remote control or for hidden parameter. like on the yamaha sy22 / tg33 for example. with some secret bits you get full control over the 2 2 operators. while the synths interface only offers a set of predefined settings like piano, bell etc.

anyhow awesome

---

<div class="post-metadata">

**Author:** ![1AM](https://avatars.discourse-cdn.com/v4/letter/1/a698b9/32.png) [@1AM](https://forum.electra.one/u/1AM)\
**Post date:** [December 25, 2022, 2:47pm UTC](https://forum.electra.one/t/sysex-dumps-updating-controls-what-am-i-missing/1025/55 "2022-12-25T14:47:39Z")

</div>

Hey Marian,

Thanks for sharing this.

Any chance you happened to document the full spec for nd3p dump with parameter byte positions, value ranges, etc. Or was this more of a proof of concept sort of exploration of its dump?..

---

<div class="post-metadata">

**Author:** ![Resyn](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.electra.one/resyn/32/727_2.png) [@Resyn](https://forum.electra.one/u/Resyn)\
**Post date:** [May 22, 2024, 1:22pm UTC](https://forum.electra.one/t/sysex-dumps-updating-controls-what-am-i-missing/1025/56 "2024-05-22T13:22:43Z")

</div>

Hi guy, have you ever actually used a Jazzmutant Lemur - which was killed as a product a few months before iPad hit the market? How about the iOS version that came soon after - and that was recently killed by [Lline.com](http://Lline.com)? So I have tried and own both. The short version is that for am musician they offer a novel way of interacting with sound sources, from synths and samplers to plugins. Not exactly what probably was in mind when Electra was born.

Will the Electra ever be able to technically handle the object-oriented mayhem the Lemur offered? And as far as I can tell still offers: just dusted off my Jazzmutant Lemur and while getting the Jazzeditor to work for new patches or changing extant ones represents a hurdle, it appears as capable as ever for controlling MIDI devices (and OSC ones). The differences between the Lemur and an ipad are non-succinctly, but perhaps accurately, touched upon here: [JazzMutant Lemur Dead – Did The iPad Kill It, Or Was It Suicide? – Synthtopia](https://www.synthtopia.com/content/2010/11/15/jazzmutant-lemur-6/)

I wonder…

---

<div class="post-metadata">

**Author:** ![oldgearguy](https://avatars.discourse-cdn.com/v4/letter/o/f4b2a3/32.png) [@oldgearguy](https://forum.electra.one/u/oldgearguy)\
**Post date:** [May 22, 2024, 1:55pm UTC](https://forum.electra.one/t/sysex-dumps-updating-controls-what-am-i-missing/1025/57 "2024-05-22T13:55:21Z")

</div>

I developed a few editors/patches for the iOS version of the Lemur.  
My main disappointment was their lack of significant sysex support for devices.

Oh, and the extremely poor customer service and lack of enthusiasm for fixing bugs. 🙂  
I did a TX-802 editor that (IMHO) still gives a very intuitive view of the operators and the envelopes that is hard to duplicate on anything other than a computer.  
I also implemented an editor+ for the Roland RSS-10 that added some LFOs and other things that were not in the RSS-10 proper.  
However, I could not implement a full surround space path editor since the Lemur didn’t support sysex messages of any decent size.

Now that some drawing/graphics capabilities have been introduced here with the Electra Mk II, there is a promising future of marrying graphics with custom controls with standard controls to create some unique environments.

I just need to either retire or to stop developing presets so I can play with these newer features.

---

<div class="post-metadata">

**Author:** ![martin](https://dub1.discourse-cdn.com/flex017/user_avatar/forum.electra.one/martin/32/145_2.png) [@martin](https://forum.electra.one/u/martin)\
**Post date:** [May 22, 2024, 3:36pm UTC](https://forum.electra.one/t/sysex-dumps-updating-controls-what-am-i-missing/1025/58 "2024-05-22T15:36:55Z")

</div>

I opt for retiring 😉

---

<div class="post-metadata">

**Author:** ![oldgearguy](https://avatars.discourse-cdn.com/v4/letter/o/f4b2a3/32.png) [@oldgearguy](https://forum.electra.one/u/oldgearguy)\
**Post date:** [September 17, 2024, 12:22pm UTC](https://forum.electra.one/t/sysex-dumps-updating-controls-what-am-i-missing/1025/59 "2024-09-17T12:22:37Z")

</div>

might be sooner than we think. 😉

[Previous page](https://forum.electra.one/t/sysex-dumps-updating-controls-what-am-i-missing/1025.md?page=2)