Sysex dumps, updating controls - what am I missing?

Developers
1

The more I work with the ElectraOne and LUA, the better I understand how to make things happen. I can now send out a patch request via the panel button, receive the sysex dump and unpack it and start using the values.

The dump is a full patch dump and Alesis uses NRPNs to change values, so the dump is NRPNs in a packed format (8 MIDI bytes to represent 7 NRPN values - first byte has the high bit for the following 7 bytes).

So far so good. BUT - I think I am missing the basic concept behind the product. My goal now is to take the full patch sysex dump and update all the displayed values for the controls.
However, I cannot see a basic way to put byte 17 from the dump into the value for the control in the 3rd position on page 2 of the preset. (for example).

I see in the documentation using the “patch”:[] definition to process the sysex message (and I am doing that) and then the parameterMap.set() to update a MIDI value, but I don’t see any clear way to determine where in the parameterMap the update should occur (the parameter number).

If I get this figured out, my next question will be – is MIDI sent back to the device every time the parameterMap is changed? Since I just received the current values via sysex dump, ideally I just want to update the ElectraOne displays to reflect those values and minimize the back and forth MIDI transmissions.

Do these questions make sense? Again, I feel like there’s some fundamental piece I am missing here.

2 Likes
2

no worries. All these questions are valid and I am sure discussing them will help others. Also, the Lua integration is a fairly recent feature, there is room to improve or change stuff.

There are basically two ways to parse patches. The first way comes directly in the JSON implementation, the second is based on Lua extension. One preset may combine both ways. The main idea is that Lua based parsing can be used in situations when JSON parsing rules would be too cumbersome. A good example is parsing out the Patch name.

I am sure @Flyweight is reading this too :slight_smile:

Parsing with JSON rules

The idea is based on making it possible to define a Patch request SysEx message and a set of parsing rules as part of the Device definition. The Patch request(s) messages are sent when the [Patch request] button is pressed. If there are any parsing rules defined, Electra applies them against incoming SysEx messages. If there is a match, the rules are used to transform SysEx bytes to parameter values. All the patch handling functionality is defined in the patch object that can be part of the device object in the preset JSON. More details on this can be found at Parsing SysEx messages | Electra One Documentation

Parsing with Lua extension

The concept is similar. The patch.onRequest() function is triggered when the [Patch request] button is pressed. The patch.onResponse() is called when there is an incoming SysEx message that matches the response header defined in the Device’s patch JSON object. In other words, even though you are parsing with Lua, you still need to have the patch defined for you device. The patch.onResponse() is actually just a substitute for the JSON parsing rules. Once the patch.onResponse() if fired, you can do whatever you want with the bytes. The most obvious task is to populate the parameterMap with the values.

An important note here is that you should be rather setting values of the parameters in the parameterMap than setting values of individual Controls. Imagine that you have two controls with the same parameter. If you use the parameterMap, both controls will get updated when the parameter value is set and their display values will be reflected, Lua formatter functions run, etc. This basically means that setting a parameter value in the parameterMap is identical to receiving an individual MIDI message.

Now, the MS bit packed format. To me, this SysEx dump format is on the edge of using Lua over the JSON rules. The reason is that every value needs to be composed out of two SysEx bytes (MSB bit + 7bits) and there is a repeating pattern in it.

With JSON rules that can be done using two rules to compose one parameter value:

{
    "type":"nrpn",
    "parameterNumber":4,
    "parameterBitPosition":0,
    "byte":7,
    "byteBitPosition":0,
    "bitWidth":7
},
{
    "type":"nrpn",
    "parameterNumber":4,
    "parameterBitPosition":7,
    "byte":0,
    "byteBitPosition":6,
    "bitWidth":1
}

These two rules are used to compose value of NRPN parameter 4 by taking 7bits out of SysEx byte at position 7 and placing them to LSB of the parameter value (the first rule) and adding one bit from the SysEx byte 0 and placing it at MSB of the parameter value (the second rule).

if I were supposed to do this with Lua, I would first convert all packed bytes to unpacked parameter values and then mapped them to the parameterMap. It means, you need to have some sort of parameter mapping implemented in the patch.onResponse() function.

I hope it helps.

2 Likes
3

Martin - every bit helps the understanding.

Currently I can receive and parse apart the sysex dump. Now I have a 100+ controls that need to be updated to display the current value before the user even attempts to change anything.

I understand that updating the Parameter Map is the way to use this, but my question remains – how do I know where in the parameter map to update the values? Is the parameter map a 1-for-1 mapping to Control ID?

EDIT - Looking at a pretty-print version of the EPR file shows me the parameter Number for each control. Time to do some coding to test out some ideas.

Thank you for being patient and answering all these questions.

EDIT 2 - success. parameterMap.set() updated the screen perfectly.

1 Like
4

Just one note, the parameterMap uses parameterType and parameterNumber as an index. So you may have CC 7 and NRPN 7 there and they will be seen as two different parameters.

5

good to know. I’ll pay attention to my types as well.

6

@martin Yeah I am definitely reading this :wink:
Trying to understand but so far I am not able to fully understand LUA and Sysex parsing.
But I have the suspicion that LUA might me required to get the AS1 fully working.

Went through the documentation.
But honestly it’s spicey and I haven’t found any online recourses describing how sysex byte decoding works on the web.
Looks more like every vendor does it differently and if its not documented your on your own.
I mean I am close I think. I had a look at the Sysex that a CTRL editor for the Yamaha Sy22/35 and TG33 spits out. But I have a hard time to understand how this bits are assembled sometimes. Especially when multiple bytes are used.

So far I had success only with single bytes.
Would be great to have a guide helping noobish users like decoding sysex.
I am willing to learn it and to write it.

small note: It looks the LUA documentation on Sysex Byte contains a v1 json example.

As I start I installed Lua for windows,

What ways do vendors use to pack 8 bits in 7byte values?
What LUA functions do we have to work with this?
Or do you need to write these yourself?

Here some C and Python examples:
https://blogs.bl0rg.net/netzstaub/2008/08/14/encoding-8-bit-data-in-midi-sysex/

2 Likes
7

to fully implement something, you need multiple pieces of the puzzle.
You need to know the type (CC, NRPN) and format (range of values, positive/negative/etc) of each parameter, you need to know how the data is sent and received (sysex dump, individual parameters) (AS-1 MIDI spec at the end of the manual provides this) and you need to know where in the dump data each parameter is located. That is the part I didn’t see yet.

You can attempt to map out where all the parameters are located, but that takes a lot of very patient exploration. The AS-1 data is 8 bit data packed in 7-bit MIDI data using the MS packing scheme.
So to explore where each parameter is located, you need to start with an Init preset, dump it, decode it to convert every 8 bytes into 7, then change one parameter on the AS-1, dump that, decode that, and compare to your baseline.

Once you do that for every parameter, you can then jump into the E1 and LUA part of it. Without the mapping of parameters into a sysex dump, life is difficult.

I have some basic LUA code to decode a packed 7-bit sysex dump into 8-bit bytes if needed.

1 Like
8

hmm, @Flyweight and @oldgearguy thanks :slight_smile: You mentioned something that really makes sense. I could extend the firmware with a function to unpack the 8bit packed data. It means, the user would work with 8bit parameters directly. Will give it some thinking…

@Flyweight, yeah, I understand, this gets quite technical. I would say that no matter if it is done in JSON or Lua, a good understanding of binary system and operations on it is needed. I will write (or steal somewher) a few lines of description and will add it to the documentation. Once, the concept is understood working with it is quite easy. The preset JSON - the rules - merely describe bitwise manipulations between SysEx bytes and parameter values.

As @oldgearguy mentioned, the biggest hassle of the AS-1 midi implementation is that the info about parameter locations in the SysEx dump is not provided. It needs to be reversed engineered…

PS: If you were willing to turn my simple howto into a more structured tutorial or something, it would be great!

1 Like
10

Deleted my post as Martin answered it. i read it too late.

Cheers

11

sorry for that :slight_smile:

I have Prophet 5 here. It uses the packed format too. I will use it to write the howto and also for the adding that to the firmware. What really sucks is that the document also does not provide info about the parameters in the sysex dump :frowning:

1 Like
12

yeah that would be great. no problem really about that post.

I can try to write this tutorial. it can help me understand.

Wow nice, really like the sound of the Prophet 5.
lucky you :slight_smile:

Decoding a sysex patch is really hard especially if there are many Parameters.
I was wondering if it can be somehow automatized.
Since the electra one I am reading sysex specs and some devices give you a nice parameter map with byte offset.
But once a Electra preset can read patches that it adds so much value. to be able to see the whole patch.

13

Here’s what I wrote to unpack the Alesis Micron data.
Note that the “74” is one past the amount of beginning bytes I need to skip in the dump because it has the standard sysex header/manufacturer id info, etc. The packed info starts at that byte (arrays number starting at 1 not 0 in LUA). I used whatever naming conventions were in my head at the moment, no standardization here.

The basic idea is to move through the buffer in 8 byte chunks. The first byte read has the collected high bits for the next 7. k is simply a linear counter for placing the expanded bytes into the array.

function patch.onResponse (device, responseId, sysexBlock)
    print ("Received sysex, length = " .. sysexBlock:getLength ())

    local deSyx = {}
    local topBits
    local highBit
    local k = 1
    local skipBytes = 74

    -- decode the received data, starting after heard data
   sysexBlock:seek(skipBytes)
    for i = 1, (sysexBlock:getLength () - skipBytes), 8 do
       topBits = sysexBlock:read()
       for j = 1, 7 do
          highBit = ((topBits << j) & 0x80)
          deSyx[k] = (highBit | sysexBlock:read())
          k = k + 1
       end
    end
    -- call functions or put code here to do useful things with the unpacked data in deSyx
end
1 Like
14

Since manufacturers do all kinds of crazy things in sysex dumps (I’m dealing with 14 bit, 8 bit and masked bits packed into bytes or that wrap around bytes), a basic function to handle the few standard packing schemes used might save user error headaches. Taking the “8 bytes to represent 7” format (I’m sure there’s a real name for it) and expanding it out into full 8 bit bytes with a single call would be nice.

However, the bulk of work that is slowing me down cannot be built in (I don’t think). Parameter value range 0 … 6, but the numbers coming from the device are negative and positive around 0 since what is on the screen is -3 … 0 … +3, so I have to two’s complement the value if it’s less than 0 and then subtract 6 for values greater than zero.

Or, 3 bits spread over 2 bytes are combined to produce the one set of values for a particular parameter.

All these special cases are not clearly documented in the minimal sysex information and you still need to have the actual device in front of you to match up the named parameter values and their order to the underlying bytes data in the dump.

Maybe by this weekend you can look through my Micron LUA code to see all the headaches conquered.

lol

2 Likes
15

I think this case is also happening for some of the parameters on the AS1.
But looks more like if bit is set in lsb value negative and positive if its not set.

Its definitely getting quite technical for me.
I am searching the web to get more info but haven’t found anything that really describes how manufacturers utilize sysex bytes and how to encode or decode them.
your post is very helpfull as well as: GitHub - francoisgeorgy/BS2-SysEx: Reverse engineering of the Novation Bass Station II SysEx format.
Guess I have to study bitwise operations… not sure.

Would it be possible to document on all cases how sysex bytes can be used? perhaps with some example?
I know there are some examples in the electra one docs, like checksums, multibyte.

16

The best way is to see some mappings and then how those translate into raw data and then into values for the end user. The link you provided for the BS II is great for someone doing a BS II implementation. When I get around to collecting up all the Micron info I have gathered, it’ll be useful for someone else doing an Ion/Micron/MiniAK implementation.

However, trying to do a direct copy/paste for another piece of gear is probably not going to work easily. The better approach is to look over the examples and see how it’s done and then apply that knowledge to your situation.

Just for accounting purposes so far, getting spun up on the Electra One hardware/firmware, learning enough scripting and LUA to get started, and getting a basic working preset (6 pages) up has taken me close to 80 hours so far and I’ll likely end up doubling that or more trying to parse and present the patch dump data in the E1 UI. The slowdown is first understanding how negative numbers, multi-byte values, and other things are represented in the MIDI dump, in the machine itself, and then in the ElectraOne. Next is all the edge cases and then trying to leverage the capabilities in the E1 to display nice info to the user as they move a control (for example delay time in Hertz rather than raw 0 to 1000 numbers).

My day job is software development and I’ve also done a fair amount of MIDI programming, so basics like hex math, masking, conversions, programming logic and structures have already been learned. This is not meant to discourage, but rather to set realistic expectations.

Presets that are fully functional and talk to the gear and pull down the current patch and send updates back are not easy to create. Creating controls that send changes to the hardware are easier to do at the start.

IMHO, after using computer editors, hw like the Novation Remote Zero/Behringer/Peavey PC1600, etc and platforms like the Lemur, the ElectraOne is the best, most full-featured, capable platform to deliver a seamless editing experience, so it’s worth investing the time to really learn it.

3 Likes
17

Yeah I know but I have come pretty far with analyzing and parsing the patch data.
It would be a pity If I am not able to ingest and parse the full patch.
But I do understand that learning is required.
I am not a developer but I am a Linux dude and can write some scripts in Bash/awk/python.

Today I was reading about masking and bitwise operations.
And now you gave me more homework. hex math :slight_smile:
Lets see how far I will get.
Anyhow thanks a lot for your replies. Very useful info!

Cheers

Tim

2 Likes
18

I’m observing all the efforts here :slight_smile: …to get an Idea on how to patch dump the Shruthi-1 and Ambika patches.

I’m also no dev but did some programming as a student and like to dive into it again with the Electra platform as a good cause.

I did the same as Flyweight and changed one parameter of the Shruthi patch and looked at the hex editor… found infos about the patch structure, initial sysex header end and changed one parameter via JSON rules afterwards but had no luck with a 1to1 value read out. I get the change but 0-127 never translates to the same value on the readout… the parameter seems to be packed. Émilie of Mutable Instruments calls it nibblized in the manual…

There is some info online about the patch format and I found a decoding function in the open source code of the IXOS editor for the Shruthi, which reads and sends patch dumps:

It’s named: “AudioProcessorShruthi::decodeSysexPatch”

And of course all the shruthi source code about the parameters is also online:

Thats’s the end of my sysex journey at the moment - it’s all a really big puzzle for me too :wink:

19

the Shruthi code doesn’t look that bad (from my perspective of course).

This is the thing – someone could have an almost fulltime job getting folks started on creating presets for their gear. Often it would just take a few back and forth sessions to get the core understanding down and you’d be on your way. Unfortunately, that’s not a paid position and therefore helping folks out in detail often takes a back seat to day job, family, personal projects, and other things life throws at you. Specific example - I’m pretty far along on my own preset creation (sending, receiving, parsing, displaying patch dumps), but there’s still a lot to do and I won’t have an hour of free time until Sunday afternoon (today being Wednesday) to look at it again. That’s just how it goes.

Getting back to the Shruthi, you should be able to change one of the parameters mapped to an early part of the dump and see the result change. The first thing I try is go the other way. Dump a Shruthi patch, define an ElectraOne control to change a value, twist the control and see if the Shruthi changes. If it does, dump that and see where the change is made. That tells you that you have proper communication and the proper CC/NRPN selected.

I’ll see if I can carve out a bit of time over the next couple days to at least get you started on changing and seeing the change for one parameter.

I love a product called HexWorkshop for looking at raw sysex dumps. It’s an older tool, but it can calculate checksums and do all kinds of other useful manipulations, but anything, even MIDI-Ox is useful

1 Like
20

I will step back a little bit for a moment. I had three goals when I designed the original preset JSON format:

  1. to be able to compose and parse SysEx messages of the majority of synths
  2. to be able to generate MIDI implementation documentation out of the JSON
  3. generate the preset JSON programatically

if the 3rd point was not part of the goal, the Lua scripting could be used for all SysEx messaging. What I wanted to achieve, however, was to make working with SysEx accessible to non-tech users. We are obviously not there yet :slight_smile:

There is a line of work that follows that goal though. @tomas released the preset versioning yesterday. It is quite a big change. One of the next steps in the editor is to add an embedded JSON editor for the Patch and SysEx data. When this is done, the irritating need to export and import presets that are manually edited should be minimized. Once the JSON editor is there, I would like to change it, so that users can choose between editing JSON as a text or using something more intuitive.

ok, the intuitive way… imagine the following workflow:

  1. you open our “future” SysEx patch dump editor in the app.electra.one.
  2. It will ask you to type in the bytes to request the patch dump.
  3. so you do so and you hit “send request”.
  4. if the request bytes are correct, the web app will receive the SysEx dump from the connected synth and will save the request bytes to the patch object in the JSON
  5. now, with the SysEx patch dump received, the received bytes can be show in the form of an indexed table. The display would include Dec, Hex, Binary, and ASCII representation of each byte. Next to that, there would be an option to register a new parameter on the screen.

image
image1920Ă—1849 414 KB

  1. so you create the parameter, eg. a filter Cutoff
  2. now, you could selected the cutoff parameter and assign the bits and bytes from the SysEx dump to it.
  3. you could also, re-request the SysEx dump to see changes if you change the parameter on the synth.
  4. once you feel the parameter is mapped correctly, you would save it to the JSON patch and would move onto the next parameter.
  5. the SysEx bytes assigned to the parameters would be highlighted in a way. To give you an indication what is used already and if everything is covered.

I am very sure that above could easily produce the the patch JSON.

If I were to take this further, I can imagine having an extra logic that could detect common patterns such as nibbelized bytes, packed bytes, etc. One would help the “learning” process by tweaking the parameters on the device. Or even the editor could do the tweaking.

All this is also closely related to another hidden gem - the instrument file. The preset file format was made and designed to describe the studio or rig setup. Say, I have Virus B, MKS-50, P5, DP/4, and MX-1 in my rig. So, I could have a preset where I have controls for all these devices and I can control them from there. That is why the preset has all that stuff around devices. The instrument JSON file, on contrary, is a description of the MIDI implementation of a particular MIDI device. It describes the parameters, patch requests, responses, defines the list of values, etc.

When creating a preset, one can start from the scratch and define everything in the preset. Another option is to load the instrument files for the devices I own, and simply drag and drop the controls to the preset layout. All the technical info would be taken out of the instrument file definition.

The time restrictions and also the ease of creating presets kind of pushed the Instrument files to the back. They are supported by the editor but there are basically no tools to create them. The good thing is that it is possible to generate Instrument files out of existing presets.

If I go back to the original topic, I would like to say that @oldgearguy said it right. Some education and better docs/tutorials are needed. Majority of my time still goes to making the firmware faster, more stable, and easier to use. I am also working in parallel on the opensource API that would other software developers a way to help me or develop their own Electra applications. But I will try to find some moments to write something every now and then. I heard @Flyweight’s offer to take part on that and I saw his new github account :). I am sure we will find a way to make something. Anybody else is welcomed to join. Pls, note that all the documentation on docs.electra.one is available on the github and can be edited and PRed.

And the very last comment. Quite a while back, @Ramyt talked to me and he was very enthusiastic about Electra as kind of a device to learn stuff. His input was one of the many that led to adding the Lua extension to the firmware. From this perspective Electra can be actually seen as a device to program, script, learn programming, and have fun in the domain of MIDI. It is not only about presets and mapping parameters, current firmware already provides a way to invent and add things such as MIDI LFOs, triggering notes, seqences, and lot more.

3 Likes
21

@martin , that’s definitely a vision of a product that really would make it easy for users to create presets for their gear and have them interact with it in a natural way.

Obviously there has been a lot of thought about the overall environment of the ElectraOne and how various pieces contribute to that approach. But, when you release things into the wild, they get used in unexpected/unplanned ways. :grinning_face_with_smiling_eyes:

So at this point, I guess I should ask about your vision versus my current approach. What I have been doing is putting all the “update the parameterMap” code in the LUA section and not doing anything with the patch[] construct for each control. I am doing it this way for a number of reasons, but it seems like that was not the original intent of all this.

For me, editing raw JSON is not fun. pure and simple. All the extra quoting and tagging and nesting and such is tedious to keep typing. What I have been doing is using your editor to arrange the controls and basically having them be “dumb”. They read and write the values given to them; they have no awareness of the structure of the patch dump itself.

In my patch.onResponse() section is where all the heavy lifting is taking place. This means that all that code is tied to the patch dump format and the NRPNs are what ties that data to the controls and parameterMap.

I read about Instrument files, but didn’t see immediately how to create them or incorporate them into what I was doing. The idea that a single preset might be able to manage multiple synthesizers is interesting. However, right now, just for one synth, my memory use is close to 70% and I’m not done with it yet.

I also still don’t see how to use the control-based patch[] construct to receive and process a full patch dump. If I put the received header sysex bytes info in it, every control would be sent the full patch dump and then they would have to unpack it and then index into the specific part they needed. To me, that seems like a lot of processing and memory overhead and it was better to do it all in one place in the single onResponse() for the device.

Comments?