Sysex dumps, updating controls - what am I missing?

[martin]

@oldgearguy

So at this point, I guess I should ask about your vision versus my current approach. What I have been doing is putting all the “update the parameterMap” code in the LUA section and not doing anything with the patch construct for each control. I am doing it this way for a number of reasons, but it seems like that was not the original intent of all this.

I have seen your preset. It is an amazing piece of work. The most important thing to say is that what yo u have done is perfectly in line where I want to go. I talked a lot about the JSON in the previous post, but it was meant to describe the way of less-technical approach. Once can say that Lua can be used in three ways:

1. not at all, just JSON
2. spice up the preset with stuff such as formatters, channel handling, etc
3. take full control

You took the full control way. And that is perectly ok. 100% supported in the preset. For instrument files, I need to thing about having namespaces for each device. Otherwise, imported functions could clash. It is definitely doable though.

@oldgearguy

I read about Instrument files, but didn’t see immediately how to create them or incorporate them into what I was doing. The idea that a single preset might be able to manage multiple synthesizers is interesting. However, right now, just for one synth, my memory use is close to 70% and I’m not done with it yet.

Current firmware gives (allocates statically) substantial amount of RAM to the controls (all 432) even though they are not used. This is being reworked completely in the upcoming major release. The change will be quite radical in terms of the memory usage.

@oldgearguy

I also still don’t see how to use the control-based patch construct to receive and process a full patch dump. If I put the received header sysex bytes info in it, every control would be sent the full patch dump and then they would have to unpack it and then index into the specific part they needed. To me, that seems like a lot of processing and memory overhead and it was better to do it all in one place in the single onResponse() for the device.

There is an option to mark a Control to trigger a patch transfer (in JSON). Such control then uses the Patch rules to compose the SysEx message and send it out whenever the control’s value is changed. I used it for the Crumar Bit99 and worked fine. It did not document that and eventually it was removed from the editor as it confused users and brought more problems than benefits.

The processing power is sufficient, the issue, say with Bit99, was that it was a bit overwhelmed with the data

But again, what you have done in the Micron preset is perfectly ok and fully compatible with the direction I want to go in the future.

[oldgearguy]

Thanks for the thoughtful replies. The real takeaway for me is that this is the first external controller that gives a user the full capabilities to make a real remote editor/controller that interacts with the hardware in a seamless way.

Other attempts have been very limited and always frustrating in the end since there were limits on the size of the sysex transfers or a lack of programmability to handle things like unpacking the format or transforming the data.

I am very excited to see where this product goes and so far have been very happy with what it can do and more importantly with the results I am getting with it.

[studiobischof]

@oldgearguy thanks for your advice and offer!
I will post my new findings in the forum when I find time to tinker again

I already have established bi-directional nrpn communication between the ambika (shruthi) and the electra and got the change dumps from 0 to 127 for one parameter (used the midi dump tool by urbanspacman). I think I am stuck with the logic of unpacking the parameter values at the moment… But because the whole workflow with the json parsing rules, code implementation via sandbox into the preset is new to me, there could be other errors on my way. I.e. I spend quite some time in the sandbox trying to find some wrong set brackets while inserting the parsing rules…
So it would be very nice, if @martin will ease the sysex workflow in the future

Nevertheless I like to work on this topic when I find time next to making sounds and every discussion here on the forum helps me to understand the sysex stuff and electra workflow better. About the unpayed time spend tinkering with the electra… I think its a question of passion for good music interfaces… in german passion is called “Leidenschaft” which in some way means “creating suffering”… for the greater cause

I also tried to just use the IXOS software editors for shruti and ambika controlled directly via my ipad touchscreen but the electra really nails the UI with its low complexity visuals and knob feel. I just can concentrate on my ears instead of thinking to much about what’s on the screen… I think @martin really found the sweetspot here

[Flyweight]

@martin definitely such tool should engage more users to get their sysex templates up and running.
But it would make sense to define the workflow in more detail based on the input of some participants.
I would think the more ( hopefully ) bidirectional templates are available the more potential buyers.

What I did so far is:

• compare patch dumps
• create bidirectional communication for NRPN’s
• parsing sysex patch data, of 7bit values
• write sysex templates, sending 2 4 byte nybbles or 1 byte messages

Where I got stuck is with multi byte values, either to send or parse them.
Exactly the picture you posted. I did come close, using a calculator with hex, bin and decimal representation.
But I did not find a way (using json ) to get the desired range of 255.
I would think that this sysex formulas cannot be that complicated and if I had an example I would be able to make it work.

This is the first time ever I had to work with binary. For me its no fun untill it works. Then the fun will come Of course its fun to learn about midi a rock solid protocol older then I am

The Elelctra One is just Awesome! The first controller that can really talk to a XV5080 and its deep sysex implementation!

I am definitely up to write a tutorial how to decode sysex.
Or perhaps the full guide howto get a bidirectional template up for any synth.
But I do needs some guidance decoding the patch data. Howto assemble the varios bits into the correct values.

[oldgearguy]

@studiobischof - attached is a quick attempt at decompressing the Shruthi sysex dump.
In theory, if you copy and paste it into the LUA editor and send it to the ElectraOne (you need to have some preset already created) and then press the patch request button, you might see info in the Electra log file.

you also need to add the patch[] construct to your .epr file in the device section:
(assuming the Shruthi OS is post 0.94)
“patch”: [{ “responses”: [{ “id”: 1, “header”: [0, 33, 2, 0, 2], “rules”: [] }] }]

(see the Patch section of the ElectraOne LUA documentation if this isn’t clear)

shruthi.zip (1.9 KB)

[studiobischof]

Wow, thanks a lot @oldgearguy! I will test it out soon

[Flyweight]

@martin

Now, the MS bit packed format . To me, this SysEx dump format is on the edge of using Lua over the JSON rules. The reason is that every value needs to be composed out of two SysEx bytes (MSB bit + 7bits) and there is a repeating pattern in it.

With JSON rules that can be done using two rules to compose one parameter value:

{
    "type":"nrpn",
    "parameterNumber":4,
    "parameterBitPosition":0,
    "byte":7,
    "byteBitPosition":0,
    "bitWidth":7
},
{
    "type":"nrpn",
    "parameterNumber":4,
    "parameterBitPosition":7,
    "byte":0,
    "byteBitPosition":6,
    "bitWidth":1
}

These two rules are used to compose value of NRPN parameter 4 by taking 7bits out of SysEx byte at position 7 and placing them to LSB of the parameter value (the first rule) and adding one bit from the SysEx byte 0 and placing it at MSB of the parameter value (the second rule).

I think I start to understand this a bit
This and @oldgearguy post with the lua code example.

so the sysexdump of a MS packed dump contains a first byte MSB that contains multiple single bits from the 7 bytes that are comming after.
so 1 packed byte with the MSB’s in it for the next 7 bytes with the LBS’s in it? the LSB’s are also the lower bits since they are on the right.
now would the binary representation of the MSB reveal its byteBitPosition? and if you discover the byteBitPosition than no further unpacking is required?

[martin]

yes, you are getting there. Let me show that on a few drawings. Imagine you have received a sysex message that consists of number of bytes:

4, 5, 6 are the positions of each byte in the message.

say that these three bytes consist of data that form the values of two parameters. One 7-bit CC 60 and one 14-bit NRPN 81.

The transformation is done with the parsing rules defined in the patch response object.

Each parsing rule has two sets of elements:

1. elements to locate the bits in the sysex bytes
2. elements to specify where the bits located with above elements should be placed within the parameter value, thus to compose the parameter value

The Rule elements to locate the bits

    "byte":
    "byteBitPosition"
    "bitWidth"

byte says which sysex bytes will be parsed
byteBitPosition tells the start position of the bits to be parsed out (counted from the right, ie. LSB)
bitWidth number of bits to be parsed out (counted towards the MSB).

The above elements are used to identify and parse out a group of bits out of the sysex data.

a few examples:
say you will want to parse the bits marked with the thick outline, then you would use

    "byte": 4,
    "byteBitPosition": 0,
    "bitWidth": 7

    "byte": 5,
    "byteBitPosition": 0,
    "bitWidth": 4

    "byte": 6,
    "byteBitPosition": 4,
    "bitWidth": 1

so to sum it up:

is translated to

    "byte": 8,
    "byteBitPosition": 2,
    "bitWidth": 4

The Rule elements to compose Parameter value

    "type":"nrpn",
    "parameterNumber":4,
    "parameterBitPosition":7,
    "bitWidth":1

type is the type of the parameter that is being composed
parameterNumber is the number (identifier) of the parameter
parameterBitPosition position where the parsed bits should be placed (counted from the right, ie. LSB)
bitWidth number of bits to be parsed out (counted towards the MSB).

These elements instruct Electra where the parsed bits (see above) should be placed within given parameter.

examples:

say, that the 4 lower bits sysex byte 5 consist of value of the 7bit CC parameter.

{
    "type":"cc7",
    "parameterNumber":60,
    "parameterBitPosition":0,
    "byte":5,
    "byteBitPosition":0,
    "bitWidth":4
}

And now the 14-bit nrpn parameter that has its value distributed between sysex bytes 4 and 6 (quite similar to MS packed format).

byte 4 consists of lower (LSB) 7 bits of the parameter value, byte 6 consists of 1 bit of MSB part of the parameter value.

{
    "type":"nrpn",
    "parameterNumber":81,
    "parameterBitPosition":0,
    "byte":4,
    "byteBitPosition":0,
    "bitWidth":7
},
{
    "type":"nrpn",
    "parameterNumber":81,
    "parameterBitPosition":7,
    "byte":6,
    "byteBitPosition":4,
    "bitWidth":1
}

because the parameter value is composed of two separate parts (7bit and 1bit), two rules are needed. These two rules, however, refer to the same parameter (nrpn 81). Electra will interpret this by putting the two parts together (logical OR operation).

And now the homework part. The Mopho user guide says:

so getting the 8bit value of C is:

{
    "type":"nrpn",
    "parameterNumber":3,
    "parameterBitPosition":0,
    "byte":4,
    "byteBitPosition":0,
    "bitWidth":7
},
{
    "type":"nrpn",
    "parameterNumber":3,
    "parameterBitPosition":7,
    "byte":1,
    "byteBitPosition":2,
    "bitWidth":1
}

by doing this, nprn parameter 3 will be set to 8bit value. if it is just a simple value 0 …255. You are done. If it was for example a number with range -128 … 127. you will need to tackle the signedness, but that is done outside the parsing rules.

[Flyweight]

@martin

{
    "type":"cc7",
    "parameterNumber":60,
    "parameterBitPosition":0,
    "byte":5,
    "byteBitPosition":0,
    "bitWidth":14
}

If I understood it a little then this should be bitWidth: 4 right ?

Diky Moc Martine! Love the artwork as wel
This is great. I will study it further and use it in the guide/tutorial.

[martin]

typo fixed. yes, it is 4 not 14. graag gedaan Tim!

[studiobischof]

Hej @oldgearguy, I tried to run your lua script next to the preset file with the patch construct inserted (and also in another test with my old patch rules instead)…

Till now I get no print in the console log when pressing the request patch button on the electra… There should be at least one line from the lua function:

function patch.onRequest (device)
print (“Requesting current patch”)

I think I have to try some basic lua stuff first, to find out why I get no feedback from the button… Also found your questions about the lua examples not working in the forum Well another riddle to solve first!

I also tried the json rules to readout the filter cutoff again and I get only the first 15 values right and than it jumps back to 0. So I understand the HEX logic about that switching back to 0 after the first 16 values on position 0 but cannot setup the right composing rules to get the other position…

I have these two dumps from the shruthi - the first at value 0 the second at value 127. So in Hex this is 0x00 to 0x7F and nibblized for the shruthi it is 00 00 to 07 0F. That part I understand now.

My rule for getting the first 0-15 values from the shruthi dumps is this:

“devices”: [
{
“id”: 1,
“name”: “Shruthi Vanilla”,
“port”: 1,
“channel”: 2,
“patch”: [ {
“request”: [“00”, “21”, “02”, “00”, “02”, “11”, “00”, “00”, “00”],
“responses”: [ {
“header” : [“00”, “21”, “02”, “00”, “02”, “01”, “00”],
“rules”: [
{
“type”:“cc7”,
“parameterNumber”:14,
“parameterBitPosition”:0,
“byte”:25,
“byteBitPosition”:0,
“bitWidth”:7
}
]
}]
} ]
}
]

So I don’t know if it is even the right way to do it and how to compose a rule for 15+ values. @martin If you have a sec to clarify, that would be great.

[oldgearguy]

If you don’t see printouts in the log file, that’s because there’s a syntax error in the script. Since I just free-typed it, I could have mis-typed something for sure. I can’t play with the hw or even the environment at work…

You also have to save your preset, manually edit the EPR file, and insert that “patch stuff” correctly.
Attached is an example preset with the patch info pasted in the correct spot.

MyPreset.epr (596 Bytes)

Let me see what I sent you and see if I can debug it now at home.

I already see one typo – in the patch.onResponse, msgSize = … line, getlength() should be getLength()

[oldgearguy]

let’s look at the dump you provided. Based on the documentation, I think the actual data starts with the bytes:
00 09 01 0A 0F 04 in the first row.
00 09 should correspond to Oscillator 1 shape (value is 9)
01 0A should correspond to Oscillator 1 param (value is 1A which is 26)
0F 04 should correspond to Oscillator 1 range (value is F4 which is 244 OR -12)
To get -12, I subtracted F4 from 256 (in hex : 0x100 - 0xF4)
and so on

Is that matching what you are doing? Also in your preset, for your controls that can go larger than 127, make sure you set the MIDI range at the bottom correctly

[studiobischof]

Thanks again for your feedback @oldgearguy! Your sysex data analysis is right - the first 8 file bytes are the header and the null argument.

I just tinkered with composing the two bytes at place 24 and 25 after the header for the filter cutoff and got it working now with this code

“patch”: [ {
“request”: [“00”, “21”, “02”, “00”, “02”, “11”, “00”, “00”, “00”],
“responses”: [ {
“header” : [“00”, “21”, “02”, “00”, “02”, “01”, “00”],
“rules”: [
{
“type”:“cc7”,
“parameterNumber”:14,
“parameterBitPosition”:0,
“byte”:25,
“byteBitPosition”:0,
“bitWidth”:7
},
{
“type”:“cc7”,
“parameterNumber”:14,
“parameterBitPosition”:4,
“byte”:24,
“byteBitPosition”:0,
“bitWidth”:7
}
]
}]

Still don’t understand the logic why “parameterBitPosition”:4 is the right position but its late and I will study hex, bits and bytes tomorrow

[studiobischof]

@oldgearguy Still no luck with your MyPreset.epr and prints from the lua script on the console when pressing down the patch request button… I will check the syntax and some lua examples from github tomorrow. When I restart the electra some lua prints can be shown on the log, so lua should work (FW 2.1.3)…

Thanks for your help! It gets me motivated to try this stuff further
And I still don’t know exactly if I can read out all shruthi (ambika) values just by patch rules and formatting of the electra faders/lists without lua…

[oldgearguy]

Did you see how I assembled the one byte from two?

A byte has 8 bits. They are typically numbered ‘backwards’ like this: 76543210
The Shruthi dump puts 4 bits into two consecutive bytes like this:

byte[24] = 00007654 byte[25] = 0003210
So to put them back together, you need to take 4 bits from each and put one group after the other.

so you take byte [25] and put it at position 0 and you take byte[24] and put it at position 4.
note that in your code above, the “bitWidth” value should only be 4 in both cases since you are only using 4 bits from each byte.

Also - in my example EPR that I attached, the LUA code does run and print the initial message to my log, so maybe import my example as a new preset and see what happens.

[Flyweight]

Hey @studiobischof I am just like you learning to screw bits and bytes.
@oldgearguy and @martin provided some good info on this in this topic.

not sure if it matter but if 1 byte is nibylized shouldn’t it be ?

bits and bytes start at 0
bitWidth starts at 1

{
“type”:“cc7”,
“parameterNumber”:14,
“parameterBitPosition”:0,
“byte”:25,
“byteBitPosition”:0,
“bitWidth”:4
},
{
“type”:“cc7”,
“parameterNumber”:14,
“parameterBitPosition”:0,
“byte”:24,
“byteBitPosition”:0,
“bitWidth”:4
}

I am not using Lua and when you add a patch construct to your template you can send out a patch request by holding the top right button.
patch request sould appear on the bottom of the screen.
you can also just add a patchrequest button to your template.
in both case it will be detected based on static header bytes.

I am not sure why to use Lua to parse the patch.
only if you need to do any kind of manipulation to the data.
perhaps its required if you work with negative values?

[studiobischof]

Ah, @oldgearguy, your explanation of the four bits into two consecutive bytes for the shruthi makes perfect sense. Thanks again! I’ll give your .erp and lua code soon another try

@Flyweight Thank you too. I got the patch request working - following your preset journey. I think I don’t need lua functions for now… I’ll try to readout some more parameters of the shruthi first. But I’m also interested to learn the logic on how it works…

[oldgearguy]

if you get stuck, you can always send me a copy of your preset in its current form. Makes it easier to work from the same page versus temporary presets. Even though I don’t have a Shruthi, I can at least see what the preset sends for various values in the log file

[studiobischof]

Yay, thanks a lot @oldgearguy! I’m mostly in family mode this weekend… I will give you an update when in electra territory again