Adding patch parsing to a preset. Step by step

Thank you again. I have contacted the manufacturer to see if they would be willing to share some info about the structure of their SysEx messages. Hopefully they will provide the needed info, although the last manufacturer I asked for SysEx info said no, go away, that’s proprietary. :confused:
Anyway, I did notice that xx & yy changes if I edit and save a patch - so it is probably some kind of checksum? I figured out that the 32 bytes before xx are for the patch name. All the bytes before that and after the header are straightforward 7-bit mappings of the NRPNs (170 bytes worth). There are a couple of masked NRPNs but they are structured like 00x0 00x1 000x and 001x, so I don’t think I can use your nifty bitmask code to separate them - I have to use a control that allows all the combinations: off/off, off/on, on/off, on/on based on the value of the corresponding NRPN. But not a huge problem - it works.
It’s also not the end of the world if I can’t get the SysEx to work for writing patches. I still have the options of saving an edited patch as a Snapshot on the E1, and of saving the loaded patch on the E1 (which is then in the buffer of the synth?) into the current patch slot on the synth using a button on the synth (although it will retain the name of the patch in that slot). In the mean time I will continue investigating using your list.

1 Like

Experimenting with the following Sysex message for patch 50:
F0 00 21 11 05 20 03 32 24 3F 3F 00 3F 00 01 00 00 7F 00 40 18 3F 3F 00 3F 00 01 00 00 7F 00 40 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 02 02 00 00 00 00 00 40 40 40 40 01 01 7F 01 01 01 00 01 01 01 01 00 00 00 7F 00 01 7A 00 00 00 00 00 00 07 01 00 00 02 18 02 03 01 00 00 00 00 00 00 00 00 00 0B 00 00 00 00 00 00 0B 00 00 00 00 00 00 0B 00 00 00 00 00 00 0B 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 06 00 00 00 00 00 00 00 00 00 00 00 00 00 14 00 0A 14 00 0A 14 00 00 0F 35 30 5F 49 6E 69 74 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 11 53 F7

This message successfully overwrites the patch in slot 50 on the synth. But if I change any data value it does not. This includes changing xx or yy.

If I try a slot other than 50 it does not overwrite the patch in that slot - which makes sense I suppose since this is patch 50. But if I change 32 to 33 in the header it overwrites the patch in slot 51.

When sending the message (including edited ones) I always receive the response:
F0 00 21 11 05 21 F7

I changed one parameter (bolded) by a lot (from 122 to 36) and saved it:
F0 00 21 11 05 20 03 32 24 3F 3F 00 3F 00 01 00 00 7F 00 40 18 3F 3F 00 3F 00 01 00 00 7F 00 40 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 02 02 00 00 00 00 00 40 40 40 40 01 01 7F 01 01 01 00 01 01 01 01 00 00 00 53 00 01 24 00 00 00 00 00 00 07 01 00 00 02 18 02 03 01 00 00 00 00 00 00 00 00 00 0B 00 00 00 00 00 00 0B 00 00 00 00 00 00 0B 00 00 00 00 00 00 0B 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 06 00 00 00 00 00 00 00 00 00 00 00 00 00 14 00 0A 14 00 0A 14 00 00 0F 35 30 5F 49 6E 69 74 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 10 51 F7

Notice that both xx and yy changed :frowning:

I changed one parameter (bolded) by a little (from 122 to 121) and saved it:
F0 00 21 11 05 20 03 32 24 3F 3F 00 3F 00 01 00 00 7F 00 40 18 3F 3F 00 3F 00 01 00 00 7F 00 40 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 02 02 00 00 00 00 00 40 40 40 40 01 01 7F 01 01 01 00 01 01 01 01 00 00 00 53 00 01 79 00 00 00 00 00 00 07 01 00 00 02 18 02 03 01 00 00 00 00 00 00 00 00 00 0B 00 00 00 00 00 00 0B 00 00 00 00 00 00 0B 00 00 00 00 00 00 0B 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 06 00 00 00 00 00 00 00 00 00 00 00 00 00 14 00 0A 14 00 0A 14 00 00 0F 35 30 5F 49 6E 69 74 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 11 26 F7

Notice that only yy changed ::confused:

Did I do everything on your list?

1 Like

I’m going for a trip away until mid next week, so don’t worry if I don’t deep dive or reply in the mean time.

For the change from 32 to 33… are you now saying you did that change manually , without changing xx and yy, and the synth accepted it ? Can you do the following then: change every subsequent bit after 32, until the synth does not accept it anymore (a way to discover the first byte that impacts the checksum). Also do the same for any byte preceding the xx and yy. I bet / hope they will all be doing nothing, which would mean we have the full scope that impatcs that checksum.

From the 2 changed values : notice byte 3 before the one in bold also changed from 7Fh (127) to 53h (83) , rendering the test less reliable. Can you try repeating while only one byte changes?

Last test : take the same 20 command message and throw away 4 or 5 bytes after the adress portion and send it. The message thus becomes too short… do you still get the same F0 00 21 11 05 21 F7 response?

1 Like

Sorry on accidently changing 2 parameters. Bad scientist. Here are the correct comparisons.
Original:
F0 00 21 11 05 20 03 32 24 3F 3F 00 3F 00 01 00 00 7F 00 40 18 3F 3F 00 3F 00 01 00 00 7F 00 40 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 02 02 00 00 00 00 00 40 40 40 40 01 01 7F 01 01 01 00 01 01 01 01 00 00 00 7F 00 01 7A 00 00 00 00 00 00 07 01 00 00 02 18 02 03 01 00 00 00 00 00 00 00 00 00 0B 00 00 00 00 00 00 0B 00 00 00 00 00 00 0B 00 00 00 00 00 00 0B 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 06 00 00 00 00 00 00 00 00 00 00 00 00 00 14 00 0A 14 00 0A 14 00 00 0F 35 30 5F 49 6E 69 74 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 11 53 F7

Small change: 7A to 79:
F0 00 21 11 05 20 03 32 24 3F 3F 00 3F 00 01 00 00 7F 00 40 18 3F 3F 00 3F 00 01 00 00 7F 00 40 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 02 02 00 00 00 00 00 40 40 40 40 01 01 7F 01 01 01 00 01 01 01 01 00 00 00 7F 00 01 79 00 00 00 00 00 00 07 01 00 00 02 18 02 03 01 00 00 00 00 00 00 00 00 00 0B 00 00 00 00 00 00 0B 00 00 00 00 00 00 0B 00 00 00 00 00 00 0B 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 06 00 00 00 00 00 00 00 00 00 00 00 00 00 14 00 0A 14 00 0A 14 00 00 0F 35 30 5F 49 6E 69 74 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 11 52 F7

Big change 7A to 24:
F0 00 21 11 05 20 03 32 24 3F 3F 00 3F 00 01 00 00 7F 00 40 18 3F 3F 00 3F 00 01 00 00 7F 00 40 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 02 02 00 00 00 00 00 40 40 40 40 01 01 7F 01 01 01 00 01 01 01 01 00 00 00 7F 00 01 24 00 00 00 00 00 00 07 01 00 00 02 18 02 03 01 00 00 00 00 00 00 00 00 00 0B 00 00 00 00 00 00 0B 00 00 00 00 00 00 0B 00 00 00 00 00 00 0B 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 06 00 00 00 00 00 00 00 00 00 00 00 00 00 14 00 0A 14 00 0A 14 00 00 0F 35 30 5F 49 6E 69 74 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 10 7D F7

Yes. I manually changed 32 to 33 without changing xx or yy and the synth accepted it. Changed the next byte from 24 to 23:
F0 00 21 11 05 20 03 32 23 3F 3F 00 3F 00 01 00 00 7F 00 40 18 3F 3F 00 3F 00 01 00 00 7F 00 40 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 02 02 00 00 00 00 00 40 40 40 40 01 01 7F 01 01 01 00 01 01 01 01 00 00 00 7F 00 01 7A 00 00 00 00 00 00 07 01 00 00 02 18 02 03 01 00 00 00 00 00 00 00 00 00 0B 00 00 00 00 00 00 0B 00 00 00 00 00 00 0B 00 00 00 00 00 00 0B 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 06 00 00 00 00 00 00 00 00 00 00 00 00 00 14 00 0A 14 00 0A 14 00 00 0F 35 30 5F 49 6E 69 74 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 11 52 F7
Then from 23 to 22:
F0 00 21 11 05 20 03 32 22 3F 3F 00 3F 00 01 00 00 7F 00 40 18 3F 3F 00 3F 00 01 00 00 7F 00 40 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 02 02 00 00 00 00 00 40 40 40 40 01 01 7F 01 01 01 00 01 01 01 01 00 00 00 7F 00 01 7A 00 00 00 00 00 00 07 01 00 00 02 18 02 03 01 00 00 00 00 00 00 00 00 00 0B 00 00 00 00 00 00 0B 00 00 00 00 00 00 0B 00 00 00 00 00 00 0B 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 06 00 00 00 00 00 00 00 00 00 00 00 00 00 14 00 0A 14 00 0A 14 00 00 0F 35 30 5F 49 6E 69 74 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 11 51 F7
Is this a pattern that will help to figure out the checksum formula?

The 32 bytes immediately before xx are for the patch name. If I change any of them without changing xx and/or yy the synth doesn’t accept the message. So, I believe the full range of bytes impacted by the checksum is from the one immediately after the patch number to the one immediately before xx. This would line up exactly with the size of the NRPN list I have. The message is 213 bytes. Is this the structure?
F0 (1 byte)
Header (6 bytes)
Patch number (1 byte)
NRPNs: MSB 5, LSB 0-127, MSB 6 LSB 0-73 (202 bytes)
Checksum (2 bytes)
F7 (1 byte)

Finally, I receive a response from the synth for any shortened message down to and including
F0 00 21 11 05 20 03 F7 . But if I remove 03 I get no response. Also, as soon as I remove any bytes the response changes from F0 00 21 11 05 21 F7 to F0 00 21 11 05 22 F7 (which may be the synth’s way of saying “I heard you, but no!”)

Please have a great trip!!

Some progress. I think I have deduced how yy is computed:

  1. Convert the 202 bytes of hexadecimal after the patch number before xx to decimal
  2. Add them up
  3. Divide by 128 and compute the remainder
  4. Convert the remainder to hexadecimal

Searching around, this seems close to the “Roland” checksum method, but without subtracting the remainder from 128 before converting back to hexadecimal?

I have tested for several parameter changes and this algorithm has worked.

But what about xx? yy always changes, but for small parameter changes it appears that xx does not change - at least, I haven’t found that yet.

1 Like

Eureka!. xx is the integer part of the quotient in step 3 expressed in hexadecimal (which is why it doesn’t usually change for small parameter changes); yy is the remainder part expressed in hexadecimal. Algorithm:

  1. Convert the 202 bytes of hexadecimal after the patch number before xx to decimal
  2. Add them up
  3. Divide by 128
  4. xx is the integer part of the quotient of step 3 in hexadecimal
  5. yy is the remainder part of the quotient of step 3 in hexadecimal
1 Like

You’ve made good progress. Yes it looks like the 22 message is an error response, and the 21 message is an acknowledgment response.
I understand you are now able to parse any byte the synth sends, including the packed parameters? And you are able to send the preset dump request from the E1 to the synth?

Which would mean we should now embark if the synth allows to receive the instruction ‘save the current buffer memory into a preset slot’. Perhaps that editor that came with the synth does send out such instruction. It would be a fairly short message anyways, as it doesn’t send any data except the slot to save it in.

As for the construction of the 20 command: when I’m back, I’ll look for some examples on how to set up a function for the checksum calculation, how to reconstruct a byte out of packed parameters, and finally reconstruct the full sysex message. I’ve done that already in some of my presets, but forgot in which ones :thinking:

2 Likes

Hello :slight_smile: . Yes I have parsed every byte the synth sends, including the packed parameters and the 32 bytes for the patch name. I have also mapped every byte to a control on the E1 (except for 13 dummy bytes that are set to 0 by default).

When I press the patch request from the E1 all the mapped controls on the E1 are populated.

Unfortunately, the synth’s editor does not appear to send a “save the current buffer to the current slot on the synth” message. Instead, if you upload a patch from the editor’s librarian page to the synth, or write a saved patch from the edit page it sends the full “20” sysEx message. Currently, the only way to save the buffer seems to be on the synth itself.

So, I agree the next steps are to:

  1. program a function for the checksum calculations.
  2. reconstruct the full sysEx message to send, and a function to send it.

FYI, because the developer didn’t fully specify the byte for each packed parameter I can’t think of a better way than to have a list control to select from all the possible combinations. For example, instead of something unique like [0 0] => A, [0 1] => B, [ 1 0] => C, and [1 1] => D, they have [x 0] => A, [x 1] => B, [ 0 x] => C, and [1 x] => D. So I have to allow for 4 combinations: 0: A&C, 1: B&C, 2: A&D, 3:B&D. That is how I have it programmed and it seems to work. But if there is a better way …
Cheers

1 Like

I think this function works for computing the two checksum bytes xx and yy. It is modified from a previous post:

function calcChecksum(payLoad)  
  local result = 0
  for i= 1, #payLoad do
    result = result+payLoad[i]
  end
  local xx = math.floor(result/128)  -- integer part of integer division
  local yy = result%128             -- remainder part of integer division
  return {xx, yy}
end

An example (using the concat function):

function concat(t1,t2) -- function to concatenate two arrays
  local table = {}
  for i=1,#t1 do
      table[#table+1] = t1[i]
  end
  for i=1,#t2 do
      table[#table+1] = t2[i]
  end
  return table
end

local patchNum = 50 -- example for patch 50 
local commandArray ={0x00, 0x21, 0x11, 0x05, 0x20, 0x03, patchNum} -- header and patch number of Sysex message`
local payLoad = {96, 58, 126}  -- parameter values in order of NRPNs (example with 3 bytes instead of 202)
local checkSum = calcChecksum(payLoad)
local array = concat(commandArray,concat(payLoad,checkSum))
for i = 1,#array do
 print(string.format("A: %X", array[i]))
end

--midi.sendSysex (port, array)

If this is on the right track then my last stumbling block is creating the payLoad array.

In my zoom MS70 CDR preset, I have implemented a fairly extensive sysex constructor, because in that device even one single control value was torn apart and spread over different bits found in several bytes.
So it is certainly too (and unnecessarily) complex to use as a the sysex constructor for your purpose, but maybe it may already give you ideas.

1 Like

In the TB-3 preset Electra One App you’ll find examples of creating midi sysex messages including the use of the roland checksum

1 Like

Here it is. Patch saving to the synth via SysEx has been implemented.
Future Impact V2
I definitely could not have done it without all the help though :pray:.
BTW, this is actually a nice little analog monosynth trapped inside a guitar pedal. It has essentially no user interface other than via its old school PC patch editor and a handful of cc mappings. Having it fully integrated with the E1 dramatically improves its usability as a synth instrument in its own right. Thanks again!

1 Like