Tutorial - first attempt

[oldgearguy]

This is a first attempt at providing an example of how to parse system exclusive dumps.

You will need to load the Sysex Example preset found in the Preset Library and unzip the attached sysex_tutorial file.

The main thing you need is the sysex_dump file.
Bring up the web editor, go into the Console, and load the file into the bottom window. Once the preset is loaded in the Electra One, send the sysex dump to it and you should see the screen change values to match what was in the dump.

This tutorial uses LUA extensively to parse the sysex data. Maybe someone can use the updated JSON editor to provide a parallel example using the same dump file eventually.

sysex_tutorial.zip (139.9 KB)

This example focuses on the “7 in 8” system exclusive packing scheme where 7 bytes of data are converted and stored as 8 bytes of MIDI sysex. Other schemes are “1 in 2” where 1 byte of data is stored as 2 bytes of MIDI sysex (4 bits per byte), and “1 to 1” where each byte of data maps directly to 1 byte of MIDI sysex. There’s also more esoteric things like the “full stream” used by some early MIDI products (Lexicon LXP series or Alesis Quadraverb - I forget exactly) where they concatenate every byte of data and then break it into 7 bits chunks.

The reason for these schemes is that internally, devices use values for parameters that are larger than the MIDI range of 0 to 127 or negative or both. All MIDI data has to fit in 7 bit bytes and if all 7 bits are used, that is the number 127 so to send and receive large/negative numbers some system to pack that data into multiple bytes has to be used.

[martin]

@oldgearguy thanks for doing this. I am going to read and give it a try. Will provide my feedback on it.

[oldgearguy]

I’m certainly open to any/every suggestion to make it useful. Definitely going to clean up the existing controls/colors to make it slightly more appealing to the eye.

[oldgearguy]

I created a second example of using LUA to parse a system exclusive dump.
This example is found in the preset library: Sysex example 2

This focuses on a device that uses 2 MIDI system exclusive bytes to hold 1 full 8 bit byte of data. So, every 2 MIDI bytes = 1 actual data byte.

This device stores the data in ‘reverse’ order. The first byte of a pair holds the lower 4 bits and the second byte holds the upper 4 bits.

The LUA code has enough comments (I think) to be helpful in this case and also in the case where the byte pair is reversed.

As a small bonus, this preset also shows how I decided to display the current note name/octave by using a format() function rather than some lists or other approaches.

This is a standalone example that doesn’t actually do any patch requesting or parsing. Just mainly here to show how one might parse another system exclusive dump format.

[Flyweight]

Hey this is great stuff @oldgearguy I will give it a try at some point.

Thanks for doing this, thanks for sharing knowledge in general.

[Flyweight]

Have added you note formatter to the Ensoniq Mirage Masos template.

Now studying the patch parser.

Weirdly the sysex returned has variable sizes.

f0h 0fh 01h 03h

I get:

Crazy stuff…

[oldgearguy]

Can you capture 2 or 3 different full (F0 to F7) sysex dumps in MIDI Ox or Sysex Librarian?

If you can, and zip them up, I can take a look and maybe see why they are different.

Don’t capture them in the E1; I want to see the full raw data, just in case there’s some timing issue or something else that is changing the total number of bytes received.

[Flyweight]

I started with the configuration data block

According the manual its send as 2 nybbles with MS clear.
low nybble first.

I tried several versions but is this correct? I think not.

function makeSwapB(first, second)
   -- the AND operation (ampersand) captures only the 4 bits of each byte.
   -- the left shift operation moves the 4 bits into the proper place
   retVal = ((second & 0x0F) << 4) | (first & 0x0F)
   return retVal
end

(header so skip 4)
F0h 0Fh 01h 02h 0Fh 

02h 0Fh 01h 02h 00h 00h 0Bh 01h 
02h 00h 0Fh 01h 02h 01h 02h 00h 
0Fh 01h 02h 01h 02h 00h 0Fh 01h 
02h 00h 00h 40h 00h 04h 04h 05h 
00h 00h 00h 00h 00h 0Fh 0Fh 0Fh 
0Eh 00h 00h 00h 00h 01h 00h 00h 
00h 0Fh 0Fh 01h 00h 00h 00h 00h 
00h 00h -- should be byte 28 counting from 00
F7h

13:30:09.887 lua: we recieved some data. length = 63 13:30:09.888 lua: Received configuration data. length = 63

is it 63 due to the sysex start upcode not being counted?

I made a clone of the preset here.

https://app.electra.one/preset/K8zKK6hvWLwwnQF09krH

What ever I tried the reported values of lua seem of.
At least I get some data loaded to the parameters.

13:45:45.284 Midi::processSysex: matched response: responseId=0
13:45:45.284 lua: we recieved some data. length = 63
13:45:45.285 lua: Received configuration data. length = 63
13:45:45.285 lua: 242
13:45:45.286 lua: #
13:45:45.286 lua: position1
13:45:45.287 lua: 242
13:45:45.287 lua: #
13:45:45.287 lua: position2
13:45:45.287 lua: 33
13:45:45.288 lua: #
13:45:45.288 lua: position3
13:45:45.288 lua: 0
13:45:45.288 lua: #
13:45:45.289 lua: position4
13:45:45.289 lua: 27
13:45:45.289 lua: #
13:45:45.289 lua: position5
13:45:45.290 lua: 2
13:45:45.290 lua: #
13:45:45.290 lua: position6
13:45:45.290 lua: 31
13:45:45.291 lua: #
13:45:45.291 lua: position7
13:45:45.292 lua: 18
13:45:45.292 lua: #
13:45:45.292 lua: position8
13:45:45.292 lua: 2
13:45:45.293 lua: #
13:45:45.293 lua: position9
13:45:45.293 lua: 31
13:45:45.293 lua: #
13:45:45.294 lua: position10
13:45:45.294 lua: 18
13:45:45.294 lua: #
13:45:45.294 lua: position11
13:45:45.295 lua: 2
13:45:45.295 lua: #
13:45:45.295 lua: position12
13:45:45.295 lua: 31
13:45:45.296 lua: #
13:45:45.296 lua: position13
13:45:45.296 lua: 2
13:45:45.296 lua: #
13:45:45.297 lua: position14
13:45:45.297 lua: 0
13:45:45.297 lua: #
13:45:45.297 lua: position15
13:45:45.298 lua: 68
13:45:45.298 lua: #
13:45:45.298 lua: position16
13:45:45.298 lua: 5
13:45:45.299 lua: #
13:45:45.299 lua: position17
13:45:45.300 lua: 0
13:45:45.300 lua: #
13:45:45.300 lua: position18
13:45:45.301 lua: 0
13:45:45.301 lua: #
13:45:45.301 lua: position19
13:45:45.302 lua: 255
13:45:45.302 lua: #
13:45:45.302 lua: position20
13:45:45.302 lua: 239
13:45:45.303 lua: #
13:45:45.303 lua: position21
13:45:45.303 lua: 0
13:45:45.303 lua: #
13:45:45.304 lua: position22
13:45:45.304 lua: 0
13:45:45.304 lua: #
13:45:45.304 lua: position23
13:45:45.305 lua: 1
13:45:45.305 lua: #
13:45:45.305 lua: position24
13:45:45.305 lua: 0
13:45:45.306 lua: #
13:45:45.306 lua: position25
13:45:45.306 lua: 255
13:45:45.306 lua: #
13:45:45.307 lua: position26
13:45:45.307 lua: 1
13:45:45.307 lua: #
13:45:45.308 lua: position27
13:45:45.308 lua: 0
13:45:45.308 lua: #
13:45:45.308 lua: position28
13:45:45.309 lua: 0
13:45:45.309 lua: #
13:45:45.309 lua: position29
13:45:45.309 lua: 112
13:45:45.310 lua: #
13:45:45.310 lua: position30

[oldgearguy]

a couple quick things – I cannot load/view any presets during the day. My access to anything “https://app.electra.one” is blocked, so I cannot view the preset.

However, based on the debug dump, the code to process the incoming sysex is not skipping enough bytes. You are processing part of the header as data. Where you say “skip 4”, it should be “skip 6” because the first two outputs lua:242 and lus:242 should only be a single lua:242.

When you go to the Mirage, what is the actual value? Is it 242 or is it 47 ?
(0xF2 versus 0x2F)

Also, the way your print statements are formatted are generating the “#”. You either need to use:

print("here is my value " .. value)

or 

print(string.format("fancy output with a number: %d and a string: %s", myNum, myString))

[Flyweight]

@oldgearguy

is the actual value? Is it 242 or is it 47 ?
(0xF2 versus 0x2F)

When I use skip = 5 it works ! yes thanks.

How can I devide deSyx[5] by 2?

parameterMap.set(1, PT_VIRTUAL, 24, ( math.tointeger(deSyx[4]) / 2 ))
parameterMap.set(1, PT_VIRTUAL, 25, deSyx[5])

I get

15:36:36.782 error running function ‘onResponse’: ctrlv2/p011.lua:304: bad argument #-1 to ‘set’ (number has no integer representation)

Parameter [21] = 31 dec on the Mirage so should be 1F…

15:12:27.022 lua: we recieved some data. length = 63
15:12:27.022 lua: Received configuration data. length = 63
15:12:27.023 lua: value 47 byte position 0
15:12:27.023 lua: value 31 byte position 1 parameter [21] matches with Mirage 31
15:12:27.023 lua: value 2 byte position 2 parameter [22] matches with 2 on the Mirage
15:12:27.023 lua: value 176 byte position 3…doesn’t make sense…also on the Mirage it says n8…?
15:12:27.024 lua: value 33 byte position 4 …parameter [24] is 16 on the mirage… 33 / 2 … ah it msut be devide by 2 so it matches
15:12:27.024 lua: value 240 byte position 5… hmmm this says On on the Mirage… expected 1 or 0
15:12:27.024 lua: value 33 byte position 6… parameter [73] is 33…
15:12:27.024 lua: value 33 byte position 7
15:12:27.024 lua: value 240 byte position 8
15:12:27.024 lua: value 33 byte position 9
15:12:27.025 lua: value 33 byte position 10
15:12:27.025 lua: value 240 byte position 11
15:12:27.025 lua: value 33 byte position 12
15:12:27.025 lua: value 0 byte position 13
15:12:27.025 lua: value 64 byte position 14
15:12:27.026 lua: value 84 byte position 15
15:12:27.026 lua: value 0 byte position 16
15:12:27.026 lua: value 0 byte position 17
15:12:27.026 lua: value 240 byte position 18
15:12:27.026 lua: value 255 byte position 19
15:12:27.027 lua: value 14 byte position 20
15:12:27.027 lua: value 0 byte position 21
15:12:27.027 lua: value 16 byte position 22
15:12:27.027 lua: value 0 byte position 23
15:12:27.027 lua: value 240 byte position 24
15:12:27.027 lua: value 31 byte position 25
15:12:27.028 lua: value 0 byte position 26
15:12:27.028 lua: value 0 byte position 27
15:12:27.028 lua: value 0 byte position 28

[oldgearguy]

what I’ve used to divide by 2 is

newValue = math.floor(deSyx[5] / 2)

the deSyx array is all numbers - they are not characters/strings.

[Flyweight]

Great thanks a lot, with your documentation I hop to manage the patch parsing.

Its very good documentation.
Thanks for this.

[oldgearguy]

I decided to put this here, since it’s more of my general approach when facing a new system exclusive dump format.

This example uses the Ensoniq Mirage system exclusive dump. From Mirage it is 1255 bytes. If sent back to the Mirage, it will be 1256 because you need to add a checksum byte before the end of sysex (F7) byte.

The Mirage packs the data into 2 bytes and the bytes are flipped, so 0000LLLL 0000HHHH becomes HHHHLLLL when unpacked and combined.

Attached are a few files showing how I might go from a raw dump in MIDI Ox to something I can start mapping to parameters. They are all packed into one ZIP file (mirage_ex.zip) to make life a bit easier.

My typical tools are MIDI-Ox, NotePad++ (excellent for editing columns), and Hex Workshop hex editor.

The first file (sysex_orig.rtf) is the full raw dump, complete with start/end sysex bytes. I color coded each section that I care about: F0/F7, the few bytes of the Mirage header, and then all the bytes associated with each section of the dump. The general layout of the dump is attached as file mirage_data.jpg.

The second file (sysex packed.rtf) is after removing the sysex start/stop and header bytes and then applied the necessary transform on each pair of bytes (swap them and remove the zeros).

There’s an intermediate step I did not show, but effectively it goes like this:

03 0B 0E 0D 03 0B 00 0E ... becomes
030B 0E0D 030B 000E     ... becomes
0B03 0D0E 0B03 0E00     ... becomes
B3 DE B3 E0

finally, I add line breaks and some numbering for each section (sysex numbered.rtf)

Naturally, the individual sections can be broken down more completely, but at this stage you should be able to see the patterns and general areas of interest.

mirage_ex.zip (111.5 KB)