I’ve been exploring if I can get all the patch names from my synth on to the E1, but am having some problems working in Lua and running out of memory. Perhaps there’s a better way to do this. Here’s the code I’m using:
function dump(o)
if type(o) == 'table' then
local s = ''
for k,v in pairs(o) do
if type(k) ~= 'number' then k = '"'..k..'"' end
s = s .. dump(v) .. ' '
end
return s
else
return tostring(o)
end
end
function midi.onSysex (midiInput, sysexBlock)
print ("Received sysex, length = " .. sysexBlock:getLength ())
local theBlock = {}
for i = 1, sysexBlock:getLength () do
local sysexByte = string.format ("%02x", sysexBlock:peek(i))
table.insert(theBlock, string.upper(sysexByte))
end
print (dump(theBlock))
end
There’s a dump request message (which I’m currently triggering with a pad in the preset) that asks for the entire user bank of 32 voices that returns response that’s supposed to be 4104 bytes long (according to the manual). The E1 doesn’t print out anything. So I tried another request that’s supposed to be 2450 bytes long, and I get the response…
lua: Received sysex, length = 12723.0
error running function 'onSysex': not enough memory
…which is over 5 times the expected size. So I expect the other one is just too big. Is there another way?
yeah, this way of printing the sysex data is very memory hungry
with 12.7 kB long sysex message, theBlock will grow to at least 25kB, string s to around 40kB, with each concat (…) on s requiring minimum size of s * 2. concat creates new strings.
The important thing is that sysexBlocks live outside of the regular RAM. seek, peek, read should be used to parse the data out.
What is strange is that you are getting longer size of the received message. I just tried to send a few larger sysex messages to your script (with print (dump(theBlock)) commented out) and I always got what I sent in.
I’ve put a ‘print’ line in the loop, and with the 2450 size SysEx dump I get a bunch of zeros before the ‘real’ response starting with F0 comes through. This might partially explain why I get a different size every time and the array gets full. The only other thing I can think of is to request each patch one by one to get the patch names, but I’m unsure how (or if) that can be done.
Do you get those extra leading zeros with the 4k SysEx dump too? 4104 bytes sounds like tx7/dx7 bank dump. Is that correct? I work with these dumps often and it works just fine.
That’s right, it’s the TX81Z. It’s the largest bulk dump response, and it’s the only one with nothing coming back, not even the sysexBlock:getLength (). I’m sure the request from the E1 is right, because the response is getting to my laptop when triggered from the E1. Is it taking too long, perhaps?
I’ve been trying more times, and the requests called VMEM (the user bank) and PMEM (performance bank) seem to get cut off. Either getting nothing or only some of the bytes in sysexBlock:getLength()
yeah, if there are leading zeros but the data is in the sysexBlock, it means that had to be F0 before the zeros.
Adjusted your script a little bit:
function dump(o)
if type(o) == 'table' then
local s = ''
for k,v in ipairs(o) do
if type(k) ~= 'number' then k = '"'..k..'"' end
s = s .. dump(v) .. ' '
if (math.fmod(k, 32) == 0) then
print(s)
s = ''
end
end
return s
else
return tostring(o)
end
end
function midi.onSysex (midiInput, sysexBlock)
print ("Received sysex, length = " .. sysexBlock:getLength ())
local theBlock = {}
for i = 1, sysexBlock:getLength () do
local sysexByte = string.format ("%02x", sysexBlock:peek(i))
table.insert(theBlock, string.upper(sysexByte))
end
print (dump(theBlock))
end
And I get a different result every time. Sometimes fewer than 4104 bytes, sometimes more, and a different amount ‘zeroed’ out. Sometimes I get the out of memory error, sometimes nothing.
Haven’t had any luck getting past the problem, but I’ve written a script that extracts the patch name. It works fine for the current voice, but the E1 seems to choke on the 4K size 32 voice user bank. As before, when the 4K response comes back to the device, it often won’t even run the print command. This time when I run other SysEx requests I can see that the first patch name in the user bank has been saved, but that’s it.
function dump(o)
if type(o) == 'table' then
local s = ''
for k,v in ipairs(o) do
if type(k) ~= 'number' then k = '"'..k..'"' end
s = s .. dump(v) .. ' '
if (math.fmod(k, 32) == 0) then
print(s)
s = ''
end
end
return s
else
return tostring(o)
end
end
-- ------------------------------------------------------
list_midiChan = controls.get (20)
patchNames = {}
function midi.onSysex (midiInput, sysexBlock)
print ("Received sysex message " .. sysexBlock:getLength() .. " bytes long.")
local byte = ""
local bytes = ""
local voice = 1
local pointer = 1
local chan = string.format("%02x", 0x00 + ( list_midiChan:getValue("value"):getMessage():getValue() ))
local VMEM_header = "f0 43 " .. chan .. " 04 20 00 " -- TX81Z User Bank header
local VCED_header = "f0 43 " .. chan .. " 03 00 5d " -- TX81Z Voice Edit header
repeat
byte = string.format("%02x ", sysexBlock:read()) -- read byte & convert to hex
bytes = bytes .. byte
pointer = pointer + 1
until pointer > 6 -- capture header (6 bytes)
if bytes == VMEM_header then
for i = 1, 32 do -- loop through 32 patches in the user bank
pointer = pointer + 57 -- skip 57 bytes (patch info before name)
sysexBlock:seek (pointer)
bytes = ""
local letter = 1
repeat
byte = string.char(sysexBlock:read()) -- read byte & convert to ascii
bytes = bytes .. byte
letter = letter + 1
until letter > 10 -- capture 10 bytes (the patch name)
patchNames[i] = bytes
pointer = pointer + 61 -- skip 61 bytes (patch info after name)
end
elseif bytes == VCED_header then
pointer = pointer + 77 -- skip 77 bytes (patch info before name)
sysexBlock:seek (pointer)
bytes = ""
local letter = 1
repeat
byte = string.char(sysexBlock:read()) -- read byte & convert to ascii
bytes = bytes .. byte
letter = letter + 1
until letter > 10 -- capture 10 bytes (the patch name)
patchNames[1] = bytes
end
print ( dump(patchNames) )
end
I was looking at it. What I found is that MIDI IO (low level serial driver) buffer gets exhausted for messages larger than 500bytes. It is a bit strange as the same messages are processed on USB interfaces without problems and at much higher transfer speeds. I put the problem on top of my work stack.
Lua is a garbage-collected language, so remember to periodically invoke its collectgarbage function, especially when working with loops. I don’t know if the firmware does it for you automatically or how often, but I’ve empirically found it can help squeeze a few more things before I get out-of-memory errors.
E1 calls garbage collector on regular basis. That takes care of anything left over from the callbacks, formatters, timers, and such. It cannot, however, fire the GC in the middle of someones Lua function. There, it is responsibility of the user to make the judgement if it is needed.
PS: my work on resolving that sysex issue (see above) is interrupted now, I have had to switch to something a bit urgent and related to MK2.
I tried writing a script that would request all 32 patches one by one and collect the patch names, but I just can’t get the loop that uses midi.sendSysex to work with the midi.onSysex function. midi.onSysex doesn’t seem to listen until the loop is finished, so I only get the last patch name. Seems like a no go with that idea.
