As there is growing number of users who use Electra One with the Claude Code or Codex, I have decided to make a few extra additions to make that process easier. The main idea is to allow the AI to easily obtain feedback from the controller after a modified preset/lua is uploaded on to the hardware. To do so electra one can now:
run, step, trace Lua script using an internal Lua debugger (@Phommed)
obtain information about system statistics (memory, threads, queues)
take screenshots of the screen
create and run unit tests (much inspired by @adamc)
Next to that, and besides the general AI knowledge, it forces the AI to follow preprogrammed knowledge about Electra JSON/Lua/MIDI (rag). The result is an information flow like this:
The biggest win here is that the agent can review and correct its own mistakes, leaving you out of the loop and not waiting for you to explain what is happening on the hardware - which might be often misleading.
While the feedback loops will work perfectly with the online version of the editor. The rag and an option to have the preset and lua files stored locally (and therefore easily accessible to the agent) on your computer is a feature of the offline version editor only.
I do work with local LLMs in my other project (the one that pays my bills) so I tried to reuse my knowledge here. I am sure, however, many of you have quite some experience with using AI for electra and it exceeds what I know. I will be happy for any info on how you work, what works for you and what does not, etc. sharing it might help other users too.
This sounds great. One of the big hassles is keeping the agent looking at the current version of the code. I do a lot of cut and pasting as a result. Having it look at the code in the editor will be very useful. Thanks. Is it specific to Claude and Codex only?
I tested it with Claude but the idea applies to any agent that can access a local folder on your computer and talk Midi. The only part that is Claude only is that the offline editor allows you to use your own Claude code API key for AI chat sessions in the editor. We want users to use their own plans and credits.
Hi Martin, your work goes further than the MCP server I posted back in May — good!
One thing I learned building widgets this way, in case it’s useful. What an agent lacks most isn’t the documentation, it’s the list of gotchas. The kind of thing you burn three hours on and that fails with no error message at all: graphics.drawText not existing on the device, coordinates having to be integers, one pot per custom tile. I’ve got about fifteen of them, verified on 4.1.4. Happy to hand them over as they are.
There are two ways to use the AI in the offline editor:
built in AI assistant (using your own anthropic key), directly modifying the preset based on users prompts
Claude Code or Codex watching the files of currently selected slot on the controller, using your plan
here is an example of simple session using the built-in AI assistant. An arpeggiator that with an option to select source (MIDI interface, port, channel) and destination (MIDI interface, port, channel) and tempo and pattern control.
the prompt:
I would like to have an arpeggiator preset created.
It should play held notes. At the very beginning
I should be able to specify interface, port, channel
of the source and interface, port, channel of the
destination. Place these on the second page called Config.
On the first page we start with two knobs: the first one
is tempo and the other one is the pattern. Add common arp
patterns.
no other prompts were needed. Working as expected. Cost 0.97 USD. Subsequent edits/prompts, eg. to add extra controls or logic, will be much cheaper as the context is set up and the hardwork done. AI strictly followed the instructions embedded in the offline editor, not guess stuff on what it finds on internet or inventing stuff.
Let’s just say I spent much more than 97c pimping out my Peak preset. But I did learn a lot about how to code more efficiently and techniques/modules that I can replicate. So a bit of a fork in the road for me. Do I continue the chatting/testing approach with lots of cut and paste, or just ask an AI agent to make the preset? I think I know the answer but don’t want to admit it.
I wanted to show that a single prompt can do a lot, but you are not restricted to work that way. You can freely combine your own work with AI and have it step in when you need it.
The beauty of it is that instead of manually copying and pasting things back and forth, the AI agent interacts directly with the preset/lua. If you change the preset or Lua manually, the agent knows about those changes and can continue from there.
Personally, I think it is a good idea to stay involved in the process and try to understand what the AI is doing. First, you will understand it when it gets more complex. Second, you will not spent tokens on trivial changes and actions (adding controls, changing labels, etc). At the same time, many users have been waiting for me to help them with things or fix issues. In a way, I have managed to transfer some of that knowledge into the agent, so it can help now without waiting for me.
But I did learn a lot about how to code more efficiently and techniques/modules that I can replicate. So a bit of a fork in the road for me. Do I continue the chatting/testing approach with lots of cut and paste, or just ask an AI agent to make the preset? I think I know the answer but don’t want to admit it.