- Open Source Cochlear Implant
- Motivation:
- For me, this is personal. I've had a cochlear implant since 1997 - Cochlear Spectra 22 model - and it's been a life changer. But the Spectra 22 is woefully outdated, and unsupported: if I manage to break this unit, I'm sorely out of luck. However, the newer upgrades of the Cochlear brand implants have left me disappointed: the sound quality is nearly unusable. When using my Nucleus 7, I have to read lips, like when I had a simple in-ear hearing aid before the implant. And, yes, I have *tried* to get used to them: the Freedom and Nucleus 7 models just don't seem to work for me. So, I'm trying to recapture the effectiveness of the Spectra 22 with my own design.
- Mission:
- Convert sound energy to RF packets that are sent to one of 20 individual channels in the implanted electrodes in the cochlea.
- Graphic
- Provide a user interface, preferably on the user's phone, to change channel response levels of each channel (mapping), as well as CI program customizations.
- Have not made any progress on this portion, yet. Still working on the implant / coil side.
-
This project only deals with the external processor aspect of the cochlear implant (CI). If you don't already have a CI device implanted, this project will not help you.
- Info:
- Factoids that may interest you:
- Cochlear implants have 20 channels on the coil implanted into the cochlea corresponding to different frequencies
- More info here, as I add it
- Contact: email
- Original FreePlane map for this site
- Download FreePlane
- Hardware:
- Current development platform
- MIMXRT1060-EVK
- Why?
- Cortex-M7, floating point hardware for DSP support
- Actual chip on the EVK: mimxrt1062dvl6a
- This has been replaced by the mimxrt1062dvl6b
-
- Specifies floating point integration
- Section 1.1, page 2
- Has audio support on-board
- Wolfson WM8960
- Similar, if not equivalent to Teensy 4.0 / 4.1
- MCUXpresso IDE for development
- Also need the NXP SDK
- You need to create an account with NXP to download an SDK. However, the account is free, and NXP doesn't really hassle you with advertising.
- NXP MCU-Link Pro
- Previous development platform(s)
- FRDM-K66F
- Why?
- Cortex-M4F, floating point hardware for DSP support
- It had audio support on-board
- Though I actually broke the audio component of my test board - probably over-voltage during some test - so now I'm using an external mic and ADC. shrug
See entry under MCUXpresso IDE for an explanation
- Runs at 180 MHz, which should be enough for sound processing
- MCUXpresso IDE
- Though I did consider mbed, at one point, because of its simplicity and library support.
- Oh! I remember, now. I ended up going to MCUXpresso because I wrecked not only the audio chip, but the mbed USB connection on the MK66. The rest of the board worked ok - and they're not cheap, so I couldn't afford to throw it out - but the only way I could download binaries to this board was through the SWD connector and a Segger j-link board. So I converted the project to NXP SDK and MCUXpresso. I shouldn't wait so long to write these notes.
- Segger j-link serial wire debugging (SWD)
- Alternative MicroControllers:
- Teensy 4.1
- Pros:
- At an operating speed of 600 MHz, it should have plenty sound processing capability
- 32/64 bit floating point processor
- Chip availability. Apparently, the MK66 has been hit by worldwide chip delays, making it harder to purchase. The 1060 on the Teensy 4.1 doesn't have this problem.
- Cons:
- Single Wire Debugging isn't natively available
- Work-around: use the NXP MIMXRT1060-EVK for development / debugging, and then transfer the binary to the Teensy 4.1
- Sony Spresense
- Pros:
- This device has six cores that can act independently, so we could process sound and produce output signals in a pipelined, round-robin format, to improve throughput. While one core is processing its sound data, the next core could start capturing data for the next packet.
- Cons:
- You can't buy this chip by itself: Sony evidently only sells it as the complete Spresense device.
- Parallax Propeller 2
- Pros:
- Similar to Sony Spresense - multiple, independent cores for pipelined processing
- Cons:
- Haven't actually tested it to see if it could handle the throughput; though at 180 MHz (or 300 MHz, depending on your spec source), it should have plenty of overhead for sound processing
- Microphone
- On-board DA7212
- Broke this one during testing
- Sample speed of 16 MHz, which should be enough for up to 8 MHz of audio, which is enough for most speech
-
- This one does not have automatic gain control (AGC)
- I use ADC sampling to get sound data from this one
-
- This one has automatic gain control (AGC)
- ADC sampling for sound input
- The Spectra 22 actually uses a filter chip with 20 channel low power switched capacitor filters with center frequencies that can be adjusted. (Cochlear Implants, Clark, pg 478.)
- Graphic:
- LCD
- This is used for testing to display the channel intensity; it will not be used in production
- Clock chip
- My initial thinking is that we'll configure one of the outputs to run at 2.5 MHz, the base frequency for the radio frequency (RF) output, and then enable the output to the coil via a transistor being on or off for a specific amount of time.
- This is a first-pass attempt (guess?) at the RF output, so it could be embarrassingly wrong.
- Current state
- Graphic:
- Power:
- On the Spectra 22, a single AA battery runs for almost a day's worth of usage. It's not necessary, but it would be nice if we could do something similar.
- Form factor:
- The Spectra 22 is worn as a side pack on the hip with a cable that runs from the hip to the ear, and then another to the coil on the scalp. The Freedom unit is a single behind-the-ear (BTE) unit. Personally, I like the hip-worn format, even though it is annoying to catch the cable, occasionally.
- Software:
- Logic:
- Board, peripheral startup
- For eternity:
- Acquire sound sample
- I'm using 2048 ADC samples into a buffer - because I broke the on-board mic during early testing - but it could also use the SAI interface at 16 KHz (2x maximum frequency due to Nyquist's Theorem)
- Mic input covers an effective range of up to 12 KHz
- Perform FFT on the sample
-
- This will sort the audio data into FFT "bins."
- I'm using 256 bins, of which half, or 128, are positive data, which means a bandwidth of approximately 93 Hz
- In a 2048 sample, with 128 bins (because only the first half of 256 have real values):
Minimum frequency: 100Hz
Maximum frequency: 12500Hz
12400 / 128 = 96.875 Hz per bin
-
-
- Bin allocation to implant channels is not linear.
— The frequency spacing of the filter channels in the nominal case is linear in 200-Hz steps from 250-Hz center frequency to 1,650 Hz, and then exponential (ie,linear on a log scale) up to 10-kHz center frequency. In this section, the spacing is by a factor of 1.149 between 1,650 and 3,300 Hz, and then by a factor of 1.172 between 3,300 Hz and 10kHz
- For now, I'm just displaying the channel energy level on an LCD screen for testing
- In reality, we'd convert the volume levels per channel to RF (radio frequency) data sent through the external coil to the internal coil
- Graphic
- Source code:
- You need the MCUXpresso SDK - see next leaf - for the source to make sense. The project source is actually only a small part of the code base: much of it is the board configuration and SDK.
- To use the source:
- Untar the source
- Typically something like:
- cd $yourSourceDirectory - where $yourSourceDirectory is where you do your project work
- cp ~/Downloads/project.tgz .
- tar xvzf project.tgz
- This will leave you with the following files:
- MK66F18_Mic_test_with_LCD.mex
- This is the hardware pin layout / definition file. You'd use it to define how the MK66 pins are used - input / output; peripheral (I2C; GPIO; ADC); etc
- At some point, I was using 20 GPIOs for LEDs that PWM flashed depending on the channel intensity. I'm not using that arrangement, but I never removed the definitions.
- TextLCD/
- This will be a new folder in your MCUXpresso project; it drives the 16x2 LCD screen
- This code was provided by Erich Styger
- source/
- The contents of this folder will go in the source/ folder of your MCUXpresso project
- MK66F18_Mic_test_with_LCD.c
- This has main(), and drives the project
- si5351.c
- This is the I2C clock chip that drives the 2.5 MHz base for the output signal
- si5351a.h
- Create a project in MCUXpresso
Mine was named MK66F18_Mic_test_with_LCD
- Select File --> Import --> File System
- Import the source files and folders into the new project
- Build the project
- Connect the Segger J-Link debug probe to the MK66 board
- You'll probably need one of these adapters:
- And one of these cables:
- Debug the project using J-Link debugging to download the binary and step through the code
- MCUXpresso IDE for development
- NXP SDK
- Current version: 2.13.0
- 2.11.0 - previous version used
- You'll need an NXP account to download the SDKs
- Project Status:
- Sound acquisition:
- I'd call this done, more or less
- Conversion from raw sound to channels:
- FFT aspect seems to be working
- Actual channel selection per FFT bins could probably be tweaked - and will likely be during actual testing - but it seems pretty accurate
- LCD display for individual channels reliably follow a frequency / tone generator as it goes up and down. The volume seems to scale properly, as well.
- I have a spreadsheet that I used to assign FFT bins to channels.
- Link:
- This is in LibreOffice format.
- Clock output:
- For what it is, it's done. Is it useful? Dunno, yet.
- RF output:
- Kinda stuck on this part. I have not been able to make much sense of the packets I've seen from the Spectra 22.
- The documentation mentions "biphasic"
- Amplitude Shift Keying
- The documentation says one thing - amplitude modulation for 0 and 1 bits - but the 'scope waves don't show this. So I'm not sure it's actually using ASK.
- This discrepancy might be because the documentation I have, that references ASK, is for the Nucleus Freedom, which is not the same as the Spectra 22.
- Bit Coding:
- Graphic:
- Frame Coding:
- Graphic:
-
- Octave home page:
- Octave is a Matlab-compatible analogue.
- RF output detection:
- We should be able to confirm that the output packets our software generates match a packet from the Spectra 22.
This is critical: we can not test a signal to the output coil until this is verified.
- Erroneous output could possibly damage the internal components, or cause biological harm.
-
- Implant mapping:
- This will be an application that runs on either a PC or a smart phone. This allows the user to read existing levels, and set custom levels for each channel.
- Channels have two critical levels:
- Threshold: the lowest level the patient can detect on that channel
- Maximum Comfortable level: the highest level tolerable on a given channel
- In mapping, the levels are set for each channel, which determines the sound quality for the user. A bad map can make speech and sound indecipherable.
- Status: haven't started on this
- "Bells and whistles":
- Features like BlueTooth connection would be nice, but secondary.
- Status: Haven't considered it beyond it being possible.
- Recent Activity:
- For much of 2023, I took a pause from this while working on domestic projects, and an article for Nuts & Volts. I have another project in spring 2024 that I want to complete, and then I'll get back to this in earnest.
- 3/13/24 - reached out to the University of Zurich for information about sCILab software
- Wanted to know if the software is available; I could read through it to learn how they captured and interpreted radio frequency (RF) packets.
- Haven't heard back. I'll try contacting the authors directly.
- 3/13/24 - reached out to Bit Scope for information about their scope "programmability."
- Wanted to know if their BitScope products could be programmed to read a cochlear implant signal and recognize RF packets. This would be helpful at the hardware level.
- Didn't hear back. I may buy one, anyway, and try that. I'm also considering the ADC (analog-to-digital conversion) on the LPC55S69-EVK module.
- 3/9/24 - inquired about '0' values in ASK (amplitude shift keying) wave
- Didn't really get much useful information from this.
- 2/1/23 - Received NXP MCU-Link Pro debugger
- As discussed in the link to mcuoneclipse.com.
- 1/11/23 - Ordered AKM5720VT chips for testing audio ADC chips
- Saw a passing reference in Art of Electronics, 3rd ed, chapter 13.
So far, I have it installed on a TSSOP16 board, but it's not connected to the 1060EVK board, yet.
- 1/18/23 - Video conference with NXP sales
- NXP recommends that I use a MIMXRT1062 due to supply channel issues. This is residual from COVID-19. I already have an MIMXRT1060-EVK, and I've implemented it as my test platform, but I haven't actually gotten to any real code, yet.
- 1/8/23 - Measured the rate of ADC samples per second on the K66
- In MCUXpresso: While a program is active, look at the Registers panel. Down near the bottom, there’s a DWT Registers section, in which you’ll find cycles and cyclesDelta.
Representative. I ran it a few times and the numbers were pretty consistent.
Start: 36993247
End: 38321652
Delta: 1328405 / 180 MHz = 0.00738 ~ 7 ms; but that’s for 256 samples ( 1024 / sizeof( float_t ) ), or approximately 34,688 samples per second.
- 4/26/22
- Tested the NAND gate output. The output is much nicer.
- Image:
- 4/20/22
- testing ADC sample size of 1024
- First issue: arm_rfft_fast_init_f32.c for case 128u problem
- You need the CMSIS_DSP_Library_Source element from the 2.11.0 SDK. For some reason, the "case" for 128 FFT bins is disabled by default in this version of the SDK. I set the #defines to enable it, and it seems to work.
- Had to shift the implant channel allocation around a bit, since each FFT bin is now worth 193.75 Hz. (12400 / 64) Using a tone generator to test frequency response - with LCD display - shows that it still follows the tone. The lowest frequencies look less than perfect - the LCD column isn't as bold at ~145 Hz - but it appears to work.
- ordered some nand gates to test the psuedo-RF output
- 4/10/22 - testing pseudo-RF output with the timer chip
- Using a transistor to limit the output of the clock chip. I'm not sure why I'm getting the smaller peaks when the transistor base isn't enabled.
- 10 nops gives me 1 µs, or 3 cycles at 2.5 MHz.
- Source image:
- Scope image:
- Schematic image:
- Stack Exchange forum thread:
- Observations:
- From the scope, I see I get two pulses in a span of 20ms, which means a throughput of 50 packets - not including the time to create and send the RF packets - per second.
- This seems kind of slow.
- I tuned out the LCD display code, to bring it down to 20 ms. The main loop is now, essentially
-- collect 2048 samples;
- perform FFT on the samples
- assign channel magnitudes
- send a test pulse - And this all takes 20 ms. Huh.
- Need to check the docs to find out how many packets the Spectra 22 normally processes per second. Are we in the ballpark with 50-ish?
- The maximum total stimulation rate depends upon the bit rate and the frame rate in the RF transmission link. Using 5 RF cycles to encode the raw bits and 6 raw bits to encode 3 actual data bits, the bit rate is 250 kBits/second in the Nucleus 22 system and 500 kBits/second in the Nucleus 24 system. The Nucleus 22 uses only the expanded mode to encode frames and has a theoretically maximal total stimulation rate of 5,900 Hz. In practice, this theoretical rate is not attainable. For realistic stimulation parameters such as pulse duration of 100 μS/phase, 30 μS phase delay, electrode 1 and 2 bipolar mode with the highest amplitude, the maximum rate is just above 3,000 Hz
- Cochlear Implant System Design Integration and Evaluation, Section IV-C
- Apparently, collection samples and using FFT to select channels is going to be far too slow. Looks like we'll need to use filters.
- I'm also going to try a sample size < 2048. I haven't yet.
- Acknowledgements
- Erich Styger
- Code for LCD screen
- AdaFruit
- They're not a sponsor, but you'll notice I buy a lot of components from them.
- SparkFun
- The components I don't get from AdaFruit probably came from SparkFun, instead.
- Web site To Do:
- Schematic
- References: a list of the references I'm using
