A quick foreward - a sincere apology up front that we disappeared for almost 2 months. As I write this our e-mail, facebook, and indiegogo are all flooded with requests for some updates, and it is certainly more than due - we're roughly 7 weeks behind with that "good news" we promised in June! As some of you have likely guessed, we have some bad news - we hit another delay, and because of that, will not be able to make shipments starting in August (we're about a month behind). The good news is that we do, in fact, finally have that good news to report, and aren't quite as far behind as our silence would suggest (4 vs 7 weeks).
It's not fair of us to ask for this much patience at such a crucial time, and to disappear for so long without giving some updates, good or bad, as to what's going on. If memory serves correctly, this is the second time we've gone an abnormally long time without an update when things weren't going well, and leaving our customers and supporters in the dark for so long is definitely NOT something we intended on doing or feel good about! Hopefully this update will give a better indication of just where we've been and why it's taken so long to get back to everyone, but ultimately an apology is due to everyone reading this - so please accept my sincere personal apologies as well as those from ReDeTec in going MIA for so long. We should be back in action at the 1-2 week update frequency until shipping now, come hell or high water!
Hello everyone, and welcome to a long and emotional roller coaster of an update! It's been a crazy 2 months, and we've cleared a number of milestones despite far more setbacks than we would have liked! Specifically, we've now finalized all of the firmware, passed our scan for CE/FCC/required approvals, ironed out a number of production challenges, completed our initial sample run, sent it off to UL, completed our shop set up, and critically started the main production run - at least the stuff that isn't UL critical. This now includes all of our externally sourced equipment like springs, hardware, etc.
The big ticket item of course, is passing our CE/FCC pre-scan. This is what we'll cover with the bulk of this post, but in essence quite a bit of the past month has been spent desperately trying to wrangle some electrical phantoms from ProtoCycler. We passed early last week and have moved straight on to the rest of testing - units are already off - so that we can get our final stamp of approval ASAP and start shipping out right after that.
We did come across quite a few other issues, however - from a server crash losing some of our design files, to a looming Canada Post strike delaying shipments, to the wiring disaster that took 4 electricians to get our Lathe up and running again, and two of our suppliers taking holidays right in the thick of it all...it seems the entire summer has been juggling last minute, critical issues as quickly as possible! With deadlines looming, the temperature rising, and more and more things falling apart, we got thrown off track a little and got caught off guard...our updates stopped, organization suffered, and some delays followed. Thankfully, all of those are now sorted out and under control, and we've finally managed to catch back up with everything else and get back on track with production!
So where exactly are we? At this point, we're just waiting on the "extra" safety stuff (beyond the minimum CE, FCC, etc) to pass, followed by a ~2 week lead time to get the first batch of 100 produced and out the door...with another 100 following every 2-3 weeks after that! Everyone seems to be moving much more slowly over the summer months but we're hammering UL to test us as quickly as possible, so we can receive our certification and begin shipping. This will push our shipments back 1-1.5 months, from early-mid August to mid-late September (based on UL timing). We're hoping to get a system set up shortly where you can find your estimated shipment date based on your order number, so you can find out exactly when your ProtoCycler is expected to ship, and receive any notifications of further delays instead of having to actively check the blog. With any luck, getting a system in place to notify everyone of delays will almost guarantee we never fall behind again, similar to how umbrella's always seem to result in sunny days!
All of that being said, at roughly a year behind we owe everyone some pretty detailed explanations on what exactly is taking so long. So without further ado, the EMC process, what went wrong, how we managed to fix it, and what we learned in the process.
First and foremost - what exactly IS "EMC" testing, and how does it relate to CE, FCC, UL, etc? Without getting too deep down the rabbit hole of international product certification, all products sold in certain markets (like Europe and North America) require certain minimum certifications to be sold legally. The vast majority of these revolve around electrical safety and "playing nice" with the rest of the world - for instance, not only does your device need to ensure it won't electrocute you, it's also not allowed to change your TV channels or cause static over the radio. It goes even further as well, in that your device must also be able to withstand other devices trying to interfere with it - sort of a redundant workaround to ensure that all of our electronics don't mess with each other, or ourselves.
These minimum certifications are collectively called "EMC testing", or Electro Magnetic Compatibility testing, and this testing is what is required to obtain CE markings (which is for sales to Europe) and FCC markings (North America) - the CE or FCC marking in turn certifies that our product is safe to use in Europe or North America respectively. Without the CE or FCC mark, it's technically illegal to sell products, but lots of people - like all of our competitors - do so anyways using some shady legal workarounds to avoid going through certification. That being said, EMC testing is only the bare minimum to ensure your product meets international guidelines. To truly ensure that the product is safe, and not just legal, requires a whole separate process: our UL testing. UL goes above and beyond EMC to cover things like fire safety, unintended operation, electrical breakdowns and failures over time that could cause issues, etc etc. Instead of making sure that a brand new perfectly operated product will meet some minimum standards, UL makes sure that even in the non-ideal real world, nothing can really ever go wrong.
Because of the nature of the certifications, the process is somewhat back and forth / iterative. We'll send the unit to EMC for a scan, pass the scan, send the unit to UL, hear word back from them, modify the unit to meet their standards...then re scan it for EMC, re send it to UL, re modify it, etc etc until both certifications are complete. EMC is also much quicker for the pre-scan, so you can quickly ensure you'll pass EMC before re sending a unit to UL to save time.
As those who have been following for a *long* time will know, we'd already done an EMC scan a few years ago in New York while we were down there for a maker faire, and had not come across any issues. While we'd made a few changes to the unit since then and our first pass at UL, the general unit hadn't really changed much - it was effectively the same circuit, though now made with UL spec components from a UL spec factory - residing in effectively the same box, a large all-metal enclosure that should make EMC testing a piece of cake by acting as a sort of "insulator" between ProtoCycler and the world (electrical interference has a very, very hard time passing through metal - this is why you usually loose reception in an elevator). We were pretty confident we would pass EMC immediately, and in late June, hinted that we'd likely have a passing result within a week. Oops.
There are three major categories for EMC - Radiated Emissions, Conducted Emissions, and Immunity. The first deals with signals we beam through the air, the second with signals that travel back through our power line into the mains network, and the third involves assaulting ProtoCycler with everything it could possibly experience in the wild - non certified units that radiated or conducted emissions, static shocks, lightning strikes, power outages or surges - and ensuring it still operates as it should. We ran into issues for all three, and scheduling conflicts meant there was dead time between each scan. Below, the individual issues and solutions:
Issue 1 - Radiated Emissions
We were very confident for this one, and it presented the least trouble. Basically, ProtoCycler gets set up on a desk in a "radio silence" room...
Then a very, very large antenna scans any emitted electrical radiation or noise in the room:
And last but not least, the results are recorded - both for 110/60 hz operation, and 220/50 hz operation.
Our first test was at 110/60, and we passed! Notice that the red line (our emissions) never exceeds the green line (legal limit)...awesome.
But, then we went for 220/50hz. Just below 200MHz, we emitted some noise above the green line...a failure. Everyone, including the EMC folk, were pretty surprised by this...what could possibly be causing the issue?
While it took almost a full day in the EMC lab to test, we managed to figure out the issue. In modifying the spooler slightly so that the motor now lived fully outside, with the plug external of the chassis (so you can swap in your own spooler if you want), we'd inadvertently brought a section of the electrical circuit outside of our big metal box that was supposed to keep everything safe. We'd also moved the plug on the circuit board to *right* beside the USB circuit on the board - which happens to be clocked right at 48 MHz, the second harmonic of which is exactly the 192 Mhz we were having issues at! While it was a little difficult to believe that our USB circuit was coupling internally to the stepper circuit, and then being brought outside of the box by the stepper wires, there was an easy way to verify if that was in fact the issue. By placing a ferrite core over the stepper wires, we completely removed the peak at 192Mhz, and got ourself a pass! 110/60 hz was also substantially improved for the FCC rating:
But the big difference by far came at the 220/50hz, where we are now safely under the line and ready for sale to Europe!
So, we'd solved our radiated emissions...but had wasted the full day of testing doing it. No matter, we'd get a second test day booked quickly, and postpone the delay until then!
Issue 2 - Rescheduling follow up tests, last minute, during the middle of summer holidays.
...except that we couldn't get a second test day booked quickly, at all. This became one of the biggest causes of our delay - we couldn't get an appointment the next day, or even the next week. We had to sit and wait, hoping we'd pass everything else. That being said, it was almost unheard of that you could pass radiated emissions, but not the much easier conducted emissions...so we would be fine!
Issue 3 - Conducted emissions (and the death of the snubber circuit).
...except that we weren't, at all. This is one of those stories where something so trivial happens you almost ignore it completely, but then 2 years later it completely comes back to haunt you. But first, some more on the actual test: Conducted emissions are when the actual mains AC signal from your wall gets interfered with from ProtoCycler - in other words it makes the power lines themselves noisy. Now, the only circuits we have that touch AC are our 12 V supply (which is independently pre-certified for all of this stuff), and our heater control circuit, which is an opto isolated triac driving the heater, using phase triggered PWM...neither of these should have caused any noise. But when we plugged into the conducted emissions test and looked at the results, we were startled to see a mass of noisy red on the screen:
Not good at ALL.
The noise was so bad the EMC techncians thought there was an equipment problem. We thought someone must have forgotten to set a parameter somewhere. This wasn't just a poor result, or excessive noise - The noise was phenomenal, and we'd just passed radiated emissions. It was as if you're standing right next to someone listening to headphones, unable to hear even the slightest hint of music - but then you put the headphones on and not only are they way too loud, they're so loud they blow both of your ear drums instantly. It just didn't make sense that there could be SO much noise "inside" the wires, but almost none coming out of them.
Now the first step in dealing with conducted emissions is to install an EMI filter, but we had already done this! And switching it out for a much bigger filter (almost the size of a brick) did practically nothing as you can see from the GIF below.
This wasn't parasitic noise or some random ground loop causing issues...somehow, ProtoCycler was managing to actively distort the mains signal in the building, and we'd yet again run out of testing time that day. We even tried turning the heater power completely off, but it made no difference - if the heater was plugged in to the circuit at all, we'd fail by around 30 dB of noise. Now even further behind, confused at what was happening, and concerned we didn't know the solution...we began looking at anything and everything that could possibly be causing this much electrical noise over our mains line, and completely forgot about posting an update on our progress.
The issue, ironically enough, turned out to be something we'd done after our original EMC test 2 years ago to try and improve our results. At the time, we'd been cutting it pretty close on our radiated emissions, so we'd tried to brainstorm ways to cut it down. We knew we would be changing things and wanted to give ourselves as much breathing room as possible, so to speak. Adding in an EMI filter was one of these (and a good idea), and adding a snubber circuit to our heater control was another (and a really really terrible idea). For those who don't know what a snubber circuit is, it effectively "snubs out" electrical noise when high impedance, high voltage loads are switched. It's relatively standard practice and a quick google of "TRIAC snubber circuit" will give the details for those who are interested, but effectively you use a capacitor to absorb sudden bursts of energy when inductive loads are switched on. We'd been using the exact same snubber circuit without any apparent issue since we first added it on over 2 years ago - now with a different style of heater, using different types of components (wire wound for UL, as opposed to more standard stuff for surface mounting we'd been using). Between the significantly reduced inductance of the heater and the new circuit layout, the snubber was now resonant with the heater, instead of damping! The very simple solution, of course, was to remove the snubber...we did, and what a difference it made! As you can see below, once we removed the out-of-spec snubber circuit, our noise disappeared. Finally, we were passing conducted emissions with ease, beating it by nearly the same amount we'd previously been failing by!
Last but not least, the immunity testing - ProtoCycler had to withstand electrical noise, radiation, shocks, surges, cut outs, and phase switches. One by one, the tests were attempted...and passed. We had now passed all of our EMC, and were ready for a final round at UL!
Now, this all happened early last week, right around the time our inbox / social media / campaign started lighting up with concerned questions about shipping dates, etc. But between catching up with all of the missed messages, getting back on track now that EMC is through, and writing this update, another week and a half seemed to disappear. Thankfully there's very little left to do before shipment at this stage - wait for UL to give the final approval, and then frantically begin final assembly of the first run! We also get our full EMC report - detailed specs, not just pass fail - at around the same time, so everything should line up nicely on the certification end of things.
While the unit is away at UL again, we'll be doing more updates on the production side of thing, as well as getting ready for some more logistics oriented updates - shipping addresses, pellet options, model and colour choices will all be locked down as we need to finalize counts for things like powder coating and electrical cables for various regions. I also suspect there will be some follow up concerns, questions, and clarifications after this update so we'll be sure to work those in as well. And as always, feel free to e-mail us! While we're still not perfect at that, we do definitely manage to check our inbox more often than the blog.
So, there you have it - it's been a tough six weeks with more obstacles than we could have expected or were really prepared to deal with. The journey to fulfillment seems to be filled with ups and downs, and after the "ups" we had in the spring this "down" felt a little tougher to take than usual - likely how you feel as well. All we can do is thank you endlessly for your patience, get back on track with our updates, and continue to keep everyone as involved and informed as we can with the process. We've got exciting times ahead as production begins, so thank you as always for your patience, and apologies again that we pushed it a little far this time - we should be back on track now :)
-The ReDeTec team