The bloodstream of civilization
Supply chains are the bloodstream of our civilization. Often, we don’t realize how complicated the process that leads us to buy a carrot in a vegetable store or a phone in an electronics store is. Before the product is in the store, it must be brought to the store. It takes cars, ships, trains, pallets, containers, forklifts, fuel, and a whole bunch of other things (and remember, these are products too). For a product to be transported it must be produced (or grown). And for this you need intermediates, raw materials, machinery, energy (that is, more products). Raw materials have to be extracted or recovered (e.g., from old products), and energy has to be generated. For this you need other machines, transportation, fuel. It all creates a very complicated system of interrelationships. Products, intermediates for products, intermediates for intermediates, products for production and transporting products and intermediates… All this circulates around the world and makes carrots and phones appear in stores, and we are just tiny cogs of this machinery. But complex systems are sometimes sensitive to disruption. Like the so-called butterfly effect. In science, we call such systems chaotic. This does not mean that chaos reigns, just that a small disturbance can quickly lead to big changes in the system.
Relationships in supply chains can get really tangled.
(Produkty – products, półprzewodniki – semiconductors, półprodukty – intermediate products, materiały konstrukcyjne – construction materials, produkcja – production, surowce – raw materials, materiały eksploatacyjne – consumable materials, maszyny – machines, wydobycie – extraction, dostępność – availability, logistyka – logistics)
Those ubiquitous chips
Semiconductors are a very special type of product (intermediate product). That is, all the diodes, transistors, processors, chips, and other components from which electronics are built. And this is because they are found in a great number of products, machines, cars, etc. They are ubiquitous and the operation of a huge proportion of devices depends on them. In addition, their production requires a very complex know-how, expensive machinery (also using semiconductors) and rare raw materials. The more advanced chips are to be produced, the more complicated and expensive it gets. And that makes the factories producing the most up-to-date processors and memory literally few in the world. And don’t think that these chips are some ultra-expensive products with niche applications. No, they are produced in millions and they are in our computers, phones, cars, the servers that provide us with the Internet and all those clouds, they control the machines in the factories. So you can see how sensitive this spot is.
Everything is starting to tear
We hear a lot about broken and torn supply chains and we’ve been experiencing them since the first lockdown – there was no paper, no cameras, computers getting more expensive, graphics cards as well. And you probably already understand that this is just the tip of the iceberg. That even a small imbalance between demand and supply of one product can trigger a whole domino of events. But the lockdown (which, from the point of view of supply chain mechanisms, was precisely a disruption of supply and demand – we suddenly stopped buying some products and started buying others) was not the only cause and not even the first. A major, though perhaps not as rapid change was (and still is) the transition of many services and softwares to the SaaS model – the famous cloud. And that means fewer computers and their components, and more servers, routers, and the rest of the network magic. This in turn means fewer semiconductors of one type and more of another. A change in semiconductor production plans means changes in ordering plans for everything needed for their production. And as you can see, it takes a lot. A whole lot of other minor or major disruptions have added to this: prolonged repairs of the transshipment terminals, the blockade of the Suez Canal caused by the container ship Ever Given, steel shortages which translated into shipping container shortages, a fire at the AKM plant (one of the few), a strike at STM (another major semiconductor maker), the US-China trade war, delayed investment in new factories. The war in Ukraine also played a role here. It forced the shifting (lengthening) of some important trade routes (Turkey unexpectedly benefited) and affected the supply of important raw materials: silicon, neon, aluminum (surprisingly – important for semiconductor production). In addition, the semiconductor industry, like many others, switched to the JIT model ( just-in-time manufacturing) – some time ago. Instead of holding large inventories, they produce in batches for the anticipated orders. And here goes the avalanche. Semiconductor manufacturers are out of supplies, generating delays, rearranging production, speculators appear buying up entire inventories of sought-after chips, prices rise, production of things dependent on semiconductors, including those needed to manufacture them, slows down, causing further delays…
Electronics – how it’s done?
What does it have to do with EXATEL? Of course, like any telecom operator, we buy a lot of routers, switches, laser transceivers for fiber optic communications and other telecom equipment. It is getting more expensive, delivery times are getting longer, and the entry of new equipment into the market is being delayed. But all this can still be planned, arranged, and the risk managed somehow. I wanted to tell you about something more interesting. As part of our various software-defined networking (SDN) activities, we are designing our own network devices (which our electronics partner FastLogic is helping us with). So that we don’t have to match the functionality of the available routers and can use the full potential of SDN.
Project cycle(on) – from each successive stage we can be turned back to one of the previous ones, sometimes even several times.
First, I will tell you how the process of designing an electronic device was for years.
We come up with an idea of what we want to build. We come up with functionalities, describe the requirements and specifications, define cost (the maximum costs) and time assumptions (when we want to have it). Then we select the key components of the device to meet these requirements: the processor, or other digital circuits, main power components, communication modules, sensors, etc. We choose those from the catalogs of the manufacturers, guided by the intention to match with the functionality, ease of implementation (because, for example, we already have libraries for a particular processor family), the horizon of support for the components (so that we don’t have a major processor withdrawn from production in a year), and, of course, the price – so as not to go beyond the budget framework of the device. With that said, not all prices and technical information can be found online, sometimes you have to contact the manufacturer or distributor, which of course extends the process to weeks. We make a very preliminary sketch of the PCB to see how big it will come out and whether we will fit in the dimensions of the device. If not, then you have to work around for smaller, or more integrated components. And since everyone always wants the fanciest, smallest, and cheapest devices possible, these sizing iterations are usually done at least a few times before everything fits into the assumed dimensions and money (or the assumptions have to be revised).
Then the research phase begins. We order these major components or their evaluation kits (ordering and waiting for them also takes some time) to test them and see if they actually meet our expectations. We do various experiments ‘in scratch’, verify the assumptions – whether it can actually be done that way, whether we hit any limitations. Not everything works out, then we have to take a step back. When all the major building blocks are gathered and we know that they work together and separately, we make the design ‘clean’ (if the device is simple) or ‘semi-clean’ if we anticipate any difficulties (in which case we add various elements that are not needed in the target product but will make testing and troubleshooting easier. We also decide whether we do the design ready for production (which takes longer) or only for a test prototype.
Then we order PCBs, all the components, assemble them (which again takes time) and we have a prototype that we can test and develop software for it (some of the software could be written before, but only on the prototype can we start doing it for real). Of course, various problems come out. Some errors in the design of the board, or the less important parts that don’t work quite as it was estimated in the documentation, or it turns out that we won’t be able to meet all the original assumptions after all. We have to take a step back, and sometimes even two or three. Make corrections in the design, and another prototype, and so on and so forth – until it works. There’s no way everything will be great the first time. But if we’re lucky, it will be on the second try. If not, sometimes and on the tenth. Each time the prototype is a little more perfect, we also make more of them. So that more people can test it, to study production spreads and destruction threshold (how high a voltage can it withstand?).
Finally, some prototype is good enough to produce a larger-scale device. First the pre-production series, only then full production. Because this is also one of the stages where the problems come out. Production techniques on a larger scale are different from prototyping, because only a larger scale reveals flaws in the assembly process, it turns out that the final costs, however, are higher than previous estimates and something has to be cut at the last minute, because the corrections resulting from certification tests have to be made (and these can often be carried out only on production copies).
Then finally our device gets to our end users, who do things that even the engineers would not have thought of in their darkest dreams. I wonder why they do this only to production copies, while they were so gentle with the prototypes…
So as you can see, there are quite a few steps in the design process, and in each step we may have to take one or more back. It takes time. As well as the communication with suppliers, waiting for parts, waiting for production.
And what about now?
What changes in the process when supply chains start to tear? We come up with ideas as we did before, assumptions and specifications are developed roughly in the same way. But it starts to get tough in the next step. Because we choose a chip perfectly suited to our needs. We ask the distributor for how much and for when can they provide us with a few pieces for testing. And the price might give you a heart attack, and the delivery time is 53 weeks (a true story from our experience). So we ask others. Same thing. But we find one that has 5 pieces and is willing to deliver to us even in a week. Only the price is 4 times bigger. Will there be more later? They don’t know.
And so it is with many parts. The delivery times are not always that long, but 3-4 months is already an often answer. Such an extension of the design process (which likes to extend itself anyway) usually cannot be afforded. So if until now, when selecting the components, we asked ourselves ‘what will give us this functionality,’ ‘what will stay in production long enough,’ ‘what will be cheap enough,’ now we ask only one question: ‘what is available?’. It narrows down the choices greatly. We’re looking for things that can be bought in a few weeks, and which the distributor has enough to last for the first production. And we only roughly check the functionalities. Because it’s better to have something a little different, less tailored, or even more expensive than to have nothing at all. We buy a few pieces, test them, and have to quickly decide if the component suits us. Because we have to buy all the copies needed for production right away. Before the prototype, even before the project. Because if we didn’t, there’s no guarantee that someone wouldn’t buy them first. And the next batch might be made a year from now (because the manufacturer has a queue of already delayed commitments). And for a price that will be dictated by the only distributor that has it in stock.
So not only is there a lot of expense at the beginning of the process (when there wasn’t any before), but there’s also the risk of all the effort going to waste – because we buy all the units before we have the time to test how they function in a complete device.
And this does not at all apply only to the ‘large’, key parts like processors or memory. For some reason, a certain type of capacitor or inverter can disappear from the market overnight. Some parts are available everywhere, at normal prices. But there is a risk that they will disappear. For a few weeks, months, a year – no one knows.
You can see for yourself – the design process has turned upside-down. You can’t quite match the parts to an idea, buy them, build, test and buy more of the same parts. You have to buy whatever parts there are and then wonder what you can actually build with them. And accept that it will take longer, cost more and that it is impossible to implement all ideas. But we manage. You have to get even more creative and flexible. It is necessary to find unconventional solutions. It is necessary to build devices that are even more modular and tolerant of last-minute changes, so that even already in the production process it is possible to replace some components.
One of EXATEL’s SDN device prototypes, the production of which greatly suffers from the bleeding in a supply chain has its own cooling system.
And what will happen next?
Will supply chains rebound? Will the bleeding stop? Is it possible to return to the old, more predictable, and less expensive design process? When it seemed that the situation was going back to normal, the war broke out (with that came a shortage of, among other things, the neon needed for semiconductor production). Now all eyes are on Taiwan and China, which is claiming rights to Taiwan’s territory (and very important semiconductor factories are located in there). Europe is trying to diversify its sources of supply and become at least partially independent in terms of production, especially for the most advanced components. But these are huge and long-term investments. I think it is going to last for a while. It will be easier for those who design and manufacture more of different types of electronics, because they can have more regular suppliers for whom they are important customers, negotiate better prices, or offset risks between different projects (a batch of inverters bought unnecessarily for one project may be useful elsewhere). It will be more difficult for those who have few product lines or, like us, design something from time to time. But for truly ambitious and innovative projects like SDN, it’s worth gritting your teeth and moving forward. It always creates a little more of demand pressure, and in the end we’ll be rewarded with a truly unique product.
