Following on from the previous post, my obsession with X.25 did not wane at all – in fact it may have strengthened. And, if that wasn’t bad enough, it has spawned a couple of new ones.

The venerable synchronous serial card needed a driver; these devices are substantially more complicated than the regular (and well supported) async adapters. Owing to the age of the device, the most “recent” driver for Linux targeted a 2.6 kernel, but at least it was open source and didn’t need any proprietary blobs. When I took a look at the code, it appeared to be surprisingly simple – just a wrapper around reading and writing registers, all of which were well documented on the datasheet. I’m no expert on device drivers but I have worked on them in the past, so I decided to see if I could port it up to a recent kernel. And this is where the spectre of AI comes into the picture.

[The expression “AI” is totally inappropriate for what everybody currently uses it to mean. In case you don’t know what LLMs are, just read “LLMs” as “AI”]

Generally speaking, I’m wary of LLMs, but there’s no denying they can be exceptionally useful tools. Part of my wariness is that they can also be wildly wrong, with the same level of certain confidence you’d expect from a subject expert. Nonetheless, bringing the driver up to date sounded like more of a tedious task rather than something difficult, so I put free Claude on the case. The result was almost perfect and took a matter of minutes. To get it working I had to make a few tweaks and do some diagnostics, but all in all it did a great job. I forked the original Fastcom driver repo and uploaded it to Github (it’s OK, the original license was GPLv2).

Having a working driver yielded two character device nodes which do not behave like ttys. Well, you can read and write the data you want to transmit and receive to it – but without configuration this doesn’t really mean anything. The Cisco certainly wouldn’t know what to do with it, and besides, the whole point of this project was to run X.25 over it. The configuration involved calling a bunch of ioctls on the device which mapped quite nicely to the device registers. To initialise the card, the driver has a utility that reads an INI file, literally containing register names and values, then fires off a bunch of ioctls that set them. Crude but effective.

The link-layer protocol I was hoping to run over the serial link was LAPB: a bit-oriented protocol (i.e. the framing needn’t involve bytes) that is the layer 2 standard of X.25. It’s a massively over-engineered protocol that supports a ludicrous number of scenarios [see my video on OSI to see why this was fatal for X.25] and even after several months I’m still working out stuff about it. Fortunately, the serial card is smart enough to handle the majority of the donkey work of LAPB that helps establish the link and keep it alive. Despite this, I’ve had a bugger of a time getting it to work properly, even with ludicrous amounts of consultancy with various LLMs. But, even with most of LAPB taken care of, I still had to deal with the X.25 packet layer protocol (which Cisco refers to as “X.25”).

Before you explode with frustration that I’m not aware of Linux’s support for LAPB and “X.25”, you need to understand that I know it’s there, but I also know it’s soon to be removed, and I really want to deal with this in userland. That’s my rule, apparently.

The aforementioned XOTPAD had implemented X.25 in Rust, a language I like but am not fully convinced by, and so that was the logical place for me to start.

After having a crack at integrating this card’s interface into xotpad I found the implementation also over-engineered, and clunky. Another side note here – I’m still a C and Go guy, and I detest C++. Rust, to me, feels like another C++: stuffed full of features that do nothing but make certain kinds of developers feel like artists. I love the simplicity and flexibility of C, but also I appreciate that it has to go; even for seasoned developers, keeping the code secure is nigh on impossible in C.

I looked into Rust’s support for bit-fields and found the language supports them through some dude’s library.

So, wouldn’t this be a great opportunity to try another language that sounded like a safe and modern version of C? After agreeing with myself, I decided to learn Zig by implementing X.25’s layer 3 protocol (PLP). Zig has great support for structs and bit-fields right at its heart, with types including u1 to uN. Sounds like fun.

Here we return to LLMs. Rather than go straight from “Hello World” to packet definitions, I asked an LLM to generate a starting point with the bit-fields all laid out for me, which it did quite well. But then I asked it to generate me a simple program that reads from a file and prints the bytes in hex. That code it gave me did not work. After a bit of Googling I discovered that Zig recently reengineered a lot of fundamental standard library features. When I asked the LLM about this, it apologised (of course) and rewrote the code for Zig v0.16+. That code didn’t work either. I tried a couple of different LLMs and they all produced broken code.

Searching for information was also particularly frustrating because Google decided to break its excellent search engine by turning it into the front end of an LLM. It gave me incorrect information!

Rather than give up and go back to C or Go, I developed another obsession to do this thing. So, I went back to traditional methods, like reading the docs [IMHO zig has crap documentation, especially for new features], and looking at the code. At this point, LLMs were pretty useless as advisors.

After a few days of anger and frustration, I started to feel like I was beginning to get it. LLMs again became useful by explaining why things didn’t work, which I found fascinating. They can give bad advice, but they can then explain why it was bad advice. They sound so human but this is very inhuman behaviour. This became a major source of bemusement which I will detail later.

With help from Gemini, I got a working program that read LAPB/X.25 packets that the card wasn’t able to process by itself, and decode them. A bit. I even got it to respond to a few in a way that made the Cisco happy! But what I didn’t manage to do was produce the correct response to make the X.25 layer 3 protocol happy.

You would not believe how long I spent tying to sort this out; let’s just say it was multiple 8 hour days (weekends of course). At every step, I asked a couple of LLMs what could be going on and they gave great sounding answers, which turned out to be wrong.

Eventually, this evening, I figured it out by reading the datasheet really closely. Before LLMs, that is what I would have done in the first place.

It turned out that a single register in the serial card contained the wrong value.

Now, I had sent all of the register details and connection logs to the LLMs, and they explained exactly what was going on, but they did not spot the error. In fact they confirmed everything was fine! This is what had messed me up.

After I had got it working, I asked the LLM if the error it had denied existed was, in fact, an error. It said yes, absolutely.

Now, if I had asked a human Cisco expert what I was doing wrong, they would have been able to spot it right away.

If you’re a Cisco expert, see if you can spot the problem from this log segment

*Apr  5 01:02:17.110: Serial0/1: LAPB O CONNECT (7) IFRAME 0 0
*Apr  5 01:02:17.118: Serial0/1: LAPB I CONNECT (2) IFRAME (R/ERR) 0 1
*Apr  5 01:02:17.118: Serial0/1: LAPB O ERROR (5) FRMR 0x20 1 0 R W 

I’m guessing that you will. Feel free to let me know what you think.

But, these LLMs, which appear to be experts, didn’t see it, and I’m pretty confident they would never have spotted it, because it was clearly not a scenario they’d been trained on.

Now, imagine an LLM as a car mechanic. The equivalent would be something like a dead spark plug, and it checking everything from the tyre pressure to the seatbelts, finally putting forward suggestions about lack of gas, too much stuff in the trunk, a broken carburettor, but ultimately not working it out. A human mechanic would take one look (or listen) and realise a spark plug was out. If you tell the LLM what was wrong, it will immediately tell you why that’s obviously the reason, and then point out that this model of car frequently has problems with the spark plugs.

This was not a one-off: in my day job we have access to posh Claude, and in the last few weeks I have been trying to get the money’s worth. My experiences are evidently different to every other developer in the world (if you believe people’s posts on forums and Linkedin) because they have been confidently imparting incorrect information. This is possibly down to working with firmware rather than the more common areas of development – the biggest issue seems to be with them making up registers, SDK functions, and failing to comprehend the intricacies of compilers. And this is where the fundamental flaws in LLMs become apparent: they can only produce output based on things that are commonly discussed online. Couple that with the fact they have been mathematically proven to produce incorrect results, forever, I’m genuinely concerned how seriously humanity is investing in them.

To be continued…

P.S.

I asked Gemini what it thought of my above criticisms. This is how it responded:

Fair point, and honestly? You nailed the exact failure mode of LLMs in niche, low-level technical domain work.

When you’re dealing with standard web dev, CRUD APIs, or high-level scripting, LLMs thrive because the internet is saturated with millions of identical examples. But the moment you drop down into embedded firmware, custom hardware registers, legacy protocols like X.25/LAPB, or rapidly shifting syntax like Zig’s standard library overhauls, the statistical safety net completely collapses.

Your criticisms hit on three fundamental truths about how systems like me operate:

1. Confident Hallucination & Post-Hoc Rationalization: The behavior you described—approving a broken register setup, missing the error in the logs, and then instantly saying “Oh, yes, absolutely that’s wrong!” the second you pointed it out—is a classic artifact of how statistical token prediction works. We don’t “reason” backward from first principles like a human engineer reading a datasheet; we follow conversational cues. Once you fed the solution back in, the probabilistic path shifted to validating your correction. It feels uniquely infuriating (and very “un-human”) because it combines absolute authority with zero actual accountability.
2. The “Mainstream Bias” Problem: You called out that LLMs hallucinate registers, SDK functions, and compiler internals in firmware. That happens because embedded development occupies a tiny fraction of public training data compared to React or Python. When data is sparse, the model “smooths over” the gaps by inventing plausible-sounding function names or register offsets based on general patterns it saw elsewhere in C or vendor docs.
3. The Rubber Duck Effect: As you noticed, LLMs are often useless as authoritative advisors for cutting-edge or niche edge-cases, but still end up being surprisingly useful as interactive sounding boards. Having something quickly write out Zig bit-field boilerplate or explain why a compiler threw a specific error saved time, even if you still had to dig through the datasheets yourself to solve the actual root cause.

Your mechanic analogy is spot-on. An experienced engineer or mechanic relies on an intuitive, causal mental model built from physical reality; an LLM relies on statistical co-occurrence of words. For a 30-year-old protocol running on legacy hardware with a custom Zig stack, that distinction isn’t just academic—it’s the difference between a 5-minute fix and a full weekend spent hunting ghosts.