UKH

Everyone is, rightly, looking at the PO scandal.  But, I can't see there is anything necessarily unique about it.  There must be other computer systems with flaws, and if as seems to be the case the law assumes computers are correct (this is insane!), there are presumably other cases of people being wrongly accused of fraud of whatever type too.  Perhaps time to review such cases more widely?

I'm a software developer. Pretty much every Enterprise level system will have some bugs in it - some you can use a workaround to avoid, others you need to patch.

Faults can be VERY hard to diagnose in complex software, expecially if they are intermittant or require a very specific set of steps to trigger them, which users don't often remember and may not be reflected in log files. It could range from a network glitch to a wrong version of part of the software to a typo in an input file or even a permissions fault - all the way to an actual error in the code.

The issue with Horizon seems to be a lack of investigation and an attempted cover-up by senior staff. If you've got an issue that's only happening at 1 site, that's one thing - when it's happening at multiple sites, then that's a big clue that there's a fundamental issue somewhere.

Loads of issues with big IT projects, just look at anything NHS related. What seems (hopefully) unique in the post office case is the absolute determination of the leadership team to hound the little people through the courts and into prison while not just ignoring, but actively hiding evidence that would've exonerated them. Truly disgusting behaviour.

It's worse than that - A problem was noted in the beta software and reported and they prosecuted 2 people who were using it in the test period. 

Isn't that a bit optimistic?  The leaders and lawyers at the PO are fairly typical of large organisations (they seem to rotate through organisations in fact).  Maybe having lots of semi-autonomous staff handling large sums of money is unusual??

Define scandal

'noun

an action or event regarded as morally or legally wrong and causing general public outrage'

I have worked in IT for the best part of 20 years. I have seen 'scandals' aplenty. Sticthplate, being in healthcare, might be referring to NPFiT which was a national scandal and wasted billions.

Being on the 'inside' so to speak I see all kinds of stuff which the public don't which would cause significant issues for some public sector bodies. And lots of embarrassment.

Right now I am involved in a national program for a specific public sector vertical (which for obvs reasons will not be mentioned). This, I kid you not, is of national importance to all of us and the way it is being managed in the background could be deemed a scandal. Wasteful, costly, inadequate, incompetent are some of the words I would use.

They all are insignificant by the post office standards though, given the human cost.

Think bigger. In recent decades Mid Staffs NHS and Nottingham Hospitals Maternity issues were organised cover ups of long reported problems with shitty treatment of whistle blowers and more people suffered (and died) than in the PO scandal. London Met Uni damaged many students and Staffs lives who complained as the organisation ignored and covered up long standing problems. Where the PO was worse was it was controlling prosecutions and withholding evidence in criminal cases.

All true, but the PO were also relying on the legal nonsense that computers are right (why not have a law saying white is black!?), which makes it almost impossible to challenge accusations. It would seem unlikely this hasn't been abused elsewhere - it's a gift to unscrupulous organisations.

https://www.benthamsgaze.org/2022/06/30/the-legal-rule-that-computers-are-p...

I think Windows Vista pretty much ripped off planet earth and got away with it

Anything Capita touches seems to be problematic. From Wiki.

Former Liberal Democrat leader Tim Farron questioned how Atos and Capita could have been paid over £500m from taxpayers money for assessing fitness to work as 61% who appealed won their appeals. Farron stated, "This adds to the suspicion that these companies are just driven by a profit motive, and the incentive is to get the assessments done, but not necessarily to get the assessments right. They are the ugly face of business."

Taking disabled peoples benefits away, can have a catastrophic effect. There have been several deaths related to the issue.

The assistant coroner issued a so-called Prevention of Future Death (PFD) report to the DWP and private contractor Capita, requiring them to explain what action they might take to improve. That was the fourth PFD sent to the DWP by coroners since 2013.

https://www.bbc.co.uk/news/uk-56819727

Not quite the same league, but serious consequences. The impact on the individual business owners, who went bust, is not noted.

The Financial Reporting Council, which regulates accountants, said the £21m fine was due to the "number, range and seriousness" of issues in KPMG's work.

Carillion's failure cost thousands of jobs and 450 building projects.

https://www.bbc.co.uk/news/uk-67087757

Dunno?

If you can come up with other examples of big corporation’s leadership teams hounding hundreds of innocent employees through the courts while withholding key evidence, then I guess I’m being optimistic.

If not, not.

If they were relying on just that they wouldn't have needed to hide evidence from the defence, nor back down when faced with experienced defence teams, nor lie to the press, nor perjure themselves, nor pervert the course of justice. As such, the relevence of the issue (although an incredibly dumb assumption in law) seems slightly overstated in the more egregious PO cases.

Many years ago there were cases of spurious calls appearing on BT phone bills. Often these were high cost calls to premium numbers. BTs stance was initially that the computerised system was foolproof and that someone at that address must have made those calls so the account holder had to pay up or be cut off.

Later it turned out that some BT operatives had worked out a loophole in the system that allowed them to make calls themselves and allocate them to other customers bills.

It happened to me in a student flat. We had proof that none of us was in the flat on the day (we'd all been at a rugby match when the calls were made) but the BT operative was adamant that the calls must have originated in the flat to the point of suggesting that one of us must have given a girlfriend a key or that the landlord had let himself in to call sex chatlines! 

I think this rule would be asuming that there was evidence that the system had performed correctly over a considerable period in the past for almost all users.  Someone was very dishonest to suggest it should apply to a newly introduced system that caused many problems from the start.  I do not think that the law would make any such presumption in such a case and I would very strongly argue that such a rule has been either fraudulently or recklessly misapplied in this case.  And it says a lot about Systemic British Incompetence in the new era that someone got away with it.

In this case the system had not been tested properly.  I have been involved in very complex systems being introduced which step-changed processes, database technology and hardware technology from glass screen legacy mainframe hierarchical (ICL, latterly Fujitsu) to relational SQL Server based client server.  It was assumed that the system would be parallel run and tested as it was mission critical and of course many bugs were sorted out this way which could never have been revealed by testing in the absence of real users.  I can think of several ways that data can disappear and I've seen it done by "professional" IT people misunderstanding the technology.

It takes the users in the roll out testing to find this out in some cases.  I find the fact that organisations can behave as Fujitsu/PO terrifying, as well as the fact that our legal system allowed it to proceed in the PO's favour for so long.

This is a pretty terrifying and jaw dropping case of systemic incompetence.  Apparently people can no longer recognise the mistakes and bad spelling/grammar that mark out a bogus doctor.

https://www.cps.gov.uk/cps/news/fake-doctor-imprisoned-forgery-and-fraud-co...

Apparently she was even involved in getting people sectioned as she worked round the NHS for many years

I worked in IT (finance systems) for a large multinational and if a supplier had behaved like Fujitsu they’d never have got through the door. We had stringent development testing followed by even more stringent user acceptance testing before anything was allowed near a production environment. The PO was criminally negligent in implementing this system and criminally culpable in its pursuit of the innocent and vulnerable. Fujitsu is criminally negligent in supplying a system not fit for purpose and criminally culpable for perverting the course of justice.

Paula Vennells left the Post Office to become CEO of Imperial College Healthcare NHS Trust…

It seems that once someone gets into a 'network' it becomes relatively simple to move to an organisation, create chaos, then move to another executive role somewhere before the mess is uncovered. Repeat that every couple of years and it's a nice little earner.

I think for the PO its a mix of factors which made it problematic. A key one being the relationship with the SPOs and also the complexity of the systems. Most franchisees although they have to pay money up the chain dont have the same sort of relationship.

Something which has been reported is the crown post offices (eg those still under direct control)  and those post offices which were part of large chains although they had the same problems had the money written off whereas the small owners got the book thrown at them.

Billing bugs arent unknown but generally get managed via the watchdog.

Rather expensive bugs include:

Knight Capital accidentally pushing some test code to prod. Lost 400 million in a few seconds and went bankrupt.

Therac-25 radiation therapy machine had a bug which killed at least a couple of patients. The manufacturer handwaved it off initially until a hospital managed to reliably reproduce it.

Seagull management.

The trick seems to be to stay somewhere just long enough so you can show the benefits of your changes eg lower costs and then get the next job before the underesourced team falls apart.

“Toby Turnaround” made a lucrative career doing just that in the NHS (allegedly).

I'd imagine that a lot of organisations aim to be quite conservative about very senior roles, preferring to hire in people with a known record in similarly-sized organizations, and that cohort isn't huge.  I guess they should also be asking why their new candidatate is so keen to be moving jobs!

I was quite astonished that this was a legal presumption.

As an ex-developer I know that however much large software systems are tested during development and production, the real testing only happens when real users are let loose on the system.

During development and production there is an assumption about how a system will be used, and however much you try to think laterally during all phases of testing, that assumption is still lurking around.

Actual users don't have that assumption, their ideas about how to use the system will be different and varied, taking the software down paths nobody expected.

Every experienced software developer knows this (I'd wonder about their "experience" if they did not).

When the software provider has a room full of people going into live data and changing that data (or adding to it) without the knowledge of the people responsible for the data then Fujitsu are fully aware there is a serious data issue and that they are acting criminally. If the PO are also aware of this then the PO are also complicit in this criminality. Those responsible need to be in court.

I completely agree with you that end users will find issues that may not have predicted by QA. Here's an example:

Thinking back to one of the trickier bugs I found (and fixed). A system I worked on many years ago allowed the user to create an initial data record with a set of information in it. There was then a mechanism to progress that record into another state, triggering various other things to happen.

Some users started reporting that when creating the initial records, some of them seemed to be being auto-progressed. Try as we might, we couldn't reproduce this and it was happening apparently randomly.

Eventually, we found that the crucial step in reproducing this was that the users had aborted the mechanism for progressing a record (there were a couple of dialogue boxes in there that allowed this) before creating the next initial record, which was then auto-progressed. Aborting the mechanism had left a background variable set when it shouldn't have been which then got picked up in the store process of the new record.

QA test scripts would have tested the creation of a record and then the acceptance mechanisms (including aborting it), usually independently, but the scenario described above was not part of a test script and so got missed.

After a minor RTA near Heathrow on a weekend evening a few years back, I was standing with the copper attending while he was trying to work his new tablet to take down the details; he was the only one on shift in West London. It just didn't work - period. I've never seen someone so able to manage his anger and frustration, although the subject of retirement did come up.

Sadly there has been a culture of expected failure in large IT projects in the UK since the 80s. The Post Office is just the worst and most recent one, but probably not the most expensive.

Here's one of the many search results:

https://beyondcommandandcontrol.com/library/failures-of-change/examples-of-...

The one shining example of fitness for purpose and continuous improvement has been the HRMC Self-Assessment Income Tax Return system. I wonder why that might be

The digitalisation generally of HMRC including live information on payroll which enabled covid payouts to be done quickly to employers was very succesful.The Tory MP- Francis Maude was a big advocate in getting these type of systems up and running.Other digitalszation services like drivers licences, tv licence s etc have also been succesfull. This shows it can be done well.There are probably other public sectors ones which work as well, you just do not hear about them. You only hear of the failures and fiasco.

However,speaking as an accountant and IT person, having a finance system that doesn't balance seems to me to be a fairly basic issue that should have been picked up very quickly indeed! 

To be fair, these are simple projects. One could get a GCSE student to knock up a database, and web front end to do either. Address, TV y/n, licence y/n, renewal date, wouldn't be too much trouble. Driving licence would be similar, but with classes of vehicle, points and banned or not. (Yes, I know these are scalable, interconnected systems, but you get the idea.)

HMRC system, is better but still not great. It remembers some information, and even fills some of it from your payroll info. However, there are a lot of areas where it could do with some serious improvements. 'Tell us if you get any benefits in kind like health insurance'. Well, you've taxed me on that already, do I need to mention it again? And so on.

They even make paying hard. They will accept a company credit card, but not a personal one. (I know there are extra charges, but they could simply add those on.) You can't simply set up a payment plan, pay over X months, in a few clicks there and then. HMRC please take my money!

These are usually the ones where it was scoped to cost £x over 3 years. But it actually cost £x++, its now 5 years and it still hasn't delivered what it was supposed to do. Yet the supplier is being paid, as we invested so much we cant cut our losses now.

We shouldn't need to congratulate ourselves when something goes to plan. That's the point of a plan

And who, if any, were the auditors?

This is one that few of the general public know about https://en.wikipedia.org/wiki/Digital_Media_Initiative

In short the BBC spent £100 million on a database that never launched.  Bit of a scandal, there was an investigation and some people lost their jobs but it never really got picked up by the press to a great extent.

Absolute shambles really....

Greater Manchester Police:  was in a meeting 4 years ago about Hate Crime and asked ‘what types of hate crime are being committed in what areas?’, (so that we can target our work). The answer was ‘we can’t tell you that coz our IT system is rubbish’.  I was stunned.  Worth a Google - multiple problems resulting in them sacking it off. Some of the problems were serious, eg. impacting on responding to domestic abuses cases. They got dragged over the coals for it, contributing to them being put under the equivalent of Special Measures iirc. £60 million. 
 

https://www.bbc.co.uk/news/uk-england-manchester-60826585.amp

I've worked on various IT systems for large organisations since the 1980s. 40 years ago there were two overriding requirements:

1) Transactions will not go "missing"

2) For each balance affecting transaction, there was some form of audit transaction hitting a ledger that ,when reconciled, would highlight discrepancies.

There would be a team whose job was to reconcile the system every day. There would be hell to pay if there was a misbalance at the start of the next business day!

In my experience this all started to change 25 years ago. IT got big into outsourcing, buying packaged solutions then trying to bolt everything together. I got helicoptered into one such project when it was in its first week after implementation. There were discrepancies in customer mortgage balances between the new sales system and the new admin system. The two suppliers were blaming each other and the outsourced IT service provider did not want to take responsibility for the shambles. It was quickly apparent that transactions were being randomly dropped between the two systems. The temporary solution was an Access database that extracted transactions from the two systems, reconciled them and highlighted discrepancies. Users could then post the missing transactions manually to the admin system. Equivalent functionality should have been designed in from the start and after a few weeks a "proper" solution was built into the interface. What was interesting about this was the reluctance of the suppliers and outsourced IT service provider to take ownership of the issue. They didn't care that customer mortgage balances were incorrect, all they cared about was protecting their reputations at all costs. They were deliberately obstructive!

Nick Wallis did quite a good podcast episode about "where did the money go?": https://audioboom.com/posts/8417438-ep-34-where-did-all-the-money-go

If I understood what was said correctly, Horizon wasn't actually designed to be a strict double-entry bookkeeping system in that it didn't necessarily always require a balancing debit for each credit (or vice versa).  It seems that a lot of discrepancies were 'fixed' with a thing called a "transaction correction", which basically adjusted one account to make things balance.  Fair enough-ish if you're confident as to how the debit/credit got missed/skipped/lost in the first place but (caveat: I Am Not An Accountant) I don't see how a system which regularly and almost as a consequence of its design allowed such a thing to happen could even be considered as suitable in the first place.  It's not as if ACID transactions in distributed systems had yet to be invented back around the turn of the century.

This is one of the main things that the first class action trial was about.  The judge ruled that the contract which every SPM had to sign, and especially the Post Office's interpretation of it, was unenforceable in the way that the PO had been using it.  (Some parts of the contract appear to have dated back to the dark early history of the Post Office more or less unchanged.)

Computers do make mistakes, but it takes human beings operating in a combination of this-is-how-we've-always-done-it-ism, blinkered groupthink and increasingly frantic arse-covering to end up jailing, bankrupting and driving to suicide hundreds of innocent people.

As you say, actual PO employees in crown offices were not treated the same way, and this was largely because they didn't operate under the onerous and legally dubious contract terms that the Post Office imposed on its SPMs.

Probably why those post masters who also used competent  solicitors were able to get allegations thrown out . Their solicitors will have been competent at reading the contracts and calling them out. 

It makes sense that the focus of the reporting has been on the shortfalls but, does anyone know if there were also mistakes in the opposite direction? 

I'm pretty sure that the developers of Horizon would have looked at you blankly if you'd asked them to describe double entry bookkeeping - or database normalisation, for that matter.

These are the fundamental building blocks of any financial systems; they're no more optional or arbitrary than a knowledge of stresses are to a civil engineer.

IT is an industry based on fads, fashion and PowerPoint; which is a shame, because it COULD deliver. It just hasn't yet.

Or asking various other awkward questions that might bring the POs house of cards tumbling down. That they folded when properly challenged is another indication, if needed, of how they knew what they were doing was wrong.

One of the, many, dodgy aspects of this affair was the post masters primary "trade union" the NSFP  was basically owned and controlled by the post office. So they couldnt get help there.

A franchise with unreal statutory historical powers. It’s not many organisations who have their own minister for example. But it also shows the historical importance of the Royal mail  for communications.  ( letters) and paying out pensions , benefits etc.  these are now historical anomalies with email and digital payments.  

Latest reminder 

https://www.theguardian.com/uk-news/2024/jan/10/infected-blood-scandal-vict...

Another Systemic Incompetence the possibility of which introduction to our society I can barely conceive of is that run by the Land Registry whose system/procedures/database allow someone's house to be stolen from them!

I find it quite hard not to feel that a lot of our society is now run by intellectual paupers who design systems that "might just work" on a good day and have not considered sources of error/modes of failure.  I can remember even when doing my Computer Science A level in 1979-80 on Tuesday evenings at Durham Techniocal college being paranoid about programs not working correctly and that is the way I spent my career.  At my last job those who came up with quick solutions which often failed unpleasantly gained the favour of management and those of us who thought hard to prevent problems and took too a long time to get it right were considered some kind of 5th columnist.

Needless to say it was we lesser mortals who had to pick up the pieces.  At one team meeting on a Monday afternoon we found out a member of the team was leaving on Friday and that I would have to pick up the systems she was responsible for.  I was specifically told not to go to her for information and ask her too many questions before she left as she would be far too busy that week to pass the information on.

You need to get with the program, grandad!

Its all about DevOps these days. Move fast and break things. If its not in the sprint, we dont care. Long term planning is now 2 weeks; time to the next sprint. Anything tricky, is considered Tech Debt. Its put into the 'to do later' pile.

Ahh, the old 'doing documentation'. Generally, peoples permissions are withdrawn, in case they steal or sabotage things. They dont want to pay them off to leave early, so they are put on 'documentation'. At one place, I was on a months notice. I was like a dead man walking. I didnt get any work, I was removed from projects and meetings. I was told to document things, but when I asked what needed documentation, there was no reply. I still had to go in every day though.

Depends a bit on how you define scandal, but the justice system in the UK has some...surprises when you dig in to the detail.

For example, the threshold for legal aid is surprisingly low. There's a large middle ground of people who do not qualify for legal aid but cannot afford to pay for legal representation. The end result is you can have someone who is completely innocent, but who either has to financially cripple themselves to have a lawyer/barrister defend them or who has to try and represent themselves in court with the subsequent risk that they'll miss something and be wrongfully convicted.

Feels pretty immoral to me.

My tuppence worth on post office/horizon bolx. I have been in siuations where systems failed or were imposed before they were ready. An pub POS system where only 1 till could be open at any time trying to deal with a Newcastle pub on a busy saturday lunch time supplied with 6 tills. A NHS warehouse system that gifted a box of contaraceptives with every order processed (mea culpa! something for the weekend?).

When designing large software systems we have a number of tools/techniques to test the validity/accuracy of software but if you think of software as recipe or plan then as Moltke the Elder said “no plan survives contact with the enemy”. Users are not the enemy.

As part of the Thatcher revolution the need for inhouse expertise was frowned upon, I was made redundant as part of this "market knows best" philosphy. Civil servants who knew how there organisations/systems worked were recruited by outside software companies/system integrators on enormous salaries, worked for a couple years providing comfort for the organisations who they used to work for and at the same time transfering there knowledge to in-house "consultants". Then after a couple of years given a retirement /redundancy package.

Without trusted advisors/knowledge of there own systems purchasers cannot make proper assessments of whats on offer and the dreaded "commercial in confidence" clause/complexity stops full evaluation of like-with-like. The UK chucked away £10's of billions when they threw out huawei router kit because they couldnt reverse engineer it to check for a back door login for Xi Jinping.

Once a purchase is made then the pressure is on to get it customised/implemented/delivered, careers are made by being on time/within budget with the supplier always looking to maximise billing for "extras".

Fujitsu had “team of eight [people] in the development team, two were very good, another two were mediocre but we could work with them, and then there were probably three or four who just weren’t up to it and weren’t capable of producing professional code”. But you can be sure a whole range of chaps/chapess's in suits who targeted the post office management with nice dinners, meetings in fancy hotels,etc to make sure the Fujitsu narraitive remained dominant. The users/post masters voices could be dismissed until a product champion/stake holder came along who couldnt be ignored. It was computer weekly in May 2009 provided an alternative view. https://www.computerweekly.com/.../Post-Office-Horizon...

So as one scandal is revealed the next disaster is just starting.

https://www.opendemocracy.net/.../nhs-palantir-contract.../

Interesting but ironically neither of your links work! 😀

Compounded by the state’s “no baxies” policy.  If your house is stolen through the incompetence of a state agency, it’s tough luck and a payout that doesn’t come close to restitution beyond the direct fiscal loss.

Well, not quite *anything* NHS related, unless by "issues" you just mean "bugs" rather than systemic flaws - in which case, yes, obviously.

In 2017 I worked on porting the legacy SUS system from proprietary software to open source ultimately saving the NHS tens of millions per year in maintenance costs. NHS Digital in particular is very highly regarded.

My wife works in the NHS and they recently implemented a medicines management system (I think it’s quite localised) that isn’t fit for purpose and actually makes her job more difficult. In basic terms as far as I understand they bought an off the shelf system that was developed in the US so doesn’t work effectively with the very different drugs used in the U.K. and is more unwieldy than what they previously used.

That's some computing form of sods law. 

A search using obvious keywords finds multiple correct links on those important stories.

Here is a new one waiting to be unpicked... likely a combination of service pressure and declining mental health support.

https://www.theguardian.com/society/2024/jan/11/number-of-women-in-uk-who-d...

I can only speak from experience at the pointy end where Byzantine is the norm and if it ain’t broke just wait for the next software update and it will be.