On Call Sure, you might use words like "boom" and "explode" when it comes to errors with your system. But could a whoopsie have the potential to render a chunk of a country uninhabitable? Welcome to On Call.

Our story comes from a reader Regomized as "Ellen" who spent the early part of the 1980s toiling away in the IT department of a company producing software responsible (in part) for running nuclear power stations.

A brand new system was in the process of being rolled out, which would keep track of which stations were online, how much power they could provide, and so on.

Commissioning was underway using a test rig connected to a new reactor under construction. "A team of Americans were in the building teaching the company's managers how to use their system," explained Ellen, "which was to sit on top of ours."

"They were providing the reactor control equipment that our system talked to."

It wasn't going well. Despite some lovely gear for the time (think a curved wooden desk with inset DEC Rainbow PCs in the control room and a bunch of VAX/VMS systems in a fail-safe cluster), there were problems getting the VAX to talk to the power station. The line was up, but there was no communication.

"Hair was being pulled out, time scales were collapsing, and I was getting white hair with stress," said Ellen, "not least because I was on a successful completion bonus."

As is so often the case, a seemingly inconsequential setting was changed and everything sprang to life. The equivalent of a ping was sent and the reactor responded: "Yes, I'm here."

"Strictly speaking, this was a reactor simulator," Ellen added, "a fact that will become important later."

However, for now, things were online, the software was working, and while there were only three days to plow through 10 days' worth of tests, the team at least had a fighting chance. Tests were set to execute sequentially overnight.

The Rainbow PC in the comms room would run them and dump the results to the printer. Ellen and co explained the approach to the site manager at the end of day meeting, which the Americans also attended. Another important point.

Yet despite the money-no-object approach, bizarrely there was no lock fitted to the computer room door. While nobody was supposed to touch the equipment, Ellen's team took no chances and stuck a cardboard box over the PC with the words "System Under Test – DO NOT TOUCH" scrawled over it.

"It was now well around 10:30pm," she recalled, "so the team and I set off for our hotel, aiming to reconvene at the 8:30am Morning Meeting."

Sadly, The Call would come in a good few hours before that morning meeting. This being before the days of the ubiquitous mobile phone, Ellen had a pager which chirped urgently at 6am. She had to attend the site NOW!

When she arrived, the tension in the atmosphere was palpable. Something had gone terribly, terribly wrong. The manager of site was also in attendance, as was the biggest of all cheeses – the Director of Power Generation.

"A sort of deathly silence fell over the room," she recalled, "the sort just before a public hanging takes place."

"It seemed that our software had experienced some sort of problem and as a result the reactor had gone offline, the control rods had slammed in, and it was now no more than an oversized kettle."

At this point we must remind readers that this was a simulator, not the real thing.

Had this been a real reactor, it would have taken months to recover, at a cost of millions of pounds.

And Ellen and her software were clearly to blame.

"No one could tell me what exactly what had happened. Just that it was my fault," she said.

Seeking to delay her execution, Ellen asked if she could review the output of the line printer to get an idea of what might have happened. The bosses agreed and gave her an hour's reprieve as she scuttled off to the comms room.

Upon entering the room (the one without a lock), she and the team were greeted by a scene of utter devastation. The box with the "Do Not Touch" lettering had been discarded. The test PC was in bits and the disk was missing entirely. The line printer had stopped mid-line when the PC had been attacked.

Alarming, but not something that would cause a reactor scram, just a delay in testing.

"I asked one of my team to connect the printer back to the VAX and dump the application logs for me," recalled Ellen. "He was told to bring them to me even if I was in a meeting – especially if I was in what was going to be a stressful meeting, to say the least."

The investigation continued and got stranger still. The other Rainbow PCs were all up and running. They shouldn't have been – Ellen's team had yet to commission them, merely setting them up for cable routing purposes. And yet there they were, humming away.

Ellen returned to the meeting with her findings. One of the US team was in attendance, and confessed to switching on the PCs.

"When asked why," said Ellen, "he responded that because us amateur-hour Brits were so far behind schedule he wanted to get started training the control room staff, so he wanted all the PCs booted and ready."

So… how was this achieved? The media to boot up the PCs was locked up in Ellen's safe store ("the back of my car," she confessed).

No problem. The US tech had simply grabbed the disk from the PC running the testing and copied it to the other computers. "Obviously it worked because they are all up and running," he said.

Suddenly, everything became clear. Had Ellen a Poirot-style mustache, some serious twirling would have been called for.

The logs arrived and were handed over. Ellen pretended to study them, but already knew what the evidence was going to show.

Ellen: "So, you cloned the disk on to all the PCs..."

US Engineer (proudly): "Yes, and saved several days".

Ellen (looking at the log): "And you went to one PC and asked for a reactor status from the power station."

US Engineer: "Yes, but it didn't work – your software is so full of bugs, it's total crap."

Ellen: "And when you cloned the disks, you changed the DECNet address on each PC?"

She, of course, knew that he hadn't. The log said as much.

US Engineer: "Err, no, what's that? Is it important?"

The protocol used for communication was designed to avoid hacking. "There were multiple control commands," explained Ellen, "to eliminate any false commands that could, quite literally, cause a bomb to go off."

In this instance, all the PCs now had the same address, meaning that when communication was attempted (for example, a simple status request from the reactor), all manner of nonsense would bounce around the network. The reactor (or, to be clear, the simulation) software decided that something weird was happening and correctly triggered its safeties. In this case, an immediate shutdown.

An extended recovery time (had this been a real reactor) was of no consequence compared to safety in the face of what might be an attack.

"After explaining all this to the now-silent room," Ellen said, "I finished with telling the Director of Power Generation that it was not our fault."

"It was someone, mentioning no names, who had disassembled our equipment and had misused our software and hardware, all before we had handed it over. The system did exactly what it was supposed to."

"And whilst simulating a reactor scram was not part of the tests, we now knew it worked."

The US contractor did the equivalent of falling on his sword. Puce-faced, he left the room, was apparently fired the same day and packed on the next plane home.

Again, this was not a real reactor and Ellen knew that the team could get back online in a matter of hours. However, "I shamelessly lied through my teeth, told the assembled team it would take me at least two weeks to reassemble the equipment, recommission all of our test and control equipment, and that I was declaring force majeure as per the contract, but I would not report the damage back to my head office."

The room was filled with apologies and gratitude that she would not be taking the issue further and that there had been no unpleasantness. The time extension? No problem – it was granted.

The team finished well ahead of time and bonuses were dispensed all round.

"And that," she said, "is how I was accused of nearly wiping [region redacted] off the map."

Ever had your bottom rescued by a fail-safe? Or been called out at an ungodly hour to deal with someone else's mistake? Of course you have, and you should share your story with an email to On Call. ®

More papers describing the orders and messages the US President can issue in the event of apocalyptic crises, such as a devastating nuclear attack, have been declassified and released for all to see.

These government files are part of a larger collection of records that discuss the nature, reach, and use of secret Presidential Emergency Action Documents: these are executive orders, announcements, and statements to Congress that are all ready to sign and send out as soon as a doomsday scenario occurs. PEADs are supposed to give America's commander-in-chief immediate extraordinary powers to overcome extraordinary events.

PEADs have never been declassified or revealed before. They remain hush-hush, and their exact details are not publicly known.

Russian crooks are selling network credentials and virtual private network access for a "multitude" of US universities and colleges on criminal marketplaces, according to the FBI.

According to a warning issued on Thursday, these stolen credentials sell for thousands of dollars on both dark web and public internet forums, and could lead to subsequent cyberattacks against individual employees or the schools themselves.

"The exposure of usernames and passwords can lead to brute force credential stuffing computer network attacks, whereby attackers attempt logins across various internet sites or exploit them for subsequent cyber attacks as criminal actors take advantage of users recycling the same credentials across multiple accounts, internet sites, and services," the Feds' alert [PDF] said.

Amazon, Apple, Google, Meta, and Microsoft often support privacy in public statements, but behind the scenes they've been working through some common organizations to weaken or kill privacy legislation in US states.

That's according to a report this week from news non-profit The Markup, which said the corporations hire lobbyists from the same few groups and law firms to defang or drown state privacy bills.

The report examined 31 states when state legislatures were considering privacy legislation and identified 445 lobbyists and lobbying firms working on behalf of Amazon, Apple, Google, Meta, and Microsoft, along with industry groups like TechNet and the State Privacy and Security Coalition.

America's financial watchdog is investigating whether Elon Musk adequately disclosed his purchase of Twitter shares last month, just as his bid to take over the social media company hangs in the balance. 

A letter [PDF] from the SEC addressed to the tech billionaire said he "[did] not appear" to have filed the proper form detailing his 9.2 percent stake in Twitter "required 10 days from the date of acquisition," and asked him to provide more information. Musk's shares made him one of Twitter's largest shareholders. The letter is dated April 4, and was shared this week by the regulator.

Musk quickly moved to try and buy the whole company outright in a deal initially worth over $44 billion. Musk sold a chunk of his shares in Tesla worth $8.4 billion and bagged another $7.14 billion from investors to help finance the $21 billion he promised to put forward for the deal. The remaining $25.5 billion bill was secured via debt financing by Morgan Stanley, Bank of America, Barclays, and others. But the takeover is not going smoothly.

Cloud security company Lacework has laid off 20 percent of its employees, just months after two record-breaking funding rounds pushed its valuation to $8.3 billion.

A spokesperson wouldn't confirm the total number of employees affected, though told The Register that the "widely speculated number on Twitter is a significant overestimate."

The company, as of March, counted more than 1,000 employees, which would push the jobs lost above 200. And the widely reported number on Twitter is about 300 employees. The biz, based in Silicon Valley, was founded in 2015.

A researcher at Cisco's Talos threat intelligence team found eight vulnerabilities in the Open Automation Software (OAS) platform that, if exploited, could enable a bad actor to access a device and run code on a targeted system.

The OAS platform is widely used by a range of industrial enterprises, essentially facilitating the transfer of data within an IT environment between hardware and software and playing a central role in organizations' industrial Internet of Things (IIoT) efforts. It touches a range of devices, including PLCs and OPCs and IoT devices, as well as custom applications and APIs, databases and edge systems.

Companies like Volvo, General Dynamics, JBT Aerotech and wind-turbine maker AES are among the users of the OAS platform.

Nvidia is expecting a $500 million hit to its global datacenter and consumer business in the second quarter due to COVID lockdowns in China and Russia's invasion of Ukraine. Despite those and other macroeconomic concerns, executives are still optimistic about future prospects.

"The full impact and duration of the war in Ukraine and COVID lockdowns in China is difficult to predict. However, the impact of our technology and our market opportunities remain unchanged," said Jensen Huang, Nvidia's CEO and co-founder, during the company's first-quarter earnings call.

Those two statements might sound a little contradictory, including to some investors, particularly following the stock selloff yesterday after concerns over Russia and China prompted Nvidia to issue lower-than-expected guidance for second-quarter revenue.

HPE is lifting the lid on a new AI supercomputer – the second this week – aimed at building and training larger machine learning models to underpin research.

Based at HPE's Center of Excellence in Grenoble, France, the new supercomputer is to be named Champollion after the French scholar who made advances in deciphering Egyptian hieroglyphs in the 19th century. It was built in partnership with Nvidia using AMD-based Apollo computer nodes fitted with Nvidia's A100 GPUs.

Champollion brings together HPC and purpose-built AI technologies to train machine learning models at scale and unlock results faster, HPE said. HPE already provides HPC and AI resources from its Grenoble facilities for customers, and the broader research community to access, and said it plans to provide access to Champollion for scientists and engineers globally to accelerate testing of their AI models and research.

HR and finance application vendor Workday's CEO, Aneel Bhusri, confirmed deal wins expected for the three-month period ending April 30 were being pushed back until later in 2022.

The SaaS company boss was speaking as Workday recorded an operating loss of $72.8 million in its first quarter [PDF] of fiscal '23, nearly double the $38.3 million loss recorded for the same period a year earlier. Workday also saw revenue increase to $1.43 billion in the period, up 22 percent year-on-year.

However, the company increased its revenue guidance for the full financial year. It said revenues would be between $5.537 billion and $5.557 billion, an increase of 22 percent on earlier estimates.

The UK's Competition and Markets Authority is lining up yet another investigation into Google over its dominance of the digital advertising market.

This latest inquiry, announced Thursday, is the second major UK antitrust investigation into Google this year alone. In March this year the UK, together with the European Union, said it wished to examine Google's "Jedi Blue" agreement with Meta to allegedly favor the former's Open Bidding ads platform.

The news also follows proposals last week by a bipartisan group of US lawmakers to create legislation that could force Alphabet's Google, Meta's Facebook, and Amazon to divest portions of their ad businesses.

Microsoft has hit the brakes on hiring in some key product areas as the company prepares for the next fiscal year and all that might bring.

According to reports in the Bloomberg, the unit that develops Windows, Office, and Teams is affected and while headcount remains expected to grow, new hires in that division must first be approved by bosses.

During a talk this week at JP Morgan's Technology, Media and Communications Conference, Rajesh Jha, executive VP for the Office Product Group, noted that within three years he expected approximately two-thirds of CIOs to standardize on Microsoft Teams. 1.4 billion PCs were running Windows. He also remarked: "We have lots of room here to grow the seats with Office 365."

The Register - Independent news and views for the tech community. Part of Situation Publishing

Biting the hand that feeds IT © 1998–2022