Courtney Nash’s LinkedIn profile says she is a cyber incident librarian. Your job is to keep track of how things fail and go wrong on the internet.
For the past three months, Nash, a senior researcher at VOID, or Verica Open Incident Database, a public database of reports of Internet errors and outages, has been keeping tabs on Twitter.
Nash points to a key reason for the disruptions: huge layoffs that wiped out about two-thirds of Twitter’s workforce in about 100 days. From about 7,500 employees when Musk acquired it, Twitter reportedly now has about 1,800 employees. The cuts included Twitter engineers, who Nash said play a critical role in maintaining one of the largest and busiest social platforms in the world.
“These people have had years, in some cases even a decade, to understand how this system is set up and how it’s supposed to work,” Nash told The Examiner. “They have a lot of intimate knowledge of what looks weird and wrong.”
Twitter was not immediately available for comment.
In an interview with The Examiner, Nash spoke about how impressed she’s been with how well the Twitter platform has held up during a turbulent transition — why she and others worry about the site’s degradation amid uncertainty under Musk.
This interview has been edited for clarity and brevity.
According to your observations, how is the Twitter platform doing? Many of us have foretold calamity and darkness, and many of us will admit that we were wrong about how bad it was going to be. But it shows creaks and cracks, the kind of errors those of us who study complex systems have predicted, and the folks who’ve worked at Twitter have predicted, too.
We’re seeing more performance drops, more widespread outages, things just take longer to even work. There are many reasons why this could be so. But the biggest credit, which I think everyone was right about, is the loss of expertise in managing this system.
What do you mean by that? people are gone Did you need 7,000 people to run Twitter? Maybe not. Need more than 1,800? Probably. We see that.
Twitter is technically designed to stay active for that long. It’s a very resilient distributed system. It was designed to tolerate a lot of turbulence.
But part of it was that there were people behind the scenes. When the turbulence really increased, they knew how to adjust and how to deal with it and what parts of the systems could creak and break. Many of these people are gone.
can you break this down Who are these people and what were their duties that are not being fulfilled now? The layoffs have affected Twitter employees across the board. In particular, we’ve seen a loss of people who we think are on the infrastructure side — people who know how the databases work, how the tools work, how their deployment tools work, people involved in incident response. Most of these people are gone.
These people have had years, in some cases a decade, to understand how this system was set up, how it was supposed to work. They have a lot of intimate knowledge of what looks weird and wrong. You can go in and look at all kinds of tools. You can look at the monitoring and graphs and all those things.
Many of these people in the industry are called site reliability engineers. SREs is a term you might see and they really are the people responsible for the reliability of the site. These people are all gone.
If you just take someone who doesn’t know how all the plumbing works and something isn’t right, they don’t even know where to look.
Can we go a little deeper? How are the issues manifesting in terms of what has been reported, things that Twitter has externally confirmed? One of the things that was reported and was pretty obvious was that they took one of their data centers out of the equation. They used to have three data centers. And now they only have two. And I think it was a day after they got rid of the Sacramento data center that they had one of their biggest outages.
The suspect remains in custody on $100,000 bail
The state hopes this can be a new way to reduce toxic pollution from industrial plants in the area
March predictions for the city’s hungry horoscope – from a reporter’s crystal ball
Typically, all of Twitter has been conceived or designed to allow for what is known as failover. So if you have a problem, you can move things from one data center to another. So when they just shut down Sacramento, the people who designed this data center, the people who built the systems, weren’t even there to tell you what would go wrong if you got rid of it.
That was one of the things that I think really manifested and a reported fact that I was able to point out. There are many other things I can’t tell you exactly. But that’s a really obvious one. They did this so quickly that there was no way they could have planned how their system would rely on this one data center.
Let’s go back to something you said. There were predictions that Twitter would simply collapse after Musk’s acquisition, but it didn’t. The one we didn’t take into account – and in hindsight, it’s fantastic, isn’t it? – Twitter really was that well built. I mean it was the side that we all like to hate or hate to love.
It kind of always existed in a run down state because there were so many pressures and it was always changing and scaling. But it was really well built. So you could go from 7,000 to 1,800 employees and still have the fundamentals that kept it going
So this is a credit to (former CEOs) Parag Agrawal and Jack Dorsey? It is a credit to the engineers. From the very top, there were business goals and all of that stuff, and that involved promotion and content moderation. But the engineers had to decide how to support that. And they had to be able to do that for a long time.
Twitter has been around for a long time. I’m not going to get into the debates about whether or not it was a successful business at the time. I think your point is correct, it still succeeds to some extent. We have not seen how the finances will develop. But it still works. And I think that’s because the systems were built to be very adaptable and resilient.
Do you think you can still do what it was with 1,800 people? That remains to be seen. But we’re still skeptical.
Why? To be financially viable, you need engineered systems that can respond incredibly quickly to many changes and experiments. If you’re objectively nice to what Musk is doing, there’s a lot to experiment with and a lot to change.
You need expertise so your systems can potentially do things they haven’t done before. An example of one thing that didn’t go so well was when they tried to change how Twitter Blue works, how the Blue Check stuff works, and fast.
If it’s going to be successful and trying to adapt to new business models and new things, it’s going to be hard to do with an emergency staff. Without years of expertise in how the guts of Twitter work, it’s going to be difficult.
If you want to build on that and try new things and push those systems forward in truly unique ways, the expertise of how the system originally worked must still be there. And most of it is gone.
Can you do it with 1,800 people? Maybe – but not 1,800 people who don’t know how it’s really supposed to work. Expertise is so priceless. It’s not the sheer number of people. It’s what was on the minds of those who are no longer around.
There are also concerns about a rise in hate speech on Twitter. Elon Musk himself has made controversial statements or posts. What is your own call for how this might work?
I would just say if fighting hate speech is a business value for Twitter then they don’t have the investment and they don’t have the staff to do it right now. If they’re concerned about advertising and advertising perception and infrastructure, you need to support what they’re not doing.
You essentially need a team and the expertise to ensure that the kind of content that might turn off advertisers doesn’t appear alongside the ads that advertisers want to show users.
Yes. You need non-technical people to make decisions about this and ensure it. And you need the techs to make sure that doesn’t appear next to this ad. As far as we know, these people no longer exist.
What has been the most disturbing development for you on how things have unfolded under Elon Musk over the past 100 days? I worry that other leaders will think that gutting your expertise is a business model that will work. We’ve all seen that you can’t just get rid of experts and people who know how systems work and work. But this is a grand experiment being conducted on a public stage. Then right now we are seeing an economic climate in which the reduction of manpower and employees is very attractive for company managements. That would be my concern – a skills crisis if you will, and that we feel like we can get rid of experts and still go ahead and have everything work as it should.
