So someone didn’t read the documentation and didn’t understand that site is used in many contexts? How is this DataDog’s fault? Blame your shitty process instead of shifting it to save face.

I worked in IAM and then IAM software dev for JPMC back in the day, so I worked with the cyber Sec people a lot. I was burned out, those people were fried. I felt so bad every time I had to bother them. Walking up to someone’s cubicle and they would be so startled and you could tell their nerves were just toast.

I ended up being an application security engineer for a while at a different company and I was the only one. After 6 months I went back to just software dev and got out of security all together honestly. I love the stuff but the lack of resources and funding but with massive expectations is a nightmare.

When I was working in that old job we had one particular fiasco that legit stresses me to remember. We have this account, no one knows what it does, but the password has never been rotated, it’s not vaulted, etc. There’s 5 apps that share the DB. I contact all the app owners, no response.

I wait a week and escalate the their bosses. No response. I send emails every single day to everyone including all the dev teams. Not one “lemme check on that” or anything. Our policy was to wait 90 freaking days for a non single user account. I’m getting yelled at to get this ticket closed when the day comes.

I go in, lock the account, change the password, and kill all DB sessions. Within 15 minutes I’m paged for a priority one incident because a trading app is down, causing the whole floor to be out and they’re losing millions every minute.

I tell them what I did and forwarded emails to everyone. The executive director is screaming at me, telling me I’m gonna be fired soon and I better fix it right now.

Sure, I can unlock the account and even force the password back to the old version. What’s that? No one knows what the old password was? Nothing I can do. Fortunately my executive director was awesome and stepped in to take the call. Overall they were down for an hour and a half. I looked at the incident later and they claimed $100 million in losses. The app owners wanted me fired. They got the uno reverse though and lost their jobs over it.

Fuck that job lol.

It’s such an accurate term. I worked in IAM for a while and when no one claimed ownership of an application account, we’d go with a scream test. Lock the account and see who screams at us lol.

I used to be a sysadmin, now a software developer. At one of my old jobs for a massive corporation, they decided to consolidate several apps’ db servers onto one host. We found out about this after it had already happened because they at least properly setup cname records so it was seamless to us. Some data was lost though, but having literally billions of records in our db, we didn’t notice until it triggered a scream test for our users. We were also running up against data storage limits

They ended up undoing the change which caused us a data merge nightmare that lasted several full workdays.