The Dumbest Mistake I Did as a Software Developer (The Biggest Lesson I Learned)
I have shared my real-life incident where I made a huge mistake in a production server while developing a product
If you are a developer joining a small startup, or a founder looking to launch a new feature, this real-life story might save you lakhs of rupees and months of wasted effort.
I still remember that day.
It is Nov 6, 2025, in Coimbatore, Tamil Nadu. I was working as a full-stack developer in a small startup company. I had just joined 1 month ago and started to work on a product.
The actual requirement is very simple.
This company has multiple products (3 in total). They needed a centralized wallet system where one customer can add money to the wallet and, with the same login, they can also access our other products and use the same wallet.
This company had one successful product which had a customer base of 1400, with 56 premium customers, and the rest of the products were just launched.
After a heavy argument, we finalized putting it as a separate backend microservice.
I decided to use Postgres and Prisma ORM to create a simple wallet.
Then came the problem.
At the beginning, they told me I could use any tech I wanted, but later they said setting up Postgres is a little too difficult as our company was already using MySQL.
We already knew the configuration, so it was easier to set up, and they asked me to change the stack to MySQL.
I developed the product and asked my DevOps engineer to give me the credentials for the MySQL dev server.
He gave the credentials, I plugged them in, and passed all the tests in the dev server.
Now the real action begins.
We had to move this to production.
So, I asked my DevOps engineer to update secrets in HashiCorp Vault.
He did that, and I tried to deploy, but unfortunately Prisma started throwing so many annoying errors.
I was surprised because it worked very well in the dev server, and I thought it would be a walk in the park.
But this prod server kept annoying me during deployment.
So, in IT we all do one thing — if something does not work, restart it first.
That's the exact thing I did.
Since it was a new feature and no one used it, I just wrote a Prisma command to drop the table and create it again and deployed it.
This time it deployed properly, and I was so happy that I moved this to prod within a week.
But you know what?
It was actually catastrophic.
I thought it was a fresh DB, but the DevOps person gave me a shared DB, which was shared with that product having 1400 customers.
What you are thinking is right.
I dropped the entire prod database with 1400 customers in it, with 56 premium customers, and the data was lost.
The product manager started calling and complaining that in the middle of her demo, the server went down and it didn't recover.
But then the DevOps person stepped in and recovered the DB from DigitalOcean Spaces.
This is surely the dumbest mistake I have ever done in software engineering.
But I wanted to ask the viewers one question.
Whose fault is this?
Is this the DevOps engineer's fault or my fault?
Because I previously worked in an MNC where they gave isolated DBs for all products. They would not give a shared DB.
But what the DevOps engineer claimed was that here we need to reduce cost and complexity.
Setting up a new DB requires too much time and energy, so they give a shared DB if the stack is the same.
Maybe he got a point.
But he could have told me it was a shared DB, or as a senior engineer, I could have asked him whether this was a new DB or a shared one.
I am just imagining what would have happened if we had no option to recover the DB.
Thanks to technology.
But I learned a lesson that day.
If anyone provides credentials or secrets, I must ask whether it is a new instance or a shared one.
And most importantly, do not drop a database for my convenience.
Only do it if there is no other option.
Because I could have solved it without dropping the DB.
The funniest part is that the founder of that company still doesn't know this yet.
Maybe one day he reads this article and tries to figure out that this incident happened in his own company.
