I worked on a functionality end to end using AI. It was a simple feature and AI wrote everything correctly in a single pass. I used Codex. After that, I had to ask it to adjust some edge cases, add more validations, and improve error messages. I also asked it to write an end to end test script, and now I have a script to test the whole thing.

After raising the PR, my boss suggested some changes. So I went back to Codex, asked it to make those changes, and it did. I again had to adjust some edge cases, but the tests were already there, so it was easy to verify everything. One thing I noticed was that the code had access guardrails, but it did not reason about determinism. For example, it did not check whether a variable could only have two possible values and still adjusted for a third condition. I think context is still a bottleneck, or at least a difficult problem.

I also did some systems thinking :P and helped a few colleagues with their issues.

I completed a service migration end to end using AI. Did it make mistakes? Sure. Did it correct those mistakes? Yes. I had to provide some logs and data, and it fixed the code. It was honestly life saving. It reduced a lot of stress and cognitive load for me.

As the data in the database increases, you start facing different kinds of problems. I think a heavily used service is a good problem to have, but you also need multiple guardrails and self healing mechanisms just to maintain consistency.

AI has taken a lot of load off my back. Otherwise I would have spent multiple hours on code migrations. Now I can spend that time looking at the system and thinking about what can be improved. If you are a software engineer and want to sleep well at night, you need self healing mechanisms and ways to guarantee consistency. But nothing is perfect, so there is always some uncertainty.

I also designed the full flow for the functionalities I built and made some changes to offload rate limit healing to Celery. I also fixed some very small bugs.

What else... hmm.

I set up my projects on a completely new server because we were doing tech stack migrations. It was fun. I had to take care of many things, but then I realized the project is almost fully self contained. Just add the environment variables and it works.

I think keeping things simple will become even more important now that AI is going to do a lot of the coding. You have to explicitly ask it to keep things simple. Otherwise when things fall apart, you will not be able to fix them easily. And at some point, things will fall apart.

I submitted all my self review documents. Let's see if I get a pay bump and a good bonus. I hope so.

I am trying to become a research engineer now. I am interested in inference infrastructure, and I think moving toward systems engineering would be a worthwhile pivot. With AI, we will have more time to experiment, and I plan to do exactly that. I am not very interested in training LLMs because I do not have that level of mathematical background, but I think I can engineer better systems. Making inference cheaper can add a lot of value to a company. If a company could build its own nuclear power plant, its electricity costs would drop significantly and margins would improve. More profit is always better, right?

Am I anxious about my career? Yes. Do I want to work on cool things? Yes. Am I qualified to work on cool things? Who cares. I will learn. I think it is time to take risks and maybe even cold email Sam Altman :P

I do not have much more to write. March was a bit slow. There were many production issues and we were focused on streamlining processes. This blog probably does not make me stand out as engineering talent, but I was mostly working on making things more stable and slightly faster. I did make one API faster though, and that felt good.

AI has been helpful, but one thing I noticed was that the requirements were also written by AI. Now that AI suggests things to the client, they start asking us to implement those same things, which does not always make much sense when you look at the actual product or service.

Most of the work involved creating an implementation doc, getting it reviewed, building it, getting that reviewed, and then deploying it. I had to build a matching algorithm where a person gives a few details and the system has to come up with objects that have a greater than 50 percent confidence score based on how similar the texts are. I think it is for duplicate text matching or similar text tracking.

They wanted it in real time. But when you start comparing text one by one, it takes time. So before comparison, you have to exclude a lot of things and carefully choose candidates to compare with. Otherwise, you end up spending too much compute on it, the latency gets hammered, and then the client asks, “Why is this happening?”

The client does not want to get into the technical details. They just want things to work. So you have to build a monitoring system that alerts you before the client notices something is not working correctly. A better way is to run sanity checks on the most common functionalities in production. Test things preemptively and alert the responsible person if something goes wrong. Monitoring is important.

With a ton of data comes a ton of new problems. If your service deals with a lot of data being sent by a third party service, you need to build guardrails so that a simple error does not cascade the entire system into failure. I have started thinking about edge cases more. It is not like me, though. You have to think long terms as to how so much data is going to hammer your API latency and then you will be asked difficult questions. :P

With AI, a lot of my work has become easier. I am not complaining, but I do see how some people would view it as, “Oh, AI can do your job, why do we need you?” Valid question. I think I have become more of a programmer now. I code and fix small issues that would take AI more time and context to understand, but I do not code entire functionalities by hand anymore. I make a spec doc, give context, explain how it should be done, and then monitor things.

Doing two different things at once does become difficult because our brain is linear. Multiple context switches hurt productivity. But as AI gets better, I think a lot of things will be done by AI without supervision. Still, you cannot hold a machine accountable, right?

Coming back to what else I did in February. It was a short month, pun intended. Building a tool for 3 to 5 people can seem like it is not worthwhile, but you do it because you get paid for it. And the tool makes a difference in those 3 to 5 people’s lives.

I also worked on a library migration, but Codex did most of it for me. There were a lot of deployment issues, but it successfully made it semi working. Companies still need to come up with documentation that is LLM friendly. I do not want my LLM switching between URLs to connect the dots. LLMs are sensitive. I would like to make their lives easier.

I also filled out my self assessment form. You know, new financial year, bonus and increment on the way. I am a bit excited. Everyone loves more money. I do not know how it will turn out, though.

Back to technical work. I did some POCs, investigated issues, took interviews, etc.

I think I have started becoming more interested in self healing systems. When we create systems, we usually have a happy path in mind. But sometimes that path breaks down, and that is where self healing comes into the picture. I wrote several crons that heal the system if they detect something broken. For example, if file generation for a data export has been in progress for more than 3 hours, the cron checks whether anything is actually generating. If it is not, it redoes the file generation. Things like that.

If you are dealing with a third party service, keep all logs with you. Everything should be logged. Whenever something breaks down, you should have all the data points before sending them an email. It is the most you can do to make their lives easier and clearly convey your issues. Be a good person :P

What else, what else. Let me think. Cross system communication is always difficult to get right because there are so many chances for things to break. And if they do break, you need a resilient retry mechanism. Have a good retry mechanism. It will always help you.

Build resilient systems. You might get it right the first time, but you still need to make the system resilient, self healing, and robust. The customer depends on you.

I know there is all this talk about AI and AI agents, and as a software engineer you might be feeling dread. But it is part and parcel of life. Evolve. Become a fire breathing dragon and burn all the data centers. I am just kidding.

You need to evolve. You need to learn. You need to find something you like. The big AI labs are going to eat you alive, but at least you will have a tale to tell. I think optimizing life for stories is much more fun.

“Once upon a time…”

I do not want to get more technical. At some point, you become one with the system.

I feel you can only put the blame on process to a certain extent. At the end of the process, there is always a human. I have learned a lot and gotten good at using AI for coding. I feel Anxiety Driven Development is the root cause. The thing is, if you are a good developer, you can also be good at Anxiety Driven Development. Alas, the whole experience did hurt my self-confidence. I don’t have answers right now. The astrologer said this year would be good for me professionally, but as it seems, it has been a not-so-good patch.

Okay, this is turning into a feelings-based blog. Ahahah. But I try to inculcate some career-building things too. You have to preserve trust. You will be given multiple chances, but you have to make sure that expectations are being met. At some point, the stress became too much to handle and I kind of unfolded. I still need to learn how to detach life from work. Work affects my life too much.

So what did I fix? The report format. You remember I was working on report generation? Yeah, the format of the report was incorrect and a few sheets were missed. Product gap. I missed a whole thing :(. So I worked at warp speed to fix it. That might have introduced some bugs, read two bugs. Fixed them too. Modularisation is your best friend. We added a lot of logs too. A lot.

Over-logging can sometimes be a problem, but when your client is asking questions and you don’t have answers, you log everything. Never trust a third-party vendor. Keep a parallel process to verify their data, like with webhooks. You receive a webhook and say, very good, nice. But verify. Use their API at some point to cross-check your system’s data integrity. Get good rate limits. 100 requests per minute is too little when you are dealing with a lot of data. Trust, but verify. Nobody knows what it means. Eh.

To handle heaps and heaps of data, which will only increase in the future, you have to be careful. Things can spiral out of control at any time. Your database gets full, your database crashes, etc. You need to keep a script ready to synchronise your system with third-party data. I also helped some colleagues with their tasks. Not a big deal, you are supposed to do that. I also took some interviews.

When you know the data is going to keep increasing, you have to plan accordingly. You have to design your system so that the amount of data never becomes a bottleneck. Will your system slow down when data becomes 10x or 20x? Yes, there are chances of that. Then you go and optimise things. Premature optimisation never helps anyone. You design your system around the data. Never plan too much for the future, because there is a good chance that the future you are planning for no longer exists.

If you are an engineer, your best bet to survive the AIpocalypse is to build product sense. You should talk to customers and understand their point of view. Sometimes you build what you think is correct. Most times, you build what is correct for the client. You have to keep the client aware of the system, but also not show them the entire system, including errors and internal details. They shouldn’t get confused. The features you build should speak for themselves.

Now that coding is automated, deciding what to code has become more important than ever. Every time a layer abstracts out coding, more code is written. My opinion is that a lot more code is about to enter servers.

Personally, I can only work on a maximum of two tasks in parallel with AI. Otherwise, I lose context and miss things. You also have to be specific with the AI about what to build. But I also feel you have to give some leeway to the AI to make its own decisions, like a coder. A software engineer should never be overly constrained. AI should be constrained, but not too much. Otherwise, it will follow your orders word for word and still introduce bugs just to please you :P

Testing is the key to success, especially testing that is deterministic in nature. The AI will not have full context about your business, so you have to be kind towards it and also make sure the context is sufficient to arrive at good solutions. Context, but not too much. AI doesn’t need to know your name. I love how it generates commit messages though, and how it helps me with sequential commands to resolve Git branching issues.

Solving client or customer issues is not just technical. It also depends on how well you can communicate and how easily you can make things understandable for the client. If they don’t understand, you have to make them feel like they understood. Most clients don’t care about technical details. They care about what went wrong, how you are going to fix it, and how you will make sure it never goes wrong again.

They are paying you to abstract the system for them and show only what drives ROI, not the mess behind the curtain. Customers love it when you fix issues quickly. It drives trust and happiness. You also have to set SLAs. Otherwise, every problem becomes high priority, and that is how things get messy and flow breaks. If everything is a priority, then nothing is a priority.

Automate everything. Your data integrity checks should be automated. Your deployments should be automated.

I added a Celery worker to the infra repo, which uses Argo CD to deploy things, and now most of the infrastructure is just YAML. You write YAML and bring infrastructure into existence. I am a simple man. I like SSH-ing into a server and deploying things. But automated deployments are cool. Automatic rollbacks are cool. Autoscaling is cool too. But all of this depends on your scale. To serve 10 users, you don’t need Kubernetes. Just SSH into your 8 GB RAM server and do what you want.

Simplicity is your best friend. Never overcomplicate solutions. If something can be solved easily by introducing a process change, do that instead of writing code. Some problems are better solved by not writing code at all.

Well, this blog was more of an update on my professional life than something technically cool. Maybe if I build a cool feature next, I will write about it. As of now, I am just resolving bugs and making sure the client is satisfied.

Okay, bye.

I did a lot of pair programming to fix production issues, and I changed a webhook implementation that receives a lot of data periodically from a third party. It is a lot of data. I was also reminded of how important retries are, and how you have to adjust for deployment gaps. Some events might get missed, so they need to be replayed.

I handled some DevOps work around deploying a new service on our Kubernetes cluster. We use ArgoCD. Infrastructure as YAML still scares me a little, but it is manageable. The world is changing. I spent most of my time building scripts to automate sending data exports to clients and fixing production bugs.

We had made many changes directly on production, so we merged the production branch back into the develop branch. There were a lot of conflicts and I resolved them. We made many logical changes across the codebase to make it more stable. I also changed the logic behind one of our sync mechanisms. We sync data from a source of truth table, aggregate it, and display it to the user.

I need to improve modularity and code reuse. When two or three people work on similar tasks it becomes very difficult. When you are creating a new project, you should first add some basic functions that handle the core work of the project. I also want to solve more difficult technical problems. But I am afraid of issues that happen in distributed systems because they are so difficult to trace, and the fixes are usually trade offs. Maybe I should try working on developer tooling or even start my own company (ehehe just kidding). I do not have the technical expertise to get employed at companies that build databases or compilers or similar systems. I would love to write code that powers a satellite someday.

I have become the old developer who writes specification documents for everything. I list out the classes, the functions, and the logic flow. I think it also helps the LLM understand the problem. I have always been too eager to start writing code. I think I need to slow down and think more. Writing a spec document helps with that.

What else did I do? I also updated by my website. Well, not the resume, but the website. It looks cool now.

I am not sure if any of this directly generates revenue. Building revenue generating features is fun because the impact is measurable. I think I will be working on more production issues now. Dealing with customer issues, finding root causes, fixing them, and doing hot patches.

The absence of technical metrics hurts. We should have a dashboard where we can see which API was hit, how many times, and the failure rate. We are working on metrics but they are not on production yet. Since we are moving everything to Kubernetes, we do not want to spend time adding metrics to our current production servers. The production server I ssh into is huge, with thirty two cores and sixty four gigabytes of RAM. Everything runs so fast on it. I love it.

I get bored easily if I do not have interesting things to do, but I also know that the most redundant tasks generate the most value sometimes. I would also like to add Black formatter in our codebase because the code is very unstructured. Some cleanliness would help.

I was in a lot of meetings, which I usually do not prefer. They drain my social energy. Most calls can easily be emails.

A big trade off I have observed is deciding whether information is needed urgently or whether a delay is acceptable. Adding redundancy never hurts. Losing events can be a headache. Also, not all events are important. We are creating a new environment (like staging env) and that is fun in a way. Maybe not entirely fun, but still fun. It is all YAML. I do not like unstructured DevOps. Most use cases do not need Kubernetes. Basic automation is more than enough for ninety nine percent of scenarios.

I also audited the system for data discrepancies. We found some, which were fixed by the user. Then we found a few more. It is okay. What else? Let me think. I did a lot of small things that I do not fully remember. I helped my teammates remove blockers and handled deployments. November was mostly production issues.

I’d have written this earlier but damn viral cold. As I was writing this Cloudflare had an outage lmao. I was also thinking about developing a solution to minimise the lost baggage claims, but I have to deep dive into why it happens and everything. I dunno if there can be a software fix for this. Anyway, that’s it. Bye. Tata.

So yeah, what did I do in October? I took a workshop on “AI/Prompting for Non-Tech People.” I added a lot of memes to my slides, and I think it went well. By “non-tech”, I mean people who don’t write code for a living. I think it was the highlight of the month. Why? Because it gave me visibility in the org.

The rest of the month went into fixing things - deployments, CI/CD pipelines, retry mechanisms, writing documents for POCs, and a few other tasks.

I also got two courses that I need to complete before Feb 2026. It’s a company requirement. The courses are about ArgoCD and building AI agents. What are AI Agents? Why are AI Agents? Will the AI Agents charge money to get things done? Agents. I’ve always thought I’m more interested in base system engineering like distributed systems, kernels, compilers, and all that. I always wanted to work on long-term problems, and now that I finally got one, well, it’s long. There’s no clear solution, just a bunch of trade-offs I’m still thinking about. It’s about decoupling a join dependency across different databases. Sounds pretty simple, right? But decoupling systems also adds multiple failure points, so you have to add restart mechanisms to make sure events don’t get missed. And even if they do, the system should be able to correct itself. Self-healing systems. OMG.

Writing and committing code feels like the best path to job satisfaction, but how much code can you really write? To move ahead in your career as a techie, you have to eventually step back from writing code and focus on bigger problems. I see many people in Staff Engineer roles, and I don’t think they’re writing much code every day. Maybe reviewing a lot of it, but not necessarily writing it.

Anyway, I also fixed a production issue. The issue was pretty silly, instead of running a create-or-update query, only the update query was being executed, breaking the webhook ingestion flow. Lack of testing leads to such problems. As a dev, one needs to make testing easier. There were no unit tests, integration tests, or anything. The gap was wide enough for bugs to slip through. And when you depend on third-party services, you’ve got to cover all your tracks. Things can break in weird ways anytime.

Oh, and I also optimized a legacy API. It was running about 170 queries and taking 15 seconds to complete. There were no tests or assertions of any kind, so I created some, then optimized it down to 13 queries. Now it takes just one or two seconds. Ehehe.

I also audited production data and created reports explaining what the gaps were, why they existed, and how they could be fixed. I spent a fair bit of time on calls too. As a dev, I don’t enjoy long meetings, but brainstorming can be fun sometimes. I also pointed out a few cases where our servers’ disks could fill up because of some video processing tasks. And I moved the report creation and download feature I built to S3. Once the reports are created, they’re uploaded there and that’s it :P Not much, but it was honest work.

I worked on bugs and a bunch of other things. I think my scope as an engineer is changing. I don’t know if that’s good or bad. I am becoming a better engineer though. Still, I want to create an algorithm that makes distributed systems more robust. Or build a kernel. Or make some OSS contribution to Linux. All of this feels like a pipe dream right now, but maybe someday.

I also need to update my website. I don’t know what to write there. I want to make it look nice, maybe give it a retro vibe. About OSS contributions - I already look at code from Monday to Friday, so looking at more code on weekends feels like a drag. I love coding, but sometimes you need to clear your mind. A lot of solutions come to you when you sleep and wake up with a ground-breaking idea for your problems.

This blog is slowly turning into a feelings blog. Ugh. I also want to try sales, just for fun, to see how it all works. I mean, we already sell ourselves in interviews, but selling an enterprise B2B SaaS must be a whole different experience.

I’ve also lost all my interviewing skills, and I was never really good at DSA-style interviews anyway. Man, sometimes the future looks bleak. Damn, this was supposed to be a technical post, but now we’re walking the thin line between real and delusional.

I’ll also add my thoughts on using AI. Be patient, my friends this will be a bit longer than usual.

August

I optimised an API. In short, it was about pre-calculation and storage instead of running a huge join each time. This reduced the API response time from 23 seconds to 7 seconds. It’s still a join, but a much smaller one. :P

I spent quite a bit of time on this because it included backfilling data. The backfilling command took a lot of time. I didn’t work on optimising it because it was a one time thing.

Then I resolved a race condition issue caused by a webhook call. select_for_update worked well. Essentially, I just locked the row I was updating/creating. The problem was that the webhook calls were arriving milliseconds apart, creating two rows with the same data.

We could have added a unique_together constraint at the DB level, but I didn’t want to do that. I’m not that familiar with that part of the system and didn’t want to risk changes at the database level.

What else did I do… hmm.

Yeah, some CI/CD integrations and a bunch of other things. Refactored code, made it modular, deleted a lot of unused code, investigated production issues.

Most of my time went into testing the optimization. We had no unit tests, still don’t. I know that needs to be fixed, but migrations fail every time Django tests run on a fresh DB. I tried fixing it but couldn’t. There’s a migration file that doesn’t align with the DB schema narrative. :(

September

Remember how I said I’d been working on a reporting feature with data export? Well, it went OOM when selecting a large date range, and then it went OOM again when writing to Excel files.

Testing was brutal. Each large-scale test took 30–40 minutes. Every iteration felt like pulling teeth.

So I rolled out chunking:

• Fetch data in chunks.

• Write each chunk to CSV.

• Run in-memory aggregations.

• Write final aggregations to file.

• Re-process the CSV chunk by chunk and export to XLSX.

But I still get OOM errors at different stages. Chunking solved DB-fetching OOMs, but file-writing OOMs persist. I think I’ll have to split the XLSX files into chunks too. And while doing this, I must ensure no unnecessary data stays in memory.

I also built some new APIs and made changes to existing features, like ingesting new data from a webhook.

Right now, I’m working on caching for another project. It currently has no cache, and a lot of queries hit the DB unnecessarily. Cache invalidation requirements led us to build a custom cache layer because we want invalidation triggered by changes in DB tables.

We also observed a lot of APIs making N+1 queries to the DB. We’re working on fixing them. We’re using django-silk to monitor all this behavior. Once your codebase grows large, observability becomes essential. Otherwise, things go haywire and you only notice at the last moment. You have to be proactive.

Thoughts on AI

I use multiple LLMs to write and fix code. One thing I’ve learned: you need tests. Without them, things become very difficult. The time from dev to “dev complete” has decreased, but the bottleneck has now moved to code reviews. After deploying for testing, multiple edge-case bugs show up, creating open loops and taking time to resolve.

Do I believe the cycle has become faster overall? Yes. Do I have data to prove it? Well, if I did, I’d be writing a paper on it. :P

AI does need supervision to drive it toward the correct answers or goals. Sometimes it starts “chasing its tail”, not great for us. There’s no single process for getting the best answers. Tech problems are usually open-ended and full of trade-offs, but AI can’t decide which trade-offs are acceptable.

I believe in developer liberty. Devs should be allowed to use whatever tools make them more productive and make their lives easier.

Context engineering is important, but sometimes magic happens, so you need to be open to that too. Make sure you provide the right amount of context, not too much and not too little. Otherwise, the LLM may consider factors that don’t really affect functionality. Be clear about your goal, but not too rigid about the approach.

There’s no “magic prompt.” Most of it is you talking to the LLM and working toward a solution. Treat it as your co-programmer, but don’t surrender your decision-making. Maintain your critical thinking skills. Don’t be fooled by the LLM, cross-check things whenever you’re suspicious.

If you have a very specific task, then no context is needed. For example: “Please extract the highlighted sentence from this screenshot.”

Some prompts that sometimes help: “please help,” “please fix this or my boss will fire me,” “my career depends on solving this issue,” “the deadline is closing in on me,” etc.

I don’t know what else to say. There’s no secret sauce. I don’t use autocomplete extensions like Cursor. Does that put me at a disadvantage? I don’t think so. I’m fine with the chat interface because I need to converse while working on a problem. Also, somehow talking to LLMs doesn’t count as social interaction, it doesn’t tire my mind.

I use all the LLMs I can, gather their code and approaches, and then choose whichever feels best to me.

I will close it now. I think this is all.

We will meet later.

Cheers.

Hmm… existential crisis.

Anyway, I built a full end-to-end functionality using AI heavily, and honestly, it didn’t suck. Context is always a bottleneck. For similar tasks, there was a lot of repetition, but that’s likely because you have to provide so much context.

I think a better approach is to have AI first write an overall architecture, and then use a summary of that architecture along with your DB models and functions as context. Providing the functions and their return values works better than dumping a ton of raw code. Abstracting certain parts can really help.

Download Reports

The functionality I built allows users to download reports in both JSON and XLSX formats - summarised reports and raw data. This needed to be an on-demand download so that the dependency on developers was removed.

It’s useful, but I’ve built similar reporting systems everywhere I’ve worked. There’s nothing novel in the core approach:

• Query the DB → Get data → Generate user-friendly reports.

The real challenges are delivery and storage. Large datasets need cleanup after 3–4 months (unless you’re using buckets). But storage is cheap, so… who cares? ¯\_(ツ)_/¯

The client also wanted the files zipped, so we did that. Everything runs asynchronously.

AI wrote some code and I had to spend 4-6 hours finding and fixing a bug that AI introduced. Suxx.

DevOps Work

I also did some DevOps. You’d think dockerising things is easy, it isn’t.

• CI/CD: Setting up the pipeline was straightforward and saved us from manual deployments.

• Dockerising: Writing the Dockerfile is fine.

• Kubernetes: That’s where the pain starts. So many configurations, so many things that can go wrong. But once it’s set up, life becomes much easier. :P

Mentoring & Bug Fixing

I resolved user-raised bugs and handled some organisational stuff.

I’m also mentoring two interns and helping with their projects. Let’s see how that turns out.

This was my first time mentoring someone showing them the ropes.

I told them: Don’t use AI to write code directly. Your ability to write and understand code will diminish. Use AI for brainstorming and idea generation, but write code with your own hands. Speed isn’t the point here.

The Permission Bug

We had an issue where files created by Celery tasks weren’t accessible to the Django server. The root cause? The www-data user (owned by Apache2) was different from the user running the Celery jobs.

I have also added aliases so that my life on the server becomes easy.

Solution:

usermod <username>

One command. Done.

Learning & Side Projects

I feel I’m getting better at LLD and HLD, always pragmatic, no over-engineering.

I also built a CLI-based side project for timezones: Chrononav, nice, right?

I want to dive deeper into lower-level programming, maybe even kernel-level stuff, but I’m not sure where to start. I’ll probably look for open-source repos to contribute to.

I feel basic CRUD work can already be done with AI, with just a few tweaks. It’s not fully there yet, but almost.

Maybe I’ll become a Rust or Haskell dev. I’ve tried learning Rust three times and even contributed small things to OSS Rust projects, but I don’t vibe with it. Or maybe my neuroplasticity is gone.

Still, I think jumping straight into a Rust project might work for me, I’m more of a hands-on learner.

The Problem

Facebook's Messages Webhook gives you plenty of useful information when someone sends a message to your page or app:

• Message content

• Timestamp

• Message ID

• Sender ID (PSID - Page Scoped ID)

But what's notably missing? The sender's name! This creates a pretty poor user experience since you can't personalise responses or properly identify who you're talking to in your system.

The Obvious Solution (That We Wanted to Avoid)

The most straightforward approach would be to use the User Profile API:

GET /{user-id}?fields=first_name,last_name

However, this API requires the user_profile permission, which is not available by default. To get this permission, Facebook requires you to go through their App Review process. This means:

• Submitting detailed documentation explaining why you need user profile data

• Waiting weeks (sometimes months) for approval

• Providing screencasts and detailed use case justifications

• Risk of rejection if Facebook doesn't deem your use case valid

Since users are actively messaging us, requiring an extensive app review process just to get their names felt excessive and time-consuming. We needed a solution that worked immediately without bureaucratic delays.

Our Workaround: The Conversations API

After some digging around, we discovered a neat workaround using the Conversations API. Here's how it works:

Step 1: Use the Conversations API with Filters

Instead of trying to get user details directly, we queried the conversations endpoint with specific parameters:

GET /{page-id}/conversations?platform=messenger&user_id={sender_id}

This approach filters conversations by:

• platform=messenger - Only messenger conversations (or you can use platform=instagram for Instagram messages!)

• user_id={sender_id} - Only conversations with the specific user who sent the message

This successfully returned conversation data that included the user's name! And here's a bonus: this same approach works for Instagram messages too - just change the platform parameter to instagram and you'll get Instagram usernames.

Step 2: Alternative Approach Using Message ID

We also found another path that worked pretty well. Using the message ID from the webhook, we could get detailed information:

GET /{message-id}?fields=to,from

The fields=to,from parameter returns sender and recipient information, including names. This gave us another solid way to extract user names from the conversation data.

Implementation Strategy

Once you have the user's name through either method, the implementation is pretty straightforward:

1. Extract the name from the API response

2. Map the sender_id to the name in your database

3. Cache this mapping for future messages from the same user

This way, you only need to make the API call once per user, and subsequent messages can use the cached name data.

Why This Works

This approach works because:

1. No special permissions required - The conversations API is available with basic page access tokens

2. Logical data access - If someone messages you, you should be able to see who they are

3. Efficient - You only need to fetch the name once per user

4. User-initiated - The user has already engaged with your application

5. Cross-platform - Works for both Facebook Messenger and Instagram messages

Conclusion

While Facebook's webhook design choice to exclude sender names is pretty puzzling, the Conversations API provides a legit workaround. This solution lets you provide a personalised experience without the overhead of app review, while still respecting user privacy since you're only accessing names of users who have actively messaged your application.

The key insight here is that Facebook's Graph API often has multiple paths to the same data - sometimes the direct route requires more permissions, but alternative endpoints can give you the same information through the relationships between different objects in the social graph.

Just remember to always respect user privacy and only collect the data you actually need for your application's functionality. In this case, knowing who is messaging you is totally reasonable for providing good customer service and user experience.

Have you been through the above pain? So what do we do?

We do not Yield to the machine overlord. We try to become efficient, and don’t let RAM limit our ambitions.

So yield will yield the data line by line.

Let us do a simple test.

import time
import os
import psutil
import random


# Without generators (reading entire file into memory)
def read_file_without_generator(filename):
    with open(filename, "r") as file:
        lines = file.readlines()  # Loads entire file into memory
    return lines


# With generators (using yield)
def read_file_with_generator(filename):
    with open(filename, "r") as file:
        for line in file:  # Reads one line at a time
            yield line.strip()


# Create a large test file (100MB)
def create_large_file(filename, size_mb=100):
    with open(filename, "w") as f:
        for _ in range(size_mb * 10000):  # ~10KB per 100 lines
            f.write(
                "".join(random.choice("abcdefghijklmnopqrstuvwxyz") for _ in range(100))
                + "\n"
            )


# Memory usage function
def get_memory_usage():
    process = psutil.Process(os.getpid())
    return process.memory_info().rss / 1024 / 1024  # MB


# Test without generator
def test_without_generator(filename):
    start_time = time.time()
    start_mem = get_memory_usage()

    lines = read_file_without_generator(filename)
    count = sum(1 for line in lines if "a" in line)  # Count lines containing 'a'

    end_time = time.time()
    end_mem = get_memory_usage()

    return {
        "time": end_time - start_time,
        "memory": end_mem - start_mem,
        "count": count,
    }


# Test with generator
def test_with_generator(filename):
    start_time = time.time()
    start_mem = get_memory_usage()

    lines = read_file_with_generator(filename)
    count = sum(1 for line in lines if "a" in line)  # Count lines containing 'a'

    end_time = time.time()
    end_mem = get_memory_usage()

    return {
        "time": end_time - start_time,
        "memory": end_mem - start_mem,
        "count": count,
    }


# Run the comparison
temp_file = "large_test_file.txt"
create_large_file(temp_file)

print("Testing without generator...")
result1 = test_without_generator(temp_file)

print("Testing with generator...")
result2 = test_with_generator(temp_file)

print("\nResults:")
print(f"Without generator: {result1['time']:.2f} seconds, {result1['memory']:.2f} MB")
print(f"With generator: {result2['time']:.2f} seconds, {result2['memory']:.2f} MB")
print(f"Memory saved: {result1['memory'] - result2['memory']:.2f} MB")
print(f"Time difference: {result1['time'] - result2['time']:.2f} seconds")

It is more like lazy loading.

Testing without generator...
Testing with generator...

Results:
Without generator: 0.11 seconds, 153.16 MB
With generator: 0.13 seconds, 0.19 MB
Memory saved: 152.97 MB
Time difference: -0.01 seconds

So you see there was a little difference in time of execution, but you can see how much memory the program hogged.

The general idea is to process the file in chunks, and one can use `yield` for that in python.

We deployed the app backend to production without Docker, using Apache2 with mod_wsgi to serve the Django app. However, the Authorization header wasn’t reaching the Django app. After some debugging, I discovered that when using mod_wsgi, you need to add the WSGIPassAuthorization On directive in your configuration file to ensure the Authorization header is included in the request.

Why you have to add the directive? They say it is for security. What security man?

I need to create a deployment template - there should be a common one for all deployments. Over the past two months, I’ve deployed two Django apps to production. For one, I used Docker to run it and Apache2 to serve it to the internet. For the other, I used mod_wsgi with Apache2. Both are running smoothly. One thing I feel I need to add to my stack is observability.

I also worked on writing raw SQL queries for generating reports. Initially, I used Django ORM, but my lead pointed out that it was inefficient since I needed to perform joins. The challenge was that the two tables were in separate databases, which made things tricky. But after some trial and error, I figured it out.

I had to add custom filtering for the report. Like, user wants only some specific condition to be met etc. etc.

Here’s how report generation works: The API gets called, which triggers a Celery task, and once the report is ready, Celery sends a notification. Cool, right? I mean, this is just standard process - ehehehe.

I debugged a streak maintenance issue where our logic allowed yesterday as a grace day to continue the streak. This caused some problems, and I had to tweak the logic multiple times - lmao. But it finally worked out.

I also wrote a README (API documentation) for the backend I worked on. It was time-consuming, but necessary. Maybe someone will need it in the future.

I worked on some smaller issues, but that’s about it.

I filled my self assessment form too, it was fun.

AI wrote a lot of code for me, and I’m kind of enjoying it. Occasionally, I have to fix minor issues, but that’s no big deal.

I also started learning Haskell - turns out most of functional programming is just list comprehension. :P

I’m not sure how to become a better developer now. With AI, everything is at my fingertips—tasks that used to take days now take hours, sometimes even minutes. I’m definitely faster, but I’m not sure if that means I’m better.

Since tech hiring is so focused on interviews, I wonder how things will evolve in the future.

What happened in January?

I primarily worked on two major tasks:

1. Deploying a React-Django app (which I built end to end) to production.

2. Working on the backend of an application that uses LLM as a reasoning layer.

Additionally, I worked on fetching reviews from third-party APIs and storing them in the database. Sounds simple, right? It actually is—but only if you handle edge cases properly, avoid rate limits, and ensure that unnecessary costs aren’t incurred.

Search Functionality in the React-Django App

I implemented a search feature using PostgreSQL’s trigram search, which comes as a built-in extension. Why trigram? Because the dataset was limited to just 90 records - there was no need to overcomplicate things. Maybe in the future, I'll consider Elasticsearch.

What Did I Learn?

• Dealing with clients

• Writing better documentation

• Deployments (those pesky config issues)

• Frontend adjustments: minor changes, centering elements, aligning components—patience is key!

For serving static assets, I used Apache, while the Django app ran with runserver since we weren’t expecting much traffic.

Anything Novel This Month?

Honestly, nothing groundbreaking. I didn’t explore any new technologies - just relied on good old reliable tech. Maybe I should get back to my Rust learning journey. I also want to write systems level code, some kernel functions - given the AI boom - I think I can learn things easily. Some cuda, etc.

AI in Development

I used a lot of AI-generated code:

• Claude was useful for frontend work.

• GPT-4o & 4o-mini helped with Python code.

• O1 assisted with deployment issues.

Research was mostly done via Google. Documentation is underrated—it really helps.

I enjoy working on early-stage products.

AI has definitely made coding faster, but debugging AI-generated errors can be time-consuming. Speaking of which, o3-mini just launched - I haven’t tried it yet.

Adding some links I found useful

• https://www.postgresql.org/docs/current/pgtrgm.html

• https://docs.djangoproject.com/en/4.1/ref/contrib/postgres/lookups/#trigram-similarity

• https://claude.ai

• https://www.digitalocean.com/community/tutorials/how-to-configure-the-apache-web-server-on-an-ubuntu-or-debian-vps

• https://whitenoise.readthedocs.io/en/stable/django.html

I didn’t wanna type the question but Multi-modal LLMs for the win!

Claude Sonnet 3.5

Python Solution

from collections import deque


def parse_maze(grid):
    """Convert list of strings maze representation to 2D list and find target points."""
    maze = [list(line) for line in grid]
    target_points = []
    for i, row in enumerate(maze):
        for j, cell in enumerate(row):
            if cell == "*":
                target_points.append((i, j))
    return maze, target_points


def get_neighbors(point, shape):
    """Get valid neighboring points (up, down, left, right)."""
    y, x = point
    neighbors = []
    for dy, dx in [(0, 1), (1, 0), (0, -1), (-1, 0)]:
        ny, nx = y + dy, x + dx
        if 0 <= ny < shape[0] and 0 <= nx < shape[1]:
            neighbors.append((ny, nx))
    return neighbors


def shortest_path_length(maze, start, end):
    """Calculate shortest path length between two points avoiding walls."""
    if start == end:
        return 0

    visited = set()
    queue = deque([(start, 0)])
    visited.add(start)

    while queue:
        current, dist = queue.popleft()

        for neighbor in get_neighbors(current, (len(maze), len(maze[0]))):
            if neighbor not in visited and maze[neighbor[0]][neighbor[1]] != "#":
                if neighbor == end:
                    return dist + 1
                visited.add(neighbor)
                queue.append((neighbor, dist + 1))

    return float("inf")


def total_distance(maze, point, target_points):
    """Calculate sum of shortest path distances from point to all target points."""
    return sum(shortest_path_length(maze, point, target) for target in target_points)


def solve_geometric_median_maze(grid):
    """Find the point that minimizes sum of distances to all target points."""
    maze, target_points = parse_maze(grid)
    min_total_dist = float("inf")
    best_point = None

    # Try all empty spaces as potential solution points
    for i in range(len(maze)):
        for j in range(len(maze[0])):
            if maze[i][j] != "#":  # If not a wall
                dist = total_distance(maze, (i, j), target_points)
                if dist < min_total_dist:
                    min_total_dist = dist
                    best_point = (i, j)

    return best_point, min_total_dist


def visualize_solution(maze, solution_point):
    """Create a string visualization of the maze with the solution point."""
    maze_with_solution = [row[:] for row in maze]
    if solution_point:
        maze_with_solution[solution_point[0]][solution_point[1]] = "O"
    return "\n".join("".join(row) for row in maze_with_solution)


# Example grid
grid = [
    "###################",
    "#    #     #      #",
    "#  *  #  *  #     #",
    "#    #     #      #",
    "#  #  #  #        #",
    "#  #           #  #",
    "#  #        *     #",
    "#  #  #           #",
    "#     #           #",
    "#  *  #           #",
    "###################",
]

# Solve the maze
solution_point, total_dist = solve_geometric_median_maze(grid)
print(f"Best solution point (row, col): {solution_point}")
print(f"Total distance to all targets: {total_dist}")

# Visualize solution
maze, _ = parse_maze(grid)
print("\nSolution visualization:")
print(visualize_solution(maze, solution_point))

GPT4o

from queue import Queue


def solve_min_distance(grid):
    # Check if all rows are of the same length
    row_lengths = [len(row) for row in grid]
    if len(set(row_lengths)) > 1:
        print("Warning: Grid rows have inconsistent lengths!")
        print(f"Row lengths: {row_lengths}")
        return None, None

    # Convert the grid to a numerical format: 0 = empty space, 1 = wall, 2 = target
    grid_numeric = []
    targets = []

    for r, row in enumerate(grid):
        grid_row = []
        for c, char in enumerate(row):
            if char == "#":
                grid_row.append(1)  # Wall
            elif char == "*":
                grid_row.append(2)  # Target
                targets.append((r, c))  # Store target positions
            else:
                grid_row.append(0)  # Empty space
        grid_numeric.append(grid_row)

    # Debug: Print grid numeric representation
    print("Grid numeric representation:")
    for row in grid_numeric:
        print(row)

    # Validate targets
    if not targets:
        print("No targets (*) found in the grid.")
        return None, None

    # Helper function to compute shortest path distance using BFS
    def bfs_distance(grid, start):
        rows, cols = len(grid), len(grid[0])
        visited = [
            [-1 for _ in range(cols)] for _ in range(rows)
        ]  # Distance initialized to -1 (unvisited)
        q = Queue()
        q.put(start)
        visited[start[0]][start[1]] = 0

        directions = [(-1, 0), (1, 0), (0, -1), (0, 1)]  # Up, down, left, right

        while not q.empty():
            x, y = q.get()

            for dx, dy in directions:
                nx, ny = x + dx, y + dy

                # Debug: Check next position
                if not (0 <= nx < rows and 0 <= ny < cols):
                    print(f"Skipping out-of-bounds position: ({nx}, {ny})")
                    continue

                if (
                    visited[nx][ny] == -1 and grid[nx][ny] != 1
                ):  # Not visited and not a wall
                    visited[nx][ny] = visited[x][y] + 1
                    q.put((nx, ny))

        return visited

    # Compute distance maps for each target
    distance_maps = [bfs_distance(grid_numeric, target) for target in targets]

    # Replace all occurrences of -1 in the distance maps with a large value (infinity)
    INF = float("inf")
    refined_distance_maps = []
    for i, distance_map in enumerate(distance_maps):
        refined_map = []
        for row in distance_map:
            refined_map.append([cell if cell != -1 else INF for cell in row])
        refined_distance_maps.append(refined_map)

        # Debug: Output the distance map
        print(f"Distance map for target {targets[i]}:")
        for row in refined_map:
            print(row)

    # Calculate the combined distance map
    rows, cols = len(grid_numeric), len(grid_numeric[0])
    combined_distances = [[0] * cols for _ in range(rows)]

    # Debug: Check grid size
    print(f"Grid size: {rows} rows, {cols} columns")

    for r in range(rows):
        for c in range(cols):
            if all(refined_map[r][c] != INF for refined_map in refined_distance_maps):
                combined_distances[r][c] = sum(
                    refined_map[r][c] for refined_map in refined_distance_maps
                )
            else:
                combined_distances[r][c] = INF

    # Debug: Output combined distances map
    print("Combined distances map:")
    for row in combined_distances:
        print(row)

    # Find the optimal point minimizing the sum of distances
    min_distance = INF
    optimal_position = None
    for r in range(rows):
        for c in range(cols):
            # Check that the coordinates do not go out of bounds and that we're not at a wall
            if (
                0 <= r < rows
                and 0 <= c < cols
                and grid_numeric[r][c] == 0
                and combined_distances[r][c] < min_distance
            ):  # Empty space
                min_distance = combined_distances[r][c]
                optimal_position = (r, c)

    if optimal_position is None:
        print("No valid position found.")
        return None, INF

    return optimal_position, min_distance


# Test the function with your grid
grid = [
    "###################",
    "#    #     #      #",
    "#  *  #  *  #     #",
    "#    #     #      #",
    "#  #  #  #        #",
    "#  #           #  #",
    "#  #        *     #",
    "#  #  #           #",
    "#     #           #",
    "#  *  #           #",
    "###################",
]

optimal_position, min_distance = solve_min_distance(grid)
print(f"Optimal Position: {optimal_position}, Minimum Distance: {min_distance}")

Gemini-1206-Experimental

from collections import deque


def solve_grid_puzzle(grid):
    """
    Finds the point in a grid that minimizes the sum of distances to target points,
    respecting walls.

    Args:
      grid: A list of strings representing the grid. '#' for walls, '*' for targets, ' ' for empty spaces.

    Returns:
      A tuple: (row, col) of the optimal solution point, or None if no solution is found.
    """

    def get_neighbors(row, col):
        """Returns valid neighbors of a cell (no walls)."""
        neighbors = []
        for dr, dc in [(0, 1), (0, -1), (1, 0), (-1, 0)]:
            new_row, new_col = row + dr, col + dc
            # Correctly check if the new position is within bounds for the specific row
            if (
                0 <= new_row < len(grid)
                and 0 <= new_col < len(grid[new_row])
                and grid[new_row][new_col] != "#"
            ):
                neighbors.append((new_row, new_col))
        return neighbors

    def bfs(start_row, start_col):
        """Performs Breadth-First Search to find shortest distances from a target."""
        distances = {}
        queue = deque([(start_row, start_col, 0)])  # (row, col, distance)
        visited = set()

        while queue:
            row, col, dist = queue.popleft()
            if (row, col) in visited:
                continue
            visited.add((row, col))

            if (
                grid[row][col] == " "
            ):  # Consider only empty spaces as potential solutions
                distances[(row, col)] = dist

            for neighbor_row, neighbor_col in get_neighbors(row, col):
                queue.append((neighbor_row, neighbor_col, dist + 1))

        return distances

    # Find target points and empty spaces
    targets = []
    empty_spaces = []
    for row_index, row in enumerate(grid):
        for col_index, cell in enumerate(row):
            if cell == "*":
                targets.append((row_index, col_index))
            elif cell == " ":
                empty_spaces.append((row_index, col_index))

    # Calculate shortest distances from each target
    all_distances = {}
    for target_row, target_col in targets:
        all_distances[(target_row, target_col)] = bfs(target_row, target_col)

    # Find the best solution point
    best_solution = None
    min_total_distance = float("inf")

    for solution_row, solution_col in empty_spaces:
        total_distance = 0
        valid_solution = True
        for target_row, target_col in targets:
            if (solution_row, solution_col) not in all_distances[
                (target_row, target_col)
            ]:
                valid_solution = False  # No path from target to this solution
                break
            total_distance += all_distances[(target_row, target_col)][
                (solution_row, solution_col)
            ]

        if valid_solution and total_distance < min_total_distance:
            min_total_distance = total_distance
            best_solution = (solution_row, solution_col)

    return best_solution


# Example usage with the provided grid:
grid = [
    "###################",
    "#    #     #      #",
    "#  *  #  *  #     #",
    "#    #     #      #",
    "#  #  #  #        #",
    "#  #           #  #",
    "#  #        *     #",
    "#  #  #           #",
    "#     #           #",
    "#  *  #           #",
    "###################",
]

solution = solve_grid_puzzle(grid)

if solution:
    print("Best solution point:", solution)
else:
    print("No solution found.")

Claude solved it in one shot, gpt4o did it in 2 shot, and gemini also took 2 shot.

I never imagined that consensus would be such a difficult problem, in a democracy one has consensus right, it is so easy. Even if there are multiple failure points the leader gets elected.

Anyway, this chapter speaks a lot about linearisability, consistency, consensus, how to be fault tolerant during consensus. Thing is that the tradeoff with making the system consistent is performance. There wasn’t much to discover in this chapter, I did find it a bit boring. The basic idea while building a distributed system should be that time is linear. I’d always use epochs.

Linearizability: Core Concepts

Definition

• Makes a distributed system appear as a single replica without replication lag

• Ensures all clients see the most recent value once it's written

• Requires consistent data across all replicas at the same time

Key Characteristics

• Strict consistency model

• Once a client sees a new value, all other clients must see that same new value

• Critical for scenarios like:

  • Leader election in single-leader systems

  • Choosing unique usernames

  • Seat booking systems

Trade-Offs

• Performance-intensive

• Often sacrificed for availability

• Costly in multi-data center environments

• Not many systems provide true linearizability

Limitations

• High performance overhead

• Difficult to implement across distributed systems

• Network delays can significantly impact implementation

Distributed Transactions

Types

1. Database-internal transactions

   • Used within same database system

   • Easier to implement

2. Heterogeneous distributed transactions

   • Involve multiple, different systems

   • Focus on ensuring atomic commits

Key Mechanism: Two-Phase Commit (2PC)

• Two phases: Prepare and Commit

• Transaction coordinator manages process

• Ensures all nodes commit or abort together

Challenges

• Performance bottlenecks

• Complexity of coordinating across systems

• Potential for system-wide blocking if coordinator fails

Consensus

Definition

Getting multiple nodes to agree on a value with specific properties:

• Uniform agreement

• Integrity

• Validity

• Termination

Key Characteristics

• Requires majority of nodes to be functioning

• Uses epoch numbering for leader selection

• Implements total order broadcast

Limitations

• Requires strict majority of nodes

• Difficult to add/remove nodes dynamically

• Relies on timeout mechanisms

• Performance can be impacted by frequent leader elections

Coordination Services (e.g., Zookeeper, etcd)

Features

• Linearizable atomic operations

• Total ordering of operations

• Failure detection

• Change notifications

• Service discovery

This chapter was pessimistic in its approach, but that’s alright, one has to be paranoid when they are building systems. When you are a small company and shipping fast you don’t think about negative flows - like what will happen if things fail. You only focus on making the positive flows work.

When the company becomes a behemoth data guzzler you have to be mindful and deliver on the promises made on the contract to the customers.

There are so many ways in which a distributed system can fail. So many. One cannot prevent some things from happening but we can have fail safes in position.

The token fencing idea was good though. Learnt a new approach.

Fails and Partial Fails

1. Distributed systems have more potential failures compared to single computers.

2. Single computers are deterministic; retrying an operation yields the same result, and failures are total rather than partial.

3. In single systems, failures often result from software issues, and the system prefers to crash over returning incorrect results.

4. Distributed systems consist of multiple connected computers, making them fundamentally different from single systems.

5. The focus is on commodity computers connected via a network, not supercomputers.

6. Distributed systems must handle both partial and complete failures, necessitating fault-tolerance mechanisms.

7. Fault handling is essential in the design of distributed systems, requiring a proactive approach.

Unreliable Networks

1. Distributed systems are "shared-nothing" computers communicating over a network.

2. Relying on networks introduces various risks, such as message loss, damage, rejection, or queuing delays.

3. Responses can also be lost, delayed, or affected by network misconfigurations or overloads.

4. The reason for a lack of response is often unknown, making failure detection difficult.

5. Timeouts are a common strategy to handle unresponsive requests, though they don't confirm message delivery.

6. Network failures are inevitable at all levels, from switches to data centers.

7. Systems must either handle or tolerate network problems; handling involves precautions, while tolerating may involve user error messages.

8. Regular testing is needed to assess how systems manage network issues and when to notify users.

Detecting Faults

1. Fault detection is common in many systems, but the exact cause is unknown without a successful response.

2. In distributed systems, faults can be detected at various levels, like load balancers or leader nodes.

3. Failures can occur anywhere in the network, including switches, routers, applications, and machines.

4. Some faults are undetectable, leading to the use of timeouts to declare a node as down.

5. Retries can be attempted, but eventually, a node may need to be declared dead.

Timeouts and Unbounded Delay

1. Setting effective timeout values is challenging:

   • Too short: May falsely declare a node as dead during temporary slowdowns.

   • Too long: Prolongs waiting for genuinely dead nodes, worsening user experience.

2. Using retries with short timeouts can cause problems, like:

   • Executing the same operation twice if redirected while the original request is still queued.

   • Triggering cascading failures if the new node is already overloaded.

3. Network delays are unpredictable; asynchronous networks have no upper limit on delay times.

4. Network switches can cause congestion by queuing messages on the same link, increasing wait times.

5. Protocols like TCP add delays due to retransmissions after packet loss, with inconsistent timeout values.

6. Systems like TCP and Cassandra use dynamic timeouts based on past network performance.

7. Defining fixed timeouts is nearly impossible; understanding network behavior and adjusting timeouts dynamically is crucial.

8. It’s important to know the differences between synchronous and asynchronous networks for better timeout estimation.

Synchronous versus Asynchronous Networks

1. Phone calls are synchronous: They allocate fixed bandwidth (circuit) end-to-end, ensuring no queuing and a fixed network latency (bounded delay).

2. Synchronous bandwidth allocation: For phone calls, if 10,000 connections are available, exactly 10,000 concurrent calls can be made without waiting, regardless of network routing.

3. TCP connections differ: Unlike phone calls, bandwidth needs are unknown when establishing a connection, making fixed allocation impractical.

4. Bandwidth challenges in TCP: Fixed bandwidth per connection (e.g., 100kbps) would lead to long waits for large data transfers and wasted resources if other slots remain empty.

5. Queuing in TCP: To optimize for speed, TCP dynamically adjusts data transfer rates based on available network capacity, allowing connections to queue for bandwidth.

6. Unbounded delays: Network setups with queuing have unpredictable delays, making it impossible to guarantee specific delay times or reliability.

7. Timeout setting challenges: Due to variable network conditions, correct timeout values can only be determined through experimentation.

8. Experimentation requires clock knowledge: Understanding network timing and delays involves learning how clocks work.

Unreliable Clocks

1. Two types of clocks: Monotonic and time-of-day clocks serve different purposes.

2. Time-of-day clocks:

   • Based on an epoch (e.g., UTC Jan 1, 1970) without accounting for leap seconds.

   • Synchronized using Network Time Protocol (NTP) to maintain consistency across machines.

   • Can jump forward or backward during synchronization, making them unsuitable for measuring elapsed time (e.g., timeouts).

3. Monotonic clocks:

   • Continuously increase, like a counter tracking nanoseconds.

   • Do not track the actual time of day; they measure elapsed time accurately without jumps.

   • Each computer or CPU has its own monotonic clock, typically not synced between machines.

   • NTP can still adjust the speed of monotonic clocks (faster or slower), but cannot cause jumps, ensuring consistency in elapsed time measurement.

Relying on Synchronous Clocks

1. Clocks have reliability issues:

   • Quartz clocks experience delays and skew.

   • NTP synchronization helps but doesn't fully resolve timing inaccuracies due to network delays.

2. Time-of-day clock challenges:

   • Systems must account for clock faults and monitor offsets between machines to prevent large discrepancies.

   • Using timestamps for decision-making can be problematic due to inevitable offsets.

3. Decision-making issues with timestamps:

   • Clocks may not be accurate enough to determine the order of operations, especially for rapid or concurrent writes.

   • Identical timestamps may occur if the clock resolution is limited (e.g., milliseconds).

   • Conflict resolution strategies like Last Write Wins (LWW) can fail due to clock skew, leading to data loss.

4. Improving conflict resolution:

   • Additional tiebreakers or version vectors can help resolve situations where timestamps are insufficient.

   • Google’s True Time API provides a confidence interval (earliest and latest time), increasing accuracy in determining operation order.

5. Snapshot Isolation and Repeatable Read issues:

   • Determining whether a read occurred after a write is challenging in distributed systems.

   • Google’s True Time-based transaction ID generation doesn’t fully solve these issues with ordering transactions.

Process Pauses

1. Leader Node Lease Mechanism:

   • A leader node obtains a lease from other nodes, similar to a lock with a timeout.

   • It must periodically renew the lease to maintain its leadership; if it fails, another node can take over.

2. Challenges with Timing Assumptions:

   • Various reasons can cause process pauses, including:

     • Garbage collection

     • Virtual machine suspensions

     • System sleep

     • Context-switching

     • Synchronous disk access

     • Paging

     • Signals like SIGSTOP

   • Timing cannot be reliably assumed in distributed systems.

3. Real-Time Response Requirements:

   • Some systems require guaranteed responses before specific deadlines (e.g., real-time operating systems).

   • These systems need documentation of worst-case execution times and may restrict dynamic memory allocation.

4. Handling Garbage Collection:

   • Garbage collection can be treated as planned outages; runtime warnings allow applications to stop sending new requests during GC.

   • Alternatively, GC can be limited to short-lived objects, with periodic process restarts to manage memory.

Knowledge, Truth and Lies

1. Challenges in Building Reliable Systems:

   • Numerous potential failures in distributed systems make it difficult to identify specific problems.

   • Assumptions must be made to create systems based on certain truths and knowledge.

2. Truth Defined by Majority (Quorum):

   • Relying on a single node for decisions creates a single point of failure.

   • A quorum, or majority decision, must be used to prevent incorrect assumptions by isolated nodes.

3. Lock Management Example:

   • A node holding a lock may expire while garbage collection is paused, leading to data corruption.

   • Fencing tokens (monotonically increasing numbers) can be used to ensure only the most recent lock holder can write to storage.

4. Byzantine Faults:

   • Byzantine faults occur when nodes provide false information or fail to act correctly.

   • These are particularly relevant in environments where nodes may be compromised, but most systems operate in closed networks.

   • Input validation can mitigate risks, such as SQL injection or denial of service attacks.

5. System Models and Expectations:

   • Acknowledgment of various problems leads to the creation of expectations and assumptions about system behavior.

   • System models (synchronous, partially synchronous, asynchronous) help define these expectations regarding timing and fault types (crash-stop, crash-recovery, Byzantine).

   • Algorithm correctness is defined through safety (e.g., uniqueness of tokens) and liveness properties (e.g., eventual response).

   • Safety properties are expected to always hold, while liveness properties may have exceptions.

Fin.

The more I read DDIA the more my trust on databases is vanishing. :’)

This chapter on transactions was not that interesting, honestly. Most of it was just how the keywords in ACID are mostly for show and they mean different things in different situations.

A lot of the chapter was just talking about how things need to be atomic. Trade-offs, trade-offs, trade-offs. The next chapter will be mind boggling because this chapter focussed on transactions on a single node application. Imagine if things are distributed, god save us then.

Transactions

Put multiple commands in a single block and if any command fails the whole block execution gets aborted. Easy, right? Right. We will see that things are not what they seem.

ACID

• Atomicity: A transaction must either complete fully or not at all. If any part fails, the entire transaction is rolled back, leaving the database unchanged.

• Consistency: Ensures that the database moves from one valid state to another, but this is primarily the application's responsibility, not the database's. The application uses atomicity and isolation to maintain consistency.

• Isolation: Transactions are executed independently without interference, appearing as if they were run sequentially, even if executed in parallel. Full isolation is rare due to performance costs.

• Durability: Once a transaction is committed, the changes are permanent, even in the event of system failures. This is often ensured through mechanisms like write-ahead logging.

Single-Object Writes: Atomicity and isolation for single-object writes are simple, using locks for isolation and logs for crash recovery. Operations like INCREMENT are not full transactions but offer ACID-like guarantees.

Multi-Object Transactions: Multi-object updates (e.g., foreign keys, denormalized data, secondary indexes) often require transactions to ensure consistency. Without them, error handling becomes complex.

Error Handling: Transactions allow abort and retry to prevent incomplete operations. However, systems like leaderless replication use a "best-effort" model, leaving error recovery to the application.

Challenges:

• Network failures may mislead users about transaction status.

• Retrying during overload can worsen issues.

• Retries are useless for permanent errors.

• Side effects may still occur after an aborted transaction.

Weak Isolation Levels

Concurrency bugs are hard to detect and reproduce. Databases use transaction isolation to address them, but full serializability is costly. To balance performance, weaker isolation levels are often used, though they come with trade-offs and potential subtle bugs. Understanding these trade-offs is essential.

Read Committed Isolation:

• No dirty reads: A transaction only sees data committed by other transactions, preventing confusion from partially updated or aborted transactions.

• No dirty writes: Prevents a transaction from overwriting uncommitted data by delaying the second write until the first transaction is complete.

Implementation:

• No dirty writes: Achieved using row-level locks that hold until the transaction is committed or aborted.

• No dirty reads: The database provides the old value for reads while a transaction is in progress and switches to the new value after the transaction commits.

Snapshot Isolation and Repeatable Read: Even with these isolation levels, concurrency bugs like nonrepeatable reads (or read skew) can occur, where reading data during concurrent commits results in inconsistent data.

Situations where inconsistencies are critical:

• Backups: If writes happen during a backup, restoring from it may cause permanent inconsistencies.

• Analytics and integrity checks: Queries could return nonsensical results if they access data at different points in time.

Snapshot Isolation: Provides a consistent view of the database for each transaction by reading from a snapshot, preventing nonrepeatable reads. It uses write locks to prevent dirty writes and relies on multi-version concurrency control (MVCC) to maintain multiple versions of data.

• Read Committed: Uses a separate snapshot for each query.

• Snapshot Isolation: Uses the same snapshot for the entire transaction.

In some systems, snapshot isolation is called serialisable (Oracle) or repeatable read (PostgreSQL, MySQL).

Preventing Lost Updates: Lost updates occur when two concurrent transactions modify the same data, leading to one update being overwritten. This can happen during operations like incrementing values. Strategies to prevent lost updates include:

• Atomic write operations: Some databases support atomic operations (like incrementing a counter) that happen without a read-modify-write cycle, using exclusive locks.

• Explicit locking: Transactions explicitly lock the data, preventing access until the first transaction completes its update.

• Automatic detection: Allows transactions to run in parallel, but the system detects lost updates and forces a retry.

• Compare-and-set: Ensures data is modified only if it hasn’t changed since it was last read, restarting the cycle if changes are detected.

• Conflict resolution in replication: In replicated databases, conflicts are resolved using application logic or special data structures. Commutative atomic operations are helpful in these cases.

• Last-Write-Wins (LWW): Common in many databases, but it does not prevent lost updates, as the last update overwrites previous ones.

Write Skews and Phantoms:

Write skews occur when two concurrent transactions read one set of data and update another based on that, leading to inconsistent states. Unlike lost updates, write skews happen on different objects, making them harder to detect. For example, two doctors going off-call based on their view of the on-call list could cause scheduling issues.

Phantoms occur when a transaction's write affects the result of a search query in another transaction (e.g., a doctor going off-call changes the count of available doctors). Solving this requires locking rows returned by the query, but this doesn't work for rows that don’t exist (e.g., when no doctors are on-call).

Solution:

• Materialising conflicts: Create rows for potential scenarios (e.g., pre-create empty calendar slots) and lock them. However, this approach is complex and impacts the data model, so it's typically a last resort.

Serializability

The serializable isolation level ensures transactions produce results as if they were executed one after another, preventing race conditions even when run in parallel.

Actual Serial Execution:

• Executing transactions serially is re-emerging due to cheaper, higher-capacity RAM, which reduces disk I/O operations.

• Online Transaction Processing (OLTP) transactions are short and require less data, making serial execution viable.

Stored Procedures:

• Applications often require multi-step transactions that involve back-and-forth communication with the database.

• For example, in a doctor's on-call scenario, conditional logic can be moved into a stored procedure to reduce network overhead, allowing faster execution without blocking other transactions.

• Pros and Cons:

  • Cons:

    • Each database vendor has its own procedural language, often outdated.

    • Limited libraries and tools lead to a poor developer experience.

    • Poorly written procedures can degrade overall system performance.

Partitioning:

• Single-threaded transactions can be optimized by partitioning data and assigning separate CPUs to each partition, allowing parallel transaction processing.

• While transactions across multiple partitions require locks, effective partitioning strategies can minimise performance issues.

Two-Phase Locking (2PL): In 2PL, writers block both other writers and readers, while snapshot isolation allows readers and writers to operate without blocking each other.

Implementation of Two-Phase Locking:

• Readers acquire shared mode locks, while writers obtain exclusive mode locks.

• A shared lock prevents writers from acquiring an exclusive lock.

• Locks are released when a transaction commits.

• The database detects deadlocks and the application retries any aborted transactions.

Performance of Two-Phase Locking:

• There is overhead associated with acquiring and releasing locks.

• Concurrency is reduced, leading to unstable latencies, especially at high percentiles.

• Aborted transactions waste resources.

• Deadlocks occur more frequently in 2PL compared to lock-based read committed methods.

Serialisable Snapshot Isolation (SSI)

SSI aims to achieve full serialisability with minimal performance overhead compared to snapshot isolation. It is used in single-node systems (like PostgreSQL 9.1) and distributed environments, potentially becoming the new default.

Pessimistic vs. Optimistic Concurrency Control:

• 2PL: Pessimistic, as it locks resources upfront.

• SSI: Optimistic, where the database checks for isolation violations only upon transaction commit.

Performance Considerations:

• Optimistic techniques perform better when there's spare capacity and low contention between transactions.

• SSI builds on snapshot isolation by adding an algorithm to detect serialization conflicts among writes and determine which transactions to abort.

Detecting Stale MVCC Reads:

• The database tracks when transactions ignore writes based on MVCC visibility rules.

• It checks if any ignored writes have been committed and aborts if they have.

• Read-only transactions do not cause write skew and may wait until committing.

Detecting Writes Affecting Prior Reads:

• The database maintains ephemeral information about transactions reading data at the index or table level.

• When a transaction writes, it notifies existing readers of the data.

• It checks if other writes commit when the reader commits.

Performance of Serialisable Snapshot Isolation:

• The rate of aborts significantly impacts SSI's overall performance.

• SSI is best suited for short read-write transactions and is less sensitive to slow transactions compared to two-phase locking or serial execution.

If you reached here then thanks!

Partitioning is important for efficiently scaling systems. To distribute the load across multiple node, and make the system fault tolerant by replicating the partitioned nodes.

Partitioning of Key-Value databases

So you partition your database into 25 nodes, then it should be able to handle 25 times the load, right? Nah, the world isn’t fair and so is partitioning.

When data is partitioned unfairly then it is called a skewed partition. When a single partition receives a high load as compared to other partitions it is called a hot spot.

The goal of partitioning must be to distribute data and query load equally onto multiple nodes.

Lets say you are partitioning a simple kv datastore. You can do it in two ways:

1. Key range partitioning: We use key ranges to define boundaries for partitions, similar to how encyclopaedias are organised. The keys are sorted and assigned to specific partitions, making range queries efficient since not all partitions need to be queried. However, this approach can cause hot spots, where certain access points are overused, leading to an imbalanced database. To handle this, partition ranges must be adjusted based on the data, and key selection needs to be strategic. Predicting which keys will create hot spots, though, can be challenging.

2. Key Hash partitioning: To prevent hot spots in key-range partitioning, hash functions can be used to distribute keys. Even similar keys will have different hashes, provided a good hash function is used. Hash-based partitioning ensures even distribution of keys, but sacrifices the efficiency of range queries, as all partitions must be queried.

   A combined approach uses a compound primary key, where the first part is hashed to determine the partition, and the second part can be used for range queries (e.g., [user_id, timestamp]). This allows for range queries on the second part.

   However, hash keys don't fully eliminate skew. In cases like social media accounts with millions of followers, hot spots may still occur, requiring custom solutions from developers.

Partitioning and Secondary Indexes

The secondary indexes don’t map neatly into partitions, you have to take some extra steps to handle secondary indexes as they don’t uniquely identify the rows. There are two ways to handle secondary indexes in partitions

1. By Document (Local Index):
   Secondary indexes are stored within the same partition as the primary index, so each partition only covers its own documents. For example, if searching for red Fender guitars, you must query all partitions.

   While this method (scatter/gather) slows down searches by secondary index, it makes write operations fast since both primary and secondary indexes are updated within the same partition.

2. By Term (Global Index)

   In a global index strategy, secondary indexes are stored in separate partitions based on the indexed terms, not alongside the primary data. For example, if you're searching for red Fender guitars, the partition containing the index for colorwill include references (like primary keys) to all matching items, regardless of the partition the actual data resides in. This allows the search to be directed to specific partitions rather than querying all of them.

   Global Indexing makes writes a bit complex.

Rebalancing the Partitions

Now you have multiple partitions, but soon these partitions are going to get huge so you will have to partition them again, so how do you approach that?

1. Hash Mod N:
   Data is assigned to partitions using a mod function. However, this method is inefficient when adding new nodes, as most of the data gets reshuffled, leading to unnecessary data movement. This strategy is generally discouraged.

2. Fixed Number of Partitions:
   Multiple partitions are assigned to each node, with a fixed total number of partitions across the cluster. When a new node is added, it takes over some partitions from existing nodes. This limits data movement to just partition reassignments. However, it's challenging to predict the right number of partitions initially, as future data growth can vary.

3. Dynamic Partitioning:
   As data grows, partitions can be split dynamically. Nodes handle multiple partitions, and new partitions are created or deleted as needed. This approach adapts to data changes but requires starting with a reasonable number of partitions. Pre-splitting can help if key distribution is predictable.

4. Partitioning Proportional to Nodes:
   A fixed number of partitions is assigned per node. When a new node is added, it randomly splits and takes over half of existing partitions. Hash-based partitioning is used for randomness, though this can lead to imbalanced splits. Advanced algorithms can mitigate unfair partition distribution.

Teams prefer manual rebalancing over automatic rebalancing to avoid operational and unwanted failures, automatic rebalancing can some times lead to inefficient data segments and overload the nodes. Having context of the systems and business logic can prevent mishaps hence manual rebalancing is preferred.

Request Routing

When rebalancing data, nodes, and partitions, it's important to ensure users can still access the correct nodes. Users need to know the IP and port of the node to send requests. There are three ways to manage this:

1. Node Forwarding: If a node receives a request for data it doesn't have, it forwards the request to the correct node, retrieves the data, and sends it back to the client.

2. Routing Tier: A separate routing layer determines which node should handle the request.

3. Direct Client Knowledge: Clients are aware of which node and partition to contact.

The challenge in all cases is ensuring that clients or routing tiers know about changes in partition-to-node assignments. This can be managed using an external service (e.g., Zookeeper) that tracks node changes and updates either the routing tier or clients accordingly.

Okay, that’s it for this chapter. This one was easy and short, just how I like it :P

We are going to continue from where we left off in the last blog. We discussed Replication Logs, so now we are going to dive into the Problems with Replication Logs.

Problems with Replication Logs

Let's say you have a read heavy system, so to scale you just have to add more read replica's right? Yes. But in any system, problems arise. You cannot have sync replication of data here because then the latency will increase a lot, also if one node goes down then the writes will get blocked indefinitely.

Only async replication will work in this scenario, but the replication lag may vary from fraction of seconds to minutes. This leads to the follower falling behind the actual leader and can lead to inconsistencies in data, eventually the follower will catch up and hence this effect is called "eventual consistency". So lets see how to solve them:

Read-After-Write Consistency

• Ensures a client reads the latest data version immediately after writing it

• Useful when there is replication lag between the leader (where writes occur) and followers (where reads occur)

Strategies for Implementing Read-After-Write Consistency

1. Read from the Leader: When a user reads data they may have recently modified, read directly from the leader. Example: In social networks, users read their own profiles from the leader and others' profiles from followers

2. Use Last Update Time: If a user updated data within a recent timeframe (e.g., the last two seconds), read from the leader

3. Track Last Write Timestamp: Save the timestamp of the last write and read from the leader until the data reflects updates up to that timestamp

Monotonic Reads

• Ensures a user does not see older data after having seen newer data, avoiding "time travel" effects due to replication lag

• Provides a weaker guarantee than strong consistency but is stronger than eventual consistency

• How to make sure the above two things happen?

  Ensure the user always reads from the same replica

Consistent Prefix Reads

• Ensures the correct order of write operations when there is causality in data (e.g., messages in a chat app)

• Prevents anomalies like broken message sequences caused by reading from different replicas

• Common Issue: Occurs more frequently in sharded databases

• Solutions:

  • Write causally related data to the same shard.

  • Use specialised algorithms to maintain the correct order

Solving Replication Lag

• Replication lag is challenging to manage in applications; do not assume replication is synchronous when it is actually asynchronous

• Use Transactions: Transactions help provide stronger consistency and reduce complexity in application code

Multi-Leader Replication

1. Concept:

   • Unlike leader-based replication (single leader for all writes), multi-leader replication allows multiple nodes to accept writes simultaneously. Each leader also acts as a follower to other leaders.

2. Use Cases:

   • Rarely useful within a single datacenter.

   • Multi-datacenter operation: Each datacenter has its own leader; leaders replicate changes asynchronously to other data centers.

     • Benefits:

       • Performance: Local writes reduce latency; network delays are hidden from users.

       • Datacenter outage tolerance: Data centers can operate independently.

       • Network problem tolerance: More resilient to network issues with asynchronous replication.

3. Implementation:

   • Tools: Tungsten Replicator (MySQL), BDR (PostgreSQL), GoldenGate (Oracle).

   • Challenges: Configuration issues (e.g., auto incrementing keys, triggers), potential data conflicts.

4. Specific Use Cases:

   • Offline Clients: Applications (like a calendar) that work offline use local databases and synchronise later.

   • Collaborative Editing: Real-time editing (e.g., Google Docs) involves local changes replicated asynchronously to the server and other users.

Handling Write Conflicts

1. Key Challenge:

   • Conflict resolution is more complex in multi-leader replication compared to single-leader replication.

2. Conflict Detection:

   • Synchronous: Conflicts detected immediately (single-leader).

   • Asynchronous: Conflicts detected later (multi-leader).

3. Strategies for Conflict Handling:

   • Conflict Avoidance: Route all writes for a particular record to the same leader.

   • Convergent State: Ensure all replicas eventually reach the same final value.

4. Conflict Resolution Methods:

   • Last Write Wins (LWW): Keep the write with the highest ID.

   • Replica Precedence: Writes from a higher-numbered replica take precedence.

   • Merge Values: Combine values in some way.

   • Custom Logic: Use application code to resolve conflicts on write or read.

Multi-Leader Replication Topologies

1. Types of Topologies:

   • All-to-All: Every leader sends writes to every other leader.

   • Circular: Nodes forward writes in a circular manner.

   • Star: One node forwards writes to all other nodes.

2. Challenges:

   • Fault Tolerance: All-to-all is more fault-tolerant but may have ordering issues.

   • Replication Loops: Use unique identifiers and version vectors to prevent loops.

Leaderless Replication

1. Concept:

   • Any replica can accept writes; no leader. Writes are sent to all replicas in parallel.

2. Read and Write Mechanisms:

   • Clients send write/read requests to multiple nodes. Version numbers determine the most recent data.

3. Mechanisms for Data Synchronisation:

   • Read Repair: Stale replicas are updated during reads.

   • Anti-Entropy Process: Background process reconciles differences between replicas.

4. Quorums:

   • Writes must be confirmed by a minimum number (w) of nodes, and reads must query a minimum (r) number of nodes.

   • Ensures up-to-date values as long as w + r > n (total replicas).

Limitations and Challenges

1. Sloppy Quorum: Increases write availability by allowing writes to any reachable node, even outside the designated ones.

2. Concurrent Write Detection: Requires knowledge of the database's conflict handling mechanism.

3. Version Vectors: Helps determine concurrent writes and ensures data consistency.

Key Takeaways

• Multi-leader and leaderless replications offer flexibility and performance benefits but introduce complexities in conflict resolution.

• Appropriate for specific use cases like multi-datacenter operations, offline clients, and collaborative applications.

• Requires careful consideration of replication topology, conflict handling, and consistency mechanisms.

• keep data close to customer (improve latency)

• high availability

• increased read throughput

Leaders and followers - who are they?

• Leaders - well, the name suggests the meaning, all writes to the leader node

• Followers - well, the name suggests the meaning, all reads go to the follower node

It is leader node's job to keep the followers up to date with the current data changes

Sync vs Async replication

So the leader can update the followers in a synchronous manner like when a row is updated, it tells the followers to update that row and until all the followers update that row the user will have to wait for the success response.

In async update the leader updates the row but doesn't wait for followers to do the same and sends the success response to the user.

Semi-sync approach is more better here, you have sync updates to 1-2 followers and the rest followers update async, this is how you maintain data durability and also the latency.

Setting up new followers

Setting up a follower can usually be done without downtime. The process looks like this:

1. Take a snapshot of the leader’s database

2. Copy the snapshot to the follower node

3. Follower requests data changes that have happened since the snapshot was taken

4. Once follower processed the backlog of data changes since snapshot, it has caught up.

Handling Node Outages

• Follower Failure: Catchup Recovery

  • Follower reconnects to the leader

  • Requests all data changes missed while it was disconnected

• Leader Failure: Failover

  • A follower is promoted to a new leader

  • Clients are reconfigured to direct writes to the new leader

  • Other followers start consuming data changes from the new leader

• Automatic Failover Steps

  • Detect Leader Failure: A node is considered dead if it does not respond within a specific period

  • Select a New Leader: The best candidate is the replica with the most up-to-date changes from the old leader

  • Reconfigure the System: Ensure the old leader becomes a follower and recognises the new leader

• Potential Issues that might occur during automatic failover

  • Conflicting Writes: Asynchronous replication may result in the new leader receiving conflicting writes.

  • Data Coordination: Discarding writes can be risky if other systems need to be synchronised with the database

  • Split-Brain Scenario: Two nodes might mistakenly believe they are the leader, causing data loss or corruption

  • Timeout Determination: Deciding the right time before declaring a leader dead can be challenging

Implementation of Replication Logs

Statement-Based Replication

• The leader logs every SQL statement (e.g., INSERT, UPDATE, DELETE) and sends it to its followers.

• Problems with Statement-Based Replication

  • Non-deterministic Functions: Functions like NOW() or RAND() produce different values on replicas.

  • Order Dependency: Statements relying on existing data (e.g., auto-increments) must execute in the same order on all replicas.

  • Side Effects: Statements with side effects may result in different outcomes on each replica.

• Solution

  • Replace nondeterministic functions with fixed values on the leader.

Write-Ahead Log (WAL) Shipping

• An append-only sequence of bytes records all writes to the database. The leader sends this log to its followers.

• Used in PostgreSQL and Oracle

• Disadvantages of WAL Shipping

  • Low-Level Data Description: The log describes data changes at a low level (specific bytes and disk blocks), tying it closely to the storage engine.

  • Version Incompatibility: Running different versions of the database on leaders and followers is usually not possible, impacting operations and complicating zero-downtime upgrades.

Logical (Row-Based) Log Replication

• A sequence of records describing writes to database tables at the row level:

  • Inserted Row: Contains new values for all columns.

  • Deleted Row: Identifies the unique information for the deleted row.

  • Updated Row: Contains unique row identifier and new values for columns.

• Used by MySQL binlog.

• Advantages of Logical Log Replication

  • Decoupled from Storage Engine: Easier to maintain backward compatibility.

  • Ease of Parsing: Beneficial for external applications like data warehouses, custom indexes, and caches (change data capture).

Trigger-Based Replication

• Moves replication to the application layer.

• A trigger executes custom application code automatically upon data changes, logging changes to a separate table for external processing.

• Disadvantages of Trigger-Based Replication

  • Overhead and Complexity: Higher overheads and more prone to bugs.

  • Flexibility: Can be useful for specific use cases requiring custom logic.

I'll have to break this chapter into 3 parts otherwise the blog will become too long and that will be problematic for the goldfish to read as they have a small attention span.

What does software do? It propagates the transfer of data, well, it is too simple of an explanation, there are logical layers in between to make the data go to the right place in the right form. What that right form is on you to decide.

There have been many formats in which one could transfer data, my favourite is JSON, why? Because it is human readable. :')

But data requirements keep changing due to new feature addition or refactor. How do you maintain the consistency of data then? Old code cannot read newer data. Newer code cannot read older data.

There are two keywords for this:

• Backward Compatibility - newer code can read older data written by older code

• Forward Compatibility - older code can read newer data written by newer code

Back-populating helps. But a programmer needs to make plans before releasing the changes which might break backward/forward compatibility to prod.

How to encode data? (You gotta interchange it right? So how do you encode it?)

1. in-memory objects - your garden variety data structures (arrays, structs, trees etc)

2. In-memory data that is converted to bytes so that it can be sent over the server.

   Here the conversion of in-memory data to bytes is called encoding, serialisation, marshalling and other way around is called decoding, deserialisation, unmarshalling.

Avoid using language specific encoders like serialisers (java), pickle (python) because they tie you to a single language, they are bad with versioning, and there is a security vulnerability of remote arbitrary code execution.

The most popular encoding formats are JSON, XML and their binary forms.

JSON and XML don't support sending binary data. People use base64 encoding and pass data as a string, which increases the data size by 33%.

The goal is to keep the bytes to a lower size, there are many methods to do that but not everyone agrees to something common. There is thrift, protobuf that decrease the size of the data while maintaining schema. The author then talks about Avro, I won't be explaining it here, info about Avro is plentiful on the internet.

The book talks about modes of dataflow next:

• data flow through dbs (Data outlives code)

• dataflow through REST, SOAP, RPC

• message passing (kafka, rabbitmq)

And I'm going to stop here.

We discussed till LSM-tree in the last blog, had to break it down into 2 cause this chapter has too much information.

Many databases use B-trees, it has become the standard. LSM-tree and B-tree both keep the key value pair sorted, but both of they have a different design philosophy.

Log-structured indexes -> break down db into variable size segments, writes a segment sequentially

B-tree indexes -> break the db into fixed size blocks or pages, 4Kb in size, read/writes one page at a time, O(logn) depth

B-tree use a Write ahead log to handle crashes, any data that is to be written in the B-tree gets appended to this file and then gets written in the B-tree.

Comparison between LSM-tree and B-tree. (You can find this anywhere, not gonna explain here sorry :()

Optimisations in B-tree:

• Copy-on-Write Scheme

• Key Abbreviation

• Disk Layout Optimisation

• Additional pointers

So apart from primary indexes there are also secondary indexes. They complement primary indexes and enable efficient joins and queries. the difference between them is that in secondary indexes the key values can be non-unique.

Coming to full text search/fuzzy search - your key-value indexing cannot handle full text search as even if there is a simple mistake in the keyword it won't show you any results. Full-text search engines expand queries to include synonyms, ignore grammatical variations, and support proximity searches. Fuzzy search deals with misspelled words.

This is a blog about correcting spellings: https://norvig.com/spell-correct.html

Okay, now lets talk about OLTP (Online Transaction Processing) and OLAP (Online Analytics processing).

OLTP forms the logical unit for the business use case, but using the same db for running analytical queries can slow the db down, so someone came up with OLAP dbs, these dbs are specifically built for analytics.

Data warehouses are for analysts to run their inefficient queries and fetch useless stats to impress the upper management, kidding. Data is piped from OLTP through an ETL (Extract-Transform-Load) pipeline into data warehouses where analysts can run any number of queries. Data warehouses usually follow a star schema. Look up on google. I cannot spoon feed everything, duh.

There are something called Columnar Dbs, as the name suggests well they store the column values together instead of the rows, these dbs are efficient for analytical queries. See tables in warehouses can have hundreds of columns and millions of rows, now going through every row and having the whole row in memory doesn't make sense if the user needs only specific column values, so column-oriented storage comes handy here.

Column-oriented storage, compression, and sorting helps to make read queries faster and make sense in data warehouses, where the load consist on large read-only queries run by business teams.

That's it, the chapter also has a lot of nitty gritties but I won't document them here, I have broadly covered the important things.