Detecting Duplicate Content

Search for a command to run...

No comments yet. Be the first to comment.
One of the methods of abstraction is to integrate another service. But can you trust someone else's service? Third-party APIs are notorious for failing at the worst possible time. Over time, I ended u

A race condition can hit you anytime, you think it won't because your system isn't that big, or doesn't process a lot of requests, but it will happen, and when it happens, you will feel a bit silly ab

The world is going through a tumultuous phase, but that doesn't stop the progress of humanity. Things go on, people move, prices fluctuate, we eat fruit and live a healthy life. April is hot, so hot,

Well, I celebrated my birthday. :P I worked on a functionality end to end using AI. It was a simple feature and AI wrote everything correctly in a single pass. I used Codex. After that, I had to ask i

Let me think. So I worked on a new functionality and raised a PR in just 2 days for the whole end-to-end correct flow. I did not focus much on performance because I wanted to first get it right and th

Detecting and classifying content as spam or duplicate poses a significant challenge for social media firms due to the vast amount of data published every second. However, limiting the number of posts, follows, and messages appears to be the simplest solution to this problem. You can also add the "report as spam" button to the posts, and unload the task to the community, but keep in mind that the community is cunning and might report posts that do not match their ideologies so you need to have a content moderation team in place. Too much work. :')
I was assigned the task of detecting duplicate content on the feed, which prompted me to delve into spam detection and research papers published by Facebook on the subject. After thorough research, I developed a pragmatic solution.
Do you know that the detection of malware in binaries relies on fuzzy hashing algorithms? Since malware has a fixed pattern of 0s and 1s, it can be detected in a binary file by identifying these patterns.
Similarly, content can be converted into md1 hash, but since even slightly different content can result in different hashes, we need to create fuzzy hashes and search for similar hashes.
Using fuzzy hashing algorithms to identify similar hashes, we can effectively detect duplicate content on social media feeds, which is essential in preventing spam and improving the user experience. Although spam is a subjective topic, context is also important to see if the post is spam or not.
I used the ctph.js library to hash the content and detect similarities.
Some good blogs around the same topic:
https://medium.com/@glaslos/locality-sensitive-fuzzy-hashing-66127178ebdc
https://blog.pythonicforensics.com/fuzzy-hashing-and-ctph-69f4a7bbfe48
Read more blogs from yours truly
Cheers.