As most of you probably know, the one important thing that separates Dark Web from Clearnet is the fact that Deep Web was unindexable and as such unreachable by search engines. As it stands this might soon become untrue since DARPA has announced that they are finishing the development of their MEMEX search engine capable of crawling the dark web.
MEMEX search engine
As reported by DARPA a total of 17 different contract teams were employed in developing and perfecting MEMEX search engine. It was launched earlier this year and was also featured at “Wait for What?” where its use and possibilities were talked about. The main use for MEMEX, as reported by DARPA will be to help human-trafficking investigators pinpoint the necessary clues they need for the investigation and which they would miss otherwise.
It was stated that several agencies are already using MEMEX to aid in their investigation, but details regarding this statement were not disclosed. What differentiates MEMEX from everyday search engines like Google and Yahoo is the fact that it is capable of crawling through a larger set of data, which includes dark web .onion sites reachable by the Tor browser. After crawling, MEMEX creates a data map that will hopefully allow investigators to connect certain data to online human-trafficking offenders.
The main problem that investigators are currently facing is not the lack of data obtained by searching the web, but rather going through all the collected hyperlinks that a certain search yields.
MEMEX helps in this case by analyzing data and making out patterns of data usage that can be connected to certain offenders or organizations. Another feature that differentiates MEMEX from other search engines, as mentioned above, is the fact that it searches deep web and sites that would otherwise be deemed irrelevant by other software. Temporary sites that have not yet been indexed by regular search engines are found on the dark web and are often used for hiding illegal pornographic advertisements or other forms of communication in the human trafficking community.
The full extent and use of MEMEX are still uncertain and while DARPA concretely states that it will not serve in deanonymizing dark web users who are not in any way connected to human trafficking, there are still those who fear the integrity of dark web after this announcement. The fact that the dark web was not searchable offered a certain feeling of protection to its users and taking that away from them has left them uneasy. Nonetheless, MEMEX creators say that their work will not serve in any way to compromise the personal data of innocent people.
MEMEX Deep Web Search Engine
They continue to claim that “If something is password protected, it is not public content and MEMEX does not search it.”It is also mentioned that the only reason why dark web markets are targeted is that they offer a good place for spreading and advertising human trafficking, despite the fact that most marketplaces distinctly forbid it. While this technology is very young and has a lot of room for development, it is certain that it offers a step in the right direction when it comes to fighting, child exploitation, and human trafficking. This, however, does not relieve the fact that it can also be greatly abused if given the chance to do so.