I remember the day well – sat in a meeting with the board discussing some work I’d done investigating people on the dark web. Little did I know that years later, the tools that started out as home-brewed scripts would be enterprise level monitoring tools indexing hundreds of thousands of hidden sites, and a ten terabyte store of passwords and leaked data. As I was sitting there, one of DarkInvaders co-founders asked me a pivotal question – What makes your scanner better than the next idiots? In his defence, the fresh faced 19 year old apprentice sat in front of him didn’t exactly cut an imposing figure, but my answer then is the same as my answer now – We are hackers ourselves. Okay, we do it legally, but our investigators grew up on these forums, chatting to hackers about hacking is second nature to them. This, of course, is a massive advantage – we already had a presence on these forums and had built reputation on them. More so than that, we understood what the black-hats were on about and could actually engage in discussions.

So, instead of coming into the weird and wonderful world of the dark web sticking out like a sore thumb (as, I can assure you, a number of the ‘blue team’ dark web investigators do), we fit in seamlessly. And we waited, we watched and we learnt. Me and, at the time, my two fellow apprentices started to build a better image of the ecosystem of the dark web. How did forums like xss.is and raidforums.com slowly funnel people to sites like payload.sh? Questions like this, and finding the answers to them, consumed us. But we answered them.

‍

All these answers finally led to the first iteration of a script dubbed DarkSky. All that script did was grab 200 popular sites, and take 5 of their pages – storing the data in text files. I knew we needed improvement, and all the learning we had done on how the ecosystem functioned fed into that passion. A fair while, a lot of iterations and only a little of Jura’s finest later – the DarkInvader crawler as we know it was finally deployed. Intelligently learning how to seek out the sensitive forums and message boards whilst also using heuristics to inform our engineers investigative efforts, backed by a colossal database of over two hundred thousand sites with over a million pages stored, all led to a technology that I could confidently say was cutting edge. I could drone on and on about speed statistics and sites hit per day, machine learning algorithms or in fact our heuristic formula – but is this the real reason I think our scanner is superior? No, it’s not.

The real reason I think our OSINT scanner is better than the rest is for the very reason I touched upon at the start – we are hackers ourselves. I recognise myself in the faceless avatars on these forums, and can empathise with their burning passion. I often think that perhaps if Gavin Watson hadn’t taken a chance on me all those years ago, I may have ended up amongst them. And so it is the human element, the people behind the screens, that ultimately make our scanner better. Because at the end of the day, there is no substitute for truly understanding why these people do what they do; it’s because they love it, just like us.

‍