The Insights and the Future of Information

A Sony Walkman was – some thirty-five years ago – a revolutionary piece of technology. Small, lightweight, and beautiful, it was music on the go. We made mixed tapes, gifted them to buddies and to our first romantic interests. Many of us still have stacks of tapes in the attic.

Most of us have since long switched to playlists on popular services like Spotify or Apple Music. Instead of buying a CD with 12 songs, of which consumer research already showed us 30 years ago that most buyers were interested in only 2 songs, you can create a playlist that matches your taste (or that of your BFF).

It is a profound shift in the way music is consumed. After long and hard years of transformation from the physical to a digital model, the music market, today, is worth more than ever. Instead of a physical asset, the asset is the insight into your music tastes: The playlist.

What if the same would hold true for data?

95% of data is never used beyond primary use. It is created in a blink of an eye, written to some hard disk, and sits there idly, never to be touched again. For public information, we use search engines such as Google to find the information. For business information, we need to open countless files across any number of systems and applications to find the document that we know is there… somewhere. Small wonder that most data is never used beyond its primary use. The fact is, it’s simply too complicated and too time-consuming to get to that information. In a world driven by ubiquitous data, this needs to change.

What if relevant data would find us?

Say, you want to know more about a client relationship. What about instead of combing through piles of often unorganized and unstructured data, the relevant insights come to you, right on time, automatically? You’d no longer be one step behind your customer’s experience. Instead, you’d be developing this customer relationship right alongside your client, in real-time.

For this future to be the new norm, we need to map a new course for information: Content, Relevance, Timing and Destination together form Insights; computational tokens, so to say. These tokens need to be enabled to find their way to the best possible destination – automatically. They need to become self-aware of the context they operate in, finding their way around the informational universe they populate in a self-propelled manner. On the way, they might collate with other tokens to form new types of Insights.

These insights need to be highly relevant to an end-user, which might be a human operator or a machine driven by software requiring this input to make its next smart move. These Insights tokens will become tradable opening up data silos much like playlists opened up the world to more music than ever before compared to the data silo known as the CD. This system will be an intelligent, self-aware software layer that will power any company in a not-so-distant future.

PS: These ideas drive the development of Squirro, and it’s likely the key reason why Gartner named us a Visionary in the recently released Magic Quadrant for Insights Engines beating such heavy-weights like Coveo, Microsoft, Google, Sinequa, etc. Get your complimentary copy of the Gartner report here.

Posted in Uncategorized | Leave a comment

Indexing is not enough

We all use search. Every day. Most of us use one search engine only. Type a keyword or two, get a result list, look at the results – most of us never go beyond the first three results. If not satisfied, redo with another keyword combination. Most of us never use advanced search forms. Over the past 20 years, we’ve all been primed to this interaction pattern. 

Search traditionally works by crawling data, creating an index, providing for a search interface, typically a single field search bar. Trillions of person-hours have been invested by the leading contestants to get the relevance of the result set right. It’s the first three results that matter. 

Back in 2004-2009, when we operated with the largest homegrown search engine here in Switzerland, our key metrics were 1.23 keywords as input. That is not enough to spell “Bern Restaurant”. It’s either “Be Restaurant” – no we don’t want to be a restaurant … or “Bern Rest” – ok, the saying is that’s what Bernese people do the whole day anyhow (me being one of them). Today with better type ahead, this has grown to a query length of about two words. 

The core principle remains: You need to go to information. 

Done efficiently, it is a wonderful way to access information. Large search providers and more focused providers such as Enterprise Search vendors have added layers of sophistication to the approach (e.g. ingestion optimization, profiling, relevance ranking, and many more strategies). The basic approach, though, remains the same. 

In a world of too much information, this approach has its limits.

You need to know what you look for. Only if you can describe – to some level of precision – what you’re looking for, will the result list be somewhat meaningful for your request. While it is straightforward to use “that restaurant” for a search, it is less so for more complex situations. Say you join a new organization. By definition, you do not know what others, before you have written, say, about products. How do you start your search not even knowing the product terms?  Or you try to find, within a large organization, who knows what about a customer situation? Most organizations have disjunct information systems – multiple CRMs, service management tools, file shares, and more. How do you find the relevant information for a quick remedy?

A different approach is required to get to the next level: instead of you going to information, the right insights need to come to you at just about the moment you need them to get your work done. You turn information provision upside down. 

What is required for this to happen: you need computationally aware informational objects. Sure, the data item will not develop cognition about itself. But the construed concept of the informational object must express in computational terms the concept and the notion of the underlying information/data. 

In essence, the information/data needs to be expanded into a concept of an informational object with lots of (probabilistic) meta information to get some level of cognition of its content and meaning. Besides, the system needs a good understanding of your current informational needs, in other words, it needs a good profile of you. 

To extract such insights of information/data and you as a user, the recent advances in AI are the catalyst of this transformation. An essential element is to approach this transformation to an informational concept or object not as yet another relational schema but as a continuous probabilistic re-compute: Information changes, situational changes, user preference changes.

The first three Bernese restaurants we rendered back in the days were not, in absolute terms, the best. They were the ones we thought – well, as the ranking algorithm we put in place provided – were the most relevant to your search and matching your search profile.  Today we extend this concept and think of this like a continuously cooked bouillabaisse and from which the system picks a bowl when it’s time for lunch, automatically. 

This information revolution that is about to take place is well summed up in this quote from Anthony Mullen et al.:In the 15th century, Copernicus introduced the shift from an earth-centric to a sun-centric view of the solar system. There is a Copernican shift underway in how enterprises handle data. The approach shifts the emphasis from relational schemas as the center of the “representation” universe to concept and object models expressed across semantic and machine learning technologies.

Plus see what we do over at Squirro: Read Gartner’s Magic Quadrant for Insight Engines, where Squirro has been recognized as a Visionary.

Anthony Mullen, Magnus Revang, Stephen Emmott, Erick Brethenoux, Bern Elliot, Jessica Ekholm2021 Strategic Roadmap for Enterprise AI: Natural Language Architecture, Gartner, December 2020

Posted in Uncategorized | Comments Off on Indexing is not enough

The public debate that does not take place

We’ll get through the Pandemic – eventually. The vaccines are effective. It looks like we’ll get over here most vaccinated by end summer.

This would be the time by our political leaders to start a debate about the future beyond the pandemic plagued last two years.

Most of my friends and my family and me actually are privileged kids. We survived the largest public health crisis relatively unscathed. We were restricted for some time in our freedom of movement, but that was about it. I acknowledge this fact.

Many others are not so privileged. People in the hospitality business, in event organization, fitness centres, shops assistants, many in smaller companies lost their jobs, their livelihoods, their perspectives in life.

We as a society have an obligation to each other.

Given the economic devastation that Covid has caused it would now be the moment to start a debate about how to shape the post-Covid future, how to help to folks that for no mistake of theirs lost their livelihoods. We will need a Marshall type program to get them back on their feet. And yes that might include that we for say 2-3 years pay a percentage point or two more in taxes (and close some of the egregious tax loopholes awarded to corporations and wealthy individuals).

Why shall we do that? Out of pure self-interest.

Provide our children a great place to live. A society that is a great place to bring up your children and you can provider them a perspective and all of that in a safe environment is an equitable society. If that social contract is broken, you get broken states populated by desperate and vulnerable people. Just ask people in formerly prosperous places like Venezuela or closer to home people in destitute banlieues and suburbs in say Manchester, Berlin or even Geneva and Zurich.

Just, there’s no such discourse by our elected leaders. They will miss the beat, yet again.

PS: Irony of it all: Such a program would enjoy popular support and would be a vote winner.

Posted in Uncategorized | Comments Off on The public debate that does not take place

Missing the beat – again

Our lovely government has been inept at handling the largest public health crisis for a century. This is not a statement with hindsight but an expression of frustration for more than a year.

After having failed at setting up a comprehensive testing and tracing regime, our government bumbled the vaccine rollout: Lonza, based in Switzerland, proposed the government our own vaccination production.

While the ‘we don’t publish numbers on weekends, because also our employees need time off, too’ comment of the ministry of health is Asterix & Obelix level parody (‘It’s 5pm, tea time. We’ll be back to battle after that’) the rejection of the vaccination production kills people.

And that is just the latest misstep. With the next one already visible on the horizon: The social and economic upheaval is significant. And instead of starting a national conversation about healing, about rebuilding, about solidarity (yes that might imply higher taxes) you hear, well, Nothing…

But then, this is something most likely – and the proof points of the past 12 months support the claim – beyond their skillset. Most of the folks in positions of relevance starting with our government minsters have been politicians all their live or most of their lives.

In a country like ours a political career is still built primarily on risk avoidance (Any too extreme position will cost you the votes across the aisle to become minister… ). A crisis is pretty much the opposite. The nature of a crisis is confusion. It requires taking calculated and at times bigger risks to resolve it. Not an operational pattern any of our leaders is used to.

Posted in Uncategorized | Comments Off on Missing the beat – again

Attitudes matter

Excerpt from an email conversation with a colleague of mine. It started with a comment on this video: Squirro App Studio.

Teammate: This video is no longer current.

Me: Can you explain to me what is not current here (except a portion referring to DSS instead of AIS)?

Teammate: Hi Dorian, That’s what I mean. It refers to DSS so it should not be distributed. Is my assumption not correct? Thank you.

Me: Dude, in a perfect world sure. Now we live in an imperfect world: So yes your assumption is not correct (until we have redone the video, btw are you volunteering?). 

Look I react to this because most recently people left and right have been nagging about things and that goes on my nerves: Startups by definition are not perfect. We told all of you (Remember me saying in your hiring interview: Startups are f**ing hard. Are you in for that?). 

Tesla produced shitty cars for the better part of the first 10 years of their existence (don’t believe me? check their resell value….). But they were after something (understand mobility in terms of software not hardware) and see where they are today…

Same with us: We understand the digital world in terms of insights not data. On that road things have edges. Life with it or else. Or to say it with Mark Watney in the Martian ““I’m going to have to science the shit out of this.”

So I would have preferred a comment of the type: ‘Use video with note only: still references DSS, will need updating’. Bonus points for saying: @author: Contact me, happy to help for an updated version.

You know like I do for requests you have for me. Instead of me saying: “Do your job, deliver results.”, I am a ping away and e.g. over the weekend help you to expand the SFDC playbook to make you successful. 

Posted in Uncategorized | Comments Off on Attitudes matter


On Friday our government announced finally a switch to a comprehensive testing strategy. Finally…. (we suggest this approach 10 months ago…)

Posted in Uncategorized | Comments Off on Finally!

Understanding mRNA Vaccines

Brilliant feed on Twitter:

Posted in Uncategorized | Comments Off on Understanding mRNA Vaccines

Trump is an enabler II

In addition to an earlier post here more items, which Trump unintentionally enables:

  • A realisation in the rest of the world that a world policeman isn’t a bad thing after all (To paraphrase former President Holland’s remarks at the SEF earlier this year)
  • That flooding the zone with s..t, aka constant lying, helps the truth win in the long run as currently nobody believes anything that Trump says anymore
Posted in Uncategorized | Comments Off on Trump is an enabler II

The renaissance of the middle ages

The world is currently undergoing a curious transformation: It’s history in reverse. We are back in the middle ages: tribes, rule of the most powerful, that is no rule of law, lies all around.

Posted in Uncategorized | Comments Off on The renaissance of the middle ages

Rethink value chains

An interview today’s Handelszeitung on how AI allows for rethinking value chains.

German version.

Posted in Uncategorized | Comments Off on Rethink value chains