Category Archives: CI

News, practices, examples and theory related to competitive intelligence and market intelligence.

Running into the Fog

I had the opportunity recently to connect with log-time friends and colleagues Arik and Derek Johnson from Aurora WDC for their new podcast “Running into the Fog.” The Brothers Johnson have done a great job opening up a series of conversations with leaders on deep topics. I’m sure they’ll explore great thoughts with other guests soon, and I’m honored to be their second guest on this podcast.

We had a broad-ranging conversation about competitive intelligence. I came out of the writing-a-book closet to reveal the small business Competitive Intelligence project I’ve got underway. We talked about industry-specific CI, how to identify and develop talent, and the relationship and overlap among tactics, strategy, and execution. We also discussed our experiences making sense in the early days of the Covid 19 pandemic, remember our last time together in person, and the best Star Wars movies and shows.

Please give this podcast a listen if you want to understand some recent, cutting-edge thinking about Competitive Intelligence in modern business. Give the entire series a listen if you want to explore emerging and challenging topics.

Legal and Ethical Questions about Internet-Based Competitive Intelligence Research

A fellow Competitive Intelligence professional emailed me recently to ask about the legality of Internet research based on a competitor’s robots.txt file and information marked “Confidential” yet available via web searches. I thought this was a topic that other Competitive Intelligence professionals may have questions about, and could foster some discussion about legality and ethics in our profession. I am sharing the body of my assessment of these questions in this post, and I hope you’ll share your thoughts and analysis in the comments.

A Competitive Intelligence professional needs to comply with their General Counsel’s advice. Even if you believe that activity is legal and ethical, your General Counsel is the final arbiter of the risk that your organization is willing to tolerate.

In the US, the Economic Espionage Act defines the legal basis for Competitive Intelligence. According to my understanding of the Act, based on conversations with legal experts, holders of sensitive information bear a substantial burden for keeping information secure.

The SCIP Code of Ethics requires researchers to be honest in their identities. Researchers can execute a sort of “general public” approach to Internet research, and the standard tools and methods available. Advanced Search falls within that framework. Using fake on-line identities, harvesting passwords, or “hacking” would violate that code.

So now we can examine each scenario on both a legal and ethical framework:

Information protected in robots.txt files: 

From a practical perspective, the methods I described in the article will not find information covered in robots.txt; Google and other search engines honor this file. 

In a legal sense, the holder of this information, placing information on a public server or platform, is not doing its part to protect that information.

In an ethical sense, it is a question whether or not looking at a robots.txt file or accessing the assets specified in that file constitutes hacking. I do not believe it does, as a researcher can use a standard browser and Internet connection. 

Corporate Counsel would be reasonable in setting a boundary against this activity, however. It’s fair to assume a competitor could bring a case based on this activity if it were to come to light.

Documents marked “Confidential”: 

From a legal perspective, marking information “Confidential” does not confer protection under the Economic Espionage Act. A holder of information marked Confidential still has an onus to take reasonable steps to protect that information, and posting the information to a public server does not meet a reasonable standard for protection.

From an ethical perspective, this varies by industry. For example, Government Contracting is one industry that takes confidentiality very seriously. Using competitor information marked “Confidential” is often the basis for charges of fraud. Aggrieved parties have made these charges even if the “stolen” information was improperly made available, e.g., on a public web server, a printed document left in a waiting room.

Again, Corporate Counsel would be reasonable to exclude this sort of information from CI research. Companies go to court when competitors have used their confidential information, even when the plaintiff did not meet a minimal standard for securing that information.