Legal and Ethical Questions about Internet-Based Competitive Intelligence Research

A fellow Competitive Intelligence professional emailed me recently to ask about the legality of Internet research based on a competitor’s robots.txt file and information marked “Confidential” yet available via web searches. I thought this was a topic that other Competitive Intelligence professionals may have questions about, and could foster some discussion about legality and ethics in our profession. I am sharing the body of my assessment of these questions in this post, and I hope you’ll share your thoughts and analysis in the comments.

A Competitive Intelligence professional needs to comply with their General Counsel’s advice. Even if you believe that activity is legal and ethical, your General Counsel is the final arbiter of the risk that your organization is willing to tolerate.

In the US, the Economic Espionage Act defines the legal basis for Competitive Intelligence. According to my understanding of the Act, based on conversations with legal experts, holders of sensitive information bear a substantial burden for keeping information secure.

The SCIP Code of Ethics requires researchers to be honest in their identities. Researchers can execute a sort of “general public” approach to Internet research, and the standard tools and methods available. Advanced Search falls within that framework. Using fake on-line identities, harvesting passwords, or “hacking” would violate that code.

So now we can examine each scenario on both a legal and ethical framework:

Information protected in robots.txt files: 

From a practical perspective, the methods I described in the article will not find information covered in robots.txt; Google and other search engines honor this file. 

In a legal sense, the holder of this information, placing information on a public server or platform, is not doing its part to protect that information.

In an ethical sense, it is a question whether or not looking at a robots.txt file or accessing the assets specified in that file constitutes hacking. I do not believe it does, as a researcher can use a standard browser and Internet connection. 

Corporate Counsel would be reasonable in setting a boundary against this activity, however. It’s fair to assume a competitor could bring a case based on this activity if it were to come to light.

Documents marked “Confidential”: 

From a legal perspective, marking information “Confidential” does not confer protection under the Economic Espionage Act. A holder of information marked Confidential still has an onus to take reasonable steps to protect that information, and posting the information to a public server does not meet a reasonable standard for protection.

From an ethical perspective, this varies by industry. For example, Government Contracting is one industry that takes confidentiality very seriously. Using competitor information marked “Confidential” is often the basis for charges of fraud. Aggrieved parties have made these charges even if the “stolen” information was improperly made available, e.g., on a public web server, a printed document left in a waiting room.

Again, Corporate Counsel would be reasonable to exclude this sort of information from CI research. Companies go to court when competitors have used their confidential information, even when the plaintiff did not meet a minimal standard for securing that information.

Getting started with STEEP analysis

Strategic analysts use STEEP to understand the complex interplay of forces over which they have no control:

  • Reduce the risk of missing important details
  • Rethink issues based on a broad understanding of the big picture
  • Identify the factors that explain events
  • Get beyond normative analysis and assumptions
  • Make sense of the overwhelming amount of information and data
  • Identify trends, uncertainties, and inflection points that will define the future

STEEP is an acronym and reflects the five meta-factors that define any situation:

S = Society
T = Technology
E = Economy
E = Environment
P = Politics

STEEP in interchangeable with other outside-in analytical models such as PESTLE (prevalent in the UK and takes a more detailed view of Law) and STEMPLE (prevalent among national security and military intelligence professionals and evaluates Military factors in detail). Analysts can adapt STEEP to add factors for more granular detail or demote factors that may not be relevant in a situation.

I will dive into STEEP in future posts:

  • Specifics of each factor
  • Standards of source reliability and data credibility
  • STEEP as a foundation for advanced analysis and problem-solving
  • Applying STEEP to current events and challenges analysts are dealing with today such as the Covid-19 pandemic

Resources:

Making Sense of a Fast-Moving World Awash in Information with STEEP Analysis