A fellow Competitive Intelligence professional emailed me recently to ask about the legality of Internet research based on a competitor’s robots.txt file and information marked “Confidential” yet available via web searches. I thought this was a topic that other Competitive Intelligence professionals may have questions about, and could foster some discussion about legality and ethics in our profession. I am sharing the body of my assessment of these questions in this post, and I hope you’ll share your thoughts and analysis in the comments.
A Competitive Intelligence professional needs to comply with their General Counsel’s advice. Even if you believe that activity is legal and ethical, your General Counsel is the final arbiter of the risk that your organization is willing to tolerate.
In the US, the Economic Espionage Act defines the legal basis for Competitive Intelligence. According to my understanding of the Act, based on conversations with legal experts, holders of sensitive information bear a substantial burden for keeping information secure.
The SCIP Code of Ethics requires researchers to be honest in their identities. Researchers can execute a sort of “general public” approach to Internet research, and the standard tools and methods available. Advanced Search falls within that framework. Using fake on-line identities, harvesting passwords, or “hacking” would violate that code.
So now we can examine each scenario on both a legal and ethical framework:
Information protected in robots.txt files:
From a practical perspective, the methods I described in the article will not find information covered in robots.txt; Google and other search engines honor this file.
In a legal sense, the holder of this information, placing information on a public server or platform, is not doing its part to protect that information.
In an ethical sense, it is a question whether or not looking at a robots.txt file or accessing the assets specified in that file constitutes hacking. I do not believe it does, as a researcher can use a standard browser and Internet connection.
Corporate Counsel would be reasonable in setting a boundary against this activity, however. It’s fair to assume a competitor could bring a case based on this activity if it were to come to light.
Documents marked “Confidential”:
From a legal perspective, marking information “Confidential” does not confer protection under the Economic Espionage Act. A holder of information marked Confidential still has an onus to take reasonable steps to protect that information, and posting the information to a public server does not meet a reasonable standard for protection.
From an ethical perspective, this varies by industry. For example, Government Contracting is one industry that takes confidentiality very seriously. Using competitor information marked “Confidential” is often the basis for charges of fraud. Aggrieved parties have made these charges even if the “stolen” information was improperly made available, e.g., on a public web server, a printed document left in a waiting room.
Again, Corporate Counsel would be reasonable to exclude this sort of information from CI research. Companies go to court when competitors have used their confidential information, even when the plaintiff did not meet a minimal standard for securing that information.
Strategic Market Intelligence 