The Blue Beacon

The Korean Internet is quite an interesting place.  Although the situation has been improving somewhat these days, for the longest time (ever since the beginning of its prime time in late 1990s) a small number or giant “portals”—the one-stop starting point for all activities—have held undue influence and dominance on most people’s Internet usage.

These portals operate technically on top of the TCP/IP network just like the rest of the Internet does, but their business strategy and mindset are little different from those of old monolithic service providers such as Compuserve and AOL: They tend to build a wall around their service, making it way too inconvenient to access whatever contents/information contained inside the wall from the outside.

Take robots.txt, the web-crawling robot exclusion policy of the top Korean portal as an example:

NAVER Blog’s robots.txt says:

User-agent: *
Crawl-Delay: 5
Disallow: /PostList.nhn
Disallow: /PostPrint.nhn
Disallow: /NBlogPostPreview.nhn
Disallow: /NBlogHidden.nhn
Disallow: /BlogInfo.nhn
Disallow: /PostExportDoc.nhn
Disallow: /PostPreview.nhn
Disallow: /NTag4Ajax.nhn
Disallow: /NWeather4Ajax.nhn
Disallow: /buddy/
Disallow: /export/
Disallow: /common/
Disallow: /post/
Disallow: /npost/
Disallow: /main/ 

This is the exclusion policy of NAVER Blog, the leading blog service provider in Korea.  The URL space /post/ stores, yes, all post contents; by excluding that, NAVER prohibits Google (or any other crawler) from indexing post text, effectively nullifying the purpose of crawling.

How about NAVER’s knowledge base service, kin.naver.com (similar to Yahoo! Answers, which was more or less modeled after it)?

kin.naver.com’s robots.txt says:

User-agent: *
User-agent: *
Disallow: /browse
Disallow: /api
Disallow: /detail
Disallow: /db
Disallow: /editor
Disallow: /expert
Disallow: /ing
Disallow: /kinac
Disallow: /knowhow
Disallow: /list
Disallow: /nboard
Disallow: /ngc
Disallow: /open100
Disallow: /openkr
Disallow: /poll
Disallow: /qna
Disallow: /search
Disallow: /wizard
Disallow: /xfile

Not much different.  Again, it prevents virtually everything from being indexed by third-party crawlers.

Why do the Korean portals have to be so draconian in their indexing policy?  It can be explained by the fact that, for example, NAVER also happens to be the number-one Internet portal and search engine in Korea—even Google has had a hard time penetrating into the monopoly.  The portals need to “protect” the vast array of contents (blog, news, knowledge base, etc) in order to force people to use their directory and search service, thus building a self-contained, vertical empire.  In fact, if you go to any Internet cafe in Korea, there is a 99% chance that the web browser has NAVER as its startup page.  It’s not that NAVER bribed Internet cafe owners or anything on a massive scale; it just happened that people would need to go to NAVER in order to find what they want to find, because NAVER’s contents will not be available through any other portal or search engine.

The beauty of Internet (or at least the web portion of it) is that everything is interconnected together so you can hop among a multitude of relevant contents and information before you realize.  In that sense, the Korean Internet does not deserve the name, in that the contents are often not inter-connected, but unidirectionally linked pursuant to the interest of a small number of corporate giants.

Fortunately, there are a growing number of “web 2.0” companies in Korea, most of them fairly new to the scene, that do seem to realize the importance of inter-contents-provider synergy; it remains to be seen whether those pioneers would succeed to change the landscape of the Korean Internet for the better.