How Some Might Bypass Paywalls…

…Like Teenagers Sneaking Into A Hotel Pool?

standard-ny-pool

Google indexes everything. Like everything. From the content on websites that require payment to access. It’s Uh-Mazingly flawed, right?! I know.

Digital publications allow discriminatory access so search engines can inspect HTTP request headers (Referer & User-Agent). It’s a bit of information websites exchange.

When you’re fifteen links deep, mining Wikipedia & you click a link to another site… a request is sent to the new web site. Referer identifies the address of web pages linking to another site’s content. For example in the past when you clicked a link during a Google search, the Referer would say https://www.google.com/. This is no longer the case.

Recently, sites began checking for another identifier as well, User-Agent. This identifies which browser or app being used. So, when Google’s web crawlers index the web it’s tells Wall Street Journal for example… the visitor is also an agent of Google. This is where the opportunity for f*ckery lies.

If information contained in request headers is edited, anyone can appear to be a Google web crawler. In fact, if someone were so inclined they could easily create a Chrome extension that does just that! Really? Really. **Then with a puff of smoke… any website’s Paywall vanishes David Copperfield or Houdini style.**

JSON is decent programming language & there are plenty of free web-based tools that create/edit JSON files. If someone were so inclined they could use them to do the following:

1. Create a new file & name it    manifest.json  & then paste into the file the following while adding any sites they desired to “read”. They’d be adding the addresses to what’s called a permissions list. 

{
  "name": "If Someone Were So Inclined Chrome Extension",
  "version": "0.1",
  "description": "If Someone Where So Inclined Chrome Extension.",
  "permissions": ["webRequest", "webRequestBlocking",
                  "http://www.ft.com/*",
                  "http://www.wsj.com/*",
                  "https://www.wsj.com/*",
                  "http://www.economist.com/*",
                  "http://www.nytimes.com/*",
                  "https://hbr.org/*",
                  "http://www.newyorker.com/*",
                  "http://www.forbes.com/*",
                  "http://online.barrons.com/*",
                  "http://www.barrons.com/*",
                  "http://www.investingdaily.com/*",
                  "http://realmoney.thestreet.com/*",
                  "http://www.washingtonpost.com/*"
                  ],
  "background": {
    "scripts": ["background.js"]
  },
  "manifest_version": 2
}

 

2. Then, they’d create a file called   background.js   & paste the following into the file:

 

var ALLOW_COOKIES = ["nytimes", "ft.com"]

function changeRefer(details) {
  foundReferer = false;
  foundUA = false

  var reqHeaders = details.requestHeaders.filter(function(header) {
    // block cookies by default
    if (header.name !== "Cookie") {
      return header;
    } 

    allowHeader = ALLOW_COOKIES.map(function(url) {
      if (details.url.includes(url)) {
        return true;
      }
    });
    if (allowHeader.filter(Boolean)==true) return header; 

  }).map(function(header) {
    
    if (header.name === "Referer") {
      header.value = "https://www.google.com/";
      foundReferer = true;
    }
    if (header.name === "User-Agent") {
      header.value = "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)";
      foundUA = true;
    }
    return header;
  })
  
  // append referer
  if (!foundReferer) {
    reqHeaders.push({
      "name": "Referer",
      "value": "https://www.google.com/"
    })
  }
  if (!foundUA) {
    reqHeaders.push({
      "name": "User-Agent",
      "value": "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
    })
  }
  console.log(reqHeaders);
  return {requestHeaders: reqHeaders};
}

function blockCookies(details) {
  for (var i = 0; i < details.responseHeaders.length; ++i) {
    if (details.responseHeaders[i].name === "Set-Cookie") {
      details.responseHeaders.splice(i, 1);
    }
  }
  return {responseHeaders: details.responseHeaders};
}

chrome.webRequest.onBeforeSendHeaders.addListener(changeRefer, {
  urls: [""],
  types: ["main_frame"],
}, ["requestHeaders", "blocking"]);

chrome.webRequest.onHeadersReceived.addListener(blockCookies, {
  urls: [""],
  types: ["main_frame"],
}, ["responseHeaders", "blocking"]);


 

Both of these files could then be placed in directory on their computer (folder). Again, these two files would be the only files in the directory. If this imaginary person were too lazy to copy & paste, they could even download the source code here.

They’d open Chrome & type    chrome://extensions/   in the browser address bar.

They could click   Load unpacked extension… while making sure Developer Mode is checked in the upper right side if the buttons are missing.

Screen-Shot-2016-02-18-at-10.49.25-PM

They could select the directory where they saved the two files…

Then, if someone were so inclined…

…they might enable a certain chrome extension & hop some paywalls like a teenager sneaking into a hotel pool.

Lesson To Be Learned Here: Any time an access point for a trusted third party is introduced… inevitably, access ends up being allowed to anybody.

 

This story is provided for entertainment, informational & perhaps educational purposes ONLY. It is provided “as is” WITHOUT WARRANTY OR CONDITION OF ANY KIND, EXPRESS OR IMPLIED, & ALL SUCH WARRANTIES OR CONDITIONS ARE HEREBY DISCLAIMED. THE AUTHORS OF LUVATFIRSTBYTE.WORDPRESS.COM, WORDPRESS.COM & IT’S PARENT COMPANIES ASSUME NO RESPONSIBILITY TO YOU OR TO ANY THIRD PARTY FOR YOUR ACTIONS OR HOW THIS STORY IS INTERPRETED OR ACTED UPON.

 
Advertisements
Standard

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s