In Capture The Flag competitions, often the goal is to simulate real world activities. So when creating CTF challenges, sometimes challenge authors will want to simulate actual users who will trigger a vulnerability.

For example there can be a Cross Site Scripting (XSS) challenge where the goal is for the participant to exploit an admin’s privileged session to obtain tokens/cookies/source code. Other examples might include a second order SQL injection triggered by another user visiting the post page, or maybe a CSRF attack in some kind of management panel.

One of the solutions to this issue is to bundle a headless browser into the challenge itself. Either through the use of a scheduled job (i.e. cron) or by directly having the challenge create a browser instance.

While this approach will work, it’s likely to cause more headaches for a few reasons:

It's very resource intensive
- Browsers are famously RAM hungry and this applies to headless browsers as well. Each headless browser can consume from 100 to 400MB of RAM - all for a single challenge. The naive solution of creating a browser tab per submission will quickly result in higher memory usage and crashes.
The challenge container/image will be complicated
- It's a hassle to identify all the dependencies required for a browser to run in a container. Modifying an existing image will often just bring in bloat.
- Since containers should be considered ephemeral, running cron jobs in a container isn't recommended.
The container image will be huge
- All the extra dependencies and additional installations will result in a big image. This will make it annoying to build, push, pull, run - you name it!

We searched high and low for a low resource solution :(

On its own the resource considerations don't look like much. One or two challenges running their own browsers isn’t going to be hard.

But at ctfd.io, as part of our workshops, we can be running hundreds of challenges simultaneously. The resource utilization and complications will quickly increase. Not to mention, challenge developers don’t really want to have to deal with browsers. It’s a secondary part of their responsibility to develop high quality, engaging, CTF challenges.

So we wrote Tourist.

The Tourist Way

Tourist is an HTTP API around Microsoft Playwright that lets us offload the work of browser interaction to a dedicated service. Instead of bundling the browser with every challenge, we can just make HTTP queries to a dedicated service from the challenge.

Tourist allows you to:

Schedule jobs to have a browser visit your challenges
Perform browser actions.
Everything you can do with Playwright (click buttons, follow links, dismiss alerts), you can do with Tourist.
Take screenshots, generate PDFs, record videos.

Tourist can schedule jobs both synchronously - when you need to block your challenge until the results are obtained, or asynchronously when you want to use a non-blocking approach.

Security

You’d be right to ask if running an exposed API to a headless browser is secure. The answer is: not exactly. However, we’ve taken precautions to ensure that Tourist is secure and can be safely run in your environment.

Sandbox

All browser actions are executed in a Node JS vm2 sandbox. The only exposed object is the page object which Playwright runs against. It’s not possible, assuming the security of vm2, to import other modules or to do anything useful.
Additionally, we recommend running Tourist in a container.

Authentication

Tourist supports JWT authentication, and has two kinds of tokens:

Issuer Tokens - obtained during the start of the application (or via CLI). Used to obtain visit tokens.
Visit Tokens - obtained via the API by using Issuer Tokens to authenticate. Used to schedule jobs which are restricted to specific target domains.

You might ask why two tokens instead of an API key?

Once a challenge is completed a participant may have access to the challenge source code as well as the Tourist token. Having the token restricted to a specific target ensures that even after the token is leaked it cannot be used to schedule jobs against any other targets.

The issuer token is just to generate visit tokens via the API instead of logging into the Tourist server directly. If your deployment of Tourist isn't publicly available you can always disable authentication entirely.

High test coverage

At the time of writing this article, Tourist is covered by almost 100 unit tests, touching on every aspect of its functionality.

Examples

Here are basic examples from the Tourist repository. They are written in Python with the requests library, but any language and HTTP library should work.

Adding Cookies to the Browser

# Go to https://httpbin.org/cookies and take a screenshot synchronously
import base64
import requests

url = "http://localhost:3000/api/v1/sync-job"
token = "<visit-token>"

headers = {
  "Authorization": f"Bearer {token}"
}

data = {
  "steps": [
    {"url": "https://httpbin.org/cookies"}
  ],
  # You can create a video and a pdf the same way by using additional options: "RECORD" and "PDF"
  "options": ["SCREENSHOT"],
  "cookies": [
    { "name": "test", "value": "test", "domain": "httpbin.org", "path": "/" }
  ],
}

response = requests.post(url, json=data, headers=headers).json()

if response["status"] == "success":
  screenshot_b64 = response["result"]["screenshot"]
  screenshot = base64.b64decode(screenshot_b64)

  with open("screenshot.png", "wb+") as screenshot_file:
    screenshot_file.write(screenshot)

Getting a Screenshot of a website

# Go to https://example.com and take a screenshot synchronously
import base64
import requests

url = "http://localhost:3000/api/v1/sync-job"
token = "<visit-token>"

headers = {
  "Authorization": f"Bearer {token}"
}

data = {
  "steps": [
    {"url": "https://example.com"}
  ],
  # You can create a video and a pdf the same way by using additional options: "RECORD" and "PDF"
  "options": ["SCREENSHOT"]
}

response = requests.post(url, json=data, headers=headers).json()


if response["status"] == "success":
  screenshot_b64 = response["result"]["screenshot"]
  screenshot = base64.b64decode(screenshot_b64)

  with open("screenshot.png", "wb+") as screenshot_file:
    screenshot_file.write(screenshot)

Asynchronously getting a website screenshot

# Go to https://example.com and take a screenshot asynchronously (by checking job status)
import base64
import time
import requests

url = "http://localhost:3000/api/v1/async-job"
token = "<visit-token>"

headers = {
  "Authorization": f"Bearer {token}"
}

data = {
  "steps": [
    {"url": "https://example.com"}
  ],
  # You can create a video and a pdf the same way by using additional options: "RECORD" and "PDF"
  "options": ["SCREENSHOT"]
}

response = requests.post(url, json=data, headers=headers).json()

if response["status"] == "scheduled":
  job_id = response["id"]
  print(f"Job ID: {job_id}")

# you probably want to do a better job awaiting the job completion than this...
while True:
  time.sleep(1)
  job_response = requests.get(f"http://localhost:3000/api/v1/job-status?id={job_id}", headers=headers)

  if job_response.status_code == 200:
    job_response_data = job_response.json()

    if job_response_data["status"] == "success":
        screenshot_b64 = job_response_data["result"]["screenshot"]
        screenshot = base64.b64decode(screenshot_b64)

        with open("screenshot.png", "wb+") as screenshot_file:
          screenshot_file.write(screenshot)

        break

Interacting with a website while recording a video

# Go to https://ctfd.io/, click on features, click on check-out demo
import base64
import requests

url = "http://localhost:3000/api/v1/sync-job"
token = "<visit-token>"

headers = {
  "Authorization": f"Bearer {token}"
}

data = {
  "steps": [
    {
      "url": "https://ctfd.io",
      "actions": [
        "page.click('#navbarResponsive > ul > li:nth-child(1) > a')",
        # DO NOT use page.waitForNavigation(), tourist will automatically await each step's completion
        "page.click('body > div.content > div.container.mb-5 > div:nth-child(2) > div > h5 > a')",
      ]
    }
  ],
  # You can create a video and a pdf the same way by using additional options: "RECORD" and "PDF"
  "options": ["RECORD"]
}

response = requests.post(url, json=data, headers=headers).json()


if response["status"] == "success":
  video_b64 = response["result"]["video"]
  video = base64.b64decode(video_b64)

  with open("video.webm", "wb+") as video_file:
    video_file.write(video)

For more check out the examples directory in the GitHub repository.

Open Source

Tourist is our solution to dealing with browser based CTF challenges. It solves the problems we mentioned above:

It’s easy to use
- You can avoid the headache of having to create an automated browser agent to visit your challenge. Just put the API call in your challenge and Tourist will visit it.
It helps you avoid containerization hassle
- You no longer have to package the browser inside your containers. You can use any base image and manage the browser with simple API calls.
Better resource usage
- Reusing the headless browser is a huge resource saver. Your images will be hundreds of megabytes lighter, the runtime will consume hundreds of megabytes less RAM and significantly less CPU time.

Tourist was born as an internal project here at ctfd.io. We use it for essentially every event that we run. We hope that other CTFs and challenge developers can benefit from it as well.

In summary, deploy Tourist once, use it with any challenge that needs a browser.

We have open sourced Tourist under the Apache 2.0 license at https://github.com/CTFd/tourist. Feel free to submit any issues, feature requests, contributions to the project!

Introducing Tourist

Miłosz Skaza