Skip to content
/ f1 Public

Automated image scraper powered by Deno KV and GitHub Actions. Scrapes a live photo of the Las Vegas Formula 1 track build every ~10 minutes.

License

Notifications You must be signed in to change notification settings

nberlette/f1

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

🏎️ @nberlette/f1

Scraping photos of the Las Vegas Formula 1 track construction

This is an autonomous image scraper developed using TypeScript, Deno, and GitHub Actions. It was purpose-built to document the historic Formula 1 track construction in Las Vegas, Nevada, slated to host the inaugural Heineken Silver Grand Prix on November 18th. The images will be stitched together into timelapse videos of the track's lifecycle, from start to finish 🏁.

Track Details and Statistics

Estimated Top Speed Circuit Length Corners Straights DRS Zones
212 mph β€’ 342 km/h 3.8 miles β€’ 6.12 km 17 3 2

πŸ“Έ Latest β€’ 🎬 Timelapse β€’ πŸ—“οΈΒ Previous Images β€’ ℹ️ Project Details β€’ 🌟 Star It!


Latest Snapshot

The latest image scraped from the Formula 1 track build in Las Vegas


Timelapse Preview

timelapse.mp4

Note: this video was created with images from 2023-08-15 - 2023-10-12


About

The first scrape happened on June 3rd, 2023. As of October 18th it has surpassed 18,500 commits, equivalent to over 1.2GB of image data. Photos are stored in the ./assets folder of this repository, and also persisted to a Deno KV database backed by FoundationDB.

The origin of the scraped images is an real-time photo feed, sourced directly from the official Formula 1 website.

⚠️ This project is for educational purposes and is not affiliated with Formula 1.

Tools Used

  • Deno v1.37.2
  • Rust-based JS runtime, sandboxed, with great TS/TSX support.
  • Provides the tools for network and file system operations.
  • TypeScript 5.2.2
  • Superset of JavaScript featuring advanced static typechecking.
  • Better type safety means more readable and maintainable code.
  • GitHub Actions
  • Provides free macOS virtual machines powering the scraper.
  • Responsible for scheduled execution of the scraper workflow
  • Temporarily stores the image artifacts
  • Deno KV (currently in beta)
  • Provides us with global data persistence and caching
  • ffmpeg (timelapse feature is unstable)
  • Leveraged to automatically generate timelapse videos

AI-generated artwork of a Formula 1 car racing down the Las Vegas Strip

AI-generated F1 art created with SDXL 1.0 and the prompt "Formula 1 cars on the Las Vegas Strip"


How it Works

The majority of the work happens in main.ts, despite it only being 3 lines of code. It is responsible for invoking the scraper located in src/scrape.ts, and is ran every 10 minutes by a GitHub Action defined by the workflow in main.yml.

Assets and Data

Images are named after their capture time as a JPEG file in UTC. For example, an image captured at 2023-07-09T04:28:57 would be saved as ./assets/2023-07-09/04_28_57.jpg. The latest image is always saved as ./assets/latest.jpg for easy access.

Scrape Process, Step-by-Step

  1. GitHub Actions runs the scrape workflow every ~10 minutes, depending on traffic.
  2. The runner checks out the repo, installs Deno, and prepares to scrape.
  3. deno task scrape is executed, which runs the main.ts file.
  4. main.ts imports scrape() from src/scrape.ts, which defines two inner functions, read and write.
    • Once it has checked that import.meta.main is set, the following steps are taken:
    1. πŸ” READ: read() is called with IMAGE_URL.
      • Internally, the Fetch API is used to download the image.
        If the request fails, it will be retried up to ATTEMPTS times, with a short pause between each successive attempt.
      • If all attempts are exhausted without success, the run will terminate.
      • Otherwise, a new instance of the Image class is returned.
    2. πŸ’Ύ WRITE: write() is called, with the Image as its only argument. Before writing, however, it runs through some checks:
      1. The Image.hash is checked against the hash "table" in Deno KV.
        If an entry exists, the image is stale and won't make it any further. If Deno KV is unavailable, the image data is checked against latest.jpg via a timing-safe equality comparison, avoiding exposure to timing-based attacks.
        • If they are equal, the image has not updated at the origin. The process starts over at step 4 and repeats until a new image is found.
        • If the maximum number of ATTEMPTS is reached and no new image was found, the job terminates unsuccessfully.
      2. If we've made it this far, we have a fresh image and we need to store it.
        • Image.write() persists the image to Deno KV.

          The key is generated by the Image API, using the image timestamp.

        • The image timestamp is indexed with its unique SHA-256 hash in Deno KV.

          This prevents later scrapes from duplicating this image. It also means if you try to instantiate a new Image from an old hash, it will always return the original image and its original timestamp.

        • Image.writeFile() saves it to the local file system.

          The filename is generated by the Image API, using the image timestamp.

        • Image.writeFile() also saves it to ./assets/latest.jpg,
      3. The setOutput helper pipes the image metadata to the GitHub Actions runner, to be used in the commit step.
    3. The scrape is now complete and the runner proceeds to the final steps.
  5. The photo is stored as a workflow artifact for 90 days.
  6. 🀝 All changes are committed + pushed to the repository.
  7. 🏁 The job finishes successfully and the runner is terminated. Hooray!

AI-generated artwork of a Formula 1 car racing down the Las Vegas Strip

Previous Snapshots

October 17th October 15th October 13th October 11th
2023-10-17T20:26:59 2023-10-15T21:38:24 2023-10-13T23:52:14 2023-10-11T21:26:47
October 9th October 7th October 5th October 3rd
2023-10-09T22:27:19 2023-10-07T22:26:51 2023-10-05T22:19:31 2023-10-03T22:20:37
October 2nd September 28th September 24th September 20th
2023-10-02T22:29:06 2023-09-28T22:24:03 2023-09-24T22:12:21 2023-09-20T22:22:10
September 16th September 12th September 8th September 4th
2023-09-04T22:20:41 2023-09-16T22:45:35 2023-09-10T22:33:07 2023-09-04T22:49:09

MIT Β© Nicholas Berlette β€’ Made with β™₯️ in Las Vegas, NV

This project is not affiliated with Formula 1 and is for educational purposes only.