Using the YouTube API to Import Videos

Today we'll be creating a PHP app to query the YouTube API for a specific user's videos. Then we'll take the data, import the videos into a SQL database, and create static-based blog posts for the Kirby CMS.

We run a cannabis based publication called WeedPornDaily, and through that brand we've created hundreds of dank YouTube videos. Recently we migrated the WeedPornDaily website from Wordpress to Kirby, and we've been in the process of importing all of our content across the entire social media spectrum (Tumblr, YouTube, Instagram, etc). If there's one thing I hate most it's unnecessary data entry, so I created this script to instantly import all of our YouTube videos from our channel to our website.

Let's get this party started.

Get Your API Key

In order to use the YouTube API, we have to register an application with the Google Developer Console.

  1. Click here to create a new application.
  2. Select a pre-existing project, or create a new one.
  3. Click create credentials, then select API Key from the dropdown.
  4. Copy this API key and save it for later!
  5. Click on the search on top and type in "YouTube". Select "YouTube Data API v3" from the dropdown.
  6. Click "Enable".
  7. You're good to go!

You can check your rate limits by clicking the Quotas tab on the "YouTube Data API v3 page". You get 1 million queries per day, so you should be fine scraping a channel or two.

Get Your YouTube Channel ID

The YouTube API requires you to query using the Channel ID, not the custom URL they allow users to set. The Channel ID is listed in your account preferences. If you're not sure where to exactly find the Channel ID, click here to see YouTube's guide on it.

Test Query

Before diving deep into any code, I like to run a test query to see we can connect to the API. Copy and paste the following URL into your browser, or an app like Postman, and insert your API key and Channel ID.

https://www.googleapis.com/youtube/v3/search?key={your_key_here}&channelId={channel_id_here}&part=snippet,id&order=date&maxResults=20

If it worked, you should see the latest 20 videos from the YouTube channel. If it didn't you might need to get a new API key, or try a different Channel ID.

Let's Scrape

Now that we have the data we need, let's build a simple application to use it. The goal is to query the API, grab the 20 posts it gives us, then loop through the rest of the pages. Each time we query, we'll save the data to a SQL database.

Pagination

The YouTube API doesn't use a simple incremental count for their pagination (e.g page 1, page 2, etc). They use page tokens (e.g. AFj94Nk). If we use the "nextPageToken" provided in the JSON response, we can query the next page, and so on -- until we run out of next page tokens.

The page tokens are applied through the query string. Add this to your API URL: &pageToken=GDAQFA - and replace with your token (next or previous page).

MySQL

I'm using MySQL for this project, but you can really hook in any kind of solution here. I'll also show you an example later on of creating static TXT based posts from the API data.

Run the following SQL statement to create a table to store the data:

Let's break down some of the columns. The id will be an auto-incrementing integer that allows for quick sorting and creating cross-table keys. The video_id will be the ID we see in YouTube URLs (e.g. http://youtube.com/watch?v=this_id). Everything else is pretty self explanatory.

Method to the Madness

We'll be creating 3 methods or functions that do everything.

loop() will loop through the JSON data and save to the SQL DB. scrape() will check if there's a next page token, and if so, call the nextPage() function. Then the nextPage() function grabs the data, calls the loop() to save the data, then calls scrape() again to check for another page. This will ideally loop until we're out of nextPageTokens.

Let's create our app.

Connecting to SQL

Open up a new PHP file and paste the following:

We define the MySQL connection variables and then run mysqli to connect to the DB. We store the connection in a global variable so that any function can access it without passing it through.

Data Grabber

Add this underneath the SQL connection above:

This is a basic cURL function that fetches a $url we provide and return the data. This will be used each time we query the API for JSON data.

The loop()

Add this underneath the get_data() function:

This function accepts a JSON object and loops through an array stored in the posts property (e.g. $json->posts = [{ id: 1, title: 'Post Name'}]). Since YouTube stores videos and other types of content, we do a quick check to see if the video type ($video->id->kind) is a video. Then we just grab all the post info from it's object properties and sling them into a SQL statement.

The scrape()

Add this underneath the loop() function:

This basically takes the JSON request object, checks for a page token, and runs the nextPage() function.

Grabbing the nextPage()

Add this underneath the scrape() function:

This function takes our $token, feeds into the $nextpage URL, calls get_data to query the API, then uses loop() to save the data, and scrape() to check for another page.

Starter Seed

We have all the functions we need to make this app work. Let's do an initial API request and start our "loop". Add this underneath the nextPage() function:

Since our scrape() and nextPage() functions require a page token, we have to do a manual API query first to "seed" them with data to get them going.

Run the app!

That's it! If you run the app, you should get a database full of your YouTube videos.

This is a fast and efficient way to quickly accrue all your YouTube content, from the video to the thumbnail. It made importing 300+ videos from a single channel a snap.

Implementing Kirby

I also had to import these video into a website running on the Kirby CMS. Kirby is a static based CMS that uses TXT files formatted to YAML blueprint standards. Our website, WeedPornDaily, already had an existing YAML blueprint for videos, so I conformed this script to that. But you can edit this to fit any post type easily.

Swap this loop() with the other:

Nothing fancy here. Just creating a folders and files, and making sure they're structured in the right way. Our goal ultimately was a folder structure like this: blog/2017/20170420-postname/video.txt. If you have issues running this, make sure the folder you're in has the correct CHMOD settings server-side.

That's just the beginning

We've only tapped into one small sliver of the YouTube API's functionality. Stuff like searching for videos is within our grasp now that we have an API key. The sky is your limit, or possibly the literal rate limits.

I hope this helps get you started integrating YouTube into your app.

Stay regular,
Oscar

Keep Reading:

Oscar

Oscar is an artist and engineer who's been creating cannabis brands and media experiences for over 10 years, and developing full-stack applications for over 15 years.