Getting Started with Google BigQuery

Introduction

Google BigQuery is an enterprise data warehouse that enables users to run SQL-like queries against extremely large-scale data in a matter of seconds. BigQuery supports both business and spatial data.

FME can help get business and spatial data into BigQuery where users can leverage analysis services on their data. Write support to Google BigQuery was added in FME 2020, enabling users to easily migrate business and spatial data from formats that FME supports into BigQuery.

This tutorial series will provide an introduction to working with the Google BigQuery format within FME.
 

Requirements

• FME 2020.0 or newer

• Google BigQuery Account

• Granted the required permissions to create tables in BigQuery
   

Terminology

Project:

Google Cloud projects are the basis for creating, enabling, and using all Google Cloud services. Users will need to have access to a Google Cloud Project and be granted any necessary permissions in order to connect to BigQuery.

Dataset:

BigQuery datasets are top-level containers for tables and views within a specific project. They are used to organize and control access to tables and views. FME is currently not able to create new datasets so tables must be written to existing datasets.

Note that the BigQuery concept of a dataset is slightly different than FME's concept of a dataset. BigQuery's definition of a dataset is similar to a database schema in a relational database or a Feature Dataset in the File Geodatabase format.

Table:

BigQuery tables contain individual records. Each table is defined by a schema that describes column names, data types, and other information. A BigQuery table is analogous to a feature type in FME.

Articles

1. Loading a Single File to BigQuery

2. Loading Multiple Files Dynamically into BigQuery

3. Cleaning and Preparing Data for BigQuery

4. Automating Data Upload to Google BigQuery from Google Cloud Storage