+ Our goal is to extract a list of Account Summaries from Google Analytics API.
+
+ The very first step is to get familiar with the data structure of the dataset we are going to extract.
+ So let's take a look at Account Summaries API documentation.
+
+ {% include template_folder ~ '/accountSummary.json' %}
+
+ {% endapply %}
+
+ {% include template_folder ~ '/propertySummary.json' %}
+
+ {% endapply %}
+
+
+ Ok, not too bad, we have one list of structures (Property Summary is a structure) and few other not nullable fields.
+ This should give us the following schema:
+
+ {% apply escape %}{% include template_folder ~ '/schema.txt' %}{% endapply %}
+
+ {% endapply %}
+
+
+ Now that we have our schema, we can start building our custom data extractor.
+ Our next step is to figure out if there is any existing SDK that we can use to interact with Google Analytics API.
+ In this case, we are going to use Google Analytics Admin SDK.
+
+ Sometimes we might not be able to find any SDK that fits our needs, in such cases we would have to build our own HTTP client and handle the API requests manually. +
+ +
+ Let's start by preparing our Extractor class. We will call it AccountSummariesExtractor.
+ First we need to make sure that our extractor is implementing the Flow\ETL\Extractor interface.
+ Technically speaking, it's all we need but to make our extractor to be compatible with Flow DataFrame::limit() function,
+ we should additionally implement Flow\ETL\Extractor\LimitableExtractor interface.
+
+ Most of the Flow\ETL\Extractor\LimitableExtractor logic is reusable so to avoid code duplications we are going to use Flow\ETL\Extractor\Limitable trait in our extractor.
+
+ {% apply escape %}{% include template_folder ~ '/extractor-01.php' %}{% endapply %}
+
+ {% endapply %}
+
+
+ Our extractor boilerplate is ready, let's try to implement the extract() method logic.
+ FlowContext $context is a container for all parameters/services that are shared between all the stages of the ETL process.
+ We can use it to access things like EntryFactory or Config however we are not going to use it in this example.
+
+ Let's start by fetching the list of Account Summaries from Google Analytics API, iterating over the list and yield each Account Summary. +
+ + {% apply spaceless %} +
+ {% apply escape %}{% include template_folder ~ '/extractor-02.php' %}{% endapply %}
+
+ {% endapply %}
+
+
+ ga_account_summary_to_row function is responsible for transforming the Account Summary structure into a row that fits our schema.
+ It could be a private method of our Extractor, however, extracting it to separated function will make our code cleaner and easier to test/use later.
+
+ {% apply escape %}{% include template_folder ~ '/ga_account_summary_to_row.php' %}{% endapply %}
+
+ {% endapply %}
+
+
+ Our final step would be to implement pagination logic, since Google Analytics API returns only up to 200 Account Summaries per page.
+ Typically, we are not going to have more than 200 Account Summaries, but it's always good to be prepared for such cases.
+
+ {% apply escape %}{% include template_folder ~ '/extractor-03.php' %}{% endapply %}
+
+ {% endapply %}
+
+
+ That's it! Our custom data extractor is ready to be used.
+ We can now use it in our ETL process to extract Account Summaries from Google Analytics API.
+
+ {% apply escape %}{% include template_folder ~ '/usage.php' %}{% endapply %}
+
+ {% endapply %}
+
+
+ We are almost done, at this point we can fetch all Account Summaries from Google Analytics API and transform them into a DataFrame.
+ Our final but also optional step would be to prepare a dsl function returning our extractor that is going to improve the readability of our ETL data processing pipeline.
+
+ {% apply escape %}{% include template_folder ~ '/usage-dsl.php' %}{% endapply %}
+
+ {% endapply %}
+