Add HTTP data

[ewdurbin]

I suggest that we add a new "http" column to linehaul which includes:

• HTTP Method (GET/OPTIONS)
• HTTP Status Code (200/206)
• HTTP Bytes served
• HTTP Range Header value (if applicable)

This would help to analyze usage more precisely and surface overall usage data better via BigQuery

[miketheman]

See #232 for extra context

[miketheman]

cc @di for assistance on the BigQuery side

[di]

Can someone give me the exact fields (name and types) we want to add here? And which tables (file_downloads and simple_requests?). Preferably a diff of the following:

The existing file_downloads schema:

[
  {
    "mode": "REQUIRED",
    "name": "timestamp",
    "type": "TIMESTAMP"
  },
  {
    "name": "country_code",
    "type": "STRING"
  },
  {
    "mode": "REQUIRED",
    "name": "url",
    "type": "STRING"
  },
  {
    "mode": "REQUIRED",
    "name": "project",
    "type": "STRING"
  },
  {
    "fields": [
      {
        "name": "filename",
        "type": "STRING"
      },
      {
        "name": "project",
        "type": "STRING"
      },
      {
        "name": "version",
        "type": "STRING"
      },
      {
        "name": "type",
        "type": "STRING"
      }
    ],
    "mode": "REQUIRED",
    "name": "file",
    "type": "RECORD"
  },
  {
    "fields": [
      {
        "fields": [
          {
            "name": "name",
            "type": "STRING"
          },
          {
            "name": "subcommand",
            "type": "STRING"
          },
          {
            "name": "version",
            "type": "STRING"
          }
        ],
        "name": "installer",
        "type": "RECORD"
      },
      {
        "name": "python",
        "type": "STRING"
      },
      {
        "fields": [
          {
            "name": "name",
            "type": "STRING"
          },
          {
            "name": "version",
            "type": "STRING"
          }
        ],
        "name": "implementation",
        "type": "RECORD"
      },
      {
        "fields": [
          {
            "name": "name",
            "type": "STRING"
          },
          {
            "name": "version",
            "type": "STRING"
          },
          {
            "name": "id",
            "type": "STRING"
          },
          {
            "fields": [
              {
                "name": "lib",
                "type": "STRING"
              },
              {
                "name": "version",
                "type": "STRING"
              }
            ],
            "name": "libc",
            "type": "RECORD"
          }
        ],
        "name": "distro",
        "type": "RECORD"
      },
      {
        "fields": [
          {
            "name": "name",
            "type": "STRING"
          },
          {
            "name": "release",
            "type": "STRING"
          }
        ],
        "name": "system",
        "type": "RECORD"
      },
      {
        "name": "cpu",
        "type": "STRING"
      },
      {
        "name": "openssl_version",
        "type": "STRING"
      },
      {
        "name": "setuptools_version",
        "type": "STRING"
      },
      {
        "name": "rustc_version",
        "type": "STRING"
      },
      {
        "name": "ci",
        "type": "BOOLEAN"
      }
    ],
    "name": "details",
    "type": "RECORD"
  },
  {
    "name": "tls_protocol",
    "type": "STRING"
  },
  {
    "name": "tls_cipher",
    "type": "STRING"
  }
]

And the simple_requests schema:

[
  {
    "mode": "REQUIRED",
    "name": "timestamp",
    "type": "TIMESTAMP"
  },
  {
    "name": "country_code",
    "type": "STRING"
  },
  {
    "mode": "REQUIRED",
    "name": "url",
    "type": "STRING"
  },
  {
    "mode": "REQUIRED",
    "name": "project",
    "type": "STRING"
  },
  {
    "fields": [
      {
        "fields": [
          {
            "name": "name",
            "type": "STRING"
          },
          {
            "name": "subcommand",
            "type": "STRING"
          },
          {
            "name": "version",
            "type": "STRING"
          }
        ],
        "name": "installer",
        "type": "RECORD"
      },
      {
        "name": "python",
        "type": "STRING"
      },
      {
        "fields": [
          {
            "name": "name",
            "type": "STRING"
          },
          {
            "name": "version",
            "type": "STRING"
          }
        ],
        "name": "implementation",
        "type": "RECORD"
      },
      {
        "fields": [
          {
            "name": "name",
            "type": "STRING"
          },
          {
            "name": "version",
            "type": "STRING"
          },
          {
            "name": "id",
            "type": "STRING"
          },
          {
            "fields": [
              {
                "name": "lib",
                "type": "STRING"
              },
              {
                "name": "version",
                "type": "STRING"
              }
            ],
            "name": "libc",
            "type": "RECORD"
          }
        ],
        "name": "distro",
        "type": "RECORD"
      },
      {
        "fields": [
          {
            "name": "name",
            "type": "STRING"
          },
          {
            "name": "release",
            "type": "STRING"
          }
        ],
        "name": "system",
        "type": "RECORD"
      },
      {
        "name": "cpu",
        "type": "STRING"
      },
      {
        "name": "openssl_version",
        "type": "STRING"
      },
      {
        "name": "setuptools_version",
        "type": "STRING"
      },
      {
        "name": "rustc_version",
        "type": "STRING"
      },
      {
        "name": "ci",
        "type": "BOOLEAN"
      }
    ],
    "name": "details",
    "type": "RECORD"
  },
  {
    "name": "tls_protocol",
    "type": "STRING"
  },
  {
    "name": "tls_cipher",
    "type": "STRING"
  }
]

[miketheman]

I think this is the right patch for both:

  {
    "name": "http",
    "type": "RECORD",
    "fields": [
      {
        "name": "method",
        "type": "STRING"
      },
      {
        "name": "status_code",
        "type": "INTEGER"
      },
      {
        "name": "bytes_served",
        "type": "INTEGER"
      },
      {
        "name": "range_header",
        "type": "STRING"
      }
    ]
  }

Two patch files, appending it to the main list beneath each schema's tls_cipher.

file_downloads.patch
simple_requests.patch

[di]

These fields have been added to the schema.

[miketheman]

These fields have been added to the schema.

Thanks @di!

I think the next bit here would be to extend the parser to account for the newer fields coming in via logs, and then update the log format to emit them - I'm not certain of the order there, it depends on how the parser handles newer fields. Likely blocked on #263 in some fashion.

[Mariajose-prog]

Done

[Mariajose-prog]

Done

[Mariajose-prog]

Done

[Mariajose-prog]

Done

[Mariajose-prog]

Done

[Mariajose-prog]

Done

[Mariajose-prog]

Done

[Mariajose-prog]

Done