V3 Notes

Try conventional commits! https://conventionalcommits.org/

Features needed for 3.0

• Pagination & filtering

  • just pass settings to adapter
  • make sure we're not opening up query injection vulnerabilities when we handle this info in the adapter. or, ideally, if we define a standard format, we can do some of this prevention before the data reaches the adapter.
  • since i don't have a format for this yet, using waterline's probably makes the most sense.

• Adapters

  • mongoose adapter should really use the update method on updates, not a select and then re-save(). (See #92.) This means that all model validations will have to operate only on the field value in question, because the whole doc may not be present (http://mongoosejs.com/docs/validation.html#update-validators-and-this).
    • See https://kev.inburke.com/kevin/faster-correct-database-queries/#highlighter_302237
  • at least we should document mongo settings for more consistency (write majority, read majority or linearizable): https://aphyr.com/posts/322-call-me-maybe-mongodb-stale-reads http://www.dagolden.com/index.php/2633/no-more-dirty-reads-with-mongodb/ https://docs.mongodb.com/manual/reference/read-concern/
  • should we change the adapter interface? possibilities...
    • fortune does something interesting where includes are actually not provided to adapter.find(). That sorta makes sense (an include is just another find(), so why make adapter.find() know about includes specifically), but it seems to take away some possibility for joins. Waterline seems to let adapters opt-in to various features, so they can support joining or not. if they do, that's used; if not, multiple find()s are used. that's cool.
    • an interesting idea would be to use waterline all the way through, but then we'd lose mongoose validations. so i could add a "waterline adapter", which itself works with any waterline adapter. then my adapters are basically orms, which talk to their own adapters.

• Transforms

  • allow user to transform the query that's built from the request, not just the resources sent to/returned from the db. E.g., we can filter on some virtuals (those that are simple aliases) by transforming the query so the query contains a where that uses the field names as they're stored in the db (#103). See also #110.
    • label mappers can be subsumed by this, and we'll get a performance boost. instead of doing a separate query for the matching ids, they just add a where clause to the query. Usually, it'll be something simple like (where status=drafts or where time>=date.now()) but, in the degenerate case, it could just be a list of ids to. this is more performant, more consistent (saves a race condition between queries) and more elegant to implement.
    • Note: this could be the solution to #104 as well: at this "transform the query" stage, we modify the query to remove requests for any protected fields, and then remove any includes for those removed fields. If the fields to remove (with a predicate fn testing whether the user has access) were defined declaratively, we could do this automatically to minimize the bugs. Maybe this could become a more full-fledged permissions system?
    • Misc links re the above new approach (using Waterline as somewhat of a reference)
      • https://docs.google.com/drawings/d/1u7xb5jDY5i2oeVRP2-iOGGVsFbosqTMWh9wfmY3BTfw/edit
      • https://github.com/balderdashy/waterline/blob/master/lib/waterline/utils/query/forge-stage-two-query.js
      • https://github.com/balderdashy/waterline/blob/master/lib/waterline/utils/query/private/normalize-where-clause.js
      • https://github.com/sailshq/waterline-sql-builder
      • https://github.com/balderdashy/sails-docs/blob/master/contributing/adapter-specification.md
      • http://sailsjs.com/documentation/concepts/extending-sails/adapters/available-adapters
      • https://github.com/balderdashy/sails-disk/blob/master/index.js
      • https://github.com/balderdashy/sails-postgresql/blob/master/helpers/select.js
      • https://github.com/balderdashy/sails-mongo/
  • for transforming response content, obv. linkage needs to be transformable, but...
    • how does the notion of transforming linkage fit with the notion of transforming the request-derived query? I'd imagine that anyone transforming linkage (i.e., types and ids) might want to transform the type/id parsed from the url in the same way
  • but what about carl's use cases of using these transforms to fail requests? is this the right way to fail a delete request? what about how to fail (e.g. 403) a request to access a single resource, in the case that a request for a collection containing that resource would succeed (just with the resource omitted)?
    • transducers can probably help here (https://gist.github.com/ethanresnick/ed0a3be0d37ab6edd3efe5b8593ef301) but, conceptually, we have to figure out a little more about what these transforms are [i.e., what their result should be-- a partial response doc? a resource or error object?]. For now, I like the resource or error object id.

• Dbs, transactions, multiple queries

  • Add a waterline adapter so we can postgres? Maybe just support postgres with knex? (But populate is nice, and this library kinda assumes it'll have a model to read... even though the downside is that waterline doesn't seem to support a lot of postgres-specific goodness, and transactions are still undocumented)
  • If we have every request map to one adapter function call, the adapter function calls can enforce atomicity as well as the backing database can. In postgres, that can be a pure transaction. In mongo, we can at least do pre-checks where possible to minimize chance that one of the requests will fail? (E.g. we could address #121 by calling validate on all docs before saving any of them.)
  • Questions:
    • Is there a more robust way to do transactions in mongo? See Adam's email
      • we can apply the idea of a two-phase commit in mongo, treating each atomic document update like we would a separate database in the normal distributed, two-phase setup. this does leave open the possibility, though, of the app seeing intermediate results.
      • See https://www.npmjs.com/package/fawn
    • What if we want to do multiple adapter methods across a single transaction? Could we make something like

      const t = adapter.transaction();
      t.do(...); t.do(); 
      t.execute().then(result => {}, err => {});
      

      • The idea behind this is that some requests might need multiple queries. See #110.
      • This is not that hard, and it's supported in Waterline: https://github.com/balderdashy/waterline/issues/62
        • Note that waterline doesn't support transactions across different adapters yet, I don't think, and that's hard to do conceptually--though there are lots of caveats in practice. All the adapters would need to support transactions individually for their portion of the larger transaction, and then you could use something like the two phase commit protocol (basically, asking all adapters whether their individual transaction worked; applying all transactions if so, and rolling them all back otherwise).
      • Fortune does something similar: http://fortune.js.org/api/#adapter-begintransaction
      • Maybe hink of it as Request -> Query[] (with transformed payload, query) -> Results -> Transformed Results -> Response

• Nested includes

• Permissions

  • Possibly useful reference: https://docs.scaphold.io/authentication/permissions/

• Add features that JSON:API has been slow to add, but that have just proven essential...

  • [create with related resource] https://github.com/json-api/json-api/issues/795

    • solution for now: go with the approach at https://github.com/json-api/json-api/issues/795#issuecomment-312362533

  • [multiple types] https://github.com/json-api/json-api/issues/862

    • solution for now: meta.labels array in each resource. extra fields present when that contains a given "subtype". no support for setting it, atm. Use base type in type key. Optionally create collections directly for these other labels, or a filter param on the base url.

  • [version field for optimistic locking] https://github.com/json-api/json-api/issues/600

  • [linking in complex attributes] https://github.com/json-api/json-api/issues/383

    • solution for now: meta isn't reserved in complex attributes so this is tricky. i'd say just use normalized data and we'll figure it out later.

  • Others? Simpler id scheme? Make a serializer that doesn't use json-api at all, to implement something like Goal app or Micro API? linking in complex attributes?