Parsing

[Bike]

I'm more convinced that parsing and type operations can and should be separated. Parsing involves the environment and some details about how the implementation deftype works.

Furthermore, for a more flexible system with possible extended specifiers, we'd want to store information about specifiers in the environment, where it conceptually belongs, rather than how we do it now with methods on symbol-specifier-ctype and such.

So here's what I'm thinking. Conceptually, type specification works a bit like form evaluation. You have macros from deftype. You have "functions" like cons, that don't need the environment because they can accept "arguments" of already specified types. And you have "special operators" like array that "evaluate" some subforms and don't "evaluate" others.

The environment can include bindings of type specifiers as either special operators, macros, or functions. All of these can just be ordinary functions. The functions for macros are just type expanders, so probably a function of a specifier and an environment that returns an expanded specifier. The functions for functions have the lambda list of the type specifier, so for example the function for cons would take two arguments, both already-parsed specifiers. And the functions for special operators take a specifier and an environment and return the parsed specifier. This way the system could be actually totally separate from ctype as the functions could return whatever other kind of parsed specifier.

Parsing should also do caching. Caching is somewhat nontrivial since caches may have to be partially invalidated when new classes or types are defined, and because they should be thread safe.

[charJe]

I understand your analogy of type definition to functions macros and special forms. It makes sense, but it also seems over complicated to naive me.

I don't really understand the need for environment since types are usually defined globally through deftype or defclass.

The environment can include bindings of type specifiers as either special operators, macros, or functions

It could, but shouldn't it be used for storing actual macros and functions. It is valid to have a deftype and a function with the same name since they live in different namespaces. Would this approach be polluting the function namespace with type expanders?

This way the system could be actually totally separate from ctype as the functions could return whatever other kind of parsed specifier.

It makes sense and is very useful to me that specifier-ctype returns a ctype. On top of that, I particularly like the ctype internal representation of different kinds of types as classes. Wouldn't the separated parser need to use some other intermediate representation; why not just use ctype?

Lastly, would this different parsing system change the API of define-extended-type? At the end of the day, we still need to do a deftype for primitive type parsers, and a more advanced expander that returns a custom ctype for extended-specifier-ctype.

[Bike]

Sorry I'm only replying to this now - I have been too busy with cvm/bytecode to think about this library.

I understand your analogy of type definition to functions macros and special forms. It makes sense, but it also seems over complicated to naive me.

Part of the idea is that you could have type functions that can be defined more simply since they don't have to worry about parsing.

I don't really understand the need for environment since types are usually defined globally through deftype or defclass.

Two reasons. First, lexical types and classes might be an interesting extension. Second, and more importantly, there might be multiple distinct global environments for e.g. cross compilation purposes - this is what Clostrum is about.

It could, but shouldn't it be used for storing actual macros and functions. It is valid to have a deftype and a function with the same name since they live in different namespaces. Would this approach be polluting the function namespace with type expanders?

I meant "functions" as in "type functions" like I described earlier, not anything to do with the function namespace.

It makes sense and is very useful to me that specifier-ctype returns a ctype. On top of that, I particularly like the ctype internal representation of different kinds of types as classes. Wouldn't the separated parser need to use some other intermediate representation; why not just use ctype?

The idea is the parser could return some non-ctype representation that exists, like SBCL's. But it could just as well return ctypes if that's what you want (and that would probably be the default or something, since I'm writing both systems).

Lastly, would this different parsing system change the API of define-extended-type?

define-extended-type is pretty much entirely about parsing, so I don't think so. It would be part of the parser system.