Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

builtin function @reify to create a type from a TypeInfo instance #383

Closed
8 of 9 tasks
andrewrk opened this issue May 30, 2017 · 46 comments
Closed
8 of 9 tasks

builtin function @reify to create a type from a TypeInfo instance #383

andrewrk opened this issue May 30, 2017 · 46 comments
Labels
proposal This issue suggests modifications. If it also has the "accepted" label then it is planned.
Milestone

Comments

@andrewrk
Copy link
Member

andrewrk commented May 30, 2017

  • for arrays, pointers, error unions, nullables: T.child_type field.
  • for functions, T.return_type field
  • for functions, T.is_var_args and
  • T.arg_types which is of type [N]type
  • @fieldsOf(T) where T is a struct or enum. Returns anonymous struct { T: type, name: []const u8 }
  • accessing a field by comptime string name
  • implement @scurest proposal below
    • @typeInfo
    • @reify

cc @raulgrell

@andrewrk andrewrk added the enhancement Solving this issue will likely involve adding new logic or components to the codebase. label May 30, 2017
@andrewrk andrewrk added this to the 0.2.0 milestone May 30, 2017
@ranma42
Copy link

ranma42 commented May 30, 2017

For functions, would it make sense to also expose the number of arguments and make them individually accessible? (cc @AndreaOrru )

@AndreaOrru
Copy link
Contributor

Yes, that would be super useful to implement the IPC in my kernel, for example.

@andrewrk
Copy link
Member Author

Added a third item for this above

@andrewrk
Copy link
Member Author

cc @hasenj

<hasenj> for comptime is there a way to perform property access by string or something similar?
<hasenj> for example, say, @field(object, field_name)

@hasenj
Copy link

hasenj commented Sep 21, 2017

@fieldsOf(T) where T is a struct or enum. Returns anonymous struct { T: type, name: []const u8 }

Is the type even needed? I think a list of names as strings should suffice, given that there are other builtin functions that can be used to reflect on the type. For example:

@field(some_struct, name); // expands to some_struct.<value of name>
// e.g.
@field(a, "foo"); // expands to `a.foo`

@typeOf(@field(a, "foo")); // gives type of a.foo

@tiehuis
Copy link
Member

tiehuis commented Sep 22, 2017

It seems like with just these features we could implement a structural based interface mechanism, similar to go.

Here is a motivating example to implement a print trait of sorts, allowing any struct which implements a print field with the appropriate type to have it called. I don't think anything is too glaringly out of place here, but feel free to correct.

const std = @import("std");
const builtin = @import("builtin");
const TypeId = builtin.TypeId;

fn getField(comptime T: type, x: var, comptime name: []const u8) ->
    (fn (self: T) -> %void)
{
    for (@fieldsOf(x)) |field_name| {
        if (field_name == name) {
            const field = @field(x, name);
            const field_type = @typeOf(field);

            if (@typeId(field_type) != TypeId.Fn) {
                @panic("field is not a function");
            }

            if (field_type.is_var_args) {
                @panic("cannot handle varargs function");
            }

            // Would need to be a bit more in-depth to handle a &T self arg
            const expected_args = []type { T };
            if (!std.mem.eql(type, field_type.arg_types, expected_arg_types)) {
                @panic("prototype does not match");
            }

            if (@typeId(field_type.return_type) != TypeId.Error
                and field_type.return_type.child_type != void)
            {
                @panic("return type does not match");
            }

            return field;
        }
    }

    null
}

// Expects a struct with a field method of type `print(self) -> %void`.
pub fn printTrait(x: var) -> %void {
    const T = @typeOf(x);
    const Id = @typeId(T);

    if (Id != TypeId.Struct) {
        @panic("expected a struct");
    }

    if (getField(T, x, "print")) |func| {
        func(x);
    } else {
        @panic("no print field found!");
    }
}

@andrewrk
Copy link
Member Author

With this example, if you were going to do printTrait(x) couldn't you instead do x.print() ?

@tiehuis
Copy link
Member

tiehuis commented Sep 28, 2017

You're right. This example would really only provide slightly more targeted error messages.

A better example would be a printDebug function which could recursively print fields of structs and enums, similar to println!("{:?}", x) in Rust. Another example as well that could be very useful would be for generic serialization code by inspection of field names.

@scurest
Copy link
Contributor

scurest commented Nov 5, 2017

Since zig can store types as regular data, I was wondering if it would be better to, rather than have magic fields that make types seem like structs, expose the reflected data in actual structs. Example:

const NullableType = struct {
    child: type,
};

@reflect(?u32) ==> NullableType { .child = u32 }

const ArrayType = struct {
    child: type,
    len: u64,
};

@reflect([4]u32) ==> ArrayType { .child = u32, .len = 4 }

// Possible example for a struct

const StructType = struct {
    field_names: [][]const u8,
    field_types: []type,
    field_offsets: []u64,
}

const S = struct { x: i32, y: u8 };

@reflect(S) ==> StructType {
    .field_names = [][]const u8 { "x", "y" },
    .field_types = []type { i32, u8 },
    .field_offsets = []u64 { 0, 4 },
}

etc. You get the idea. You could also add a souped-up version of @typeId that returns an enum with variants like Nullable: NullableType, etc.

Some pros: possibly easier documentation (you can now lookup what fields you can access just like for regular structs), fewer special cases in the field-lookup code.

There could also be a @deReflect (deflect? unreflect?) that turns one of these reflected structs into a type, so you could generate eg. a struct programmatically at compile time.

Of course, @reflect could also be implemented as a regular function in userland (I hit #586 when trying it) as long as some reflection mechanism exists.

@andrewrk
Copy link
Member Author

andrewrk commented Nov 6, 2017

I like this idea. I ran into an issue which is related which I will type up and link here.

@andrewrk
Copy link
Member Author

andrewrk commented Nov 6, 2017

I think #588 has to be solved before the idea @scurest outlined here can be implemented.

@andrewrk
Copy link
Member Author

andrewrk commented Nov 6, 2017

We could potentially even remove these functions:

  • @sizeOf
  • @alignOf
  • @memberCount
  • @minValue
  • @maxValue
  • @offsetOf
  • @typeId
  • @typeName
  • @IntType

Instead these could all be fields or member functions of the struct returned from @reflect, or replaced with @MakeType (un-reflect, deflect, whatever it's called).

@Ilariel
Copy link

Ilariel commented Nov 6, 2017

I like the idea of @reflect since it exists in another namespace as a builtin and can exist as part of the language instead of being a standard library hack.

The @MakeType name could be @reify given the meaning of the word: "to regard something abstract as if it were a concrete material thing"
However whether it is fitting to use it given the CS meaning related meaning is another question
https://en.wikipedia.org/wiki/Reification_(computer_science)

With the "maketype" functionality we could write type safe compile time type generators that can give proper compiler errors and flexibly generate types. I think we need to be able to name the types too if we want to export them to C as a visible structs. On Zig side we can just use aliases to address the generated types if we want to.

@andrewrk andrewrk added accepted This proposal is planned. proposal This issue suggests modifications. If it also has the "accepted" label then it is planned. labels Nov 7, 2017
andrewrk added a commit that referenced this issue Nov 7, 2017
see #383

there is a plan to unify most of the reflection into 2
builtin functions, as outlined in the above issue,
but this gives us needed features for now, and we can
iterate on the design in future commits
@andrewrk andrewrk modified the milestones: 0.2.0, 0.3.0 Nov 7, 2017
@andrewrk andrewrk modified the milestones: 0.3.0, 0.4.0 Feb 28, 2018
@tgschultz
Copy link
Contributor

tgschultz commented Mar 8, 2018

I had this same idea while looking over some older reflection code I'd written and threw together an example of what I imagine a TypeInfo struct (returned by @reify) might look like. This is with no knowledge of compiler internals, it's just spitballing.

This example uses pointer reform syntax as described by #770

pub const TypeId = enum {
    Void,
    Type,
    NoReturn,
    Pointer,
    Bool,
    Integer,
    Float,
    Array,
    Slice,
    Struct,
    Union,
    Enum,
    ErrorSet,
    Promise,
    Function,
    Literal,
    Namespace,
    Block,
};

pub const LiteralTypeId = enum {
    Null,
    Undefined,
    Integer,
    Float,
    String,
    CString,
};

pub const PointerTypeId = enum {
    Single,
    Block,
    NullTerminatedBlock,
}

pub const TypeInfo = struct {
    isNullable: bool
    isErrorable: bool
    Id: TypeId,
    Type: type,
    Name: []const u8,
    Size: usize,
    Alignment: u29,
    Detail: TypeInfoDetail,
};

pub const TypeInfoDetail = union(TypeId) {
    Void: void,
    Type: void,
    NoReturn: void,
    Pointer: PointerTypeInfoDetail,
    Bool: void,
    Integer: IntegerTypeInfoDetail,
    Float: FloatTypeInfoDetail,
    Array: ArrayTypeInfoDetail,
    Slice: ArrayTypeInfoDetail,
    Struct: StructTypeInfoDetail,
    Union: UnionTypeInfoDetail,
    Enum: EnumTypeInfoDetail,
    ErrorSet: EnumTypeInfoDetail,
    Promise: PromiseTypeInfoDetail,
    Function: FunctionTypeInfoDetail,
    Literal: LiteralTypeId,
    Namespace: void,
    Block: void
    Opaque: void,
};

pub const PointerTypeInfoDetail = struct {
    Id: PointerTypeId,
    Child: *TypeInfo,
};

pub const IntegerTypeInfoDetail = struct {
    isSigned: bool,
    bits: u8,
    maxValue: usize,
    minValue: usize,
};

pub const ErrorTypeInfoDetail = struct {
    ParentSet: *TypeInfo,
    value: usize;
};

pub const FloatTypeInfoDetail = struct {
    bits: u8,
    maxValue: f64,
    minValue: f64,
    epsilon: f64,
    //potentially maxExp, hasSubnorm, etc?
};

pub const ArrayTypeInfoDetail = struct {
    isNullTerminated: bool,
    Child: *TypeInfo,
    length: usize,
};

pub const StructTypeInfoDetail = struct {
    isPacked: bool,
    memberNames: [][]const u8,
    memberOffsets: []const usize,
    Members: []const *TypeInfo,
};

pub const UnionTypeInfoDetail = struct {
    Tag: *TypeInfo,
    memberNames: [][]const u8,
    Members: []const *TypeInfo,
};

pub const EnumTypeInfoDetail = struct {
    isErrorSet: bool,
    Tag: *TypeInfo,
    memberNames: [][]const u8,
    MemberValues: []const usize,
};

pub const PromiseTypeInfoDetail = struct {
    //???
};

pub const FunctionTypeInfoDetail = struct {
    Return: *TypeInfo,
    Args: []const *TypeInfo,
};

@shawnl
Copy link
Contributor

shawnl commented May 18, 2019

For functions, would it make sense to also expose the number of arguments and make them individually accessible? (cc @AndreaOrru )

I think the solution to all of these type of things would be to expose the LLVM bindings, and allow them to be used at comptime on Zig code.

@eira-fransham
Copy link

eira-fransham commented Jun 7, 2019

I don't really like having syntax to emit fields from an inline for. I'd prefer to just be able to manipulate the typeinfo, either by having the returned TypeInfo be mutable or by having a @reify builtin. Here's what the former would look like.

var outType = struct {
  // ... everything that doesn't need to be generated dynamically ...
};

const inFields = &@typeInfo(inputType).Struct.fields;
const originalFields = &@typeInfo(outType).Struct.fields;
var outFields: [originalFields.len + inFields.len]StructField = undefined;

for (originalFields) |field, i| {
  outFields[i] = field;
}

for (inFields) |field, i| {
  outFields[i + originalFields.len] = do_something_with(field);
}

@typeInfo(mytype).Struct.fields = &outFields;

@Sahnvour
Copy link
Contributor

Sahnvour commented Jun 7, 2019

I'm not sure types should be mutable, in your example (I assume the mytype at the end is in fact outType) what would happen if outType is used before and after its fields get reassigned ?

Generating fields, definitions and such inline has the advantage that they happen when the type is created and you can't end up with a mismatch that would necessitate more language rules. @reify is also immune to this problem.

Also, I don't really see what it adds that wouldn't be otherwise possible; yet it's introducing a new way of defining types on top of the already existing one. (same goes for @reify)
I think it's tempting to use the fact that we already have compile-time introspection and code working with @typeInfo(foo).Struct.fields and the like, because it really is handy for consuming this data. However for producing types it is just a more verbose way of using the natural language constructs. Consider a function that has many attributes for example, I'd rather read its definition expressed purely in Zig than multiple lines modifying a comptime object, because it's clearer and closer to the "normal" language.

@marler8997
Copy link
Contributor

Not knowing about this proposal, I just created a duplicate proposal for the @reify function here: #2906

Only difference being that I called it @infoType instead of @reify. The thought being that it's the inverse of @typeInfo.

I created it because I found a use case for it which I describe here if you want to take a look: #2782 (comment)

@andrewrk
Copy link
Member Author

andrewrk commented Jul 16, 2019

I created it because I found a use case for it which I describe here if you want to take a look: #2782 (comment)

I think it's a good use case, and I think it asks the question, "Even if we don't go full bananas on @reify, what if it works for simple types such as integers, slices, and pointers, so we can get rid of @IntType, and so that @marler8997's use case is solved better?"

I'm going to make a separate issue for that.

@andrewrk
Copy link
Member Author

This issue is now blocking on #2907.

@Sahnvour
Copy link
Contributor

Another random thought on what @reify could achieve: help exchange zig types (think optionals, slices, unions...) accross ABI boundaries. We could create helpers to convert from/to C types representing them automatically and their instances accordingly, so that in the end we can make dynamic zig libraries easier to use. I understand and agree that static linking is to be preferred, but the usecase exists. Makes sense?

@cshenton
Copy link
Contributor

Related to AoS to SoA transforms is creating an archetype based ECS. For example, given a user defined archetype with some component fields, I'd like to be able to generate an appropriate concrete "storage" type (here storage is shown by simple slices).

const MyArchetype = struct {
    location: Vec3,
    velocity: Vec3,
};

// I then write some function Storage() that uses the @reify feature
// const MyArchetypeStorage = Storage(MyArchetype);

// I want to generate something like this:
const MyArchetypeStorage  = struct {
    location: []Vec3,
    velocity: []Vec3,
}

Modern game engines tend to jump through lots of hoops (and runtime costs) to pay for this abstraction, because it enables things like batched compile-time dispatch (if I have a bunch of storages I can run some function on each one that contains a particular set of batched field).

Would be great to be able to generate things like this.

@Sobeston
Copy link
Sponsor Contributor

I recently came across a problem which @reify would be very useful for. There's a very long C struct I'm working with where large blocks of fields are selectively present for different platforms using macros.

There's no neat way of currently solving this in zig without resorting to code generation, or writing massive amounts of code.

With @reify I could split these blocks into their own structs, and neatly "join" them based on comptime values to form the struct I'm looking for.

This would greatly simplify and reduce the footprint of the code, which would be great for my sanity and for spotting bugs.

@az5112
Copy link

az5112 commented Jun 18, 2020

I would recommend having a look at Circle (https://www.circle-lang.org/). It's a compile-time machinery on top of C++. It's sometimes complex (not saying it in a bad way - C++ itself is very complex) but it does pass the printf test. (It's possible to create printf without compiler hooks.) And it does much more than that.

@bb010g
Copy link

bb010g commented Jul 7, 2020

I'm running into a desire for full-strength @reify from writing argument parsing machinery; it'd be nice to generate structs & tagged unions which represent the byte slices parsed. https://github.com/MasterQ32/zig-args manages to avoid this currently by restricting its API to a struct with duck-typed, specially-named metadata fields, but this restricts access to other aspects of Zig's rich type system in the API. You can't use a struct to store a few different fields about an option, or a tagged union to clearly show how choices are structured. Trying to avoid unintuitiveness with @reify leads to complexity getting shoved into (comptime) dynamically typed trees that are weakly typed by default.

@ikskuh
Copy link
Contributor

ikskuh commented Aug 3, 2020

After some discussion with @alexnask in IRC we came to the conclusion that allowing @Type(.Struct) would be more sane than all the hacks and workarounds based on @TypeOf(.{ … }) and similar techniques.

People are already working around the restrictions that you cannot create struct at comptime creating APIs that are convenient to use, but really hard to understand. Allowing @Type(.Struct) would make the code more straightforward, readable and maintainable.

The question that was not clear is whether to allow creating structs with .decls set to a non-empty set or not. Allowing it would probably open up possibilities for really convenient interface APIs similar to interface.zig, but making code harder to understand, but still easier than the current implementations

@alexnask
Copy link
Contributor

alexnask commented Aug 3, 2020

For reference, the following is achievable today:

const std = @import("std");

// Create a tuple with one runtime-typed field
fn UniTuple(comptime T: type) type {
    const Hack = struct {
        var forced_runtime: T = undefined;
    };
    return @TypeOf(.{Hack.forced_runtime});
}

/// Types should be an iterable of types
fn Tuple(comptime types: anytype) type {
    const H = struct {
        value: anytype,
    };
    var empty_tuple = .{};

    var container = H{
        .value = empty_tuple,
    };

    for (types) |T| {
        container.value = container.value ++ UniTuple(T){ .@"0" = undefined };
    }
    return @TypeOf(container.value);
}

pub fn StructType2(comptime names: []const []const u8, comptime types: anytype) type {
    std.debug.assert(names.len == types.len);
    const Storage = Tuple(types);

    return struct {
        const field_names = names;
        const field_types = types;
        const Self = @This();

        storage: Storage,

        pub fn create(literal: anytype) Self {
            comptime std.debug.assert(std.meta.fields(@TypeOf(literal)).len == field_names.len);
            comptime std.debug.assert(std.meta.trait.hasFields(@TypeOf(literal), field_names));

            var self: Self = undefined;
            inline for (field_names) |name, idx| {
                self.storage[idx] = @field(literal, name);
            }
            return self;
        }

        fn fieldIndex(comptime name: []const u8) ?comptime_int {
            var i = 0;
            while (i < field_names.len) : (i += 1) {
                if (std.mem.eql(u8, name, field_names[i])) return i;
            }
            return null;
        }

        fn FieldType(comptime name: []const u8) type {
            const idx = fieldIndex(name) orelse @compileError("Field '" ++ name ++ "' not in struct '" ++ @typeName(Self) ++ "'.");
            return field_types[idx];
        }

        pub fn field(self: anytype, comptime name: []const u8) if (@TypeOf(self) == *Self) *FieldType(name) else *const FieldType(name) {
            const idx = fieldIndex(name).?;
            return &self.storage[idx];
        }

        pub fn format(
            self: Self,
            comptime fmt: []const u8,
            options: std.fmt.FormatOptions,
            writer: anytype,
        ) !void {
            try writer.writeAll(@typeName(Self));
            if (std.fmt.default_max_depth == 0) {
                return writer.writeAll("{ ... }");
            }
            try writer.writeAll("{");
            inline for (field_names) |f, i| {
                if (i == 0) {
                    try writer.writeAll(" .");
                } else {
                    try writer.writeAll(", .");
                }
                try writer.writeAll(f);
                try writer.writeAll(" = ");
                try std.fmt.formatType(self.storage[i], fmt, options, writer, std.fmt.default_max_depth - 1);
            }
            try writer.writeAll(" }");
        }
    };
}

pub fn StructType(comptime fields: anytype) type {
    var field_names: [fields.len][]const u8 = undefined;
    var field_types: [fields.len]type = undefined;

    for (fields) |f, idx| {
        field_names[idx] = f[0];
        field_types[idx] = f[1];
    }

    return StructType2(&field_names, &field_types);
}

pub fn main() void {
    const FooStruct = StructType(.{
        .{ "a", usize },
        .{ "b", bool },
    });

    var foo = FooStruct.create(.{ .a = 0, .b = false });
    foo.field("a").* = 1;
    std.debug.print("{}\n", .{foo});
}

This is a simple implementation of a @Type(.Struct)-type construction (which could be improved with default field values and other things).
In my opinion it is clear that @Type(.Struct) would be better solution to this (including .decls although I could see how this is somewhat controversial)

@andrewrk
Copy link
Member Author

andrewrk commented Aug 7, 2020

After some discussion with @alexnask in IRC we came to the conclusion that allowing @Type(.Struct) would be more sane than all the hacks and workarounds based on @TypeOf(.{ … }) and similar techniques.

I think this is a strong argument. I'd like to "accept" @Type supporting struct, enum, etc, but keep the decls as an open proposal, with no conclusion yet. Hopefully that feels like progress.

@ghost
Copy link

ghost commented Aug 22, 2020

Note on decls: perhaps we could sidestep the issue by requiring method definitions, and in fact all decl values, be written by value? That is, (in the case of methods) use a function which has already been defined, rather than constructing it from scratch. #1717 will make this more symmetric for methods vs variables/constants. (No examples as I'm on mobile -- hopefully I'm being clear enough.)

@tadeokondrak
Copy link
Contributor

This is now implemented for every type, and can generate everything that syntax can except

I think this issue should be renamed to be about allowing declarations, or closed and replaced with a new one for that.

@andrewrk
Copy link
Member Author

I think this issue should be renamed to be about allowing declarations, or closed and replaced with a new one for that.

sounds good 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
proposal This issue suggests modifications. If it also has the "accepted" label then it is planned.
Projects
None yet
Development

No branches or pull requests