How to register Flink table schema with nested fields?

Question

I'm working on registering a DataStream as a Flink Table in StreamingTableEnvironment.

The syntax I'm using is like so:

StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);

tableEnv.createTemporaryView(
       "foo",
       dataStream,
       $("f0").as("foo-id"),
       $("f1").as("foo-value") 
)

In this case "foo-id" and "foo-value" are primitive types. In the Datastream I also have Json objects that I would like to ideally register as a Row Type similar to the suggestion here for Flink CREATE TABLE command:

CREATE TABLE input(
             id VARCHAR,
             title VARCHAR,
             properties ROW(`foo` VARCHAR)
) WITH (
            'connector' = 'kafka-0.11',
            'topic' = 'my-topic',
            'properties.bootstrap.servers' = 'localhost:9092',
            'properties.group.id' = 'python-test',
            'format' = 'json'
);

See: Get nested fields from Kafka message using Apache Flink SQL

Is there a similar way to define nested types using Expressions?

I'm not registering streams using the Create Table command with connector because the data format is custom, which is why I resorted to registering a stream.

score 1 · Answer 1 · answered Feb 25 '21 at 00:40

I played around with return types and nesting some more and this solution seems to work:

DataStream<Row> testDataStream = env.fromCollection(Arrays.asList(
     Row.of("sherin", 1L, Row.of(100L, 200L)),
     Row.of("thomas", 1L, Row.of(100L, 200L))
)).returns(
    Types.ROW(
             Types.STRING,
             Types.LONG,
             Types.ROW_NAMED(
                  new String[]{"val1", "val2"},
                  Types.LONG,
                  Types.LONG)
));

tableEnv.createTemporaryView(
    "foo",
    testDataStream,
    $("f0").as("name"),
    $("f1").as("c"),
    $("f2").as("age"));

Table testTable = tableEnv.sqlQuery(
     "select name, c, age.val1, age.val2 from foo"
);

DataStream<Row> result = tableEnv.toAppendStream(
    testTable,
    TypeInformation.of(Row.class)
);

result.print().setParallelism(2);

I'm still open to ideas, if there are better ways of doing this.

How to register Flink table schema with nested fields?

1 Answers1