spark sql中支持sechema合并的操作。

直接上官方的代码吧。

val sqlContext = new org.apache.spark.sql.SQLContext(sc)// sqlContext from the previous example is used in this example.// This is used to implicitly convert an RDD to a DataFrame.import sqlContext.implicits._// Create a simple DataFrame, stored into a partition directoryval df1 = sparkContext.makeRDD(1 to 5).map(i => (i, i * 2)).toDF("single", "double")df1.saveAsParquetFile("data/test_table/key=1")// Create another DataFrame in a new partition directory,// adding a new column and dropping an existing columnval df2 = sparkContext.makeRDD(6 to 10).map(i => (i, i * 3)).toDF("single", "triple")df2.saveAsParquetFile("data/test_table/key=2")// Read the partitioned tableval df3 = sqlContext.parquetFile("data/test_table")df3.printSchema()// The final schema consists of all 3 columns in the Parquet files together// with the partiioning column appeared in the partition directory paths.// root// |-- single: int (nullable = true)// |-- double: int (nullable = true)// |-- triple: int (nullable = true)// |-- key : int (nullable = true)

也就是说df1和df2都保存在data/test_table目录下了。 

df1列名分别为single,double,key 

df2列名分别为single,triple,key。

然后df3直接读取test_table后，会将df1,df2的列都加在一起，那么dfs的列分别就是single,double,triple,key

然后将df3.show。结果就 是：

single double triple key3      6      null   1  4      8      null   1  5      10     null   1  1      2      null   1  2      4      null   1  8      null   24     2  9      null   27     2  10     null   30     2  6      null   18     2  7      null   21     2

　

大家看，是不是df1和df2合起来的集成呢（不需要做关联）