sparksql語法，通過對映方式讀txt

inferring the schema using reflection（通過對映指定schema）

--宣告乙個類
scala> case class person(name: string, age: int)
defined class person
--讀取txt檔案內容，然後對映到類的字段，轉化成dataframe
scala> val people = sc.textfile("hdfs://node1:8020/test/input/people.txt").map(_.split(",")).map(p => person(p(0), p(1).trim.toint)).todf()
people: org.apache.spark.sql.dataframe = [name: string, age: int]
--變數註冊成表，表名為people
scala> people.registertemptable("people");
--執行sql
scala> val teenagers=sqlcontext.sql("select name,age from people where age>=13 and age<=19")
teenagers: org.apache.spark.sql.dataframe = [name: string, age: int]
scala> teenagers.show
+------+---+
| name|age|
+------+---+
|justin| 19|
+------+---+
scala> teenagers.select("name").show
+------+
| name|
+------+
|justin|
+------+
scala> teenagers.count
res19: long = 1
--通過位置訪問
scala> teenagers.map(t => "name: " + t(0)).collect().foreach(println)
name: justin
--通過字段訪問
scala> teenagers.map(t => "name: " + t.getas[string]("name")).collect().foreach(println)
name: justin
--轉化成map對
scala> teenagers.map(_.getvaluesmap[any](list("name", "age"))).collect().foreach(println)
map(name -> justin, age -> 19)

結果集對映通過xml

create table grade gid number primary key,gname varchar2 20 create table student sid number primary key,sname varchar2 20 gid number references grade ...

hive對映hbase，通過phoenix查詢該錶

rowkey key 生成規則待定 列族 單列族 cf1 列名 id,name,code 1在hive中建表對映到hbase create table tmp.test hbase phoenix key string,id int,name string,code string stored by...

Spark SQL 新增第三方依賴包

最近在使用spark sql 執行hive語句時碰到異常資訊如下 開始以為時hive環境有問題，把sql語句拿到hive環境單獨跑沒有問題，排除掉這項。若有問題，參考解決方案。使用spark submit jar 提交掉時候已經引入了json serde 1.3 jar with dependenc...