使用在hdfs上的檔案匯入到hive中

2021-08-04 16:32:00 字數 1831 閱讀 4066

hive要開啟,另一邊root@hadoop-virtual-machine:/# hdfs dfs -ls /user/hive檢視hdfs上都有些什麼,

然後把資料檔案匯入到hdfs上 ,root@hadoop-virtual-machine:/# hdfs dfs -put /opt/os.txt /user/hive/

hive那邊用show databases;檢視當前都有什麼資料庫

然後建立表(此表是用來裝匯入進來的資料的)

hive> create table os

> (pathid string,nodeweight int,pathweight string,nodeid int,shutdown string)

> row format delimited fields terminated by ' ' lines terminated by '\n'

> stored as textfile;

要保證建立的表的格式要和資料檔案中的格式一樣,例如欄位間的間隔符是什麼行與行間的間隔符都要進行限定,

匯入資料 hive> load data inpath '/user/hive/os.txt' into table os;

hive> select * from os;

ok["12->22"]9638[52]12false

["19->78"]11895[51]19true

["16->17"]8608[58]16true

["16->17"]10240[54]16false

["12->22"]9638[52]12true

["2->22","2->3"]9638[52,44]2true

["19->78"]11895[51]19false

["16->17"]8608[58]16false

["16->17"]10240[54]16true

["1->65"]13271[53]1 true

["11->17","11->13"]10383[50,56]11false

["6->21","6->22","6->19","6->7","6->44","6->10","6->9","6->11"]5398[54,58,54,55,50,59,55,55]6false

["0->52","0->91","0->53"]9814[52,58,54]0true

["1->65"]13271[53]1 false

["11->17","11->13"]10383[50,56]11true

["6->21","6->22","6->19","6->7","6->44","6->10","6->9","6->11"]5398[54,58,54,55,50,59,55,55]6true

["0->52","0->91","0->53"]9814[52,58,54]0false

time taken: 0.114 seconds, fetched: 17 row(s)

進來了其實我在匯入的時候下面顯示loading data to table default.os

failed with exception unable to alter table. for direct metastore db connections, we don't support retries at the client level.

failed: execution error, return code 1 from org.apache.hadoop.hive.ql.exec.movetask

但是之後我想檢視一下表中到底有沒有資料,資料是匯入進來的,但是為什麼低下有failed那幾行我也不太清楚,如果有知道的可以告訴我,學習知識大家互相討論互相進步嘛

把hdfs上面的檔案定時匯入到hive裡面

建立分割槽表 create external table ext startup logs userid string,starttimeinms bigint,activetimeinms bigint,city string partitioned by y string,m string,d ...

將hdfs資料匯入到hbase中的官方案例

首先我們在本地建立乙個檔案資料,資料中間是tab分隔符 root hdp 4 vi mrhbase.tsv 1002 pear yellow 將資料put到hdfs上 root hdp 4 hadoop fs put root mrhbase.tsv 建立hbase表 hbase main 016 ...

使用Flume監控檔案並匯入到HIVE表中

首先因為hive的儲存是基於hdfs的,所以目標等同於,flume監控檔案並上傳hdfs上 hive建表 create table test name string,gender string row format delimited fields terminated by flume配置檔案如下...