sqoop自動增量匯入 報錯

2021-08-22 10:36:01 字數 2322 閱讀 8716

在sqoop的查詢案例中可能會出現以下的報錯:

[admin@hadoop102 sqoop-1.4.6.bin__hadoop-2.0.4-alpha]$ bin/sqoop import \

> --connect jdbc:mysql://hadoop102:3306/company \

> --user root \

> --password root \

> --table staff \

> --target-dir /user/company \

> --delete-target-dir \

> --fields-terminated-by "\t"

warning: /opt/module/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/bin/../../hbase does not exist! hbase imports will fail.

please set $hbase_home to the root of your hbase installation.

warning: /opt/module/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/bin/../../hcatalog does not exist! hcatalog jobs will fail.

please set $hcat_home to the root of your hcatalog installation.

warning: /opt/module/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/bin/../../accumulo does not exist! accumulo imports will fail.

please set $accumulo_home to the root of your accumulo installation.

18/08/07 22:12:04 info sqoop.sqoop: running sqoop version: 1.4.6

18/08/07 22:12:04 error tool.basesqooptool: error parsing arguments for import:

18/08/07 22:12:04 error tool.basesqooptool: unrecognized argument: --user

18/08/07 22:12:04 error tool.basesqooptool: unrecognized argument: root

18/08/07 22:12:04 error tool.basesqooptool: unrecognized argument: --password

18/08/07 22:12:04 error tool.basesqooptool: unrecognized argument: root

18/08/07 22:12:04 error tool.basesqooptool: unrecognized argument: --table

18/08/07 22:12:04 error tool.basesqooptool: unrecognized argument: staff

18/08/07 22:12:04 error tool.basesqooptool: unrecognized argument: --target-dir

18/08/07 22:12:04 error tool.basesqooptool: unrecognized argument: /user/company

18/08/07 22:12:04 error tool.basesqooptool: unrecognized argument: --delete-target-dir

18/08/07 22:12:04 error tool.basesqooptool: unrecognized argument: 1

18/08/07 22:12:04 error tool.basesqooptool: unrecognized argument: --fields-terminated-by

18/08/07 22:12:04 error tool.basesqooptool: unrecognized argument: \t

try --help for usage instructions.

其實,我們可以一眼就看出來報錯的資訊是在 import 上面,出錯的原因,就是因為引數輸入錯誤。漏掉了– import。正確的**輸入是這樣子的。

sqoop 增量匯入

在隨著系統的執行,每天都會產生新的資料,在進行資料匯入的時候,由於之前已經匯入過資料,所以在產生新的資料的時候,就可以直接在之前導過的資料的基礎上繼續匯入資料,沒有必要之前的所有資料都重新匯入一遍 增量匯入資料的使用 以上為增量匯入的引數check column 檢查列,檢查資料庫表中的索引列,一般...

sqoop增量匯入

執行以下指令先將我們之前的資料匯入 sqoop import connect jdbc mysql master 3306 test username hive password 123456 table customer m 1使用hdfs dfs cat檢視生成的資料檔案,發現資料已經匯入.然後...

sqoop增量資料匯入

1.使用sqoop按欄位增長將mysql中資料抽取到hdfs 2.使用sqoop按時間增長將mysql中資料抽取到hdfs jps start all.sh2.開啟mysql服務。sudo service mysql start登入mysql mysql u root p3.在mysql庫中,建立s...