mysql千萬級測試1億資料的分頁分析測試

有什麼問題可以互相討論:[email protected]

於堡艦

現在我們繼續進行乙個測試相同的表結構插入1億條資料這次用到的是innodb表引擎,表名有些變化,這裡為甚要新建乙個表的很重要元素是原來的那張表是每個uid=1來做的索引,這次uid是1...10不等的數每種1千萬條記錄

create table `ipdata` (

`id` int(11) not null auto_increment,

`uid` int(8) not null default '0',

`ipaddress` varchar(50) not null,

`source` varchar(255) default null,

`track` varchar(255) default null,

`entrance` varchar(255) default null,

`createdtime` datetime not null default '0000-00-00 00:00:00',

`createddate` date not null default '0000-00-00',

primary key (`id`),

分頁排序查詢

select id from ipdata where uid=1 order by id asc limit 1,10; 0ms;

select id from ipdata where uid=1 order by id asc limit 10,10; 0ms;

select id from ipdata where uid=1 order by id asc limit 100,10; 0ms;

select id from ipdata where uid=1 order by id asc limit 1000,10; 0ms;

select id from ipdata where uid=1 order by id asc limit 10000,10; 47ms;

select id from ipdata where uid=1 order by id asc limit 100000,10; 266ms;

select id from ipdata where uid=1 order by id asc limit 1000000,10; 1.594s;

select id from ipdata where uid=1 order by id asc limit 5000000,10; 5.625s;

select id from ipdata where uid=1 order by id desc limit 5000000,10; 11.235s;

select id from ipdata where uid=1 order by id asc limit 10000000,10; 11.562s 無返回結果

select id from ipdata where uid=1 order by id asc limit 9999990,10; 11.719s;

select id from ipdata where uid=1 order by id desc limit 9999990,10; 18.719s;

結論是如果單查詢id,order by的時間比較可觀,但是可見正序和倒序時間不同.

返回全部結果查詢"*"

select * from ipdata where uid=1 order by id asc limit 1,10; 109ms;

select * from ipdata where uid=1 order by id asc limit 10,10; 0ms;

select * from ipdata where uid=1 order by id asc limit 100,10; 16ms;

select * from ipdata where uid=1 order by id asc limit 1000,10; 63ms;

select * from ipdata where uid=1 order by id asc limit 10000,10; 356ms;

select * from ipdata where uid=1 order by id asc limit 100000,10; 2.969s;

select * from ipdata where uid=1 order by id asc limit 1000000,10; 30.766s;

select id,uid,ipaddress,source,track,entrance,createdtime,createddate from ipdata where uid=1 order by id asc limit 1000000,10; 29.953s;

...下面的就不測試了,已經難以接受了

結論select id 要比select *快了不少至少在大的結果面前;

結果count測試

select count(*) from ipdata where uid=1; 12.281s;

select count(*) from ipdata where uid=2; 12.250s;

....

select count(*) from ipdata where uid=10; 11.453s;

count級別大概是10多秒左右返回都是1000萬;

count(id)測試

select count(id) from ipdata where uid=1; 10.281s;

select count(id) from ipdata where uid=2; 10.531s;

....

select count(id) from ipdata where uid=10; 12.531s;

count(id)這裡我不知道是機器原因可能測試不是十分準確,總之相差不大,不知道是否mysql預設通過唯一主鍵來count,如果*和id差不多都方便我還是推薦id,呵呵

這樣我們可以通過select id 來得到id列表,然後通過in來得到相應的記錄,可見是可行的;還有在這次測試中我們通過uid這個屬性來過濾掉了90%的結果集,如果根據95%過濾理想化可能還有點欠缺,但是根據80%過濾原則就不同了,至少這個索引還是理想的，過濾掉的內容看來mysql就可以算到千萬級別的用時了。其實這裡面的時間不代表真實時間,畢竟機器也是我們辦公室一台pc電腦,資料也比較小,這裡我只是有時間來測試一下千萬條乃至上億條資料的處理能力,到伺服器上應該要比這個快很多,畢竟磁碟io差距大,而且cpu也有差距,

總結

接下來我將要測試一些關於1億+的使用者資料表的解決方案,及大資料的搜尋方案通過lucene/solr+mysql

mysql千萬級測試1億資料的分頁分析測試

mysql生成千萬級測試資料

mysql億級資料遷移

mysql千萬級大資料SQL查詢優化

mysql千萬級測試1億資料的分頁分析測試

mysql生成千萬級測試資料

mysql億級資料遷移

mysql千萬級大資料SQL查詢優化

相關推薦