生產(chǎn)環(huán)境下， 我們一般使用?esrally來做es的基準(zhǔn)測試。 但是畢竟和真實生產(chǎn)的請求場景可能有差異的。?

成都創(chuàng)新互聯(lián)公司長期為成百上千家客戶提供的網(wǎng)站建設(shè)服務(wù)，團隊從業(yè)經(jīng)驗10年，關(guān)注不同地域、不同群體，并針對不同對象提供差異化的產(chǎn)品和服務(wù)；打造開放共贏平臺，與合作伙伴共同營造健康的互聯(lián)網(wǎng)生態(tài)環(huán)境。為章丘企業(yè)提供專業(yè)的成都網(wǎng)站制作、網(wǎng)站建設(shè)，章丘網(wǎng)站改版等技術(shù)服務(wù)。擁有十年豐富建站經(jīng)驗和眾多成功案例,為您定制開發(fā)。

因此，有必要考慮使用真實流量來壓測ES集群。

常用的流量復(fù)制工具有tcpcopy之類的，這里我們介紹一個更簡單易用的工具 gor 。

github地址：?https://github.com/buger/goreplay

TIPS：

我在測試環(huán)境的 192.168.2.4 上實際測試用的命令(放大1倍流量)：

#?192.168.2.4?為老es集群節(jié)點 #?192.168.2.188?為新es集群節(jié)點 wget?https://github.com/buger/goreplay/releases/download/v1.0.0/gor_1.0.0_x64.tar.gz tar?xf?gor_1.0.0_x64.tar.gz?-C?./ ./gor?--input-raw?:9200?--output-http?"http://192.168.2.188:9200|200%"

下圖是gor 輸出到stdout的內(nèi)容：

下圖是我在老es集群更新文檔后，可以看到新的es集群也自動更新了文檔

更多的關(guān)于gor 流量復(fù)制的介紹，可以看下面的內(nèi)容：

參考文檔：

https://testerhome.com/articles/21180

https://github.com/buger/goreplay/wiki

https://www.jianshu.com/p/57e058ad4995

1、流量實時復(fù)制引流（--input-raw 攔截端口配合--output-http輸出），例如將本機9200端口的HTTP流量實時復(fù)制到 es-dest:9200。

gor?--input-raw?:9200?--output-http?"http://es-dest:9200"

2、如果目標(biāo)服務(wù)器使用的庫與線上機器一樣，且只需要引流Get方法的請求。

gor?--input-raw?:9200??--http-allow-method?GET?--output-http?"http://es-dest:9200"

3、當(dāng)需要對線上服務(wù)進行整體性能壓測時，可將線上請求擴大N倍，進行引流，將請求擴大1倍，也可縮小，調(diào)整"|"后面的百分比即可。

gor?--input-raw?:9200??--http-allow-method?GET?--output-http?"http://es-dest:9200|200%"

4、只復(fù)制某個URL請求，--http-allow-url參數(shù)，-http-allow-url參數(shù)可用正則表達式（--output-http-url-regexp在gor 0.16已經(jīng)過期，使用--http-allow-url代替）。

gor?--input-raw?:9200?--http-allow-method?GET?--output-http?"http://es-dest:9200"?--http-allow-url?mall.*hotword

5、多目標(biāo)服務(wù)器的流量復(fù)制引流，有點類似nginx的mirror。

gor?--input-raw?:9200?--output-http?"http://es-dest:9200"?--output-http?"http://es-dest:9200"

6、將流量轉(zhuǎn)發(fā)到多個端點，默認情況下，它會將相同的流量發(fā)送到所有輸出。

gor?--input-tcp?:2920020?--output-http?"http://staging.com"??--output-http?"http://dev.com"

7、將相同的流量發(fā)送到多個站點，并且平分所有流量。

gor?--input-raw?:9200?--output-http?"http://staging.com"??--output-http?"http://dev.com"?--split-output?true

8、HTTP超時，默認情況下，http請求和響應(yīng)的超時時間為5秒。你可以像這樣覆蓋它：

gor?--input-tcp?replay.local:2920020?--output-http?http://staging.com?--output-http-timeout?30s

9、基本身份驗證，如果您的開發(fā)或登臺環(huán)境受基本身份驗證保護，那么可以在重放期間注入這些憑據(jù)：

gor?--input-raw?:9200?--output-http?"http://user:pass@staging.com"

10、性能壓力測試

可以將流量復(fù)制到文件，然后再對他們進行回放。回放的時候，流量會維持原始的時間間隔。如果你使用了百分比來進行速率限制，那么回放的速率會相應(yīng)的增加或減少。有了這種速率限制，gor就可以用來進行壓力測試。

gor?--input-file?"requests.gor|200%"?--output-http?"staging.com"

目前，input-file僅在使用基于百分比的限制器時才支持此功能。與默認限制器不同input-file，它不會降低請求速度，而會減慢速度或加速請求發(fā)射。

11 、錄制與回放

./gor?--input-raw?:9200?--output-file=requests.gor

執(zhí)行錄制命令后，將會創(chuàng)建新文件并不斷向其寫入所有捕獲的請求。

12、流量回放，重播來自文件的請求。

./gor?--input-file?requests.gor?--output-http="http://localhost:92000"

您應(yīng)該看到所有記錄到http://localhost:92000的請求，并且它們將以相同的順序重播，并且與錄制的時間完全相同。

gor的用法也遠不止上述提到的這些，更多的應(yīng)用測試場景，還請讀者自行挖掘。

./gor --help? 可以列出更全面的幫助信息

Gor?is?a?simple?http?traffic?replication?tool?written?in?Go.?Its?main?goal?is?to?replay?traffic?from?production?servers?to?staging?and?dev?environments. Project?page:?https://github.com/buger/gor Author:?<Leonid?Bugaev>?leonsbox@gmail.com Current?Version:?1.0.0 ??-copy-buffer-size?int ????Set?the?buffer?size?for?an?individual?request?(default?5M)?(default?524289200) ??-cpuprofile?string ????write?cpu?profile?to?file ??-debug?verbose ????Turn?on?debug?output,?shows?all?intercepted?traffic.?Works?only?when?with?verbose?flag ??-exit-after?duration ????exit?after?specified?duration ??-http-allow-header?value ????A?regexp?to?match?a?specific?header?against.?Requests?with?non-matching?headers?will?be?dropped: ?????gor?--input-raw?:9200?--output-http?staging.com?--http-allow-header?api-version:^v1 ??-http-allow-method?value ????Whitelist?of?HTTP?methods?to?replay.?Anything?else?will?be?dropped: ????gor?--input-raw?:9200?--output-http?staging.com?--http-allow-method?GET?--http-allow-method?OPTIONS ??-http-allow-url?value ????A?regexp?to?match?requests?against.?Filter?get?matched?against?full?url?with?domain.?Anything?else?will?be?dropped: ?????gor?--input-raw?:9200?--output-http?staging.com?--http-allow-url?^www. ??-http-basic-auth-filter?value ????A?regexp?to?match?the?decoded?basic?auth?string?against.?Requests?with?non-matching?headers?will?be?dropped: ?????gor?--input-raw?:9200?--output-http?staging.com?--http-basic-auth-filter?"^customer[0-9].*" ??-http-disallow-header?value ????A?regexp?to?match?a?specific?header?against.?Requests?with?matching?headers?will?be?dropped: ?????gor?--input-raw?:9200?--output-http?staging.com?--http-disallow-header?"User-Agent:?Replayed?by?Gor" ??-http-disallow-url?value ????A?regexp?to?match?requests?against.?Filter?get?matched?against?full?url?with?domain.?Anything?else?will?be?forwarded: ?????gor?--input-raw?:9200?--output-http?staging.com?--http-disallow-url?^www. ??-http-header-limiter?value ????Takes?a?fraction?of?requests,?consistently?taking?or?rejecting?a?request?based?on?the?FNV32-1A?hash?of?a?specific?header: ?????gor?--input-raw?:9200?--output-http?staging.com?--http-header-limiter?user-id:25% ??-http-original-host ????Normally?gor?replaces?the?Host?http?header?with?the?host?supplied?with?--output-http.??This?option?disables?that?behavior,?preserving?the?original?Host?header. ??-http-param-limiter?value ????Takes?a?fraction?of?requests,?consistently?taking?or?rejecting?a?request?based?on?the?FNV32-1A?hash?of?a?specific?GET?param: ?????gor?--input-raw?:9200?--output-http?staging.com?--http-param-limiter?user_id:25% ??-http-pprof?:8181 ????Enable?profiling.?Starts??http?server?on?specified?port,?exposing?special?/debug/pprof?endpoint.?Example:?:8181 ??-http-rewrite-header?value ????Rewrite?the?request?header?based?on?a?mapping: ????gor?--input-raw?:9200?--output-http?staging.com?--http-rewrite-header?Host:?(.*).example.com,$1.beta.example.com ??-http-rewrite-url?value ????Rewrite?the?request?url?based?on?a?mapping: ????gor?--input-raw?:9200?--output-http?staging.com?--http-rewrite-url?/v1/user/([^\/]+)/ping:/v2/user/$1/ping ??-http-set-header?value ????Inject?additional?headers?to?http?reqest: ????gor?--input-raw?:9200?--output-http?staging.com?--http-set-header?'User-Agent:?Gor' ??-http-set-param?value ????Set?request?url?param,?if?param?already?exists?it?will?be?overwritten: ????gor?--input-raw?:9200?--output-http?staging.com?--http-set-param?api_key=1 ??-input-dummy?value ????Used?for?testing?outputs.?Emits?'Get?/'?request?every?1s ??-input-file?value ????Read?requests?from?file:? ????gor?--input-file?./requests.gor?--output-http?staging.com ??-input-file-loop ????Loop?input?files,?useful?for?performance?testing. ??-input-kafka-host?string ????Send?request?and?response?stats?to?Kafka: ????gor?--output-stdout?--input-kafka-host?'192.168.0.1:9092,192.168.0.2:9092' ??-input-kafka-json-format ????If?turned?on,?it?will?assume?that?messages?coming?in?JSON?format?rather?than??GoReplay?text?format. ??-input-kafka-topic?string ????Send?request?and?response?stats?to?Kafka: ????gor?--output-stdout?--input-kafka-topic?'kafka-log' ??-input-raw?value ????Capture?traffic?from?given?port?(use?RAW?sockets?and?require?*sudo*?access): ????#?Capture?traffic?from?9200?port ????gor?--input-raw?:9200?--output-http?staging.com ??-input-raw-bpf-filter?string ????BPF?filter?to?write?custom?expressions.?Can?be?useful?in?case?of?non?standard?network?interfaces?like?tunneling?or?SPAN?port.?Example:?--input-raw-bpf-filter?'dst?port?9200' ??-input-raw-buffer-size?int ????Controls?size?of?the?OS?buffer?(in?bytes)?which?holds?packets?until?they?dispatched.?Default?value?depends?by?system:?in?Linux?around?2MB.?If?you?see?big?package?drop,?increase?this?value. ??-input-raw-engine?libpcap ????Intercept?traffic?using?libpcap?(default),?and?`raw_socket`?(default?"libpcap") ??-input-raw-expire?duration ????How?much?it?should?wait?for?the?last?TCP?packet,?till?consider?that?TCP?message?complete.?(default?2s) ??-input-raw-immediate-mode ????Set?pcap?interface?to?immediate?mode. ??-input-raw-override-snaplen ????Override?the?capture?snaplen?to?be?64k.?Required?for?some?Virtualized?environments ??-input-raw-realip-header?string ????If?not?blank,?injects?header?with?given?name?and?real?IP?value?to?the?request?payload.?Usually?this?header?should?be?named:?X-Real-IP ??-input-raw-timestamp-type?string ????Possible?values:?PCAP_TSTAMP_HOST,?PCAP_TSTAMP_HOST_LOWPREC,?PCAP_TSTAMP_HOST_HIPREC,?PCAP_TSTAMP_ADAPTER,?PCAP_TSTAMP_ADAPTER_UNSYNCED.?This?values?not?supported?on?all?systems,?GoReplay?will?tell?you?available?values?of?you?put?wrong?one. ??-input-raw-track-response ????If?turned?on?Gor?will?track?responses?in?addition?to?requests,?and?they?will?be?available?to?middleware?and?file?output. ??-input-tcp?value ????Used?for?internal?communication?between?Gor?instances.?Example:? ????#?Receive?requests?from?other?Gor?instances?on?2920020?port,?and?redirect?output?to?staging ????gor?--input-tcp?:2920020?--output-http?staging.com ??-input-tcp-certificate?string ????Path?to?PEM?encoded?certificate?file.?Used?when?TLS?turned?on. ??-input-tcp-certificate-key?string ????Path?to?PEM?encoded?certificate?key?file.?Used?when?TLS?turned?on. ??-input-tcp-secure ????Turn?on?TLS?security.?Do?not?forget?to?specify?certificate?and?key?files. ??-memprofile?string ????write?memory?profile?to?this?file ??-middleware?string ????Used?for?modifying?traffic?using?external?command ??-output-dummy?value ????DEPRECATED:?use?--output-stdout?instead ??-output-file?value ????Write?incoming?requests?to?file:? ????gor?--input-raw?:9200?--output-file?./requests.gor ??-output-file-append ????The?flushed?chunk?is?appended?to?existence?file?or?not.? ??-output-file-flush-interval?duration ????Interval?for?forcing?buffer?flush?to?the?file,?default:?1s.?(default?1s) ??-output-file-max-size-limit?value ????Max?size?of?output?file,?Default:?1TB?(default?-1) ??-output-file-queue-limit?int ????The?length?of?the?chunk?queue.?Default:?256?(default?256) ??-output-file-size-limit?value ????Size?of?each?chunk.?Default:?32mb?(default?33554432) ??-output-http?value ????Forwards?incoming?requests?to?given?http?address. ????#?Redirect?all?incoming?requests?to?staging.com?address? ????gor?--input-raw?:9200?--output-http?http://staging.com ??-output-http-compatibility-mode ????Use?standard?Go?client,?instead?of?built-in?implementation.?Can?be?slower,?but?more?compatible. ??-output-http-debug ????Enables?http?debug?output. ??-output-http-elasticsearch?string ????Send?request?and?response?stats?to?ElasticSearch: ????gor?--input-raw?:9200?--output-http?staging.com?--output-http-elasticsearch?'es_host:api_port/index_name' ??-output-http-header?--output-http-header ????WARNING:?--output-http-header?DEPRECATED,?use?`--http-set-header`?instead ??-output-http-header-filter?--output-http-header-filter ????WARNING:?--output-http-header-filter?DEPRECATED,?use?`--http-allow-header`?instead ??-output-http-header-hash-filter?output-http-header-hash-filter ????WARNING:?output-http-header-hash-filter?DEPRECATED,?use?`--http-header-hash-limiter`?instead ??-output-http-method?--output-http-method ????WARNING:?--output-http-method?DEPRECATED,?use?`--http-allow-method`?instead ??-output-http-queue-len?int ????Number?of?requests?that?can?be?queued?for?output,?if?all?workers?are?busy.?default?=?1000?(default?1000) ??-output-http-redirects?int ????Enable?how?often?redirects?should?be?followed. ??-output-http-response-buffer?int ????HTTP?response?buffer?size,?all?data?after?this?size?will?be?discarded. ??-output-http-rewrite-url?--output-http-rewrite-url ????WARNING:?--output-http-rewrite-url?DEPRECATED,?use?`--http-rewrite-url`?instead ??-output-http-stats ????Report?http?output?queue?stats?to?console?every?N?milliseconds.?See?output-http-stats-ms ??-output-http-stats-ms?int ????Report?http?output?queue?stats?to?console?every?N?milliseconds.?default:?5000?(default?5000) ??-output-http-timeout?duration ????Specify?HTTP?request/response?timeout.?By?default?5s.?Example:?--output-http-timeout?30s?(default?5s) ??-output-http-track-response ????If?turned?on,?HTTP?output?responses?will?be?set?to?all?outputs?like?stdout,?file?and?etc. ??-output-http-url-regexp?--output-http-url-regexp ????WARNING:?--output-http-url-regexp?DEPRECATED,?use?`--http-allow-url`?instead ??-output-http-workers?int ????Gor?uses?dynamic?worker?scaling.?Enter?a?number?to?set?a?maximum?number?of?workers.?default?=?0?=?unlimited. ??-output-http-workers-min?int ????Gor?uses?dynamic?worker?scaling.?Enter?a?number?to?set?a?minimum?number?of?workers.?default?=?1. ??-output-kafka-host?string ????Read?request?and?response?stats?from?Kafka: ????gor?--input-raw?:9200?--output-kafka-host?'192.168.0.1:9092,192.168.0.2:9092' ??-output-kafka-json-format ????If?turned?on,?it?will?serialize?messages?from?GoReplay?text?format?to?JSON. ??-output-kafka-topic?string ????Read?request?and?response?stats?from?Kafka: ????gor?--input-raw?:9200?--output-kafka-topic?'kafka-log' ??-output-null ????Used?for?testing?inputs.?Drops?all?requests. ??-output-stdout ????Used?for?testing?inputs.?Just?prints?to?console?data?coming?from?inputs. ??-output-tcp?value ????Used?for?internal?communication?between?Gor?instances.?Example:? ????#?Listen?for?requests?on?9200?port?and?forward?them?to?other?Gor?instance?on?2920020?port ????gor?--input-raw?:9200?--output-tcp?replay.local:2920020 ??-output-tcp-secure ????Use?TLS?secure?connection.?--input-file?on?another?end?should?have?TLS?turned?on?as?well. ??-output-tcp-stats ????Report?TCP?output?queue?stats?to?console?every?5?seconds. ??-prettify-http ????If?enabled,?will?automatically?decode?requests?and?responses?with:?Content-Encodning:?gzip?and?Transfer-Encoding:?chunked.?Useful?for?debugging,?in?conjuction?with?--output-stdout ??-split-output?true ????By?default?each?output?gets?same?traffic.?If?set?to?true?it?splits?traffic?equally?among?all?outputs. ??-stats ????Turn?on?queue?stats?output ??-verbose ????Turn?on?more?verbose?output

另外有需要云服務(wù)器可以了解下創(chuàng)新互聯(lián)cdcxhl.cn，海內(nèi)外云服務(wù)器15元起步，三天無理由+7*72小時售后在線，公司持有idc許可證，提供“云服務(wù)器、裸金屬服務(wù)器、高防服務(wù)器、香港服務(wù)器、美國服務(wù)器、虛擬主機、免備案服務(wù)器”等云主機租用服務(wù)以及企業(yè)上云的綜合解決方案，具有“安全穩(wěn)定、簡單易用、服務(wù)可用性高、性價比高”等特點與優(yōu)勢，專為企業(yè)上云打造定制，能夠滿足用戶豐富、多元化的應(yīng)用場景需求。

標(biāo)題名稱：使用gor復(fù)制真實流量壓測ES集群-創(chuàng)新互聯(lián)
網(wǎng)站路徑：http://www.chinadenli.net/article46/dcdphg.html

成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián)，為您提供軟件開發(fā)、手機網(wǎng)站建設(shè)、網(wǎng)頁設(shè)計公司、小程序開發(fā)、企業(yè)建站、網(wǎng)站排名

廣告

聲明：本網(wǎng)站發(fā)布的內(nèi)容（圖片、視頻和文字）以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主，如果涉及侵權(quán)請盡快告知，我們將會在第一時間刪除。文章觀點不代表本網(wǎng)站立場，如需處理請聯(lián)系客服。電話：028-86922220；郵箱：631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載，或轉(zhuǎn)載時需注明來源： 創(chuàng)新互聯(lián)