Algorithms and data structures to accelerate network analysis

As the sheer amount of computer generated data continues to grow exponentially, new bottlenecks are unveiled that require rethinking our traditional software and hardware architectures. In this paper we present five algorithms and data structures (long queue emulation, lockless bimodal queues, tail...

Full description

Saved in:
Bibliographic Details
Published in:Future generation computer systems Vol. 86; no. C; pp. 535 - 545
Main Authors: Ros-Giralt, Jordi, Commike, Alan, Cullen, Peter, Lethin, Richard
Format: Journal Article
Language:English
Published: United States Elsevier B.V 01.09.2018
Elsevier
Subjects:
MATHEMATICS AND COMPUTING
ISSN:0167-739X, 1872-7115
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract As the sheer amount of computer generated data continues to grow exponentially, new bottlenecks are unveiled that require rethinking our traditional software and hardware architectures. In this paper we present five algorithms and data structures (long queue emulation, lockless bimodal queues, tail early dropping, LFN tables, and multiresolution priority queues) designed to optimize the process of analyzing network traffic. We integrated these optimizations on R-Scope, a high performance network appliance that runs the Bro network analyzer, and present benchmarks showcasing performance speed ups of 5X at traffic rates of 10 Gbps. •A New queuing algorithm to reduce packet drops in hardware queues.•New lockless bimodal producer–consumer queue to eliminate multi-thread contention.•Algorithm to dynamically shunt traffic while maximizing information entropy.•Lockless hash table with low false negatives to eliminate memory contention overheads.•Multiresolution priority queues to reduce the complexity of a priority queue down to O(1).
AbstractList As the sheer amount of computer generated data continues to grow exponentially, new bottlenecks are unveiled that require rethinking our traditional software and hardware architectures. Here in this paper we present five algorithms and data structures (long queue emulation, lockless bimodal queues, tail early dropping, LFN tables, and multiresolution priority queues) designed to optimize the process of analyzing network traffic. We integrated these optimizations on R-Scope, a high performance network appliance that runs the Bro network analyzer, and present benchmarks showcasing performance speed ups of 5X at traffic rates of 10 Gbps.
As the sheer amount of computer generated data continues to grow exponentially, new bottlenecks are unveiled that require rethinking our traditional software and hardware architectures. In this paper we present five algorithms and data structures (long queue emulation, lockless bimodal queues, tail early dropping, LFN tables, and multiresolution priority queues) designed to optimize the process of analyzing network traffic. We integrated these optimizations on R-Scope, a high performance network appliance that runs the Bro network analyzer, and present benchmarks showcasing performance speed ups of 5X at traffic rates of 10 Gbps. •A New queuing algorithm to reduce packet drops in hardware queues.•New lockless bimodal producer–consumer queue to eliminate multi-thread contention.•Algorithm to dynamically shunt traffic while maximizing information entropy.•Lockless hash table with low false negatives to eliminate memory contention overheads.•Multiresolution priority queues to reduce the complexity of a priority queue down to O(1).
Author Ros-Giralt, Jordi
Commike, Alan
Cullen, Peter
Lethin, Richard
Author_xml – sequence: 1
  givenname: Jordi
  orcidid: 0000-0003-4450-609X
  surname: Ros-Giralt
  fullname: Ros-Giralt, Jordi
  email: giralt@reservoir.com
– sequence: 2
  givenname: Alan
  surname: Commike
  fullname: Commike, Alan
  email: commike@reservoir.com
– sequence: 3
  givenname: Peter
  surname: Cullen
  fullname: Cullen, Peter
  email: cullen@reservoir.com
– sequence: 4
  givenname: Richard
  surname: Lethin
  fullname: Lethin, Richard
  email: lethin@reservoir.com
BackLink https://www.osti.gov/servlets/purl/1538310$$D View this record in Osti.gov
BookMark eNqFkD1PwzAURS1UJNrCP2CI2BPsOK0dJJCqii-pEksHNuvVfqEuqY1sF9R_T6IwMcD0lnOu7rsTMnLeISGXjBaMsvn1rmgO6RCwKCmTBa0KyqsTMmZSlLlgbDYi4w4TueD16xmZxLijlDLB2ZjcLto3H2za7mMGzmQGEmQxhYPuA2OWfAZaY4sBEmYO05cP7x0J7THaeE5OG2gjXvzcKVk_3K-XT_nq5fF5uVjlmos65bwRuq5LQaU0jJcgKikM1ps5wsYI4PMaGwlM15TNwHDNZak5x4pKs2lkyafkaoj1MVkVtU2ot9o7hzopNuOSM9pBNwOkg48xYKM6DpL1LgWwrWJU9WOpnRrGUv1YilaqG6uTq1_yR7B7CMf_tLtBw-75T4uhL4dOo7Gh72a8_TvgG1ZaiQg
CitedBy_id crossref_primary_10_2166_wst_2023_215
Cites_doi 10.1109/HPEC.2014.7040975
10.1145/114005.102808
10.1016/S1389-1286(99)00112-7
10.1145/2830318.2830319
ContentType Journal Article
Copyright 2018 Elsevier B.V.
Copyright_xml – notice: 2018 Elsevier B.V.
CorporateAuthor Reservoir Labs, Inc., Portland, OR (United States)
Reservoir Labs, New York, NY (United States)
CorporateAuthor_xml – name: Reservoir Labs, New York, NY (United States)
– name: Reservoir Labs, Inc., Portland, OR (United States)
DBID AAYXX
CITATION
OIOZB
OTOTI
DOI 10.1016/j.future.2018.04.034
DatabaseName CrossRef
OSTI.GOV - Hybrid
OSTI.GOV
DatabaseTitle CrossRef
DatabaseTitleList

DeliveryMethod fulltext_linktorsrc
Discipline Computer Science
EISSN 1872-7115
EndPage 545
ExternalDocumentID 1538310
10_1016_j_future_2018_04_034
S0167739X1830222X
GrantInformation_xml – fundername: US Department of Energy, United States
  grantid: DE-SC0017184; DE-SC0006343; DE-SC0004400
  funderid: http://dx.doi.org/10.13039/100000015
GroupedDBID --K
--M
-~X
.DC
.~1
0R~
1B1
1~.
1~5
29H
4.4
457
4G.
5GY
5VS
7-5
71M
8P~
9JN
AACTN
AAEDT
AAEDW
AAIAV
AAIKJ
AAKOC
AALRI
AAOAW
AAQFI
AAQXK
AAXUO
AAYFN
ABBOA
ABFNM
ABJNI
ABMAC
ABXDB
ABYKQ
ACDAQ
ACGFS
ACNNM
ACRLP
ACZNC
ADBBV
ADEZE
ADJOM
ADMUD
AEBSH
AEKER
AFKWA
AFTJW
AGHFR
AGUBO
AGYEJ
AHHHB
AHZHX
AIALX
AIEXJ
AIKHN
AITUG
AJBFU
AJOXV
ALMA_UNASSIGNED_HOLDINGS
AMFUW
AMRAJ
AOUOD
ASPBG
AVWKF
AXJTR
AZFZN
BKOJK
BLXMC
CS3
EBS
EFJIC
EFLBG
EJD
EO8
EO9
EP2
EP3
F5P
FDB
FEDTE
FGOYB
FIRID
FNPLU
FYGXN
G-Q
G8K
GBLVA
GBOLZ
HLZ
HVGLF
HZ~
IHE
J1W
KOM
LG9
M41
MO0
MS~
N9A
O-L
O9-
OAUVE
OZT
P-8
P-9
PC.
Q38
R2-
RIG
ROL
RPZ
SBC
SDF
SDG
SES
SEW
SPC
SPCBC
SSV
SSZ
T5K
UHS
WUQ
XPP
ZMT
~G-
9DU
AATTM
AAXKI
AAYWO
AAYXX
ABDPE
ABWVN
ACLOT
ACRPL
ADNMO
AEIPS
AFJKZ
AGQPQ
AIIUN
ANKPU
APXCP
CITATION
EFKBS
~HD
ABPIF
OIOZB
OTOTI
ID FETCH-LOGICAL-c379t-3f7c9927088d132a7487de9b6eabd7a369ef8a1c9015ad3c382c33e408dbf823
ISICitedReferencesCount 3
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000437555800044&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN 0167-739X
IngestDate Mon Nov 20 03:50:45 EST 2023
Sat Nov 29 07:28:05 EST 2025
Tue Nov 18 21:38:41 EST 2025
Fri Feb 23 02:35:14 EST 2024
IsDoiOpenAccess true
IsOpenAccess true
IsPeerReviewed true
IsScholarly true
Issue C
Language English
LinkModel OpenURL
MergedId FETCHMERGED-LOGICAL-c379t-3f7c9927088d132a7487de9b6eabd7a369ef8a1c9015ad3c382c33e408dbf823
Notes SC0004400; SC0006343; SC0017184
USDOE Office of Science (SC)
ORCID 0000-0003-4450-609X
000000034450609X
OpenAccessLink https://www.osti.gov/servlets/purl/1538310
PageCount 11
ParticipantIDs osti_scitechconnect_1538310
crossref_citationtrail_10_1016_j_future_2018_04_034
crossref_primary_10_1016_j_future_2018_04_034
elsevier_sciencedirect_doi_10_1016_j_future_2018_04_034
PublicationCentury 2000
PublicationDate 2018-09-01
PublicationDateYYYYMMDD 2018-09-01
PublicationDate_xml – month: 09
  year: 2018
  text: 2018-09-01
  day: 01
PublicationDecade 2010
PublicationPlace United States
PublicationPlace_xml – name: United States
PublicationTitle Future generation computer systems
PublicationYear 2018
Publisher Elsevier B.V
Elsevier
Publisher_xml – name: Elsevier B.V
– name: Elsevier
References Google Performance Tools. [Online]. Available
Mosberger, Jin, Laboratories (b5) 1998
J. Ros-Giralt, A. Commike, D. Honey, R. Lethin, High-performance many-core networking, in: Proceedings of the Second Workshop on Innovating the Network for Data-Intensive Science - INDIS ’15, 2015.
(Accessed 17 September 2017).
Paxson (b1) 1999; 31
Cormen (b7) 2009
J. Ros-Giralt, A. Commike, R. Rotsted, P. Clancy, A. Johnson, R. Lethin, Lockless hash tables with low false negatives, in: 2014 IEEE High Performance Extreme Computing Conference, HPEC, 2014.
Leon-Garcia (b3) 2011
Jordi Ros-Giralt, Alan Commike, Peter Cullen, Jeff Lucovsky, Dilip Madathil, Richard Lethin, Multiresolution priority queues, in: Presented at the IEEE High Performance Extreme Computing Conference, Boston, USA.
Herlihy (b4) 1991; 13
Cormen (10.1016/j.future.2018.04.034_b7) 2009
10.1016/j.future.2018.04.034_b2
Herlihy (10.1016/j.future.2018.04.034_b4) 1991; 13
Paxson (10.1016/j.future.2018.04.034_b1) 1999; 31
Mosberger (10.1016/j.future.2018.04.034_b5) 1998
10.1016/j.future.2018.04.034_b6
10.1016/j.future.2018.04.034_b8
10.1016/j.future.2018.04.034_b9
Leon-Garcia (10.1016/j.future.2018.04.034_b3) 2011
References_xml – year: 1998
  ident: b5
  article-title: Httperf: A Tool for Measuring Web Server Performance
– year: 2009
  ident: b7
  article-title: Introduction to Algorithms
– volume: 31
  start-page: 2435
  year: 1999
  end-page: 2463
  ident: b1
  article-title: Bro: a system for detecting network intruders in real-time
  publication-title: Comput. Netw.
– volume: 13
  start-page: 124
  year: 1991
  end-page: 149
  ident: b4
  article-title: Wait-free synchronization
  publication-title: ACM Trans. Program. Lang. Syst.
– year: 2011
  ident: b3
  article-title: Probability, Statistics, and Random Processes for Electrical Engineering
– reference: J. Ros-Giralt, A. Commike, R. Rotsted, P. Clancy, A. Johnson, R. Lethin, Lockless hash tables with low false negatives, in: 2014 IEEE High Performance Extreme Computing Conference, HPEC, 2014.
– reference: Jordi Ros-Giralt, Alan Commike, Peter Cullen, Jeff Lucovsky, Dilip Madathil, Richard Lethin, Multiresolution priority queues, in: Presented at the IEEE High Performance Extreme Computing Conference, Boston, USA.
– reference: . (Accessed 17 September 2017).
– reference: Google Performance Tools. [Online]. Available:
– reference: J. Ros-Giralt, A. Commike, D. Honey, R. Lethin, High-performance many-core networking, in: Proceedings of the Second Workshop on Innovating the Network for Data-Intensive Science - INDIS ’15, 2015.
– ident: 10.1016/j.future.2018.04.034_b6
  doi: 10.1109/HPEC.2014.7040975
– ident: 10.1016/j.future.2018.04.034_b9
– ident: 10.1016/j.future.2018.04.034_b8
– volume: 13
  start-page: 124
  issue: 1
  year: 1991
  ident: 10.1016/j.future.2018.04.034_b4
  article-title: Wait-free synchronization
  publication-title: ACM Trans. Program. Lang. Syst.
  doi: 10.1145/114005.102808
– volume: 31
  start-page: 2435
  issue: 23–24
  year: 1999
  ident: 10.1016/j.future.2018.04.034_b1
  article-title: Bro: a system for detecting network intruders in real-time
  publication-title: Comput. Netw.
  doi: 10.1016/S1389-1286(99)00112-7
– year: 1998
  ident: 10.1016/j.future.2018.04.034_b5
– year: 2009
  ident: 10.1016/j.future.2018.04.034_b7
– ident: 10.1016/j.future.2018.04.034_b2
  doi: 10.1145/2830318.2830319
– year: 2011
  ident: 10.1016/j.future.2018.04.034_b3
SSID ssj0001731
Score 2.2238922
Snippet As the sheer amount of computer generated data continues to grow exponentially, new bottlenecks are unveiled that require rethinking our traditional software...
SourceID osti
crossref
elsevier
SourceType Open Access Repository
Enrichment Source
Index Database
Publisher
StartPage 535
SubjectTerms MATHEMATICS AND COMPUTING
Title Algorithms and data structures to accelerate network analysis
URI https://dx.doi.org/10.1016/j.future.2018.04.034
https://www.osti.gov/servlets/purl/1538310
Volume 86
WOSCitedRecordID wos000437555800044&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
journalDatabaseRights – providerCode: PRVESC
  databaseName: Elsevier SD Freedom Collection Journals 2021
  customDbUrl:
  eissn: 1872-7115
  dateEnd: 99991231
  omitProxy: false
  ssIdentifier: ssj0001731
  issn: 0167-739X
  databaseCode: AIEXJ
  dateStart: 19950201
  isFulltext: true
  titleUrlDefault: https://www.sciencedirect.com
  providerName: Elsevier
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwtV07b9swECYMp0OXNn2hadKCQ7dAhSXaIjkaRtIHgqCDB28qSZGNAlkKHDfIz-9RPEpu0zbt0EUwaNEyeB-Pd6e77wh5y1KTG6vyRMw0OCiZniTaJzmaNGU6N-BBu65Q-Iyfn4vVSn4ejb7EWpibmjeNuL2VV_9V1DAGwvals_8g7v5HYQA-g9DhCmKH618Jfl5_bcHjv1gH9mWfAnocWGK_bQKfgzIGDhvPEXHchCxwuDNwk-zaqqcd3YjvsWwRJgZbQCD_85Am314n76uNqsOLDfBnq_7dRrteVyEBaF4PSFx41u_mToLwmd1eVD8X_MegRCr6rCuMlOGxvhu4BIXMWdc2F86doGwFB-s-DeWcURsjMXZA3eKXSj7EGy7fBdYVn54nOrpajIr-SJ_tNTrzNXh7GZ9J0Ht7848nq0_9aZ1y7FmJ_y-WV3Y5gHef8DvzZdyCRt6xTJb75BG6FHQeoPCEjGzzlDyO7Tooau9nZAcZFJBBPTLogAy6bemADIrIoBEZz8ny9GS5-JBg94zEMC63CXPcwK7jcIyUKcsUB9e0tFLnVumSK5ZL64RKjTcIVckMExlsTTudiFI7kbEXZNy0jX1JKIwI6aQRpVNTZ2daMgd-uWNSaQWb-oCwuCaFQWZ53-CkLmIK4WURVrLwK1lMpgWs5AFJ-llXgVnlnvt5XO4CrcNg9RWAi3tmHnrp-FmeGNn4DDKYhsh49cdvD8nDAd9HZAxCsa_JA3Ozra43bxBM3wEpa5Cb
linkProvider Elsevier
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Algorithms+and+data+structures+to+accelerate+network+analysis&rft.jtitle=Future+generation+computer+systems&rft.au=Ros-Giralt%2C+Jordi&rft.au=Commike%2C+Alan&rft.au=Cullen%2C+Peter&rft.au=Lethin%2C+Richard&rft.date=2018-09-01&rft.pub=Elsevier&rft.issn=0167-739X&rft.eissn=1872-7115&rft.volume=86&rft.issue=C&rft_id=info:doi/10.1016%2Fj.future.2018.04.034&rft.externalDocID=1538310
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0167-739X&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0167-739X&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0167-739X&client=summon