siddhartb_

MStream and MIDAS are more accurate than previous baselines for unsupervised anomaly detection. However, there can be scenarios where some labels (ground truth information) are known. In such cases, a semi-supervised algorithm might work better. We are currently working towards building a semi-supervised approach for anomaly detection in real-time.

To the best of my knowledge, MStream and MIDAS are the fastest and detect anomalies in real-time.

Hi, I am one of the authors of the work. MStream detects anomalies, intrusions, DoS and DDoS attacks in real time and constant memory. It is built on top of MIDAS (https://github.com/Stream-AD/MIDAS/) and works in a multi-aspect data setting i.e., entries having multiple dimensions such as event-log data, multi-attributed graphs etc. MStream is two orders of magnitude faster while achieving higher accuracy on several publicly available datasets.

Github Repository: https://github.com/Stream-AD/MStream

Existing GAN based approaches excel at generating realistic samples, but seek to generate typical samples, rather than extreme samples. We propose ExGAN to generate realistic and extreme samples.

ExGAN allows the user to specify both the desired extremeness measure, as well as the desired extremeness probability they wish to sample at. Generating increasingly extreme examples can be done in constant time (with respect to the extremeness probability), as opposed to the exponential time required by the baseline.

Paper: https://arxiv.org/abs/2009.08454

Extreme Value Theory is extensively used in anomaly detection as well. This can be a first step towards generating anomalous data which is usually quite difficult to find.

There is a recorded presentation of the paper at https://youtu.be/Bd4PyLCHrto The first 5-10 minutes or so should be quite explanatory. Please feel free to let me know if you have any specific doubts. Thanks.

Detecting Intrusions, Denial of Service (DoS) attacks, Distributed Denial of Service (DDoS) attacks. It can also be used to detect fake profiles in Social Networks like Twitter, Facebook, Amazon reviews, and Financial Frauds. Basically any suspicious similar group of edges in time-evolving/dynamic graphs.

Hi, I'm the author of the MIDAS algorithm. We choose the number of hash functions and bucket according to the maximum error we can tolerate and the false positive probability theoretical guarantee we want. Please refer to the AAAI paper here: https://www.comp.nus.edu.sg/~sbhatia/assets/pdf/midas.pdf Let me know if you need more details.

Code is quite neat! What are the changes it will need for including fit and predict API?

Yes, there is a Python implementation available. MIDAS has also been converted to Rust and Ruby. Please check out the Github page for the links.

This is an R wrapper around the C++ implementation https://github.com/bhatiasiddharth/MIDAS

MIDAS can help social networks like Twitter and Facebook detect fake profiles used for spam and phishing in real-time, at a speed many times greater than existing state-of-the-art models.

Great question, it will be interesting to try it out. Temporal relations should be affected a bit but MIDAS should be able to detect anomalies.

In addition to detecting intrusions, it can detect fake ratings and frauds. Basically finding anomalous and suspicious behavior in any dynamic (time-evolving) graph.

Yes, we take expected count of a particular user/source node into consideration.

We handle locality in terms of both source and destination, therefore we should be able to handle both DoS and DDoS attacks.

Code is available in C++, Python, Ruby, R, and Rust at https://github.com/bhatiasiddharth/MIDAS

Currently MIDAS is available in Rust, Python, Ruby and R at https://github.com/bhatiasiddharth/MIDAS. If someone is interested to convert MIDAS to other languages, please feel free to do so and let me know so that I can add a link in the repository.

We assume (like any anomaly detection algorithm) that the majority is normal sample. In your context, the normal samples will be considered as outliers and therefore caught by the algorithm. One way to mitigate this is to either swap the labels. Another way is to sample a subset of the anomalies and then try.

Definitely. It will need only very small changes to the code. I would love to add it as a plugin. Can you point to some resources that can help in incorporating MIDAS into Gephi.

Sounds interesting. Can you elaborate on what the data is like?

Interesting question. With an increase in dimensions, we consider the correlation between the features in addition to considering them individually. The work is currently under review. Feel free to get in touch and I can update you once we release the MStream work.

Nice suggestion. Will definitely try to refactor. Thanks!

In most of the cases, timestamps should be with the data itself (assuming its a dynamic graph). If timestamps are to be chosen, one can select in a way seeing how many edges usually come in one time tick (second/minute etc.)

Timestamps don't affect any parameters other than alpha (temporal decay factor). You may want to check out how to decay the contribution of the past edges in the anomalousness of the current edge. If there is lot of granularity in the timestamps, a smaller alpha should be chosen. Hope it helps.

Thanks, MIDAS can be used to detect intrusions, fake ratings, frauds. Basically finding anomalous and suspicious behavior in a dynamic (time-evolving) graph.

We have also extended MIDAS to detect group anomalies in higher-dimensional records e.g. event-log data or multi-attributed graphs. We will release it soon.

We detect suddenly appearing bursts of activity which share many repeated nodes or edges, which we refer to as microclusters. E.g. denial of service (DoS) attacks in network traffic data and lockstep behavior.

Also, we detect scenarios where an individual edge may not be anomalous but along with other edges it acts as an anomalous community. For example, in the animation at https://github.com/bhatiasiddharth/MIDAS/ it may be possible that an individual edge is not anomalous but together the three malicious entities do a coordinated DoS attack.

Thanks. We give theoretical guarantees on the False Positive Probability which can be useful to decide the parameters. Some use cases of the project include detecting: 1. Intrusions 2. Fake Ratings 3. Financial Fraud

Code and Datasets we used are available at https://github.com/bhatiasiddharth/MIDAS