5

I can understand the need for real-time data in 'bidding' and 'automated' scenarios. I can also see the point for direct customer interaction and response through social media (maybe).

However, beyond this, real-time often just seems to mean 'Not too out of date for my use case'. This means, quite often, data for yesterday, or this morning. It doesn't mean the user requires a real-time 'live' streaming system at massive cost.

However real-time is a fashionable buzzword at the moment.

I would love to know what real value generating business use-cases there are, that involve people, which require real-time data?

6

Learning: This is huge and can be applied to anything.

Imagine trying to learn how to play poker. You can only play one hand every hour. How long will it take you to learn how to play poker?

"Not too out of date for my use case"

If the company is smart and whose goal is to make a profit, they will change their "use case" or even completely revamp the point or meaning of the use case so they can derive insights like what their competitors are doing, and much more outside the scope of this response to, for lack of a better word, win.

With a realtime source feed this doesn't even cost much money. I actually am 100% broke at the moment ($0) but I could program a few things on my computer or even (lol) go to the public library, rent out an amazon web server for 25 cents an hour that I remote desktop into, and start gathering insights with the correct programming and know-how. Then, with correctly applied statistics, start deriving insights.

This concept mostly involves repeatedly testing different products or permutations of ideas (which is essentially what everyone does), but in the accelerating marketplace of the online world (and to the retail world as well since they are beginning to blend), competition is not going to go away.

Whomever learns the quickest wins. If you have direct competitors, whomever has the economic advantage will win, even if it's an accumulation of several different economic advantages.

You could say they got that economic advantage in a different way because maybe they:

  1. Already know more about the market
  2. Initially had a longer time frame to learn more
  3. Had better connections/got luckier/hired the smartest people
  4. Are able to learn much faster than their competitors

The internet changes VERY rapidly. So would you agree with me that numbers 1, 2, and even 3 are becoming more irrelevant nowadays? I didn't list money/resources because, usually, all they let you do is learn more quickly in some form or fashion. Sometimes there are barriers to entry (example: oh no, we can only do 5,000 API requests per day and can't pay more for anymore?). Learn more and you'll figure out a way around anything.

Colleges don't even teach the skills that are in most demand today! - unless its a programming language or something. And if they do, then the teachers most likely don't know much more than the students! Think about "social media."
Who teaches that?
How many people are looking for experts in it?
Now look at my poker learning analogy.

If you think money is the biggest problem, then you're not learning quickly enough how to potentially point out trends that could persuade someone to invest, or you're not able to see all the little miniature data points that reveal the hidden picture - the one that could be lowering costs by a factor of 4. That is, assuming you're able to analyze them.

You'd also not see how the real-time feed allows one to quickly test ideas and throw them out or keep them. What if you wanted to test permutations of ideas? Assuming sample size is large enough, you'd only be able to test one permutation (of which you'd probably have lets say 3x3x3=27 of) every day or whatever interval. How would you sell that to inner management? Think they'd be more easily persuaded if you could get the answer within an hour and do it cheaply?

What about factors that may change during your 27 days of permutation testing that would affect the outcome like market factor forces, seasonality, etc.

It could be argued that getting the data over a longer time frame is better because you'd hope all uncontrollable factors would be included in the analysis as an average; but what if you wanted to control one of those factors and, say, just test at 4pm-5pm with all the permutations? Or any other innumerable cases I could think of.

The factor of time here is most important in my opinion to be honest. Why? You may sacrifice a bit of accuracy by not doing the test for a lengthy period, but that sacrifice will lead you to conduct more experiments with better accuracy...ones that you could immediately implement.

Now you could also argue your sample size isn't large enough.

Most people assume you need a massive sample size. Something to consider though is the validity of your analysis increases with sample size, BUT it increases at a quickly decreasing rate. Something to consider - and apply with realtime data.

I love massive sample sizes too, but how do you think drugs get approved in clinical trials or medical research studies are done with 20 people? They can't test them out on thousands and thousands of people. They use statistical methods.

Example: They look at all results of EVERYTHING, even things you may think are completely unrelated they record. Blood pressure, heart rate, hair growth, skin color change, rashes, change in color preference perceived mood, sleep time, computer time, perceived interest, the combination of each factor and the combination of each factor on each other combination of factors comparing that to known full population averages are...factoring in time..etc..The data in small sample sizes is there (ie. real-time data would be considered a small sample size if you're using little snapshots of it), it's about setting up for it and digging deep.

Many, many other statistical methods are available like fractional factorial, non-parametric, design of experiments, deriving simulated numbers from your numbers based on standard deviation and probability and seeing how they fit into your model, etc. methods as well.

Ok enough belaboring on my part. Here are a few examples too:


IP Addresses & What Their Attributes are at that Time

  1. GEO location to IP services such as Maxmind constantly update their database because IP addresses change so frequently. For example, for a day or even 3 hours you could have your IP be used as a Tor exit node meaning alot of websites will block you, and do it quickly, or list you as using a "proxy." This is based upon data from GeoIP databases, which somehow will pick up on you being a tor exit node and will change the attributes of your IP quickly and accordingly to that information.

  2. An attempt to geolocate a potential customer based on the results of a GEO-IP database and thus discovering potential facts about them using a big enough sample size that you regress into a big database of known facts about that area (like avg level of income) using census data. =x This involves the central limit theorum.

  3. Thwarting fraud and identity theft. If your IP is listed as a "proxy" in GEO-IP databases, the merchant will likely pick up on that (by doing behind the scenes requests to the their GEO-IP database provider) and reject your attempt to purchase anything. Essentially they are assuming your attempt to hide your identity must mean you're also hiding other things. Plus, statistics from transactions reveals this to overwhelmingly be true anyway.

Imagine you are a thief and stole someone's CC number and are trying to buy stuff online. Are you going to use your home IP address? Probably not.


Online Media Buying

I can actually add more to this since I buy media online. Albeit I admit that much of that does involve bidding and you admit the realtime factor being important for.

However, there's MUCH more to it than that. One thing that REALLY bothers me when I'm buying media is the lack of real time RESULTS being published as well. Say I want to raise my bid price really high and see what happens - I'm testing the market. Most systems will only charge you when you do this as if you were bidding slightly higher than the next person, but sometimes they won't, or in my experience they actually just say they do and charge you some strange inflated ratio.

In our example lets say I get my results every 30 minutes. During that 30 minutes my bid price was $10 dollars per click. A very high bid on average. I need to know the results of this test to judge all sorts of things about the market (and by market it could mean a specific website, or a specific keyword, anything really) like competition, size, potential of market, conversion rate, etc.

I get my results back 30 minutes later and, oh crap, I just spent $10,000 because I was blind during the test. Had I been able to see it realtime, I could of just done it for 30 seconds and known a good outcome.

You could argue that I should have gradually raised my bid. If so, I point you back to the poker analogy up top and remind you of all the permutations I have to test as well.


This also gets even more complex when you're working with sales reps at a digital marketing firm and the client or your boss wants answers quickly. This means you have to conduct a lot of tests on a very, very wide permutation of variables in order to be able to give them the information they seek or just to be able to found out what the profitable combination is.

I could tell a client or anyone else alot in about 10 minutes with realtime data, using statistical analysis (a skill very few in my field have unfortunately), and trying different permutations...mostly involving macro changes (macro changes let you cover more ground at once - it's essentially a compensation for the inability to do everything real-time). Without that, it could take a week. Time is money.

Being able to make a move every 30 minutes or every morning won't work well when you have so many permutations to test (bid, landing page, source, offer, etc. etc. etc.).

As another example, a company I worked for had a tech team that could only change the landing page every 2 weeks - whereas it would take me 30 seconds to make the change I wanted...but I wasn't allowed to do it. In fact, I wasn't even allowed to talk directly to the person making the change. It had to go through 3-4 different and usually unnecessary people.

Imagine being able to make a move every 2 weeks and during those two weeks having to run the campaign and deal with the pressure and constant reminder that the campaign was not profitable. Back to the poker learning analogy.

  • Great content, thank you. I am going to have to digest that fully - there is a lot of meat. – CodeBeard Aug 14 '13 at 14:42
  • I'm usually not the best at describing things in a concise or poetic manner as I'd like to be - but I try my best. It's a quality I'm working on. Basically I tried to just throw my brain onto paper. – Taal Aug 14 '13 at 15:11
3

Three examples:

  1. There is a startup that generates stock market signals from real-time social network data
  2. There is a logistics consultant who visualizes lots of inventory data in real time (link leads to a German promo video).
  3. Here is someone who predicts visitors to South Tyrolean using amazon sales rankings of travel literature (ok, maybe this project may not need real time data)

Disclaimer: I am not related in any way to these projects

  • Interesting, So this is contextual information for finical trading? Someone making a buy decision in real-time. I can see that they would need real time. – CodeBeard Aug 13 '13 at 21:32
2

I would love to know what real value generating business use-cases there are, that involve people, which require real-time data?

I'm going to focus on savings, such as lives and property:

In the case of weather reports, knowing as soon as possible when something is forming or approaching that might affect your business can give you more time to properly secure the place and/or evacuate people so that you can bring everything back up quickly after the storms. Of course, other people would argue that it also means that you shut things down earlier or when it's not 100% necessary, but improved data also means that you can get earlier estimates of if your property is going to be affected.

Other uses cases are tracking resource costs and inventory levels ... if you know that costs are going up even when you're not yet due for a re-order, you can make a decision to purchase out-of-cycle if you suspect that the costs will go up further and remain high through when you'll next need to order. Tracking inventory levels is important in this case to know how long you can hold out without a new shipment and if you're able to take an early delivery.

I've also heard of e-commerce companies keeping track of various metrics as indicators that something's going wrong while they can still deal with it. Think of it as the difference between an oil pressure gauge and a 'low oil' light in your car -- if you know that sales are down today, yet people are still adding the same value of items to the online shopping carts ... that might be a sign that something's wrong that's keeping people from being able to complete the sale. The earlier you can figure out what's wrong and fix it, the less money you lose from an outage.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.