Public transport users know the pain. You get to the station, stand in your usual spot on the platform and when the train rolls in, there aren't any seats in the carriages close to you. But by using some sensors that weigh the carriages, Sydney trains will be able to let you know which carriages have the fewest people, helping you find a seat for the long commute.

Lots of discussions about complex topics start with the premise that there are two types of people. That's where Symantec's Chief Technology Officer Hugh Thompson began his discussion on the challenges facing the security industry. He began his entertaining security keynote at this year's CeBIT event in Sydney telling the story of a bird that flew into a commercial aircraft as the plane was being loaded by ground staff. It was trapped in the passenger cabin, only becoming known when the trans-Atlantic flight was in the air. The reactions to the story are indicative, he said, about differing attitudes to security risks.

I recently attended a press and analyst summit run by UK events company NetEvents. The event, held in San Jose this year, introduced me to a few companies I’d not come across before that offer interesting products and services. Here are five I think are worth keeping an eye on.

Most of us breeze past the terms and conditions that come with free Wi-Fi networks and just hit "yes" so we can get online. When you sign into a shopping centre's Wi-Fi, though, you might be agreeing to allow them to track you.

Vizio recently got in trouble for collecting data on TV customers who opted out. This brings up an interesting question: Can my TV collect and share my data? Yep, the option is hidden in the settings of most smart TVs as "viewing information" or "internet-based advertising". If you don't want to be tracked, opt out.

It's not news that companies mine and sell your data, but the ins and outs of how it works aren't always clear. The US Federal Trade Commission recently published a report that explains how it works in a little more detail.

Want skills that can instantly turn you into a rockstar job applicant? Then you need The Essential Data Analysis Bundle. This 3-course package, which teaches practical lessons in Excel, Business Analysis And Data Science, is available this week at 81% off the retail price for Lifehacker Australia readers.

Knowledge is power. So if data equates to knowledge, having a lot of it will naturally allow us to make better decisions that will lead to wealth and success, right? Certainly many organisations are using big data to bolster revenue and bring about overall improvements. But big data isn't the silver bullet to business woes and you can still make bad decisions even with all the right information at your fingertips. We take a look at how big data can lead to bad decisions.

Big Data is changing the way we do science today. Traditionally, data were collected manually by scientists making measurements, using microscopes or surveys. These data could be analysed by hand or using simple statistical software on a PC.

Big data is not a new concept. It has been used in business for decades. The reason it's becoming a buzz word now is because companies are beginning to understand that if they capture all the data that is coming into their business, they can apply analytics and get significant value from it.

Information is a valuable asset for businesses as it helps them make well-informed decisions. Fantastic! Data is being generated at an unprecedented rate and organisations are hording it like there's no tomorrow, creating mammoth data sets we call big data. But is big data really helping these companies or is it just complicating the decision-making process? We find out.

The Microsoft Distributed Machine Learning Toolkit (DMTK) has been made open source by the vendor's Asia research team. The DMTK will make machine learning tasks on big data more scalable and efficient with a smaller cluster of computers. This is particularly useful for machine learning researchers and developers that work with large datasets.

Microsoft has released a Linux version of its Hadoop distribution, HDInsight, used to process streams of unstructured data through its cloud computing service, Azure. Here are the details.

The "Internet of Things" (IoT) is the new darling of the technology world thanks to our love of being tethered to our electronic devices. Watches, fridges and even dog collars are connected to the internet these days and these devices are generating an unrelenting wave of data. This has inevitably led to some privacy concerns over how the data is collected and used. How is one of the world's biggest cloud companies dealing with this IoT conundrum? We find out.

HP has ramped up efforts in the open source big data and analytics space, adding extensive support to open source technologies in the latest release of its HP Vertica analytics engine. Here's what you need to know.

Big data gets a lot of attention from media, industry and government. Companies and labs generate massive amounts of data associated with everything from weather to cell phone usage to medical records, and each data set may involve hundreds of variables. How does one begin to make sense of it all? The answer lies in "rubber sheet" geometry.

While women sitting on company boards remains a much-discussed topic, there is something new waiting to take a seat at the table: artificial intelligence, computers with company voting rights.

With the Senate passing the Federal Government's data retention bill last week, there has been a great deal of discussion of "metadata", what it is and whether the government ought to have access to it. However, metadata is just the tip of the data iceberg. The debate about data retention is only just beginning, and the outcome could touch on many aspects of our behaviour and society at large.

Looking for a practical example of the internet of things actually being useful? The Sydney Harbour Bridge is here to answer your call.