Bayesian Modelling in Python


Mark Regan

Section 0: Introduction


Welcome to "Bayesian Modelling in Python" - a tutorial for those interested in learning Bayesian statistics in Python. You can find a list of all tutorial sections on the project's homepage.

Statistics is a topic that never resonated with me throughout my years in university. The frequentist techniques that we were taught (p-values, etc.) felt contrived and ultimately, I turned my back on statistics as a topic that I wasn't interested in.

That was until I stumbled upon Bayesian statistics - a branch of statistics quite different from the traditional frequentist statistics that most universities teach. I was inspired by a number of different publications, blogs & videos that I would highly recommend any newbies to Bayesian stats to begin with. They include:

I created this tutorial in the hope that others find it useful and it helps them learn Bayesian techniques just like the above resources helped me. I'd welcome any corrections/comments/contributions from the community.

Loading your Google Hangout chat data


Throughout this tutorial, we will use a dataset containing all of my Google Hangout chat messages. I've removed the messages content and anonymized my friends' names; the rest of the dataset is unaltered.

If you'd like to use your Hangout chat data whilst working through this tutorial, you can download your Google Hangout data from Google Takeout. The Hangout data is downloadable in JSON format. After downloading, you can replace the hangouts.json file in the data folder.

The json file is heavily nested and contains a lot of redundant information. Some of the key fields are summarized below:

conversation_idConversation id representing the chat threadUgw5Xrm3ZO5mzAfKB7V4AaABAQ
participantsList of participants in the chat threadMark, Peter, John
event_idId representing an event such as chat message or video hangout7-H0Z7-FkyB7-H0au2avdw
timestampTimestamp2014-08-15 01:54:12
messageContent of the message sentWent to the local wedding photographer today
senderSender of the messageMark Regan

The below code loads the json data and parses each message into a single row in a pandas DataFrame.

Note: the data/ directory is missing the hangouts.json file. You must download and add your own JSON file as described above. Alternatively, you can skip to the next section where I import an anonymized dataset.

Loading output library...

We now have a data model that we can work with more easily. The above table shows a single row in the pandas DataFrame. I'm interested in how long it takes me to respond to messages. Let's create some plots that describe my typical response times.

Loading output library...

The above plots give a monthly and an overall perspective of the length of time (in seconds) that it takes me to respond to messages. At this point I have a lot of questions that I want to ask of the data. For example: 1. Is my response time affected by who I am talking to? 2. Are there environmental factors that affect my response time (day of week, location, etc.)? 3. What is the best and worst day to get in touch with me?

Before we try and answer some of these questions, lets take some baby steps by estimating some parameters of a model that describes the above data. That'll make it easier for us to understand the data and inquire further.

In the next section, we'll estimate parameters that describe the above distribution.

Export data for usage throughout tutorial



Loading output library...