Python Open Labs – Web Scraping

For those who have been following this blog series, sorry for a late post on the updates about Python Open Labs.

Last week we covered some basics about web scraping with python, but before I start let me make a customary disclaimer.

Make sure that any of the websites that you want to scrape have granted you the required permissions to do so. Make sure you are not violating any terms of use by doing so.

So, getting along with the updates. In a nutshell web scraping can be described as a way of extracting useful relevant information from web pages i.e html pages. This can be abstracted into following steps:

  1. Downloading the web page content (user urllib or requests module in python)
  2. View page source in a web browser to examine the html structure of web page and locate information of interest for your task at hand
  3. Try to figure out the html structuring such as class, id, html tag etc that will help your python script locate the information.
  4. Use the beautifulsoup python module to parse and reach as close as possible to the relevant information in the html page structure and then extract the information using string methods.

The steps 2 – 4 go hand in hand, i.e one helps you build more upon the other. For example, the more you understand about the html structure surrounding your page the more specific inputs you can provide to beautifulsoup methods to extract out the information.

For the previous I have uploaded the sample python files with commented code lines on the Google Drive link mentioned below which you can access under Session – 16 folder. Make sure you work through those. Doubts, queries, feedbacks are always welcome 🙂

All of the course slides and examples are made available on: https://goo.gl/YP0c2E

We will be continuing with the web scraping lecture on march 31, 2017 after which I will also upload a comprehensive document with some additional relevant sources and more interesting code.

Happy Scraping !

See you next Friday from 1:30 PM – 3:30 PM at DSSC (Room – 215), Lehman Library at Columbia SIPA !

R Open Lab – Merge and Filter Data

During the first 20-30 minutes of yesterday’s open lab, we talked about how to merge datasets and filter data using base R and dplyr package. The rest of the open lab were free discussions between participants and instructors.

Thank you to all who showed up!

Welcome to explore the materials I used for the open lab:

https://drive.google.com/open?id=0B4uO8w_uGtsCWjRId0p6QllKZlE

Enjoy the spring!

DSSC Extended Walk-in Hours

We’ve extended our walk-in hours from two hours to four hours per day between March 27th through April 27th, Monday – Thursday.

We’ve also added a calendar listing the type of help you can expect during walk-in hours as well as some of the other activities in the DSSC such as the Open Labs or workshops.

The new hours are as follows, although best to consult the calendar in case of changes. Outside of these hours, you can always request a one-on-one consultation with one of our staff.

Monday 12pm – 4pm: help with R, Stata, SPSS and SAS

Tuesday 12pm – 4pm: help with GIS

Wednesday 12:30pm  – 4:30pm: help with R, Stata, SPSS and SAS

Thursday 12pm – 4pm: help with GIS

R Open Lab – ggplot

Data visualization is an integral part of data exploration and presentation. Yesterday, we talked about ggplot2, a package which provides a mature and consistent system for plotting in R.

We explored the advantage and disadvantages of ggplot2, the syntax and usage of the package.

As always, thank you to everyone who showed up.

Materials I used for the open lab can be found here.

https://drive.google.com/open?id=0B4uO8w_uGtsCMUVoLXl1Z0JsSlU

Enjoy spring! ❤️

Python Open Labs – Format Strings

Hi

In the 15th session of Python Open Labs, this week we looked at some miscellaneous topics and revision of basic concepts of file reading and string handling from previous sessions. We also briefly looked into format strings / format specifiers for string construction in Python. The relevant slides are available on the Session – 15 folder on the google drive link mentioned below.

All of the course slides and examples are made available on: https://goo.gl/YP0c2E

As always, please keep up with your programming practise, a suggested link for the same is: https://www.hackerrank.com/domains/python/py-introduction

See you next Friday from 1:30 PM – 3:30 PM at DSSC (Room – 215), Lehman Library at Columbia SIPA ! We will be covering some basics about web scraping.

R Open Labs – Apply Family

This Wednesday we talked about apply function family in base R. We covered apply(), tapply(), lapply(), sapply() and vapply(). We also briefly introduced the concept of factors in R.

As always, thank you to all who showed up! Next two weeks’ R Open Lab will be cancelled due to midterm week and spring break.

Good luck on your midterms and have a wonderful spring break! 😄

Python Open Labs – Object Oriented Programming

Hi

In this session we looked at simple concepts of object oriented programming, including the basics of creating classes, object instances, attributes and methods.

All of the course slides and examples are made available on: https://goo.gl/YP0c2E

Meanwhile, continue to practise and brush up your skills on: https://www.hackerrank.com/domains/python/py-introduction

See you next Friday from 1:30 PM – 3:30 PM at DSSC (Room – 215), Lehman Library at Columbia SIPA !

R Open Labs – readr Package

Today we introduced readr package. It is a package used for reading csv/xls/txt etc. data. It is designed to flexibly parse many types of data found in the wild, while still cleanly failing when data unexpectedly changes. 

We covered the functionality of the package and the difference between this package and base R.

Next week we will talk about apply family.

See you next Wednesday from 10 am – 12 pm at DSSC (Lehman Social Science Library Room 215)!

R Open Labs – Linear Regression

Thank you to all who showed up last week!

Last Wednesday we went over doing linear regression in R, including: how to fit and interpret linear models, evaluate model validity and select among competing models.

See you this Wednesday from 10 am – 12 pm at DSSC (Lehman Social Science Library Room 215)!

Materials I used for this open lab is posted here. https://drive.google.com/open?id=0B4uO8w_uGtsCQTJ6aUNFMUJpZVE

Python Open Labs – Session 12 (Review)

Hi

Today we had a brief review session of all the basics of programming that we have covered so far in the Python Open Labs series. During this review we went over reading and writing files, conditional statements, for loops and while loops and various other specifics of programming with Python. This marks a major milestone in the series as all of the material covered so far should be sufficient for basic programming/scripting tasks that you may need

All of the course slides are made available on: https://goo.gl/YP0c2E

Meanwhile, continue to practise and brush up your skills on: https://www.hackerrank.com/domains/python/py-introduction

In the next session I will introduce Object Oriented Design with Python.

For those who are getting started with Python, please watch this space for a concise blog post on basics of Python Installation, IDE set up etc in the coming week !

See you next Friday from 1:30 PM – 3:30 PM at DSSC (Room – 215), Lehman Library at Columbia SIPA !